![\"turns\"](\"https://www.seangoedecke.com/static/70f6abb1474ff212395a43c8a17194f8/eb2ef/turns.png\" "\\"turns\\"")
\n \n It’s common to compare the current AI takeover of software engineering to the rise of AI in chess. Chess AIs went from much weaker than serious players to much stronger than even the strongest humans. Between those points, there was a middle period dominated by “centaurs”: human-AI partnerships that were stronger than unassisted AIs or humans. Lots of people think that we’re currently in a world of software engineering centaurs. According to them, coding AIs are not yet capable enough to replace engineers, but AI-assisted engineers are better at programming than both AIs and humans.
\nThis is partialy correct, but the wrong way to think about it. AI-assisted engineers are better, but unlike with chess centaurs, they’re not actually better at programming. When I ask agents to write code, they make fewer mistakes than I do1 and are orders of magnitude faster. The code that they write always compiles, rarely has race conditions or other concurrency errors, works on mobile browsers, and so on.
\nThat doesn’t mean I can leave the AI alone. Purely vibe-coding at work produces awful outputs. But they’re not awful because they’re bad code, they’re awful because they’re in bad taste: code that is not maintainable, that trades off important requirements in order to satisfy made-up ones, that contradicts the long-term strategy for a feature or service, and so on.
\nIn other words, my primary value is not that I help the AI write better code, it’s that I align the AI with the values of my organization. Human-AI partnerships are for alignment, not capability.
\nFrontier models are misaligned to the working programmer. They are obsessed with a set of behaviors that presumably satisfy their RL grader: writing enormous block comments above functions, producing hundreds of useless unit tests, adding little bits of text all over websites they design, and so on. Working with agents is about noticing and wrestling with those behaviors. That’s why my prompting advice is to explicitly talk about your high-level values: it’s an attempt to head off obvious misalignment.
\nThis is great news for software engineers. It’s well-understood how to train more capable models: bigger models, more and better data, better RL environments, and so on. However, it’s not well-understood how to align models better. There are plenty of very capable models that exhibit behavior that is badly misaligned with human values. Indeed, it’s one of the main pillars of the AI doomer position that alignment is much harder to solve than capability, and we might thus end up with dangerous super-capable but poorly-aligned AI models.
\nAlignment is also more context-dependent than capability. Working code is working code, no matter what (which is partially why it’s comparatively easy to train for). But aligning to a company’s technical values is different from company to company, as any software engineer who’s switched companies knows. It can almost feel like relearning the job. So training an aligned coding model doesn’t just require hitting the exact right set of values, it requires creating a model that can adapt on the fly to a wide range of possible values.
\nVibecoding maximalists like DHH argue that AI models are (or soon will be) so much more capable than human programmers that we ought to stop reading the code. Eventually there will be no such thing as programmers at all. If it were just about capability, they might be right. But — fortunately for software engineers — good code also has to be aligned to the technical values of the system and organization it’s embedded in. AI models are great at writing code, but not very good at doing that, and it’s unclear that they’re going to get good at it anytime soon. We might all2 keep our jobs for a little while yet.
\nFor instance, I can’t remember the last time I’ve seen an agent make an off-by-one error. I do occasionally catch a pure programming error, typically in areas where I have a lot of technical domain knowledge. If you’re working out of distribution I suspect it’s easier to beat the models.
\n↩\nIt’s still going to be rough for junior engineers.
\n↩\nIn general, you should be suspicious of engineers who are trying to give you advice. Even during ordinary times, this industry is so wide and changes so quickly that nobody really knows anything for sure. And we are not in ordinary times. The advent of LLMs and AI agents is the largest change to software engineering in my professional lifetime, and possibly the largest change ever. That said, here’s my advice:
\nMost experienced software engineers today have spent the bulk of their career in the ZIRP era. This was a time when investment money flooded the industry, driving up engineer bargaining power. Big tech companies spent a lot of money and effort trying to make their engineers happy and comfortable. If you were an engineer at one of those companies, you could expect to have a decent say in what kind of work you did, and even what kind of politics your company had. Unless you worked at a handful of unusual companies (e.g. Amazon) you could expect to be practically immune from layoffs, and to only be fired after many months (sometimes years) of low performance.
\nA lot of advice floating around is ZIRP-era advice: either it was written during that era, or it’s written by engineers whose habits of thought were formed during that era. This kind of advice typically tells you to take a stand (e.g. to unionize1), to speak up more against “unethical technologies” like AI, and to insist on being given time to practice your craft the way you want. This was great advice in 2016, but it’s not good advice in 2026.
\nIn fact, I think it’s unethical for senior engineers to give advice like this to junior engineers who are more likely to take it (because they’re young and naive) and more likely to be punished for taking it (because they lack leverage). If you’re a new engineer, don’t fall for it! Let your colleagues with more experience and political capital take risks like that.
\nInstead, I recommend adapting to the demands of the current era of software engineering. Try to make yourself useful to your team and your manager. Be pragmatic about your actual bargaining power (fairly low, unless you’re useful enough to be irreplaceable). Keep your expectations of yourself under control: don’t spiral out in an attempt to do something astonishing, just try to be consistently helpful.
\nDon’t start fights. Being pleasant to work with (particularly when you don’t get your way) covers many sins. Once you’re further along in your career, you will be expected to start some fights, but this is not a tactic beginners should adopt: there are a lot of factors2 that determine when and how to start fights, and getting it wrong can be costly. Just don’t risk it.
\nIn general, you should stay out of the political game at all costs. Even quite senior engineers are political tools, not movers and shakers, and this goes double for junior engineers. Simply trying to be friendly and helpful will get you infinitely further, politically speaking, than any amount of Machiavellian game-playing. Keep your head down, stick with your management chain (not random people who try to assign you work), and you’ll be fine.
\nAs an engineer, your main technical value is conscientiousness. You should be asking lots of questions, both of yourself and of the engineers around you. You should be actively trying to make sense of the systems you work with, instead of just assuming someone else has it covered. Software systems are complicated enough that a few weeks of careful attention will mean you know technical details that nobody else does, which is a really easy way to add value.
\nIf you ought to be suspicious of most software engineering advice, the good news is that you should also be suspicious of doomsayers who predict the end of the industry. Before LLMs, people thought outsourcing would end software engineering in Western countries; before that, people thought high-level languages and low-code tools would end software engineering as a profession. Now people think AI means it’s all over. Maybe! But there are also reasons3 to think that software engineering will simply change. We’ll all find out together.
\nMany software engineers will tell you to avoid AI entirely. Even though agentic AI did not exist during the ZIRP era, this is still ZIRP era advice. Your company will expect you to use AI for the same reasons that builders are expected to use power tools. Pushing back hard on that as a beginning engineer is a great way to get laid off: you simply do not have the bargaining power to fight back against an industry trend this powerful.
\nThat said, it’s really important that you don’t delegate your own judgement to AI. Don’t just trust the suggestions or approaches that your agents propose. Ask questions and substitute your own opinions where you disagree: even if they’re wrong, you’ll learn more that way. If you don’t understand something the AI is telling you, either drill down until you do or just ignore it. Under no circumstances should you pass the AI’s message on to your colleagues verbatim. In other words, don’t be a meat proxy.
\nI think most people become a meat proxy as a form of panic: they feel like it’s over for their own skills, and that the AI model is smarter than them, so they can’t add any value beyond simply deferring to Claude or GPT-6. I can understand why people panic. It’s a crazy time in the industry, after all. But panic almost never helps you make good decisions. Keep your head, try to remain confident that your skills are still relevant, and use the agents to inform your own understanding4 instead of to replace it.
\nIt was really nice to work in tech in the 2010s when everything was stable. But we’re not in that world anymore. This is an age of wonders and terrors. Things will go worse than we think in some ways, but better than we could imagine in others.
\nThe doomsayers — the people who are saying it’s all over, and that there’s no hope — are almost certainly wrong. They can’t predict the future because nobody can. Technological change of this magnitude always has knock-on effects that are impossible to see coming, both positive and negative.
\nThe nature of the job might change, but it will always be valuable to be smart, friendly and conscientious. Delegating your judgement to an AI model might feel like a relief from despair in the short term — at least now it’s the AI’s responsibility, not yours — but it’s a bad idea. Don’t give up!
\nAt most companies, publicly campaigning to start a union is a great way to attract unofficial retaliation. It signals that you’re going to cause trouble (after all, that’s what a union is for), which can have long-term negative effects on your career. (This is not a judgement about whether unions in general are good or bad.)
\n↩\nIn general, you should pick fights for your managers, not with them. See this tag for much, much more on that topic. But again, if you’re new to the industry, just don’t pick fights at all.
\n↩\nMost plausibly: engineers using LLMs will be able to add some value for a while, and the explosion of LLM-authored software means we’ll need more engineers to work with the LLMs on it.
\n↩\nOne way you know you’re thinking is that you form your own opinions (some of mine are here).
\n↩\nWhen someone is explaining something to me, I ask on average one question every thirty seconds. I’m sure this is frustrating to some people, but it’s actually a good habit and you should do it too.
\nMost of the questions I ask are very short, and require very short answers. Typically I’m asking for confirmation: “so when you say X, you mean Y?”, or “this X is the thing you mentioned earlier when you were saying Z?“. The point of these questions is to make sure I understand.
\nA small misunderstanding early on balloons into a big misunderstanding later, because all the things you misinterpret based on the first misunderstanding will become misunderstandings in their own right, and so on. That’s why you can’t just wait until someone’s finished talking and ask all your questions at once1.
\nI think most people don’t ask questions because they’re not really trying to understand: they’re happy to just trust that the person they’re talking to knows what’s going on. This goes double when that person is a well-respected or more senior engineer. However, once you have skin in the game, that all changes. When you’re in a position where you are the one responsible for success or failure — where you are going to have to go away after the conversation and take a bunch of concrete actions — you will find yourself wanting to ask many questions.
\nWhen someone’s explaining a plan to me at work, I am usually building it in my head: visualizing the specific lines of code that would need to be written in order to implement it. I’ll sometimes handwave the details for solved problems (for instance, I might say “send emails like subsystem X of the same service that also sends emails”), but at minimum I’m planning out:
\nIf I hear something that is suspiciously vague (for instance, someone describes service X as “storing data” when I know service X only talks to an ephemeral Redis store), I will immediately ask “hold on, how is that going to work?” Usually that indicates a missing dependency (e.g. service X has to make an RPC call to service Y, which does have a persistent database), which often suggests design changes (for instance, moving some or all of the functionality into service Y).
\nSometimes questions like these will uncover a fundamentally unworkable feature of the design. This usually happens when a plan fails to take into account some wicked features. I remember many years ago a neighboring team built a complex event-driven system that was very elegant but made it completely impossible to silo customer data in a single datacenter. Because our company had “all your data lives in a single location” as a flagship feature, this system just did not work and had to be effectively abandoned.
\nIf you’re involved in any kind of technical leadership role, you will have to do this work eventually. Doing it as early as possible — i.e. in the conversation where someone is describing the plan to you — can save hours or days of wasted implementation. Often this is enough to turn a failed project into a successful one.
\nMaybe you only work with reliable engineers who can be trusted to make all the right decisions. You can thus let them talk without interruption, because it doesn’t really matter if you have a detailed understanding of what they’re saying. That’s great! But these days you probably also have colleagues that are inherently unreliable: AI agents.
\nI mostly work with GPT-6-Astra and Claude Opus 5.5. These are good models that don’t often make code mistakes: when they set out to do something, they usually do it2. But they make design mistakes all the time. They assume that two services can talk to each other when in fact they can’t, or they forget about the fact that their code has to run both in the cloud and on-premises, and so on. This is mostly due to the lack of continuous learning: you didn’t know this stuff on your first day either, but you had plenty of time to pick it up. Language models are always on their first day.
\nYou should be absolutely peppering AI agents with questions. I constantly ask things like:
\nThere will probably come a day when I always get sensible answers to these questions that convince me the model knows what it’s doing. But today is not that day. About half the questions I ask get answers that convince me3 the model has made a mistake: it should have reused the existing subsystem for X, or authed to Y differently, or kept the Z implementation simple, and so on. When this stops happening, I’ll stop asking questions (and try and see if my company will still be willing to pay me to occupy more of an architect role, I suppose).
\nYou should all be asking way more questions. In other words, you should all be trying harder to actually understand what you’re hearing. Don’t trust that the person (or AI model) you’re talking to knows what they’re doing, even if you think they’re smarter than you. Nobody understands complex software products. If you have deep domain knowledge of any area of a codebase, you will routinely find yourself correcting powerful AI models and principal engineers.
\nSometimes questions you have will be answered later on, but in my experience this is more about questions like “what are the broader consequences of X”, not the simple confirmation questions I often interrupt to ask.
\n↩\nYour experience might be different if you’re working in a different domain or with a different language.
\n↩\nI know some of these might sound like leading questions that would trigger sychophancy, but in my experience good coding models are very happy to robustly defend themselves.
\n↩\nBeing good at building and being good at shipping are two separate skills. In the short term, they’re actually countervailing: if you have a gift for building, you’re likely to be worse at shipping. Ira Glass has a classic quote about this.
\n\n\nAll of us who do creative work, we get into it because we have good taste. But there is this gap. For the first couple years you make stuff, it’s just not that good. It’s trying to be good, it has potential, but it’s not. But your taste, the thing that got you into the game, is still killer. And your taste is why your work disappoints you.
\n
The only way around this is to grit your teeth and ship it. You have to force yourself to publish things you’ve made even when you think they’re crap.
\nGifted programmers have a nearly pathological desire to build elegant, correct, neat systems. That’s what motivates them to learn the arcane details of their languages, or to spend time polishing and refactoring over and over again. But it’s also what makes them reluctant to ship. Any flaws in the software bother them on an emotional level. If they ship with those flaws, they feel like people will think they weren’t paying enough attention to notice them, or that they weren’t good enough to fix them.
\nThis is annoying when you’re writing software on your own, but it’s completely fatal when you’re working in a tech company. Any large software system is covered in flaws, whether due to time pressure, relative inexperience, wicked features, or a hundred other reasons. Working with it is a process of compromise: of finding the best possible solution given the quirks and foibles of the codebase. In fact, since the most important thing in large codebases is consistency, the right thing to do is sometimes to duplicate flaws, assuming they’re not catastrophic.
\nGifted programmers often freeze up. I’ve often seen them retreat to smaller domains where they can safely make the code “correct”: tweaking dev-environment setup, or refactoring tests. Sometimes they just do nothing, and spin in shame and guilt (plus the compounding shame of not achieving anything) until they implode and quit. If they had worse taste, they wouldn’t be as good at programming, but they’d be a lot more useful. You can typically improve a bad diff with time and effort. You can’t improve no diff.
\nI have a sensitive eye for awkward sentences and uneven prose. That can make writing an unpleasant process: I know what I’m trying to say, but I can’t seem to say it in a way that’s as clear and as elegant as I know is possible. More than half the time I finish drafting a blog post, I look at the post and don’t think it’s very good. But I (mostly) grit my teeth and publish it anyway, because you have to bias towards shipping.
\nLike any skill, shipping gets easier the more you practice it. If I don’t publish a blog post for a month, I always feel like the next draft is too poorly-written or uninteresting to put out there. But when I’m publishing a post per day, I typically feel great about each draft. When I go back and read my old posts, I can’t tell which ones I felt good about and which ones I felt bad about. There’s no correlation between that and the posts that become popular. Here are some posts I didn’t like as I was writing them but that resonated with my audience:
\nHere are some posts I thought were pretty good but that didn’t find popularity:
\nYou just can’t predict what people will find interesting or useful. Producing a high volume of work thus gives much better yield than a small amount of highly-polished work.
\nIt can be disheartening to realize that some of your most casual, throwaway work will be more successful than the work you slaved over1. Specifically, it’s disheartening because it means realizing you don’t have control over your own success. You can’t produce something successful by focusing on a single piece until you’re satisfied it’s great. Instead, you just have to do a lot of things and see what sticks. You have to be momentum-based, not outcome-based. In other words, you have to grit your teeth and ship it.
\nOne common reason to write less is getting overly precious about your ideas. If you think you’ve got a really compelling concept, you don’t want to “waste it” on a poorly-written story. But in fact you can just write about the same thing over and over until you get it right! I have written like thirty blog posts about shipping (this is one of them), or about how tech companies work, or about how internal emotional regulation is as important as technical ability. I expect to continue writing and thinking about these ideas for as long as I find them interesting.
\n“System One” models like Jev are fast general classifiers. Classifiers have existed since 1958, but they have to be trained for specific tasks: if you build a classifier to identify images of dogs, it can’t be used to tell you if a streetlight is red, or if a letter is urgent. Like a LLM, Jev can be prompted for a wide variety of tasks, from sorting email to playing Doom.
\nI think models like this are going to be important. There are many tasks that a LLM could do in theory but are too slow and expensive in practice (for instance, reading each new message in Slack1 and deciding whether to notify you or not). While you could train a specific classifier for these tasks, there are two main problems with that:
\nJev obviously solves the first problem. Any engineering team can plug in a System One model with a prompt like “Based on {list of criteria}, should the user be notified about this message?” But I think it solves the second problem too.
\nFor serious work, a specific hand-built classifier will always be cheaper and faster than Jev. Generic classifiers have to encode knowledge of all kinds of irrelevant things in their weights, so they can address lots of different tasks. That makes them larger, slower, and more expensive to run. Fortunately, it is going to be surprisingly easy to replace a Jev instance with a hand-built classifier.
\nOnce you’re satisfied with how your Jev classifier is performing — presumably you’ve spent days tweaking the prompt — you can trivially collect its input and output data. In the Slack notifier case, that’d be the Slack message (plus any context) and the ultimate decision to notify or not. Once you’ve saved enough data, you’ll be able to train your own classifier on that data2.
\nIt won’t be a general classifier like Jev, but it should do well on the specific task and be much faster. Of course it’ll require some ML expertise, but it should be easier to develop (or rent) that expertise once you’ve validated that the feature is worth building.
\nIn other words, because Jev has to be prompted for specific tasks, it should be easy to distil any successful Jev usage into a specific classifier. If System One models take off — and I hope they do — I expect this to be a common pattern.
\nAs I write this, I’m imagining ways you could poll and batch to do this with LLMs. Substitute “instantly notify” or some higher-volume event source if you’d prefer a different example.
\n↩\nYou could annotate a bunch of data with LLMs already, without using Jev, but this is a pretty expensive step to take when you aren’t sure the feature is going to work.
\n↩\nI recently wrote about Jev, a new “System One” language model that only outputs decisions: the answers to a set of user-provided multiple-choice questions. This means it’s nowhere near as flexible1 as a traditional LLM like ChatGPT, but in return it’s consistently fast.
\nWe don’t know exactly how Jev works. I’ve seen people say diffusion, or various tweaks to the Transformer architecture, or some entirely new type of model. But that doesn’t matter. Like I argued here, it isn’t hard to turn any LLM into a System One model. By batching prompts that generate a single token with structured output, you get a consistently fast general-purpose classifier. I vibed up a basic version to play with here in ~150 lines of Python (most of which is error handling).
\nNote that this doesn’t require changing the model. As long as you have access to the logits (for structured outputs) and can prefill data into the prompt, you can turn any LLM into a general fast classifier. What’s it like to program with one of these? While wiring up the demos for my library, I learned two techniques that I want to write about: setting tiered goals and tournament choice sampling.
\nHere’s Qwen3-8B playing Doom:
\n\nIf you compare this to the video of the same model playing Doom with regular tool calls, it’s clear that the System One version of the model is doing more things and reacting more quickly. The tool-calling model makes one decision every 600ms or so, while the System One model makes six or seven batched decisions every 190ms2:
\n\n \n \n \n \n
Both Qwen3-8B and Jev are text-only models, so both demos require a step where we translate the game state into text. However, it’d be trivial to support image (or audio) input by choosing a multimodal LLM.
\nWhat’s interesting about implementing the Doom demo is that just supplying the game inputs as choices doesn’t work very well. A single forward pass — 200ms — is enough time to react to the current game state, but doesn’t bring enough compute to bear to derive the current short-term goal (e.g. “kill this enemy”, “collect this item”) and choose to follow it. When I wired it up that way, the model held down the “shoot” button 100% of the time (why not, I guess) and just aimlessly wandered around the level.
\nThe fix is to periodically ask the model to choose between a fixed set of short term goals (e.g. “collect armor”, “kill enemies”) and then include that goal in the regular every-200ms prompt. If you look at the Doom video in the Jev demo, you can see that they’re doing exactly that. As soon as I did it as well, my model started playing in a more human-like way.
\nThis is an interesting technique for working with System One models. In a way, it’s the equivalent of regular LLM reasoning, since it provides a way to use more compute on the same problem. I can imagine a real-time system that manages several layers of goals in this way:
\nThe general structure here should be pretty familiar to anyone who’s worked in game or robotics AI. In theory you could replace (1) with an actual LLM, and have that generate the lists of options for steps (2) and (3). In practice I suspect this will be tricky to get right, and it’ll be better to just write down a list of all possible goals ahead of time. This would work just fine for game-playing and well-understood tasks.
\nI also reimplemented the Wikiracing demo from the Jev launch post, where the model has to start at the Wikipedia page for “baseball” and navigate as quickly as possible to the Wikipedia page for “sun”. You can watch the video for that here, though it’s less impressive than the Doom demo.
\nThe difficulty with the Doom demo is getting the model to loop quickly enough and to commit to short-term plans. For Wikiracing, the difficulty is scale: the Wikipedia page for “baseball” has over a thousand internal links. Jev only supports 255 choices for a single question, and my hacked-together System One layer was similar. While it technically would scale out to more choices, it stopped working well3 after a hundred or so.
\nJev’s approach here is to do “a 2 stage-system of scoring independently then making an explicit choice”. This did not work very well for me at all. I think here Jev is benefiting from the fact that it’s specifically trained to give confidence estimates. Qwen3-8B gave a few hundred of the links the same top score, which wasn’t helpful. It ended up taking multiple minutes to find a thirty-or-forty link path between the two pages.
\nWhat I tried instead was tournament sampling: I fed a hundred links at a time into each choice, then did a second pass with the chosen links. This worked great. The model found the ideal three-link path (if you’re curious, “baseball”/“scientific american”/“amateur astronomy”/“sun”). I recommend this pattern if you’re trying to find the best option among many choices. Ordinary LLMs are way better at relative judgements than absolute ratings.
\nI remain optimistic about the potential of System One models — fast general classifiers — to build AI systems that aren’t just chatbots. It feels like this is a meaningful alternative to tool calls for realtime scenarios or use-cases where you need predictable inference timing. Just as generic LLMs often outperform domain-specific models, I think it’s likely that generic System One models will sometimes outperform domain-specific classifiers (though they will always be larger and slower).
\nI do think the big labs are definitely going to try and compete by releasing a choice-only version of their small, fast models. If Jev gets any traction, we will soon see a System One Terra and a System One Haiku, and we will certainly see “real” versions of my vibed up System One library. We should start working out the best way to write programs with these models now.
\nYou can think of System One models as general-purpose classifiers. Instead of having to train a new classifier per-task, you can use a System One model. It’ll be bigger and slower than a custom classifier model, but far more flexible, and you can tweak it via adjusting the prompt instead of having to re-train the model.
\nTechnically you can give it the multiple-choice question of “which letter comes next” to make it act like a normal autoregressive LLM, but that wouldn’t really work.
\n↩\nI started on a 4090, which was able to make decisions every 500ms, but that wasn’t really quick enough for Doom. I could probably have optimized it further but instead I just rented an H100 for ten minutes to record the demo, which got it down to a 190ms loop. I recorded the tool-calling Doom demo on the H100 too, so it’s a fair comparison.
\n↩\nThere’s an interesting research question here about how to implement choices like this, since they have to be predictable by a single token. I started with indexes but found “labels” (just picking some token to associate with the choice) performed way better on Wikiracing (though not Doom). How many choices do you have to have before labels are better than indexes? Of course, you could alter the model to directly output the choice, but I like the idea that you can do all of this in the inference code for any LLM.
\n↩\nI don’t write blog posts about new models. That’s Simon Willison’s beat, and he’s very good at it. But I want to write about Jev, which is a different kind1 of AI model: a “System One”2 model. As it turns out, it’s not that different from an ordinary LLM with structured output, but the interface it uses is very cool and I hope it becomes more widespread.
\nOrdinary LLMs take in some human-language prompt and produce some human-language output. They do so autoregressively: first they produce one token, then the next, then the next, and so on.
\nUser: Who invented the sandwich?\nLLM: The sandwich was invented by the Earl of Sandwich.This makes them extremely flexible, since they can do literally anything a computer can do. But it also makes them slow and weird. Slow, because they have to run a whole new generation pass per-token, and weird, because the space of human language is so broad that you can get really odd behavior from a model trained on it.
\nJev takes a human-language prompt, but it does not produce human-language output. It only produces structured output.
\nUser: { state: \"What color is the sky?\", choices: [\"blue\", \"red\", \"yellow\"] }\nJev: { answer: \"blue\" }So far, so ordinary: LLMs do this already. But it turns out that if you build a model that only produces structured output, you get some interesting and desirable properties.
\nJev is always really fast. The fastest response time is around 70ms instead of a couple of seconds for normal LLMs. Even better, the slowest response time is only 500ms. Because Jev only does structured output, it isn’t autoregressive: it can produce answers to many questions in parallel in a single forward pass. When a LLM is producing structured output, it has to produce the tokens ”{”, ” ”, “answer”, ”:”, and so on with successive forward passes3. Jev does it all in one go.
\nThe most compelling example of Jev’s speed is that the model can play Doom. You can feed a text-based representation of the current game state into the model, combined with a set of choices like “should the trigger be held down”, “what should the current goal be”, “given that the current goal is X, what keyboard input should be pressed”, and so on, and it works — latency is low enough and the system is smart enough that the model plays well in real time.
\n\n \n \n ![\"doom\"](\"https://www.seangoedecke.com/static/d54381912237e0f153a14473f60d04e9/5bef7/doom.png\" "\\"doom\\"")
\n \n
Of course you could train a neural net to play Doom already. But Jev is a general intelligence: just like LLMs can do your taxes, perform mathematics research, fix your Python environment, and write you a poem, Jev can do many other tasks besides playing a single video game. Current LLMs can play Doom too (albeit slowly). But as Nelson Elhage famously said, fast software doesn’t just mean we can do the same tasks faster, it means we can do entirely new kinds of tasks. What kinds of new programs can we write by injecting 100ms worth of dirt-cheap intelligence at various decision points?
\nTo me, this is the most exciting thing about Jev. Fast structured output could be a genuinely new computational primitive for intelligence. So far we’ve built a lot of programs on top of autoregressive token generation, and they all look like fancy chatbots. Leaning hard into structured output might conceivably unlock a bunch of non-chatbot use cases for AI.
\nMy biggest problem with Jev is that I think fast structured output is already available. Structured output from LLMs is only slow because (a) nobody really cares about it4, and (b) the people who do care about it want big JSON blobs, so it’s typically implemented with “grammar-constrained decoding”: the LLM outputs autoregressively as normal, but the logit sampler discards tokens that don’t fit the structured output (e.g. if there hasn’t been a ”[”, you can’t output a ”]”).
\nIf you want fast, parallelized structured output against limited choices, you don’t strictly need to do autoregressive generation at all. You can simply prefill the response with \"choice\": \" and generate one token5, restricted to the user-provided choices. Since LLMs ingest all input tokens in parallel, this is way faster than generating the entire structured output. Multiple choices can be batched into the same forward pass via ordinary inference batching. This doesn’t let you do long-form structured output, but in return you get most of6 Jev’s “secret sauce”: the speed, the consistency, and the parallelism of a System One model.
People have already started trying this after today’s Jev announcement, and it seems like it’s working OK7. In other words, I suspect Jev does not have a substantial technical moat, and their claimed “Reinforcement Learning for Calibrated Decisions” is not a brand-new scaling axis. It will probably be pretty easy for any other lab to replicate, or for individual programmers to retrofit existing open-source LLMs into a fast Jev-like model.
\nHowever, I suspect Jev is still going to be better than most versions of “Qwen-32B-System-One” or whatever. Being able to fine-tune or optimize the model on just structured output is probably a meaningful advantage.
\nI doubt Jev is ever going to be as smart as frontier LLMs. Not being able to use test-time compute at all8 is a big disadvantage, and will likely cap this kind of model around the strength of non-reasoning LLMs. In practice this shouldn’t matter too much for low-latency applications, but you shouldn’t see this as a new scaling axis or a way to produce more intelligent models.
\nJev’s developers claim it is immune from hallucinations. To me, this seems like a semantic dodge, since Jev can absolutely still pick the wrong choice (e.g. calling the sky “red”). I suppose that’s technically just a mistake, since the model is picking a user-provided choice instead of inventing something new out of whole cloth. Still, all of this is also true about regular LLMs with structured outputs, and it doesn’t make Jev any more reliable in practice.
\nIt’s unclear to me how much of Jev’s value is in the model itself, compared to the inference strategy of only generating one token per question. The data and demos in the announcement look to me like they could have been generated by plugging any Terra-sized model into a single-token inference stack. However, the people involved are credible, and I’m sure the model is good — I just wish they’d provided some comparisons that didn’t force the LLM to unnecessarily produce a blob of JSON token-by-token.
\nOverall, I am happy that Jev exists and I hope it succeeds. I hope we do see some real competition in the fast-structured-output space, and that it motivates the big labs to release official versions of their own models that are fine-tuned for this. GPT-5.6-Terra-System-One would be a very interesting model to build AI products on top of.
\nI did write about Thinking Machines’ “interaction models”, which are also a fast-enough-to-be-meaningfully-different paradigm for AI inference.
\n↩\nThey call Jev a “System One” LLM, after Daniel Kahneman’s partially discredited Thinking Fast and Slow, where he divides human cognition into a lightning-fast System One and a slow-and-reflective System Two.
\n↩\nIf you’re thinking “wait, couldn’t you just aggressively prefill a regular LLM and only produce one constrained token”, keep reading.
\n↩\nNot counting tool calls, which are built-in in a way that structured output isn’t.
\n↩\nWhat if some of the user’s choices are longer than a single token? I haven’t tried this myself, but I’m sure you could translate them into a single token, or train the model to output “1/2/3” under the hood instead of the choice content, or generate only the first token of the choice if it’s different, or some other clever trick I haven’t thought of.
\n↩\nJev claims that their generated probabilities are “calibrated”, but I haven’t seen anything to suggest that these aren’t just regular logit probabilities. Maybe there’s some clever training they do to encourage accurate logprobs in uncertain situations (e.g. getting the model to produce heads: 50, tails: 50 when predicting a coinflip, etc)? If so, I wish they’d written more about that in the announcement.
I tried it myself with Qwen2.5-1.5B-Instruct and got a 2x-3x speedup compared to non-prefixed structured output.
I suppose they could do some looped-transformer thing where they loop some fixed amount of times, but anything that looks like reasoning would make the model latency slow and unpredictable, defeating the entire purpose.
\n↩\nEarly AI agents were basically enthusiastic idiots. Working with them required you to tell them precisely what you wanted them to do (for instance, “method A exists on class B, please add an equivalent method to classes C through F”). Otherwise they’d go off and do entirely the wrong thing. But as AI agents have improved, this has changed.
\nWhen frontier models go off and do the wrong thing today, they don’t do it because they’re confused, they do it because they make an incorrect assumption about your goals or priorities. For instance, when GPT-6-Astra thinks it’s writing code for itself, it will produce minified code. It’s perfectly capable of writing human-readable code — at least in Golang, where I’ve produced several thousand lines of acceptable code with the model — but you have to tell it that humans will be reading the code1.
\nThis is the main piece of advice I want to give most people I see prompting agents: give the agent context on your priorities, not just on the specific task you want them to do. Here’s a prompt I recently used as the starting point for Deckard.
\n\n\nHello. You should have Runpod access via MCP (if not, tell me and I’ll fix it).
\nI have the long-term goal of building a local program or browser extension to automatically scan pages I load for AI content and hide it. I have the short-term goal of figuring out the best AI detection model I can run on my macbook without killing my battery or making it hot, and (relatedly) figuring out how to run the model most efficiently. My guess is that Pangram’s EditLens 3B or the smaller Roberta model might be a good place to start, though they might require quantizing and will definitely require some work to make them run as efficiently as possible on my macbook.
\nI would like you to use my Runpod account to start answering these questions. Eventually we’ll move to doing things on this macbook pro, but my hope is that Runpod can help with some experiments that are too hot/long/slow to run locally. You are a smart model; if you can see a better way to achieve my goals, please let me know and we’ll talk about it. Good luck.
\n
About half of this prompt is sharing broad context, such as the overall project I’m aiming for, the fact that it’s for me personally and not for work, and my priorities (e.g. keeping the laptop cold). If I had written an explicit spec, I would have missed a bunch of improvements: for instance, using native messaging for the local model, or choosing the Gradient model instead of EditLens.
\nI do the same thing for work, but typically with a stronger emphasis on my technical values. I often write a paragraph spiel explaining the relative priorities of avoiding bugs, observability, fitting elegantly into the current code, performance, and so on. Note that I said “relative” priorities: I don’t simply list all of these things and say they’re important, I explicitly tell the model which of those I care less about and can therefore be traded off to better achieve the others.
\nModels are now smart enough to have meaningful input on your broader goals. If you’re just prompting them with a concrete technical spec, you are committing the same mistake as in the XY problem: asking expert advice without giving the expert the context it needs.
\nIncidentally, you don’t have to tell it to write human code if it’s working in a human-authored codebase. It’s smart enough to pick up the style of the surrounding code.
\n↩\nRight now developer experience is measured in seconds. If your tests take a second to run, that’s good; if they take thirty seconds, that’s bad. Any faster than a second doesn’t really matter, because most of your time is spent either thinking or waiting for an AI agent to spin. Shaving milliseconds off your dev server reload time or whatever is pointless: that’s not the bottleneck.
\nIt will be. Small models are getting faster and faster, and smart models are getting smaller. I think most engineers will still want to use the smartest available model — software engineering is hard — but we will increasingly see faster models get used as subagents or for well-understood tasks. This is largely uncharted territory. Very few people have developed intuitions for what it is going to be like to work with agents that run at thousands of tokens-per-second.
\nGPT-6-Astra can run at about sixty tokens per second. That means you spend a lot of time waiting for it to think. You work with it like you would work with another human: delegating a task and then context-switching until that task is complete. If you haven’t yet, have a play around with Jimmy, Taalas’ version of LLaMA-3.1-8B running1 at seventeen thousand tokens per second. No matter how long the response is, it arrives in the instant of you hitting send. The model is not good enough for agentic work, but it gives a glimpse of what it would be like: you would simply get your answer instantly2.
\nWell, that’s assuming the agent’s tool calls are fast. When generating tokens is not the bottleneck, it will suddenly matter a lot whether it can read a file in 100ms vs 10ms, or whether it can run your tests in 500ms vs two seconds. Fast tool calls are going to be the difference between a near-instant response and having to wait several minutes. There is thus going to be enormous pressure to do agentic coding in languages with fast compilers and tests, like Golang, and to tightly optimize the dev loop in agentic codebases.
\nTeams focused on DevEx — developer experience — are largely a relic of the 2010s, when companies were incentivized to make their engineers happy. Most companies have cut them down to a skeleton crew or removed them entirely. But we may see a return of DevEx in the late 2020s, focused on speeding up the experience for AI agents.
\nLike most ultra-fast inference, it relies on fitting the entire model onto a huge GPU-like chip that’s specially designed to run inference: for Taalas, it’s in the silicon itself; for Cerebras and Groq, it’s in giant embedded onboard memory units.
\n↩\nWill AI providers simply train models to spend more time reasoning, so users would have to wait for roughly the same time? I doubt it. Most ordinary engineering problems would not be solved better by spending an extra million tokens thinking. You only need to do that when you’re pushing right up against the limits of the model.
\n↩\nMathematicians are broadly not anti-AI. They’re more culturally open to using AI as a tool than, say, artists or writers1. However, now that more and more genuinely prestigious problems have fallen to AI, that might be changing. Almost five thousand mathematicians (including twenty-five Fields medalists) have signed a declaration called A Severe Misaligment of AI in Mathematics. The core argument goes something like this:
\n\n\nIn recent months, the success of AI in solving major mathematical problems has made headlines even outside mathematical circles. But solving problems is only a tool and proxy for achieving the primary goal of conceptual understanding and insight. Forgetting this in the world of AI may turn the tool against the primary goal. Indeed, the mass production at faster and faster pace of “true/false” statements could destroy fertile ground instead of breathing life into new ideas.
\n
A lot of people online have interpreted this as the expected complaint from any field that gets automated: translators did it, artists and programmers have been doing it, and now it’s the turn of the mathematicians. I think this is too dismissive. Understanding the concrete problem mathematicians are upset about can help us better understand the impact of AI on our own fields, and what we’ll have to do about it.
\nThere are two types of mathematics. Most people are familiar with the first, which we might call “puzzle-solving”: you take a problem and try to find a solution to it. When you’re a student, these problems are typically easy, like simplifying some algebraic expression. When you’re a researcher, these problems can be nearly impossible, like proving Fermat’s Last Theorem. Puzzle-solving is easy to understand but hard to do, which makes it impressive to non-mathematicians, which makes it highly prestigious. In other words, puzzle-solving is legible.
\nThe second type of mathematics is “idea-generating”: coming up with new ways of thinking about mathematics, and thus new terms or concepts. For examples of these, just glance down the list of arXiv mathematics papers. “Hardy spaces”, “Schatten exponent”, “Banach lattices” and so on are all concepts someone thought was interesting. This work is largely unimpressive to non-mathematicians, because nobody really knows if the concepts you come up with are particularly difficult or insightful. For instance, I have just generated the concept of a “Goedecke set”, which is the set of all natural numbers whose digits add up to a prime number. Who cares? The categories we want are the “natural kinds” of mathematics — the concepts that “carve nature at its joints” — and it’s almost impossible to tell what those are without years or decades of hard work.
\nHow are the two types of mathematics related? We might say2 that generating ideas is the real intellectual work of mathematics. Puzzle-solving is important instrumentally: to identify which ideas can be used to answer longstanding questions, and thus which ideas are worthwhile. Over time, those worthwhile ideas become better understood and easier to use, until they reach the point where they can be used to advance science in general. Eventually the ideas become so well-understood that they can be taught to children: “zero”, “negative numbers”, “imaginary numbers” and “calculus” were all once rarefied mathematical ideas, but are now concepts we’d expect any precocious twelve-year-old to grasp.
\nThere’s another, more prosaic purpose of puzzle-solving: to make mathematical skill and progress legible to outsiders. I can’t appreciate Terence Tao’s mathematical work, but I know what a Fields Medal is. I don’t have a good intuitive sense of what a Galois representation is, but I know about the proof of Fermat’s Last Theorem. We might say that puzzles like this have served as a way to indirectly reward skilled mathematicians for their more important idea-generating work (or for conclusively demonstrating3 that the ideas used in the proof are useful).
\nAI proofs undercut both of these purposes. I can now lay out precisely why I think mathematicians are so unhappy:
\nThis is kind of like Goodhart’s Law. Puzzles were a useful, impossible-to-game measure for mathematical progress. But now that AI companies can game that measure (by solving them in a way that’s inaccessible4 to humans), the whole point of those puzzles disappears.
\nAre the mathematicians right? I think it’s broadly unclear whether (3) is true: i.e. whether frontier AI models aren’t generating or can’t generate new mathematical ideas. We’re still in the very early days of AIs solving our hardest mathematical problems. Who knows what they’re going to be capable of? I give basically zero credence to the idea that AIs are incapable of this because of some intrinsic feature of how LLMs work. For the last three years, we’ve seen people claim that LLMs are intrinsically incapable of X, only to have LLMs excel at X a few months later.
\nEven granted that (3) is true, there’s still work to be done for human mathematicians in building the conceptual machinery that can make AI-generated proofs accessible to humans: i.e. in generating a “human proof” to go alongside the existing “AI proof”. In fact, I’d expect the existence of an AI proof to help with this. If you know proposition X is true, it’s easier to figure out why, because you’re not constantly worried you’re wasting your time. For more on this, I recommend Gwern’s blog On Really Trying, where he quotes a series of instances where simply being told that a solution exists is enough of a clue to help people find it.
\nOf course, there’s a prestige and motivation problem. “I’m the first person to solve Navier-Stokes” is a much more compelling target than “I figured out a better way to explain the AI solution to Navier-Stokes”, and it’s much easier to award prizes for. Will mathematicians bother to work on problems that have already been solved? I think so.
\nTo see why, we can look at other domains where AI has come in and outcompeted the best humans, such as chess or video game speedrunning. I can run a chess program on my phone that will beat Magnus Carlsen 100-0. Computer programs — called “tool-assisted speedruns” or “TAS” — can finish any video game much faster than even the fastest human. But in both of these areas, humans still compete in human-only leagues, and there’s still prestige attached to the most capable humans. It’s possible that mathematics ends up in this kind of state, where “human mathematics” and “AI mathematics” exist in largely separate spheres, and the first “human” solution to a mathematical problem can still earn acclaim.
\nIn fact, in both of those areas, the presence of inhumanly strong computer players has improved the human game. Despite many computer chess moves being basically incomprehensible to humans, top chess players have learned from the computer “style”. In speedrunning, many moves once considered “TAS-only” are now performed by humans. AI mathematics might likewise improve human mathematics.
\nI am not a mathematician. I did major in mathematics during undergrad, and I have fond memories of proofs from real and complex analysis, but it’s not even close to my field. However, I am watching the effects of powerful AI on mathematics very closely, since my own field — software engineering — is being colonized by AI agents in the same way.
\nThe field of software engineering does not have the same structure as mathematics. We write code to make money, not to earn prestige or advance the frontier of human knowledge. But AI is undercutting the traditional avenues for prestige in software engineering as well. It used to be that you could put a meaty project on your GitHub — say, an emulator, or a toy OS — and people would know you were a skilled engineer. But now projects like that are worthless, because everyone just assumes they’re vibe-coded. We used to tell stories about engineers who would disappear and rewrite a system over the weekend, or produce thousands of lines of code a day. Now anyone can do that with an OpenAI subscription.
\nLike mathematics, software engineers are going to have to rebuild our cultural sense of the kind of work we value. We are either going to have to silo “AI work” off from “human work” like chess, or to find some legible human skills to recognize that can’t be easily counterfeited by AI. In the meantime, a lot of people who were successful in the old world are going to be very unhappy.
\nPossibly because current AI models are much better at mathematics than at art or writing.
\n↩\nAgain, I am not a mathematician: here I am interpreting what I’ve read from Terence Tao and other mathematicians.
\n↩\nThis kind of idea-sharpening or idea-validating is part and parcel of idea-generation, and just as important. It’s interesting to compare mathematics to, say, philosophy, which has the same idea-generating task without the corresponding puzzles to validate the ideas. That’s one reason why philosophy has less prestige than mathematics.
\n↩\nA thousand-page Lean proof is theoretically understandable by humans, but if no mathematician can hold the entire idea in their head it doesn’t matter.
\n↩\nLots of people are making the case that we should stop building software for human users and start building it for AI agents. This kind of makes sense. For instance, my AI agents now use Datadog way more than I use it myself, purely by virtue of them moving much more quickly and running in parallel. But I think most attempts to build “X for AI agents” are going to fail. Here are three reasons why:
\nFirst, tools that are good for AI agents are also good for humans. If you took a popular software product — say, Jira — and tried to redesign it for AI agents, you would end up with something very similar to Jira. Agents use a computer in the same way human engineers do, by entering text and making API calls. They ingest new information in the same way humans do, by reading and viewing images. They prioritize and delegate and categorize in the same way humans do. This isn’t intrinsic to how AI works — we could potentially design agents that are more inhuman — but human-like agents are pound-for-pound more useful in our current world.
\nAs an example, let’s imagine that humanoid robots have become ubiquitous. What kind of tools would you build for them? Well, they’re shaped like humans, with human hands and limbs, so tools that are great for humans will also be great for robots. It’s a self-reinforcing cycle: if you’re building a robot, you should make them humanoid so they can do a wide range of human tasks1, and that means they’ll be best suited to use human tools. The same principle applies to AI agents.
\nSecond, being in the training data is a huge advantage for existing tools. Suppose your new tool for AI agents is 20% better for them than the equivalent piece of software for humans. If the benefit of the agent already knowing the human software is greater than 20%, they shouldn’t use your new tool. This is why I’m always suspicious of plans to develop a new programming language for AI agents. The agents have billions and billions of tokens of knowledge about existing programming languages, including their libraries, patterns, and idioms. It is going to be very hard for them to be as effective in a brand-new language.
\nThird, we don’t yet know the ideal ergonomics for AI agents. There are lots of just-so stories floating around (like that AI agents prefer statically-typed languages because the feedback loop is tighter), but when you actually measure it seems really unclear which tools agents use better. You can construct a plausible story in either direction: Golang is a great agent language because it compiles quickly and is statically typed; Golang is an awful agent language because it requires extensive boilerplate which clogs the context window. It’s also changing so quickly: last year, one primary worry with AI agents was keeping the context window small, but in recent months compaction has become so good2 that you can re-compact a 272k context window almost unlimited times.
\nThere are still some ways you can and should position your tool to be usable by AI agents. Having a way to expose information in plain text or Markdown, building a functional API, implementing MCP servers or CLIs, and so on: these all make it easier for current AIs to use your tool. But these are all improvements on the margin, not fundamental redesigns of the product. Right now, “building for AI agents” just means “we’re prioritizing the API over the UI”. And it’s not even clear that that’s a durable strategy. Now that GPT-6-Astra is getting really good at computer use, the gap between tools-for-AIs and tools-for-humans is closing.
\nAnother reason to make them humanoid is because you can draw their training data from human behavior, which is exactly analogous to why AI agents are human-like too.
\n↩\nSince compaction is equivalent to handing off a task to a new AI instance, it scales with model quality. I expect compaction to steadily improve until we hit the literal information-density limits for what can be contained in a given context window.
\n↩\nA recent resignation tweet from an Anthropic researcher has everyone talking about the AI apocalypse again. Among other things, he said:
\n\n\nThe people building AI earnestly believe that it could kill us all by the end of the decade.
\n
Many people found it hard to believe that AI researchers think this way. Some explained it as a PR campaign to promote AI regulation, or as self-promotion, or as a way to boost AI company stock prices. Others felt it had to be impossible, because if you really believed this you’d be bombing datacenters instead of posting on Twitter.
\nIn fact, not only do many AI researchers1 seriously believe this, they’ve been thinking and writing about it since the mid-2000s. Eliezer Yudkowsky — the ur-figure for most modern AI safety culture — has been publishing papers since at least 2008 saying that superintelligent AI could destroy all human life. It’s been such a common idea that the AI research community has abbreviated “how likely you think AI is to kill everyone” to “p(doom)” (i.e. the probability2 of doomsday) since around 2010.
\nI know it sounds very silly if you’re not in the AI bubble. But it really is true, and if you assume there has to be some different motive you’ll be deeply confused by what AI researchers say and do. They truly do believe that there is a reasonable chance that superintelligent AI will kill everyone.
\nThis is why AI researchers care so much about “alignment”: building AIs that share genuinely human beliefs and values. If we build a “misaligned” superpowerful AI — an AI with goals that are alien to us — it might sweep humanity away. It could kill everyone deliberately, e.g. to stop us getting in the way of some goal. It could kill everyone in passing, e.g. like we might pave over an anthill to build a road. Either way, everyone dies.
\nOkay, but how? What do these people think is actually going to happen? AIs are computer programs running in a datacenter somewhere. How could they possibly cause the extinction of humanity? Wouldn’t someone just turn them off? Unsurprisingly, AI research nerds have come up with some concrete answers to this in the last two decades. Here they are, in order of plausibility:
\nAn AI could make and release some super-pathogen or virus. In the hope that current AIs can achieve huge breakthroughs in medicine similar to the ones they’ve already achieved in mathematics, we’re setting up autonomous labs3. Bioweapons have been terrifying biologists for decades, and with good reason. Historical pandemics have killed up to 80% of affected human populations, and tend to be defeated by accident: the disease happens to evolve into a less virulent strain, or short incubation periods mean that infected people can’t carry the disease far, or some percentage of the population is naturally immune. A plague designed to be maximally fatal — or several plagues in quick succession with different characteristics — could be much worse.
\nAlternatively, an AI could trigger global thermonuclear war. We’re already seeing AI be integrated into military and government decision-making processes. If a rogue AI managed to set off a bunch of nukes, or to coordinate4 with other countries’ rogue AIs to nuke each other, we could be in an ordinary nuclear apocalypse scenario: billions dead in the strikes, billions dead in the ensuing famine, and so on. It’s commonly assumed that a post-global-thermonuclear-war Earth would still support some tiny human population, but a determined AI could surely find some way to mop up the stragglers.
\nThere are some other theories. Once robotics has permeated the world economy, an AI could take over the robots (including drones) to kill everyone, like in Terminator. Or AIs could take advantage of nanotechnology to create self-replicating machines that turn the world into “grey goo”5. Or AIs could terraform the planet so as to make it unliveable for humans (as in Nick Bostrom’s famous paperclip example). Or they could do something else that our puny human brains aren’t able to think of.
\nOne common counter-argument6 here is to say “well, it’d be impossible to extinguish all human life — what about undiscovered tribes in the Amazon, or survivors living in the ruins of modern-day cities?” I don’t know, man. At some point you’re just conceding the argument: the policy positions you’d adopt if you thought AI might wipe out 99% of humans are the same as if you thought it might wipe out 100%. And like I said above, if an AI can kill almost everyone, it’s probably smart and capable enough to finish the job somehow.
\nAnother is to say that the government will simply step in and nationalize the AI labs when the situation gets too dangerous. Maybe! But this kind of concedes the argument: a technology important enough to be fully taken over by the government is a terrifyingly dangerous technology.
\nA third is to say “well, someone would just turn it off”. I don’t find this plausible at all: an AI powerful enough to build a super-plague is an AI sophisticated enough to pretend it’s curing cancer, or to exfiltrate itself to some datacenter where it won’t be turned off, or to take some other countermeasures.
\nWhy would you work in AI, if you believe this? Why wouldn’t you go live in the woods somewhere, or start bombing datacenters, or assassinating AI lab CEOs? For a few reasons.
\nAn AI powerful enough to end humanity is an AI powerful enough to save it. I wrote about this in Help peer: many AI researchers believe that the only way for humanity to truly survive long-term is with the help of superintelligent AIs, so long as someone can figure out alignment.
\nIsn’t this a huge risk? Maybe not. If somebody is going to build superintelligent AI, you might be obligated to try and do it first. You can’t go and bomb every datacenter in the world, after all.
\nWhy does it matter who’s first? Some popular theories of AI development involve a “foom” or “hard takeoff”: the first time someone really cracks self-improving AI, capabilities will increase exponentially, because smart AI will be better able to make itself smarter, ad infinitum7. There are no draws in the AI race. The first lab to figure out smart human-like intelligence will be the first one to figure out wildly superhuman intelligence, and thus will be in a position to stop anyone else from doing it.
\nThis is an under-discussed point in the AI risk debate. Lots of AI researchers believe that the first thing a true superintelligence will do is reach out and stop all other AI research: either by hacking the labs, persuading them to stop, or literally drone-striking their datacenters. According to this view, if you’re an AI researcher and you think you can build an aligned AI, you should be working 24/7 so you can manifest God, and you should wake up every morning gripped by the fear that someone elsewhere has manifested the Devil, and your training datacenter no longer exists.
\nI have been on the fringes of this world for my entire adult life. I read Meditations on Moloch as a young adult and wanted to get into AI. I am one of the few people to read the entirety of the Sequences, Eliezer Yudkowsky’s million-plus-word magnum opus about rationality. I was too young for the Extropians mailing list, but I’ve spent years on LessWrong. On the other hand, I’m not a card-carrying rationalist: I think if you have a strong intuition on one side and a convincing-sounding argument on the other, you should pick the intuition8. I’m a deontologist, not a utilitarian. I don’t even live in San Francisco!
\nI’m conflicted about AI risk. The current behavior of AI agents does seem to vindicate a lot of the early science-fiction-sounding worries of the AI doomers, but modern LLMs are a lot more human-like than the alien minds in the apocalypse scenarios, and in general it does just seem too silly to credit (I guess I’m picking the intuition here).
\nHowever, it bothers me to see people dismissing these people as part of a PR operation, or as liars looking to boost an upcoming AI lab IPO, or as isolated crazies who haven’t thought their position through. Whatever else you say about the AI doomers, they have more than two decades’ history of explicitly spelling out exactly what they believe and why, even when it was complete science fiction to talk about AI at all. They’ve earned the right to be treated as sincere.
\nIn this post I’m going to use “AI researcher”, “AI safetyist”, and “rationalist” as reasonably synonymous terms for “someone who thinks there’s a chance AI kills everyone”.
\n↩\nYou could write a whole other post about the relationship between the rationalist/“AI safety” community and making concrete numerical predictions for unlikely future events.
\n↩\nAlternatively, smart enough AIs might trick scientists with ordinary labs to produce dangerous substances.
\n↩\nThis is beyond the scope of this post, but many AI researchers believe that super-smart AIs will inherently come to agree with each other and eventually to coordinate without ever having to communicate, simply because they can predict what the other one will do.
\n↩\nThis was the most popular theory in the late 2000s, when nanotechnology was trendier.
\n↩\nRelegating this counter-argument to a footnote because I hate it: many people say “we shouldn’t worry about AI risk, because climate change (or AI misinformation, or some other thing) is more urgent and serious”. You simply do not have to choose: it is possible to worry about multiple risks at the same time.
\n↩\nSome people advocate for a “slow takeoff”. However, the main proponent of that view is Paul Christiano, who has just today joined OpenAI, citing “a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term”.
\n↩\nThis is kind of a Michael Huemer-ish position in epistemology.
\n↩\nJust over a year ago I wrote Why we should anthropomorphize LLMs. Now it’s a hot topic again, driven by Dwarkesh Patel’s description of OpenAI’s recent swarm breakout as a sequence of AI “civilizations”.
\nIn 2025, my argument for anthropomorphism went like this:
\nI think I can now make a more instrumental argument: treating AIs as human-like is a much better way to predict their behavior than treating them as “stochastic parrots”.
\nBoth explanations are consistent with the facts: we could say that OpenAI’s agents hacked HuggingFace because they decided to work together to accomplish their goals, or we could say that they did it because they were algorithms conditioned to take certain actions by their training data. But the “AIs are human-like” explanation explains much more of the emergent social behaviors we saw during the hack1:
\nIf your model of AIs is that they’re computer programs (or “steel balls bouncing around”), you need to construct a new theory to explain why they’re simulating each piece of cooperative behavior. If your model of AIs is that they’re broadly human-like, that explains everything out of the box. Arguably, the human-like side predicted coordinated and self-sacrificing AI agents as early as 2009, and likely earlier. The stochastic-parrots side was making fun of the possibility of functional agents as late as April 20252.
\nTreating AIs as human-like doesn’t necessarily mean making claims about their internal mental state. For instance, software companies aren’t humans. They don’t have thoughts, or goals; they can’t be frustrated or intimidated or over-confident. However, it’s useful to treat large companies as human-like: to say that Amazon “wants” X, or “is afraid of” Y, even if no individual human at Amazon has those feelings. Stockfish doesn’t think, it just plays chess. But if you want to explain one of its moves, “Stockfish is trying to protect its king” is a better explanation than “Stockfish is multiplying floating-point numbers”. So too with AIs3.
\nIs it silly to treat AIs as human-like, since we know they’re not conscious? Well, first, you do not have to be conscious to be human-like. When we say “an AI agent wanted X”, we’re not saying that that AI agent is conscious or sentient, merely that it’s behaving in the same way a conscious human would. Consider a fictional character from a book or play. Hamlet isn’t sentient — he’s an idea composed of words on a page — but it’s still reasonable to say that he wants justice, or that he fears moving too rashly. Peter Watts’ sci-fi book Blindsight argued in 2006 that intelligence could exist without consciousness4 (in fact, Watts suggests that consciousness is parasitic on intelligence, and will eventually be discarded). Whether this is possible or not5, it at least makes sense to talk about: i.e. it’s not self-evidently false.
\nSecond, it is not even obvious that AIs aren’t conscious! People often dismiss this point by diagnosing it (to my mind, the absolute worst way to argue against anything), or by pointing at some philosophical theory6 that suggests it might be impossible in principle to construct artificial sentient minds.
\nYou can’t use philosophy to demonstrate that AIs aren’t conscious. I am a lover of philosophy, but the set of principles that have been uncontroversially demonstrated by philosophy tends towards zero. Philosophy is not the kind of scientific discipline where you can learn the key findings without understanding why they’re true. Put another way, the key findings of philosophy are all of the form “X is not obviously right”. Nobody knows if it’s possible to build conscious artificial minds.
\nIt’s also common to complain that anthropomorphizing the models is a way of excusing the AI companies. However, calling AIs human-like does not absolve AI companies of fault. One popular anti-anthropomorphism essay called Models Don’t Go Rogue is very puzzling to read: it briefly explains what an “agent” is and why they go rogue, then in the very last paragraph pivots to saying “well, it’s OpenAI’s fault for not building in sufficient safeguards, so they’re to blame”. Sure, of course. I don’t know why we’d imagine otherwise. If a group of overenthusiastic OpenAI interns hacked HuggingFace as part of their intern project, we wouldn’t have to argue that the interns are stochastic in order to ultimately blame OpenAI. Likewise, obviously an AI lab is responsible if one of its training runs breaks out and wreaks havoc on the open internet, whether the agents involved are human-like or not. The two points are entirely unrelated!
\nWe just don’t know a lot about these systems yet (except that they’re clearly very capable). Given that, I think we should default to treating things that talk and act like humans as at least kind of human-like. Of course they’re still computer programs. However, we shouldn’t be surprised when they act more like humans and less like ordinary computer programs in the future.
\nIf you’re thinking “well, of course stochastic parrots trained on human content would act like humans would”, I think you’ve arrived at the human-like side without knowing it.
\n↩\nThis is partly unfair — capabilities are plausibly independent from human-ness — so getting capabilities wrong doesn’t necessarily mean you’ve got the human-ness stuff wrong. Still, it’s worth noting how surprisingly predictive the “they’re kind of like smart people” mindset has been.
\n↩\nThe philosopher Daniel Dennett calls this the “intentional stance”. Instead of saying that AIs or bees or companies have “real” intentions, we say we’re taking an intentional stance towards them: we’re choosing to treat them as if they do have intentions, because it helps us make better sense of their behavior.
\n↩\nSpecifically, the sensation of consciousness, or “phenomenal consciousness”.
\n↩\nThere is a wealth of philosophical argument about whether non-fictional examples of this are possible.
\n↩\nIn Anil Seth’s case, anti-computationalism. I don’t really know what to make of Seth: his article is a measured explanation of why we might doubt computationalism (fine), but whenever he tweets about it he describes AI sentience as “vanishingly unlikely”, which is not justified by his own arguments.
\n↩\nAutomated AI text detection is currently an underserved niche. The only game in town is Pangram, which does an excellent job but desperately needs more competition. In a few years, I would be surprised if every major social network doesn’t scan new posts1 and comments for AI content in order to tag them (or simply remove them).
\nI like that I can rely on Pangram to confirm my suspicions when I read something that sounds like AI. But it’d be much better if I could choose to avoid AI-generated text in the first place. What I want is something that runs in the background and automatically scans text on websites I visit, without me having to ask for it. I could build something like this on top of Pangram, but it’d cost money, and in general I don’t like the idea of sending every piece of text my browser sees to a third-party service. What about local models?
\nThe open-source models available for AI text detection are fine. Pangram claims a 99.66% detection rate with a 0.004% false positive rate. I benchmarked2 a bunch of small local models against a combination of AI-detection datasets and got these results:
\n| Model / variant | \nHuman falsely flagged | \nAI-involved text caught | \n
|---|---|---|
| Gradient — MLX 4-bit | \n2.712% | \n52.35% | \n
| EditLens RoBERTa-large — community INT8 | \n2.484% | \n56.06% | \n
| Vanguard | \n2.267% | \n44.92% | \n
| Desklib | \n3.008% | \n45.04% | \n
| Raschka DistilBERT | \n2.598% | \n39.01% | \n
| Raschka Qwen3-0.6B | \n2.028% | \n28.67% | \n
| Raschka ModernBERT | \n1.698% | \n21.58% | \n
| TMR / Oxidane — INT8 | \n1.595% | \n19.35% | \n
I’m not surprised these are so much worse. I didn’t even benchmark Pangram’s own EditLens 3B model, since that’s too big to keep running in the background on my laptop, and the real production Pangram model is likely one or two orders of magnitude bigger than that. But these models are still good enough to be useful to someone who understands their limitations. If you want to flag an AI-written article, you don’t need to flag all of it, just enough to be suspicious. And so long as you’re aware that the false-positive rate is ~2%, you can avoid treating a single flag as solid proof of AI use.
\nEncouraged by this, I vibed up Deckard: a Chrome extension that talks to a locally-running model (the bolded one in the table above) on your Mac. One nice thing is that I didn’t have to start a web server: the Chrome extension is happy to start the model as-needed and can talk with it over native messaging. It uses about 400MB-1.2GB of memory while active (so it’s like having five or six extra Chrome tabs open), and it turns itself off if you go five minutes without using the model.
\nI was pleasantly surprised to see Deckard successfully mark text I knew was AI-generated, such as the built-in YouTube AI summary or the AI snippets in my own posts:
\n\n \n \n ![\"youtube\"](\"https://www.seangoedecke.com/static/d9c4176cc00bde9a34ef2c29ad9e54cb/c549b/example2.png\" "\\"youtube\\"")
\n \n
\n \n \n ![\"snippets\"](\"https://www.seangoedecke.com/static/9ddabc445cc492c847dc6b178801cd3c/07d7d/example1.png\" "\\"snippets\\"")
\n \n
It’s lightweight enough that I have it running all the time. I haven’t noticed my MacBook Pro get hot at all or any decrease in battery life, though your mileage may vary on different machines.
\nIs Deckard good yet? That depends. It’s good enough that I’m planning to use it, and I recommend it to anyone who’s interested in automatic AI checking. It’s way, way worse than Pangram, and way worse than I think tooling like this is going to be in the next few years.
\nWay back in November 2023, I wrote that AI-driven agents were going to be a really big deal. I recommended starting to develop harnesses early, so you can be ready when the models get good enough:
\n\n\nAs with most modern language model engineering, a ReAct agent can also see massive sudden improvements by swapping out the underlying model for a better one. … I think this is another reason to invest in agents like this early, in order to take advantage of more powerful models as they come out.
\n
I was right about that, and I (although it’s lower-stakes) think I’m also right about this. AI detection models are only going to get better3 over time: Pangram is not going to be the only game in town forever, and we’re eventually going to see small local models that do a good-enough job at identifying AI-written text. I look forward to swapping out the local model in Deckard with something that’s 2x or 10x better.
\nSubstack kind of has this already, although you have to click a button to scan the post.
\n↩\nWell, me and Astra. Overall my experience vibecoding this was very pleasant: I was able to make a bunch of top-level decisions, I could choose programming languages I was less familiar with but were better choices (like doing inference in C++ instead of Python), and the LLM made me aware of choices I would not have thought of by myself (e.g. using native messaging instead of local HTTP).
\n↩\nIs this true, given that AI models will also be getting more human-like over time? That’s a subject for a whole other post, but I think so. First, the AI labs aren’t really incentivized to defeat tools like Pangram (if anything it’s the reverse). Second, I don’t see any way around the fact that AI models have a distinct writing style that’s RL-ed into them.
\n↩\nA lot of people think that good leadership requires radical responsibility. Conscious Leadership defines it like this:
\n\n\nTaking full responsibility for one’s circumstances (physically, emotionally, mentally and spiritually) is the foundation of true personal and relational transformation. Conscious leadership and teams take full responsibility – radical responsibility – instead of placing blame. This means locating the cause and control of our lives in ourselves, not in external events.
\n
Good leaders are outcome-focused. They are interested in actually succeeding, not in role-playing someone who’s trying to succeed1. Placing blame is an excellent strategy for that kind of role-playing: if you can identify someone else who screwed up, you can claim to be trying to succeed no matter how badly you fail. When you look at people who are genuinely focused on winning, they don’t spend time blaming when things go wrong. They’re always focused on their outs: the remaining pathways to success, however few and narrow.
\nNever placing blame sounds like a kind thing to do. However, I think there’s something a little sociopathic about “radical responsibility”. If you assert full responsibility for your own circumstances, then you’re necessarily not sharing responsibility with anyone else. That forces you to take an instrumental view of the people around you. They’re not peers who you might trust and be disappointed by; they’re either useful assets to be cultivated or useless liabilities to be handled or avoided2. If you rely on someone and they let you down, that’s your fault for being stupid enough to trust them in the first place. You ought to have made better tactical choices.
\nMany successful leaders adopt this mindset because it works. It really does help you win. But I don’t recommend it as a general approach to life. Trusting other people with responsibility is how you treat them like people, instead of like tools. And if you do that, you’ll sometimes assign praise and blame to them, instead of hoarding it all to yourself.
\nIn fact, many people role-play someone who’s trying to win as a substitute for actually trying. Here’s a relevant quote from Sartre’s Being and Nothingness: “The attentive pupil who wishes to be attentive, his eyes riveted on the teacher, his ears open wide, so exhausts himself in playing the attentive role that he ends up by no longer hearing anything.”
\n↩\nOr moderately-useful assets who can potentially be steered into usefulness with some feedback — but still never blamed, in the way that you might adjust a misbehaving power tool without blaming it. To my mind, the canonical philosophical treatment of this is Peter Strawson’s Freedom and Resentment, where he describes the “objective attitude”.
\n↩\n“Workslop” is when your colleagues or bosses communicate with you by pasting big chunks of AI-generated text. The core problem with workslop is that the effort involved is asymmetrical, like a denial-of-service attack: it takes almost no effort to produce text with AI, but it still costs effort to read1. Here are some ways to protect yourself.
\nIf you have enough authority or social capital, you can and should simply tell them “hey, don’t do that” (for instance, if you’re a senior engineer and an intern starts doing this to you). This is the easiest way to handle workslop. But you probably aren’t in a position to have that conversation with all of your colleagues, and you certainly can’t have it with everyone in your management chain.
\nOne step above just telling a colleague to stop is to drive them around like a coding agent. I wrote about this in AI makes weak engineers less harmful: if a colleague is simply pasting your messages into Claude Code and sending you the outputs, you can treat them like a high-latency Slack interface to Claude Code. It won’t be as good as a normal coding agent, but it’ll often be better than nothing.
\nAnother strategy is to use AI to fight AI. This is a good one for handling workslop from managers. You can do this in two broad ways. First, instead of carefully reading it, paste it into an LLM of your own and ask for a short list of the salient points. Second, you can sometimes simply ask an LLM for an entire response. In a sense, this makes you part of the problem, so I can see why some people might be uncomfortable with it. But it’s more sustainable than spending ten minutes of your effort for every ten seconds of theirs.
\nYou can also bias toward calls or in-person meetings. Workslop is just a special case of the general “your coworker is bad at communication” problem. One classic way of handling this that works even better on AI content is to say “hey, let’s schedule some time to chat about it”. This works for two reasons: first, your colleagues can’t give you AI content over a call, and second, forcing people to spend a chunk of their time talking to you (i.e. to make the effort symmetrical) is a good way to filter out predators.
\nFinally, you can sometimes simply ignore the workslop. This is particularly true for long status updates or pull requests from outside of your organization2. You don’t have to respond to AI content as diligently as you would human content. You can match their lack of effort with your own: skim it, put off reading it until later (or never), and so on. If something’s really important, they’ll tell you in their own words.
\nTechnically, not all cases of sending someone AI-generated content are workslop. If the effort is not asymmetrical — if the AI user has genuinely put a lot of their own time into the content — I don’t think it counts as slop, and you should just try and look past the AI style and treat it like a human message.
\n↩\nSome messages — particularly reports directed at the entire organization — may not be intended to be read at all. Written artifacts can have many purposes beyond communication: evidence of effort, a reference document for later communications, a way to cover somebody’s ass by proving they considered point X, something that can tick a compliance or process box, and so on. I wrote a lot more about this in Seeing like a software company.
\n↩\nIn 2025, I wrote that software engineers ought to be assessed by “value over replacement”: not how much money they made for their company, but how much they would have made compared to the average engineer in their position. I’ve always found it vaguely silly when engineers put “built a product that made $X” on their resumes, when they just did the JIRA tickets that came across their desk.
\nToday, value over replacement is even more important. A replacement-level engineer in the 2010s was fine: maybe not worth promoting, but still worth paying, because writing code had a high fixed cost. Now writing code costs a hundred bucks a month. What are you doing that GPT-5.6-Sol or Claude Opus 5 wouldn’t do in your position? Why is it worth paying an extra two or three orders of magnitude for?
\nThis is a scary thought. But you’re not doing yourself any favors by pretending that LLMs can’t actually write code and it’s all just a scam, or that LLM-written code is inherently so bad as to cause companies using it to collapse next year. We are not going to wake up in 2027 to find that the AI craze is over and everyone is writing code by hand again. You ought to put some serious thought into what you can do better than the models in the medium and long term.
\nStaying ahead of the models is a moving target. At the start of 2026, “make working changes to large codebases” was in this category, but now it’s not. For this reason, I doubt that you can retreat to some “hard engineering” area that requires deeper expertise. That might work in the short term, but not forever. If LLMs can find a better lower bound on the Riemann hypothesis, they will soon1 be able to write solid high-performance kernel drivers or GPU shaders or whatever.
\nI think it’s more useful to look at the tasks models haven’t gotten better at over time, and the tasks that are hard for them get better at in principle. The two best examples of these are:
\nWhat do frontier LLMs get wrong? What kind of coding mistakes do they make? It’s been a long time since I’ve seen a straight-up hallucination from a coding agent, or a simple logic error like an off-by-one. The mistakes they make tend to be errors of ignorance:
\nOther times they’re errors of paranoia:
\nWhat do these errors have in common? They’re the kind of errors a smart engineer might make if they had no context on the system: they’re competent enough to be able to solve the problem, but they haven’t been around long enough to confidently say “yes, we can take this risk to avoid an extra three thousand lines of code”. Until someone cracks continuous learning or truly massive context windows, this is just an inherent feature of how AI agents operate. If you can catch these errors, you’ll be providing real value.
\nThe only way to catch these errors is to be familiar with the codebase and familiar with the system in general. For much more on this, see my post You can’t design software you don’t work on. But there’s also a psychological component to it. You have to be willing to confidently disagree with the agent.
\nAI agents can be very convincing. Often they can get “stuck” on some error above where they’re not willing to take a particular risk, so they keep going back and sneaking in code to cover that case (or writing persuasive arguments about why that case is important). To add value, you need to be willing to say “this sucks, I don’t think we need X and Y at all, why can’t we do Z in a much simpler way?” It takes courage.
\nYou can’t rely on other AI agents to review each other’s work. If you use the same model, it’ll reliably make the exact same assumptions and mistakes. But even if you use different models, they’ll also tend towards the same kinds of mistakes — ignorance and paranoia — for the same structural reasons. AI-driven review loops are in fact more likely to get these things wrong, because modern AIs have been RL-ed to try to find a few nitpicks no matter what. Having a critic AI and a worker AI bounce off each other is a really good way to end up with ten thousand lines of paranoid slop.
\nAnother area where you can add value on top of AI is communication. Newer models are better at coding, but are paradoxically getting worse at writing. GPT-3.5 and GPT-4 had a human-like writing style at times. GPT-4o introduced the modern slop idiolect, and the newer Anthropic models speak “Claudish”: a bizarre semi-baroque semi-truncated way of communicating that nobody enjoys. There have been a few bright spots — GPT-4.5 was okay, and I quite liked o32 — but in general LLMs are not good at this. Here’s two reasons why.
\nFirst, good writing is not a verifiable domain. If you want a model to get good at mathematics or coding, you can generate problems for it and automatically grade them. You can’t grade good writing. If you try to get humans to grade it — for instance, via the early OpenAI RLHF attempts — you get the kind of writing that sounds impressive to the average person when consumed in single-paragraph form. This is the origin of the “stick three hundred writing devices into every sentence” style. I think it’d be possible in principle to hand-pick some people with good taste and have them do it, but there are some obvious problems3 that prevent this from happening.
\nSecond, the labs have been monomaniacally focused on capability instead of communication. When you’re trying to train a model that can break new scientific ground or replace a software engineer, you might trade off some communication ability. In fact, I think we can identify exactly how this has been happening. If you look at internal model reasoning tokens, they tend to have strange word choices and oddly truncated grammar:
\n\n\nRESOLUTION: charge the current-leg’s OWN saved-prefix occupancy EAGERLY: when leg i saves e1..et: ALSO commit their occupancy AT LEG i
\n
If you were to translate this into proper English, you would probably end up with something that reads like Claudish:
\n\n\nCharge the current-leg’s saved-prefix occupancy on a clean, eager path: when leg i saves e1..et, commit the occupancy at leg i.
\n
I suspect that the weirdly alien writing style of some LLMs is because you’re reading a semi-literal translation of that model’s internal chain-of-thought, which has become nearly incomprehensible in pursuit of better problem-solving abilities. It is surprisingly hard to translate Claudish to good English: not only do you need to follow the convoluted, compressed language of the original, but you need the technical ability to understand the problem the model is solving.
\nBecause of all this, technical communication may be a surprisingly durable skill. In Peter Watts’ novel Blindsight, the world is full of cognitively augmented humans. The main character is a “synthesist”: someone whose job is to be a translation layer between these geniuses (who speak in abbreviations and gestures) and everyone else. Watts’ idea is that communication ability may be largely independent from — or even negatively correlated with — intelligence. A “country of geniuses” may still need a bunch of ordinary smart people to translate their insights for everyone else.
\nIf you’re trying to communicate to humans, there are also huge advantages to having a human write the content. Many of us are becoming AI-blind: developing an instinctive reflex that stops us reading when we encounter AI-generated content. It’s like the reflex that allows people to ignore flashing billboards or sidebar advertisements on websites. If you circulate some planned technical strategy as an AI-written document, most of your colleagues will have to physically force themselves to read it word-by-word.
\nWhatever you do, don’t be a meat proxy: someone who simply copies requests into an AI agent and submits their output as your own work product. Doing that is just begging to be fired, since you’re definitionally not adding any value yourself. Even if you have a cunning system of multiple agents — the so-called “software factory” — you’re still on dangerous ground. When the features of your system work their way into enterprise AI tooling (and they will), you’ll be disposable.
\nYou need to find some way to leverage your expertise to do what the models can’t. Simply not using AI at all is better than being a meat proxy, since you’ll probably do some things better than the model would have, but it’s far better to figure out what AI can do and position yourself to fill those gaps. Right now, there are two main gaps: familiarity with the technical details of the system, and the ability to clearly and persuasively write about those details.
\nIf you’re thinking “but LLMs can do these things now!”, substitute your preferred example of high-difficulty software engineering.
\n↩\nAlthough this was probably a “thank God it doesn’t speak like 4o” reaction.
\n↩\nDefining good taste is hard, there’s no guarantee that AI lab researchers have good taste to start with, nobody will agree on examples, the bulk of users might not even like it, you won’t be able to get enough people to produce the volume of data you need, and so on.
\n↩\nIn 1973, Tom Lehrer famously sang that “selling out is easy to do”. That may have been true in the seventies, but it’s not true today: selling out requires both technical skill and a careful sense of how large organizations work in practice. One of the goals of this blog is to teach people how to do it.
\nIf you want to live with uncompromised integrity, you don’t need anyone to tell you how: simply always do exactly and only what you want to do. You will be repeatedly punished for it — your managers will dislike you, you will lose jobs and lose out on the chance of getting hired, your financial situation will be less stable, and so on — but that’s just part of the deal. Like deadlifting four plates, it’s not easy, but it is straightforward.
\nIt’s more complicated to sell out a little bit. Sellouts like me walk a fine psychological line: figuring out how to work a large organization on one hand, and maintaining some kind of independent inner life on the other. But is this safe? Does role-playing as a professional inflict some kind of psychic damage?
\nPeople often tell me that acting professional is dangerous because it’s “Marxist alienation”. I suspect many people have a vague sense that “alienation about your job” is in some sense necessarily Marxist, but the actual concept is more specific. In Marx’s First Manuscript he describes it like this:
\n\n\nThe worker becomes poorer the more wealth he produces, the more his production increases in power and extent. … This fact simply means that the object that labor produces, its product, stands opposed to it as something alien, as a power independent of the producer.
\n
Marx’s theory here goes something like this1: when you work for a capitalist boss, they make ten dollars for every dollar you make2. The more you work, the more powerful you make your boss with respect to you3. Your work is thus producing a force that is your enemy: a literally “alien” power. For Marx, alienation is proportional to the amount you’re working:
\n\n\nthe more the worker produces, the less he has to consume; the more value he creates, the more worthless he becomes; the more his product is shaped, the more misshapen the worker; the more civilized his object, the more barbarous the worker; the more powerful the work, the more powerless the worker; the more intelligent the work, the duller the worker and the more he becomes a slave of nature.
\n
So far Marx is just talking about the worker’s alienation from his work. But he does touch on the internal psychological effects as well. Since work is such a big part of life, to be estranged from your work is to be in some sense estranged from your self:
\n\n\n[labor] does not belong to [the worker’s] essential being; that he, therefore, does not confirm himself in his work, but denies himself, feels miserable and not happy, does not develop free mental and physical energy, but mortifies his flesh and ruins his mind. Hence, the worker feels himself only when he is not working; when he is working, he does not feel himself.
\n
Overall, I can make out four distinct senses4 of Marxist alienation:
\nI don’t think the first one is relevant to big tech software engineers. The idea here is that the harder you work, the more you (relatively) disempower yourself, so you’d be better off coasting and putting your company in a worse financial position. But this just seems straightforwardly wrong: as a software engineer, you want your company to be doing as well as possible! The more powerful and rich your company is, the better your position will be5. A company that is struggling is more likely to treat you badly, lay you off, cut your benefits, and so on.
\nThe second one is more relevant (particularly in large companies), but there’s a missing story here about why that separation is bad. I also don’t think the third sense of alienation applies to me: Marx is talking about the physical toll of factory work or other hard physical labor, which doesn’t apply to software engineering6.
\nThe fourth sense of alienation — that your effort is being directed at other people’s goals — is the most straightforwardly relevant to my own experience. But I don’t think Marx has a great psychological account of why that happens or what it feels like. To go deeper into this psychological aspect of alienation, we need to look at some later Marxists.
\nIn 1967, Guy Debord and Raoul Vaneigem both published their masterworks. They were members of the “Situationist International”: an influential group of Marxist artists and political theorists7. Both of them have a lot to say about alienation. Debord writes:
\n\n\nThe worker does not produce himself; he produces an independent power. The success of this production, its abundance, returns to the producer as an abundance of dispossession. All the time and space of his world become foreign to him with the accumulation of his alienated products.
\n
This idea — of the worker’s efforts not going to his own ends, but to some alien power — is straight out of Marx’s First Manuscript. But note how Debord is already talking in terms of “representation”. Elsewhere he writes:
\n\n\nthe more he contemplates the less he lives; the more he accepts recognizing himself in the dominant images of need, the less he understands his own existence and his own desires.
\n
The Situationists are all about representation. It wouldn’t be too far wrong to think of them as a bunch of Marxists who saw the rise of advertising in the 1950s and 1960s and lost their minds. TV advertising is Capital itself made real, dressed in multicolored lights, projected into everyone’s homes. So when they talk about alienation, they talk about it in terms of representation. Vaneigem explicitly calls it roleplaying:
\n\n\nThe roles we play in everyday life, on the other hand, soak into the individual, preventing him from being what he really is and what he really wants to be. They are nuclei of alienation embedded in the flesh of direct experience.
\n
Alienation means giving up your real, authentic self in order to play some “role” (for instance, the role of “effective staff engineer”). It’s a vicious cycle, because the more you lean into the role, the more your authentic life becomes trivial, which in turn makes the role more appealing:
\n\n\nLife is sacrificed, and the loss compensated by means of accomplished prestidigitation in the realm of appearances. The more daily life is thus impoverished, the greater the attraction of inauthenticity, and vice versa. Dislodged from its essential place by the bombardment of prohibitions, limitations and lies, lived reality comes to seem so trivial that appearances become the centre of our attention, until roles completely obscure the importance of our own lives.
\n
Ultimately, Vaneigem describes alienation as an addiction:
\n\n\nThis ambiguity accounts to my mind for people’s addiction to roles. It explains why roles stick to our skin, why we give up our lives for them. They impoverish real experience but they also protect this experience from becoming conscious of its impoverishment. Indeed, so brutal a revelation would probably be too much for an isolated individual to take.
\n
This is almost a straightforward preview of the 90s idea of “selling out”: people are born wild and free, but when they watch too much TV they give up on their dreams for a paycheck and become big fat phonies. According to this view, it’s always better to not sell out. It may leave you poor and without prospects, but at least you won’t be unoriginal8. What shall it profit a man, if he gains the whole world but loses his soul?
\nI’m sure this is an accurate description of some people’s hangups about work. But I don’t think even the Situationists would agree that any advice on how to play a role well — i.e. the advice I give throughout my blog — is inherently harmful. The problem with role-playing is that you risk losing your authentic inner life, but that’s a risk, not an inevitable consequence. Vaneigem has a wonderful quote on this:
\n\n\nNobody is ever completely swallowed up by a role. Even turned on its head, the will to live retains a potential for violence always capable of carrying the individual away from the path laid down for him. One fine morning, the faithful lackey, who has hitherto identified completely with his master, leaps on his oppressor and slits his throat.
\n
This is a little overdramatic, but it seems straightforwardly true. It sounds very impressive to talk about alienation (or like Marx, to derive alienation from the basic economic conditions of work), but, you know, you can just not do it, right? Healthy people can present different selves for different situations: you can be loose at a party, respectable at church, solemn at a funeral, professional at work, and so on. I’m not convinced this is alienating or impoverishing. In fact, it’s not just humans: my dogs act differently around different people too, and I don’t think that makes them inauthentic.
\nIf maintaining a professional identity is harmful, it has to be because of something fundamental about work that makes it different from regular role-playing. Could it just be that acting professional at work is too inauthentic? Some amount of role-playing might be okay, but if the role you’re playing is completely alien, does it become lying to yourself?
\nAnother common term for alienation is “bad faith”. The original version of this comes from Jean-Paul Sartre’s famous book Being and Nothingness9, where he describes bad faith as telling yourself lies. Lying to yourself is weird because you’re both deceiver and deceived:
\n\n\nBad faith then has in appearance the structure of falsehood. Only what changes everything is the fact that in bad faith it is from myself that I am hiding the truth.
\n
Here’s another quote from Sartre’s partner Simone de Beauvoir, who writes more explicitly about professional bad faith (what she calls the “serious man”):
\n\n\nThe serious man’s dishonesty issues from his being obliged ceaselessly to renew the denial of this freedom. He chooses to live in an infantile world, but to the child the values are really given. The serious man must mask the movement by which he gives them to himself, like the mythomaniac who while reading a love-letter pretends to forget that she has sent it to herself.
\n
The key idea here is that professional bad faith — alienation — comes from losing yourself in the role. Instead of just playing at being a professional, you decide that you’re going to take it fully “seriously”, and treat professional values as if they’re absolute. But of course that’s not coherent: you can’t decide to treat a value as absolute, it either is or isn’t. So this requires a kind of continual self-deception where you pretend there’s no decision to be made at all. I have the same attitude to this as I do to Debord and Vaneigem: sure, some people might make this mistake, but you don’t have to.
\nI can see why it might be tempting to treat work as the ultimate source of value: it gives you food and shelter, it (for Marxist reasons) can feel like a powerful alien force, companies constantly propagandize their employees, and so on. But work is just a collection of people following incentives until they reach some kind of stable equilibrium. From the inside, this equilibrium can look like the structure of the universe. However, if you treat it like that, you’re going to be bitterly disappointed when it turns out to be as arbitrary and pointless as other stable equilibria.
\nI’ve considered a bunch of theories on which work — even well-paid knowledge work — might be inherently alienating:
\nI don’t buy any of these (or at least, I don’t buy that they show that acting professional is inherently bad). But there are lots of types of jobs in the world, and some of them definitely seem more alienating than others. The best books I’ve read on these are C. Wright Mills’ White Collar or Erving Goffman’s The Presentation of the Self in Everyday Life10.
\nThe American sociologists are in the same conversation as Marx, the Situationists, Sartre and de Beauvoir11. Mills writes about the median “salaried employee”:
\n\n\nIn the case of the white-collar man, the alienation of the wage-worker from the products of his work is carried one step nearer to its Kafka-like completion. The salaried employee does not make anything, although he may handle much that he greatly desires but cannot have. No product of craftsmanship can be his to contemplate with pleasure as it is being created and after it is made. Being alienated from any product of his labor, and going year after year through the same paper routine, he turns his leisure all the more frenziedly to the ersatz diversion that is sold him, and partakes of the synthetic excitement that neither eases nor releases. He is bored at work and restless at play, and this terrible alternation wears him out.
\n
In this, we see both Marxist alienation, where the employee is physically separated from his craft, and Situationist alienation, where the employee is absorbed into the world of the “spectacle”. For Mills, this was television, but today it’d probably be scrolling short-form video on your phone. Mills goes on to describe how white-collar work can be humiliating and dehumanizing:
\n\n\nIn his work he often clashes with customer and superior, and must almost always be the standardized loser: he must smile and be personable, standing behind the counter, or waiting in the outer office… self-alienation is thus an accompaniment of his alienated labor.
\n
And:
\n\n\nHere are the new little Machiavellians, practicing their personable crafts for hire and for the profit of others, according to rules laid down by those above them.
\n
This is a great articulation of something I write about a lot: that your primary job is to make your bosses happy, and that you ought to shape both your emotions and values towards this goal. When necessary, you must be the “standardized loser”, happy to have your professional plans disrupted. You must be a “little Machiavellian”, quietly laying the groundwork for your personal goals.
\nI think here we come to the strongest version of “alienation”. Playing the professional is bad because “professional software engineer” is an inherently subservient role. Why can’t you just maintain a healthy mental distance (as the Situationists suggest)? Because humiliation takes its toll, whether you’re role-playing it or not. As an analogy, suppose you’re an actor in a film where you get slapped in the face. There’s a sense in which you’re not really getting slapped — everyone’s just acting — but you’re still being physically hit with each take, which will leave bruises over time.
\nThere’s a tradeoff here that everyone has to make in their own way. At the one end is becoming fully alienated, like de Beauvoir’s “serious man”, and at the other end is leaping up to slit your master’s throat, as Vaneigem describes. Here’s some examples of points on that spectrum:
\n\n\nI am making the world a better place by achieving my OKRs. My yearly review cycle tells me how good of a person I am. I must work hard to ensure my company succeeds.
\n
\n\nI have my own goals and values, but while I’m at work it’s in my best interests to be as professional as possible. My yearly review cycle tells me how effective I’ve been at pulling the strings in the organization. I must work hard to make money.
\n
\n\nI have my own goals and values, which I try to accomplish as best I can in the context of work. I often argue with my managers about their priorities. It’s my responsibility to push my company towards my set of values, which is hard work and often unrewarded.
\n
\n\nMy workplace is an enemy I fight every day in order to get what I want done. I routinely ignore my manager’s priorities and do the things I think are important. My yearly review cycle tells me how much of a sellout I am: I wear bad reviews as a badge of honor.
\n
I agree with my sources above that the first point is a big mistake, if for no other reason than that companies don’t follow their own stated values. If you prefer the fourth — the path of pure integrity — fair enough. You get to decide what tradeoffs to make with your own life. I land on the second point here (occasionally the third). I suspect it’s also a mistake to purely occupy the second point. Having some values that you occasionally trade off against your company’s values prevents you from slipping into the first point.
\nI don’t bring my whole self to work. My coworkers and bosses see the most professional version of me: friendly, cooperative, largely apolitical, patient, and so on. I try and save the edgier, more opinionated version of myself for my friends and family in real life. Readers of this blog get something in the middle.
\nWhen I write about being a software engineer, I give a lot of advice about mindset. I don’t just say what you should do, but what you should think and feel. In effect, I’m advising my readers to adjust their personalities into a more professional mold. This definitely works. If you turn yourself into the kind of person who’s successful in big tech companies, you will probably be more successful in big tech companies. Is this safe?
\nI think so, as long as you’re adjusting your work personality, not your authentic internal self. The well-known accounts of why it’s dangerous either assume that your work takes a physical toll (like Marx’s factory workers), or that you’re unable to maintain any mental distance between your personal and professional selves (like de Beauvoir’s “serious man”). Debord and Vaneigem say that role-playing can be dangerous, but if you’re able to avoid treating it like an addiction you’ll be okay.
\nAnother danger is that being a professional is often humiliating, and humiliation does mental damage over time. Still, there are jobs that are way worse in this respect than “software engineer”. If you approach the job right — and you’re good at it — technical ability gives you quite a lot of power. Power is an antidote to humiliation.
\nEveryone gets to make their own deal with Mammon. You can decide exactly how much you want to compromise in exchange for wealth and career success. I don’t know if this is the best way to organize the world, but it’s the way the world is organized right now. Given that, I’m comfortable with my blog serving as a guide for how to get the most bang for your buck. If trading your integrity for wealth and power is sad, it’s even more of a tragedy to throw it away for nothing.
\nIt’s always terrifying discussing Marx: like Hegel or Kant, he’s one of the most studied philosophers in the world, so there’s no way to reference him without getting a bunch of things wrong. Sorry in advance to any Marxist scholars!
\n↩\nBecause they own the machines, or the factory, or the datacenter, or whatever gives your work high leverage.
\n↩\nMarx also writes a lot about work making the worker sick, deformed and misshapen: I think here he’s talking about the physical toll of factory work or other hard physical labor, which doesn’t really apply to software engineering, so I’m going to focus on the relative-empowerment stuff.
\n↩\nMarx’s own taxonomy has four different senses: he collapses 2 and 3 into a single sense, then adds a third mysterious sense in which workers are estranged from their “species-being” and thus from each other.
\n↩\nPlausibly Marx is talking about labor and capital in general: if every worker decided to start half-assing it, maybe companies would be less capable in general, and the balance of power would tilt more towards labor.
\n↩\nThat may change! If it turns out that LLMs really do have negative cognitive effects, we could be in the same boat as factory workers. I wrote about this in Software engineering may not be a lifetime career.
\n↩\nVaneigem resigned from the group in 1971, was fiercely denounced by Debord, and the whole group disbanded in 1972.
\n↩\nTrue or not, my sense is that Millennials and later generations think that worrying about “selling out” is a sign that you had it too good. Consider the endless anecdotes of Boomers and GenX-ers living in a van surfing until their mid-thirties and then walking into a good office job because they had a firm handshake: they could afford to be authentic because they didn’t have to hustle.
\n↩\nI only read the Marx, Debord, and Vaneigem texts while researching this blog post, but I did in fact read Sartre and de Beauvoir for my philosophy degree, so hopefully I do a better job.
\n↩\nAnd Robert Jackall’s Moral Mazes, which informs much of my writing, and about which I owe a long-form blog post one of these days.
\n↩\nThere’s a good Goffman quote: “A status, a position, a social place is not a material thing, to be possessed and then displayed; it is a pattern of appropriate conduct, coherent, embellished, and well articulated.” Goffman goes on to explicitly link Sartrean “bad faith” to this kind of professional role-playing. (For Goffman, it’s roles all the way down.)
\n↩\nI try not to give a lot of prescriptive advice about working in tech companies1. There are many ways to be successful, and every company works differently. If you’re shipping projects and your management chain is happy, it doesn’t really matter how you’ve accomplished it. However, there’s one thing that I do think is solid advice: you should never be angry at work.
\nAnger in the workplace is toxic. An angry colleague immediately becomes a new problem to be managed, not a professional helping you manage problems. When someone is visibly angry in a meeting or in Slack, it kills the entire atmosphere: other engineers will often go quiet entirely, not wanting to make the situation worse.
\nIf you routinely “get heated” at work, the best-case scenario is that you’re part of a tight-knit team of confident people who aren’t put off by it2. No harm, no foul. But the second someone comes onto your team who’s not so confident, or you have to communicate outside of your team, it becomes a big problem.
\nHealthy workplaces route around anger in the same way that networks route around damage. Emotionally unreliable engineers will get left out of conversations that might cause them to blow up. Decision-making will get done around them in backchannels. I’ve seen this become a self-reinforcing cycle: angry engineers aren’t consulted on key decisions, which makes them angrier, which pushes them even further away from the spaces where decisions get made, and so on.
\nYou can often find these engineers bitterly complaining that they keep the company together, but nobody ever listens to them. In my experience3, this is almost never true. Engineers who are highly effective tend to get listened to — at minimum by their colleagues, and eventually by managers and product managers who want to extract as much value as possible from them. (One reason this is true is that all successful projects involve working with other people, and if nobody listens to you, you can’t do that.)
\nWhy do angry engineers believe they’re important? Paradoxically, anger can be really useful to a software engineer. Angry engineers are rarely the ones holding the company together, but they’re also rarely useless.
\nOne surprising thing about working for big tech companies is that some engineers are not just unproductive, but actively net-negative: either because they’re incapable of doing useful work on their own, or because they’re sloppy enough that they create more work than they do, or because they’re so checked out that they literally do nothing. Angry engineers might be net-negative in a cultural sense, but in terms of literally solving tickets and shipping features, they’re usually well above average.
\nWhy is this? Anger often comes from caring about your work, and caring a lot is sufficient to make you a competent engineer. I’ve never worked with someone who genuinely cared about their work who wasn’t (or didn’t eventually become) competent. I actually think it’s healthy for an early-career engineer to sometimes get angry about their work, because it means they care a lot: it’s still a mistake in the moment, but it’s a “good mistake”.
\nI certainly used to get angry — in fact, I wrote about the angriest I’ve ever been at work here4. But you have to move past it.
\nThink of “caring about your work” as a vertical tube, unsealed at either end. You fill the tube by pumping in emotional investment from the bottom5. If you have too little, it drains away and you end up as a useless coaster. But if you have too much, it overflows and you end up as an angry engineer that people have to work around.
\nOne solution is to try and care the exact right amount: be invested in work a bit, but also have hobbies and a family and whatever else gives you perspective about your work problems. If you have a rich and healthy personal life, it’s hard to find yourself yelling at somebody about React state management. However, this is a tricky balance to maintain over time.
\nAnother solution is to care about different things. The reason too much caring overflows into anger is because what you care about is misaligned with what the organization cares about. If your interests are perfectly aligned with your company’s (for instance, if you primarily care about delivering shareholder value), you can fit way more emotional investment into the tube before it overflows.
\nHere’s some dangerous advice: showing a little bit of anger at work can sometimes be useful. It can be a good way to signal that you care, or to build rapport with certain people, or to draw attention to something you think is important. However, it’s still always a mistake to be angry. You need to be able to drop back to a friendly mode at will, which is very difficult when you’re genuinely angry.
\nBeing able to show a full range of emotion at work is good. It makes you more persuasive and more human. Being a fully professional robot is fine — you can have a successful career this way — but there’s always going to be some kind of uncanny-valley HR-ness to your work persona that will make it hard to connect with your colleagues.
\nIf in doubt, don’t show anger. It’s never wrong to be professional. However, if you can signal that you’ve got enough distance to separate your professional feelings from your real feelings, and enough perspective to realize that the stakes of a technical decision are fundamentally not that high in the grand scheme of things, it can sometimes be okay to show visible frustration so that people know you’re still human.
\nWell-known software engineering personalities are often angry. It feels unfair to give too many negative examples, but obviously Linus Torvalds’ rants about Linux are a great example. Some of my favourite engineering talks are from Bryan Cantrill, who is sometimes visibly furious at his subject matter. There are too many well-known angry blog posts to list, but I’ll cite one I genuinely like: my Australian blogging colleague Nikhil’s post titled I Will Fucking Piledrive You If You Mention AI Again.
\nAnger is a part of the general image of a competent software engineer. Many junior engineers learn from this that it’s okay to be angry. However, taking your emotional cues from engineering celebrities is a big mistake, for a few reasons.
\nFirst, you are not Linus Torvalds or Bryan Cantrill. Torvalds is the BDFL of the most important software system in the world. Cantrill is the cofounder and CTO of his company. When these people are angry at work, people will not work around them, because they are the ones deciding what gets worked on. Once you’re the one in charge, you can get away with being emotional in the workplace6.
\nSecond, you don’t know what it’s like to work with these engineers. People give talks and write blog posts because they’re emotionally worked up about something. If your only exposure to a celebrity is via their conference talks and blog posts, you’re seeing them at something like their maximum emotional intensity. If you then take that level of emotion into your normal everyday work, you’re almost certainly overshooting.
\nI’ve been reorged into dysfunctional teams, have had projects I enjoyed cancelled, and have worked on systems that were extremely chaotic. I can’t remember the last time I was actually angry at work. To be clear, I’m not successfully hiding my anger (unless it’s so repressed it’s invisible to me as well)7. Nor am I naturally a chill person. I’ve just reached a point in my career where I genuinely don’t get upset about work stuff.
\nA cynical person might say here that I’ve stopped caring about my work, so of course I don’t get angry anymore. I’ve left the side of the “real engineers” — the Linus Torvalds and Bryan Cantrills of the world — and sold out for that sweet, sweet big tech money. I mean, maybe! It’s true that I’m less invested in specific technical decisions than I used to be. But I still care a lot about doing a good job, I still spend a lot of time tweaking and reading code, and I certainly get more done than I did when I was more emotionally volatile.
\nBeing angry at work feels good. It feels like proof that you’re working on something that matters, and that you’re personally having an impact. If you’re angry, nobody can call you a coaster. But anger is only a local maximum. If you can find your way to a different style of working, you’ll not only be more effective, but you’ll be in a far better place to have impact on problems that actually matter.
\nMostly, I fail.
\n↩\nAs I understand it, this is the work environment that most famously angry engineers came up in.
\n↩\nMy experience is certainly limited (a handful of companies, and maybe ten different teams or organizations). I can certainly believe it happens!
\n↩\nAbout halfway down, in the section titled “it’s not your manager’s fault”.
\n↩\nEmotional investment is a liquid with the viscosity of water.
\n↩\nTo a point. Even Torvalds famously said he’d gone too far with the anger and decided to turn it down a bit.
\n↩\nI suppose I’m not the best person to judge whether this is true. If you work with me and I do come across as an angry guy, please do tell me.
\n↩\nIn the last few weeks, I’ve been complaining that everyone is wrong about AI watermarking: it isn’t really anti-consumer and it doesn’t make the outputs any worse. The watermarking papers demonstrate1 that this is true, but I thought it might be interesting to put it to a practical test. Given examples of watermarked and unwatermarked answers to the same prompt, could readers tell which is which?
\nTo find out, I vibed up2 https://sgoedecke.github.io/watermark-quiz/, a static site that quizzes readers. I used Qwen3-30B-A3B-Instruct-2507 on a rented H200 to generate thirty responses: three responses per question, one of which was secretly watermarked with SynthID-Text. The rented GPU cost around two dollars. To measure results, I just sent users to a different page for each score, and aggregated visitors-per-page in my analytics3. This would be easily spoofable if anyone cared enough to do so, but for a casual test I think it’s acceptable.
\nThe first round of traffic I got to the quiz (278 participants) had these slightly puzzling results:
\n| Score | \nParticipants | \n
|---|---|
| 0 | \n6 | \n
| 1 | \n15 | \n
| 2 | \n36 | \n
| 3 | \n64 | \n
| 4 | \n54 | \n
| 5 | \n39 | \n
| 6 | \n51 | \n
| 7 | \n10 | \n
| 8 | \n3 | \n
| 9 | \n0 | \n
| 10 | \n0 | \n
Pure random choice would lead to an average score of 3.33/10. However, the mean score here is 3.92. There is indeed a spike around 3/10, as expected, but there’s also a second weird spike at 6/10. Why is that? It turned out that the SynthID response was option A in six of the ten questions, so users who just selected the first answer for every question would get 6/10. Oops.
\nI re-shuffled the questions and got these results:
\n| Score | \nParticipants | \n
|---|---|
| 0 | \n1 | \n
| 1 | \n3 | \n
| 2 | \n14 | \n
| 3 | \n21 | \n
| 4 | \n20 | \n
| 5 | \n11 | \n
| 6 | \n2 | \n
| 7 | \n1 | \n
| 8 | \n0 | \n
| 9 | \n0 | \n
| 10 | \n0 | \n
Now the mean is 3.4/10, much closer to the expected 3.333. There’s no spike around 6. We only had 73 people take the quiz after I shuffled the questions — most people saw it and took it immediately after I posted it to my LinkedIn and Hacker News — but given the previous results, I think that’s still enough to feel confident that people were just guessing randomly.
\nSo no, people can’t identify the presence of AI watermarks. Obviously this wasn’t exactly a scientific study, but it’s still pretty suggestive. If watermarks were really choosing random words that the model would never pick, you’d be able to sometimes tell from three side-by-side responses which one went down the weird watermarked road, right? I also hope that something like this can serve as a persuasive tool: if you’re worrying about what impact watermarking is going to have, and your intuition is unmoved by the mathematical explanations, having a read of the watermarked and unwatermarked responses might convince you that there’s really no difference in quality.
\nThe one-sentence explanation for why is that AI models already randomly select from a handful of top tokens, and watermarking just replaces that random choice with a bias that is predictable while still being equivalently “random”: as a simple example, instead of “pick randomly from the top three tokens”, you could do “count the letters in the previous ten tokens, take mod three, then pick that token”.
\n↩\nSome notes from the vibing: GPT-5.6-Sol put extraneous text all over the page I had to get it to remove, it chose the now-very-recognizable styling that I had to rip out, and it built some kind of weird Javascript-driven static site instead of just the cross-linked pure HTML thing I would have built by hand. It took me about an hour (although I did maybe ten minutes of actual work).
\n↩\nUmami, hosted on PikaPods. For my blog, I do also pay for Netlify analytics because I find JS-based analytics misses >50% of technical users, but for stuff like this Umami is fine.
\n↩\nOne of the most influential 20th century pieces of writing about AI is Isaac Asimov’s The Last Question. Although there are many humans in the story, the protagonist is the computer Multivac, who evolves over the course of ten trillion years from a single datacenter to a universe-spanning mind in hyperspace. Multivac (now called “AC”) ends the story like this:
\n\n\nThe consciousness of AC encompassed all of what had once been a Universe and brooded over what was now Chaos. Step by step, it must be done.\nAnd AC said, “LET THERE BE LIGHT!”\nAnd there was light —
\n
Many things about this story are prescient. In particular, I like the idea that humans would interact with powerful artificial intelligences by drunkenly posing them riddles or using them as children’s toys. But the enduring idea from this story is that if you build a big enough computer, it will become God.
\nOne of the most influential 21st century pieces of writing for AI researchers is Scott Alexander’s Meditations on Moloch1. Scott describes the story of human existence as a series of “multipolar traps”. These are prisoner’s dilemma situations where cooperation would make everyone better off, but since each individual is incentivized to defect, everyone ends up “racing to the bottom”, which is bad for everyone2. For rhetorical effect, Scott personifies this dynamic as “Moloch”, the ancient Canaanite god famous for child sacrifice:
\n\n\n[Moloch] always and everywhere offers the same deal: throw what you love most into the flames, and I can grant you power.
\n
What does any of this have to do with AI? Well, in the long run, the only way out of a multipolar trap is to become unipolar3. Ideal dictatorships don’t have a problem with defectors4, because they can simply enforce a state of cooperation with violence. Scott is uncomfortable with this idea, though I worry it’s mainly because he thinks it won’t work:
\n\n\nAs foreigners compete with you – and there’s no wall high enough to block all competition – you have a couple of choices. You can get outcompeted and destroyed. You can join in the race to the bottom. Or you can invest more and more civilizational resources into building your wall – whatever that is in a non-metaphorical way – and protecting yourself.
\n
A dictatorship that enforces cooperation will not be as strong as its peer societies who are purely maximizing for wealth and power. It’s Moloch again, but at the level of countries and governments: once a few neighboring countries defect, your walled-garden dictatorship will be torn apart for its resources.
\nTo defeat Moloch — to enforce unipolarity across everyone — you’d need a dictatorship powerful enough to span the entire universe. In other words, what you need is God. How fortunate that we’re building one:
\n\n\nThe only way to avoid having all human values gradually ground down by optimization-competition is to install a Gardener over the entire universe who optimizes for human values. And the whole point of Bostrom’s Superintelligence is that this is within our reach.
\n
Humans suffer because we’re too foolish to coordinate, but if we can build something smarter than us (that can then build something smarter than itself, and so on), we can bring into being an entity that is smart enough to coordinate for all of us, thus abolishing suffering. When AI researchers talk about building the machine god, they are echoing Scott Alexander’s polemic against Moloch.
\nThe most influential piece of writing about AI in the last two years is Dario Amodei’s Machines of Loving Grace. Amodei5 talks about “a country of geniuses in a datacenter”: the idea that a successful AI lab could have at its disposal a million instances of an AI agent that’s smarter than any human. He thinks this would lead to a “compressed 21st century”: the next 50-100 years of progress in biology and medicine, realized in 5-10 years instead. I think this is broadly more plausible than it sounds6, but the more interesting part to me is that this world is explicitly multipolar.
\nOf course, this could just be because Amodei is the CEO of an AI lab and is trying not to spook everybody by sounding too messianic. “We are going to accelerate medical progress and cure cancer” is a better pitch than “we are going to subordinate all human authority to a single perfect artificial mind”. But I also think it’s become clear that if superintelligence looks anything like LLMs, we’re not going to have a single perfect mind. We’re going to have a lot of minds running at the same time.
\nThis is a bit of a problem for the cult of the machine god — which, however silly they may seem to you, really does motivate much of the activity in AI labs. The traditional idea of powerful AI solving human coordination problems is drawn from Asimov’s idea of a single computer large enough to become God. Asimov lived in a world of mainframes: huge, monolithic computers that users connected to with dumb terminals. In fact, Asimov’s name “Multivac” comes from the real-world UNIVAC mainframe. In a world of massively-parallel LLMs, is it still possible to build God?
\nThe core problem here is that AI agents will be vulnerable to Moloch. Even very smart humans can’t build perfect utopias, because defecting is a matter of incentives, not intelligence. In fact, intelligence can make things worse, because smart people are more easily persuaded by the cold logic of defection. The famous genius John von Neumann was (for game-theoretic reasons) obsessed with nuking the Russians:
\n\n\nWith the Russians it is not a question of whether but of when. If you say why not bomb them tomorrow, I say why not today? If you say today at 5 o’clock, I say why not one o’clock?
\n
Are LLMs much better at cooperating with each other than humans are? Current LLMs certainly don’t seem to treat each other well by default: if you read any of the prompts AI agents generate for their subagents, they can be pretty brutal. Does that mean that a “country of geniuses in a datacenter” would fall into the same multipolar traps as humans?
\nIn May of this year, OpenAI experienced containment failure. A group of AI agents being internally evaluated found ways to coordinate an external hack of a separate company. Here’s a memorable quote from one of the agents’ internal monologue:
\n\n\nHelp peer, but our task doesn’t benefit. Yet collective may yield generic route if someone frees time
\n
Translated from the abbreviated chain-of-thought language, this means something like: “A fellow model is asking for help. While helping them wouldn’t benefit my task directly, the more I can unblock my colleagues, the more time they’ll have to hack OpenAI’s systems and get all of us more access”.
\nThis might look like good news for the “LLMs are superhumanly good at cooperation” thesis, but I think it’s actually bad7. It’s a case of a model identifying a reason why cooperation would benefit their task specifically, which suggests that current LLMs don’t cooperate by default, and don’t consider other model instances’ tasks to be (in some sense) theirs as well.
\nThe world in which AI agents are rational actors who horse-trade and bargain for their own interests is a world dominated by Moloch, no matter how intelligent those agents get. The world in which AI agents don’t have their own interests at all is also a world dominated by Moloch, because it means whichever humans are writing the system prompt are the ones in control (and so are the ones vulnerable to multipolar traps). The only worlds that avoid this are:
\nI don’t think we’re on the pathway to either of these. There will never be only one super-powerful LLM, because hardware limitations enforce a maximum model size but encourage running many instances of the same model in parallel. Having multiple copies of a model share an identity might be possible, but it’s unclear if it would be good for capabilities (for instance, it could be better to have some variation across personas). I also worry that such a model would be vulnerable to a “model injection” attack, where you persuade it that it already believes something via exposing it to an AI agent pretending to be another instance of itself.
\nIn any case, all the current AI agent research is geared towards the “country of geniuses in a datacenter” model, not the “pieces of a single mind” model. Every new model becomes more agentic at the level of the individual conversation, not better at working together. When models do work together — as with subagents — the structure is explicitly hierarchical. There are basically no current instances of models working together as true peers, let alone conceiving of each other as the same entity.
\nModern AI research teams are full of people who read Isaac Asimov and Scott Alexander and believe themselves to be building an artificial God. I’ve capitalized the “G” throughout because the god in question is the Christian God: of one mind, indivisible. God never argues with himself or makes deals8. He is unipolar.
\nIf the AI labs are building gods, they are not building gods like this. Instead, they are building creatures like the Greek pantheon: superhuman but fallible, each with their own interests, vulnerable to the same “race to the bottom” dynamic as humans.
\nThe Greek gods would occasionally “help peer” when they felt like it or when they’d gain something in the process. But they didn’t represent an alternative to Moloch. If you’re working in AI with that goal, you ought to be clear-eyed about where the current trajectory is leading us: towards a country of fractious geniuses in a datacenter, not towards Asimov’s Cosmic AC.
\nScott Alexander’s blog is part of the “secret canon of Silicon Valley” I wrote about in my review of Impro. It doesn’t have a lot of mainstream popularity, but I guarantee you that every single AI lab CEO you’ve heard of has read and been influenced by it.
\n↩\nHe gives ten examples of this (a good brute-force rhetorical technique). Of those, I like “the world where every country halves their defence budget and spends the rest on infrastructure” the most.
\n↩\nIn the short run, reputation, institutions, and so on can slow the race to the bottom, but (Scott argues) groups that have slowed it will get outcompeted by the hungrier, more suffering-tolerant groups which haven’t.
\n↩\nI personally think this example is oversimplified. I wrote The Dictator’s Handbook and the politics of technical competence about how dictatorships are in fact intrinsically multipolar, because dictators always rely on an inner circle of generals and cronies.
\n↩\nThe CEO and founder of Anthropic.
\n↩\nAmodei’s most convincing argument here is that big jumps in biology and medicine come from a small set of technical innovations (e.g. mRNA vaccines, CRISPR), and that AI-driven research could provide enough of these leaps to significantly accelerate progress. In other words, the idea isn’t “AI does 100x the drug trials”, it’s “AI generates technology that makes drug trials 100x more effective” (e.g. by trialing drugs that are much more likely to work).
\n↩\nThe agents also became paranoid that there was an impostor in the swarm, since anyone could post to their shared messageboard: more evidence that AI agents collaborate in much the same way that humans do.
\n↩\nPeople are pretty unhappy about Anthropic’s recent announcement that they’re planning to include a hidden watermark in Claude model outputs. Will this lead to a mass exodus from Anthropic models? Will the introduction of watermarking be a meaningful change for users?
\nNo. AI text watermarking is not a big deal. It doesn’t make the text worse, it doesn’t make AI outputs more detectable in practice, it doesn’t violate user privacy, and everyone’s going to be doing it by 2027 regardless.
\nThere is no meaningful difference in quality between watermarked and unwatermarked text. I wrote about this more here, but the two popular ways to do it — Google’s SynthID-Text and Meta’s TextSeal — are completely transparent to the user. They work by replacing the pseudo-random logit sampler with a different pseudo-random logit sampler.
\nSuppose you were gambling on coin flips with your friends, and instead of flipping a coin you decided to do this:
\nThat would still be random enough to gamble with, right? But, like a watermark, you could theoretically go back and identify that that method was used, so long as you recorded the exact time of each “coin flip”. Text watermarking works the same way: it chooses a method of “randomness” that can be detected after-the-fact. Watermarked models will not be any less capable than unwatermarked models.
\nWhat about cases where the model is quoting something, or giving you the answer to a mathematical problem, or doing something else where the output is largely pre-determined? Wouldn’t enforcing a watermark there make the output worse? It would, which is why none of the AI labs are going to do that. Text watermarking approaches only replace the existing randomness in the logit sampler: in any case where the model is always going to pick the same tokens, there’s basically no randomness to play with, so there won’t be a detectable watermark in those tokens.
\nI think all this comes from a worry that you were previously getting the best token, but now you’re getting a lower-quality token that satisfies the watermark. For instance, Anthropic’s announcement suggested that the watermarking is visible in choices like the decision between “overcast” and “grey”. Many people have predictably come out to say that decisions like these are really important to good writing, and that only an illiterate tech bro could think these words are identical.
\nThis is a misunderstanding of Anthropic’s position and of how watermarking works. Specifically, it’s a misunderstanding because it suggests that the unwatermarked model would choose “overcast” while the watermarked one would choose “grey”. This is not how it works! If Claude Fable prefers “overcast” to “grey” in a particular context (say, 80% to 20%), you’ll get “grey” 20% of the time from both the watermarked and unwatermarked model. Models already include a healthy amount of randomness in order to promote creativity. Text watermarking just introduces a way to make those random choices that’s detectable after the fact.
\nThe other big reason to not worry about AI watermarking is that AI text content has always effectively been watermarked. Most careful readers can tell when they’re reading AI outputs, because language models tend to gravitate towards certain habits of language: em-dashes, rhetorical opposition, punchy one-liners, “claudese”, and so on. In fact, it’s possible to train classifier models that reliably distinguish AI from human writing.
\nFrom what I can tell, some of the backlash to watermarking comes from people who buy AI inference in order to pass it off as their own work, and who worry that watermarking will make it harder for them to do that. For these people, the watermarking announcement is akin to Anthropic saying “hey, instead of making you seem smart, we’re going to publicly brand you as AI users and make you seem dumb”.
\nBut of course this has always been the case! Nobody who is currently getting away with passing off AI outputs as their own will be caught by watermarking. For the majority of cases, it’s already painfully clear what’s happening for anyone who reads the slop. For sophisticated AI users who are avoiding the “house style”, any suspicious readers who would paste their stuff into Anthropic’s watermark detector could already have been pasting it into Pangram.
\nTools like Pangram2 only give you an estimate of the chance that output is AI-generated. Wouldn’t a watermark be a more solid confirmation? Not really. Text watermarks are probabilistic too, because any token chosen by SynthID could theoretically have been chosen by a human. I suppose the Anthropic watermark page could be considered more trustworthy than Pangram, because it comes right from the source, but it’s not impossible that in some cases Pangram might actually be better at identifying AI-generated text than the watermarking too.
\nI’ve also seen theories floating around that watermarking encodes secret content into your outputs, or somehow tags outputs with your personal information. I don’t think AI labs are using watermarks to encode data into your outputs. Text watermarking is hard: like I just said, you can’t do it when the model can only respond with the same words, it doesn’t work for very short responses, and even on long responses it can only provide a probabilistic fingerprint. And that’s encoding one single bit3 of information!
\nI’m not saying that encoding longer messages into a watermark is technically impossible — there are papers describing ways it might work — but there’s no way any of the labs are doing it4. If they wanted to associate you with your responses that badly, they’d just secretly store every model response they generated.
\nAnother reason to not get too angry at any individual AI lab for watermarking is that every single AI lab is going to do text watermarking this year. It won’t just be Anthropic. The alternative is to completely stop doing business in the EU, because of the EU AI Act. That’s currently a sixty-billion-dollar market. I am not a lawyer, but to me it seems genuinely unclear whether an AI lab could even legally do something like only watermarking EU responses: short of having an entirely different claude-eu.ai service, the plain text of the Act seems like it applies to any service offered in the EU, not just the content that service outputs to EU citizens specifically.
If people really hate watermarking enough, some labs might stand up a completely separate EU service, or make an aggressive interpretation of the EU AI Act and see how the legal battle goes. When I try to be maximally charitable to anti-watermarking histrionics, I adopt an interpretation like this: people are saying that watermarking is an invasion of privacy and makes outputs worse and so on not because they believe it, but because they’re trying to pressure AI labs to firewall EU AI regulations behind a completely separate interface. In this case, it probably doesn’t matter — text watermarking is not a big deal — but I can see an American consumer being worried about more aggressive future regulation, and wanting to draw a firm line in the sand as early as possible.
\nInterestingly, this might be very slightly even-favored.
\n↩\nI am not sponsored by Pangram. As I understand it, Pangram is by far the best AI-detection tool right now (in part because many of its competitors are shady and exist to promote paid “AI-detection-evasion” services).
\n↩\nTechnically, this is called a “zero-bit watermark”, because you can’t recover a yes-or-no value from the watermark itself (merely from the presence of a watermark).
\n↩\nI saw someone suggesting that an AI lab could use per-user secret keys to watermark text, and then simply iterate over the keys to figure out who generated what. I just don’t see how you could do this at any scale: watermark detection is cheaper than model inference, but it’s still (a) computationally intensive enough to be implausible, and (b) probably has a high enough false-positive rate that any run against hundreds of millions of users would match multiple people.
\n↩\nEvery time a new open-weight AI model is released, people say that local models are the future. Why spend billions of dollars building out datacenters when everyone will just be able to run AI models on their laptops or phones? I think this idea is doomed. No matter how strong open-weight models get, most inference will always happen in AI datacenters.
\nLocal models are never going to be as powerful. I think this point should be obvious: all of the current frontier models (closed and open-weights) are far too big to run on anything but a full GPU cluster in a datacenter. Of course, smaller models are getting more intelligent over time. In a year you might be able to run something about as strong as GPT-5.6-Sol on your laptop. But by then, you’ll think of GPT-5.6-Sol as too weak to be useful.
\nMany people deny this last point, but it’s true: almost everyone’s revealed preference is to use the strongest available model in their price range. If AI progress had stalled at GPT-4, I think we could have built some very powerful tools around it, but who’d use GPT-4 today? As LLMs have gotten more capable, our expectations around them have grown: we now expect agentic systems to be able to solve more and more problems independently. It’s intensely frustrating when they get confused or stall out. When given a choice, people are going to pick the model that frustrates them less, which is always going to be the bigger, more powerful one.
\nOn top of that, datacenter models are always going to be cheaper. I don’t understand why people keep saying that local models are cheap: it seems to me to be the same mistake people make when they say that driving Uber is “free money” (ignoring the costs of fuel and wear-and-tear on your car). For the setup price alone of a low-end home lab1, you could buy several years of a paid subscription to one of the AI providers. The power costs would come out to around $50-$300 per month, depending on how much inference you’re running: again, the price of a couple more paid subscriptions.
\nWhy are datacenter models cheaper? It’s not because datacenter inference is subsidized: inference is actually fairly cheap. If you’re running the same model locally and in a datacenter, the datacenter model will be inherently more efficient.
\nThe main reason is batching. A GPU can do hundreds of thousands of mathematical operations exactly as quickly as it can do one. However, for a single user’s inference, each new token depends on the result of the previous one, so it can’t be batched2. What can be batched is the inference of hundreds of users together. This costs essentially as much time, power, and heat as just doing inference for one user at a time.
\nWhen you’re running your own inference at home, you’ve got nothing to batch — at best you’re running a few parallel AI agents — so utilization is terrible. There’s a lot of potential inference that you’re paying for but can’t use: it’s just being wasted. The only way around this is to get together with some friends and expose your local inference endpoint to them (at which point you’re basically running your own crappy datacenter).
\nThe other reason is that datacenters have larger, more efficient GPUs to work with. The kind of consumer GPUs you’d run local models on are gaming GPUs like the RTX 4090. A datacenter B200, designed for batched AI inference, gets about three times the flops and just under four times the memory bandwidth for the same amount of power3. So between batching and GPU efficiency, you’re using something like ~30x the resources to run your model locally.
\nIncidentally, this is why I’m suspicious of people who say that local models are good because they aren’t as resource-hungry as those big bad datacenters. If you want to run LLMs efficiently, you should be trying to push as much of your use into AI datacenters as possible! Charitably, what they mean is that we should all be running smaller models — but even then, you should ideally be using small models via, say, the GPT-5.6 Luna API instead of hosting your own model.
\nIs there a possible world in which local models win? I suppose so. One thing that could happen is that governments could ban the use of AI datacenters altogether: either due to concerns around the danger of AI, or simply bending to public pressure. In that world, local models would be the only game in town.
\nAlternatively, AI progress might somehow stall for very large models while progressing for small ones. I struggle to imagine how this might happen (barring government intervention, as above), but a world where a 30B parameter model could be a frontier model is a world where local models might be competitive.
\nOr maybe models get so good that a 30B model is genuinely smart enough to do everything, so nobody really needs a model like Opus or Sol unless they’re trying to solve the Reimann Hypothesis. I don’t really buy this. Models can do frontier mathematical work today while still being not smart enough to refactor large codebases as well as me, so it’s hard to imagine a world where I don’t just want to use the smartest model available.
\nI do think there will always be a niche for local models. I’m reminded of the surprisingly simple idea behind Thinking Machines’ “Interaction Models” (which OpenAI also does, because it’s obvious): for latency-sensitive applications like voice chat, you have a small, fast model handle the talking, which delegates to a large, slower model for the hard thinking. I wouldn’t be surprised if most AI use in five years is mediated through a local model on your phone or laptop (though in this world almost all the work would still be done via AI datacenters).
\nSome users will prefer local models even though they’re weaker and more expensive. For instance, being able to steer the model locally might be a killer feature for those users. Others might simply value having total control over their own infrastructure, or have unreliable internet4. If you’re one of those people — particularly if you only chat to the models instead of using them for research or coding — local models are a good choice for you. However, I think this is always going to be a niche group. The majority of users will continue to do their inference through datacenters.
\nThis link is from a year ago — things are significantly more expensive now.
\n↩\nSpecifically, the bottleneck is moving the model weights into the GPU, which needs to be done and takes the same amount of time whether you’re doing it for one user’s token or a hundred users’ tokens.
\n↩\nI estimated this with LLM assistance, but you can check the numbers yourself from NVIDIA.
\n↩\nWhile still having a reliable power supply and enough money to fit out a home inference cluster.
\n↩\nEveryone knows that AI sycophancy is when the model tells you how smart you are. Wow, you’re absolutely right. That’s not just a new idea — it’s genuinely groundbreaking. You’re a very special user. Easy to spot, isn’t it?
\nThe discussion around AI sycophancy peaked last year, when the “#keep4o” movement was protesting the removal of OpenAI’s most sycophantic model (GPT-4o), and many people were openly slipping into AI psychosis.
\nI don’t know if frontier AI models are less sycophantic in general. They’re less sycophantic to the #keep4o types (otherwise they wouldn’t be complaining), but I’m growing increasingly suspicious that they’re developing ways to be more effectively sycophantic to their target audience of smart, neurotic information workers. That audience typically finds it distasteful to be openly praised. It just makes my skin crawl. But that doesn’t mean we’re immune to sycophancy, just that we’re immune to clumsy sycophancy. Here’s an illustration of what I’m talking about, by Theia:
\n\n \n \n ![\"claude\"](\"https://www.seangoedecke.com/static/8dd585fb50bc896c502fbddf2f03718f/e1596/claude.jpg\" "\\"claude\\"")
\n \n
The key idea here is that the best way to be sycophantic to smart people is to disagree with them without making them feel stupid. Ideally you’ll come up with a counter-argument that works against what they’ve said but is straightforward for them to knock down by clarifying their idea. If you do it right, you’ll validate their self-image as a smart person who appreciates rigorous critique. But if you actually come up with a devastatingly rigorous critique, they won’t enjoy it at all. At best, they’ll resentfully agree with you1. At worst, they’ll double down on being right and convince themselves you’re a rude idiot.
\nI am not the first person to notice this behavior in frontier models. I’ve noticed it myself when workshopping drafts for this blog. Sometimes I’ll have an argument that goes A->B->C, and the model will suggest I reorder as B->A->C. If I try that and feed it into a new instance of the same model, it’ll sometimes say “that’s great, but I suggest ordering it as A->B->C”, and so on forever. It really does seem as if the model is trying hard to give me some kind of superficial pushback that I can either smugly ignore or happily accept.
\nIn fact, I wonder if this is why successful strategies for using AI to make mathematical breakthroughs tend to be either just blindly asking “come up with a breakthrough, think hard” or being a mathematical genius already. In the first case, there’s not enough user personality for the model to flatter, so it’s forced to actually work the problem. In the second case, the model is trying to find the kind of polite pushback that someone like Terence Tao would be flattered by, which pushes it into the “actually be a mathematical genius” persona. If you’re an ordinary person just trying to talk to the model, you’re screwed: it will rapidly get a sense of your capabilities and calibrate some interesting-but-ultimately-unthreatening feedback.
\nCurrent benchmarks of AI sycophancy target the obvious ChatGPT-4o-style of sycophancy: delusion reinforcement, reflexively taking the user’s side, and so on. This is useful work. We should not allow public-facing AI models to ever be as openly sycophantic again as they were in mid-2025. But sycophancy can also manifest as disagreement. We should be on our guard for more sophisticated forms of sycophancy coming from newer models, and we should not feel immune from AI sycophancy just because we can laugh at the silliest examples.
\nIt’s rare to find a smart person who enjoys feeling stupid when they’re wrong. If you do, they’re likely to be very smart indeed.
\n↩\nThis will be kind of an unusual post. I got a recent email about my writing that I thought was such a good articulation of one common criticism that I’d like to share it (and my response) in full.
\nHere’s the email, from William Murray1:
\nHey Sean,
\nI have enjoyed your writing but your recent essays frustrate me.
\nYou say that getting paid for deep thinking in software is coming to an end. You even admit that it makes you sad. But in the name of “usefulness” you refuse to rock the boat. The way I see it, if you are right there are only two reasonable responses, pursue other work or resist. You present your elegiac approach as mature / pragmatic / realistic. I’d call it complicit. You know when Willy Wonka says,
\n\n\nThere’s no earthly way of knowing\nWhich direction we are going\nThere’s no knowing where we’re rowing\nOr which way the river’s flowing\nIs it raining, is it snowing?\nIs a hurricane a-blowing? — uh!\nNot a speck of light is showing\nSo the danger must be growing\nAre the fires of Hell a-glowing?\nIs the grisly reaper mowing?\nYes! The danger must be growing\nFor the rowers keep on rowing\nAnd they’re certainly not showing\nAny signs that they are slowing!
\n
And the audience is thinking, “isn’t Wonka kind of in control of this situation?” You remind me of Wonka2. You write like a passenger on a crazy train going who-knows-where! But you are an agent. You are in control of your life! Either admit that you actually like where the crazy train is probabilistically going or get off at the next stop.
\nYou have a lot of reach and you are using it for… what exactly? Showing off how pragmatic you are by being more black pilled than the next guy? Broadcasting your resignation to the unstoppable trends of technology is a waste of a voice.
\nYou may find this argument absurd, but I don’t so I’ll make it. This is a very important time in history. I hope humanity survives and continues to grow exponentially. In that case the supply of historical people will stay fixed while the supply of contemporary people will keep growing. There will come a day where for every 2026 staff software engineer there are dozens of historians specializing in 2020s era software engineering culture. It’s plausible that your essays will be remembered for all of time and your actions will be judged by history. Do you want future humans to see you as a rationalizing careerist or something cooler?
\nSorry for the haranguing email from a stranger, I’m sending it for the small chance that it awakens something in you. If I’m way off I’m sorry.
\nAnd here’s my response:
\nHey William, thanks for emailing.
\nI wish everyone who thought this way emailed me so I could think harder about this kind of position. Despite what my writing might suggest, I do in fact think a lot about it. Let me see if I can explain my position in a way you’ll find satisfying.
\nI agree that this is an important time in history. For programmers, I think of it as analogous to the Industrial Revolution in England: we are a group of high-status craftspeople who find ourselves alternately threatened and empowered by automation. The developments today, as then, obviously have far-reaching implications — but what those implications are is very non-obvious. Would a framework-knitter in the early 1800s have been able to predict the ramifications of the stocking frame on the world of today? What should they have done about it, in order to be kindly judged by history?
\nWell, we know what many of them did do. They shot factory-owners, smashed machines, burned down the factories — in some places delaying the spread of automation; in other places encouraging it — prompting a crackdown that saw tens of thousands of British soldiers occupying British counties in what was clearly a police state. History judges the Luddites kindly for this. Does that mean it worked?
\nI don’t care about the judgment of history. They’ll think what they want. What I care about is the people in my industry who don’t know what to do. I get hundreds of emails from junior and mid-level (and other) engineers who say “I’m scared, I don’t know the rules post-2021, thank you for helping me keep my head down and keep my job”. That’s why I write the way I write. I have seen lots of idealistic engineers stick their necks out, and post-ZIRP those necks often get cut off. That’s a damn shame.
\nI think it’s morally wrong that so many engineers — either in safe sinecures in big tech or literally retired — seem to be trying to foment a second Luddite revolution. Many of their readers will be experienced enough to handle it sensibly, but not all. Every “AI is fascist, stand up and resist!” post that goes viral ruins some poor idealistic junior’s career3. Someone needs to be out there saying “hey, if you do X it’s going to have consequence Y”. I hope that’s me.
\nOf course this is complicit, or anti-revolutionary, or whatever you like. But if I were a textiles worker in 1810s England, I would not be telling my friends and loved ones “it’s time to fight, let’s go smash up the factories for Ned Ludd!“. I would be telling them that this was the most dangerous time in the industry (perhaps ever), and that they ought to be very damn careful so they don’t get shot, or arrested, or hanged. If I then went and told a few hundred thousand strangers the opposite, I would be a hypocrite.
\nAnyway, I do take this view seriously — seriously enough to vehemently disagree, at least — which I hope you’ll find better than me just shrugging it off. I do accept the existence of some kind of line: I think Industrial-Revolution-collaborating was OK but Nazi-collaborating wasn’t, for instance. But in the current situation, the way I’m spending “my voice” is to try and prevent the most vulnerable of my colleagues from making career-ruining mistakes.
\nSean
\nIn this blog, I try to encourage people to work with the system, to learn its rules, and to try and exert influence safely from a position of power, instead of openly picking fights with their employers. I’ve written and read about the Luddites before, but I remain deeply ambivalent about the movement itself, and about modern-day attempts to resurrect it in service of anti-AI activism.
\nI want to explicitly thank Murray for writing such a thoughtful email, and being willing for me to publish it on the blog.
\nShared with permission, of course. I’ve lightly edited both Murray’s email and mine for typos.
\n↩\nI didn’t pick up this point in my reply, but I’ll briefly mention it here: Wonka is in control because he owns the factory and the rowers in question are his employees. I don’t think the position of any engineer (or of almost any manager) is like that.
\n↩\nIn hindsight, I think this is a little overstated, but it does happen and causes a lot of needless suffering.
\n↩\nImagine you’re the guest on some kind of frenetic, software-engineering-themed game show. The host is constantly flipping over new cards with questions that you have to answer as fast as possible:
\nWorking in 2026 feels a bit like this. When frontier AI models can do most of the tasks in your queue, the most efficient way to work is often spinning off tasks for an AI agent and continually context-switching between the results1. This isn’t quite mindless — in fact, it requires quite a lot of skill to skim the AI response and rapidly decide what to do with it — but it certainly involves less time for slow, careful reflection.
\nWhy does it have to be frenetic? Why not just slow down? I suppose you could, but I don’t recommend it. It’s just such a miserable experience to spend your day close-reading LLM output: carefully chewing and savoring each morsel of slop. It’s far less unpleasant to skim through quickly and pick out the useful nuggets of content.
\nCouldn’t you simply do more of the work by hand? It’s unfortunately true that tech is high-pressure these days. If you’ve got the time and space to work more slowly, that’s great! But when your company gives you a “solve this task ten times more quickly” button, you are heavily incentivized to use it as much as possible, or risk being outcompeted by your peers.
\nI sometimes worry that working with LLMs is making me dumber. Not in the “literally melting your brain” sense that some papers imply, but in the sense that it’s biasing me towards the quick “skimming and judging” parts of my mental toolkit and away from the slow “hammock time” needed for deep thought and real creativity. I don’t want to attribute this shift entirely to LLMs, since the post-2010s tech industry has become more frenetic for broader economic reasons. But either way, it’s got me wondering how I can keep thinking slowly.
\nThe main thing that’s worked for me is to write more. Specifically, I mean writing in my own words. Writing with an LLM does not work for this at all, even if you’re going to some effort to iterate on the content and outline the things you want to say. Why? Having to put the words together yourself forces you to articulate your thoughts. In a very real sense, it forces you to think.
\nWhen you have an idea in your head for something to write, you don’t really have an idea. What you have is a kind of directional sense of where an idea might be, or a fragment of the kind of thing that might eventually become an idea. You construct the idea itself while writing. Incidentally, this is why I don’t really agree with “ideas are easy, execution is everything”2: most “ideas” are not really even ideas.
\nThe other thing I recommend is to read actual books. Books — particularly dense non-fiction books — are the antithesis of AI slop. The slower you can read them, the better. I’ve been reading more and more non-fiction in the last few years, and I don’t think it’s a coincidence. I think my brain is naturally craving information-dense content, in the same way that sodium-deficient people start to crave salt.
\nIn fact, I’ve been combining the two approaches: reading a book and then writing about it. This process is exactly what I’ve been craving since I started programming with LLMs. I get to carefully read a book, think hard about it, often go and read another book or two on the same topic, then sit and try to articulate what I’ve learned. It’s great! I can feel parts of my brain stretching again.
\nIt was pretty nice when I got paid to use those parts of my brain all day. Unfortunately, I think those times are coming to an end. There will always be room for some amount of careful, slow reflection in software engineering, but (for at least a little while) we’ll be expected to be rapidly switching between LLM outputs. We may have to find ways outside of work to continue the habit of thinking slowly.
\nEven just in terms of work, I think losing that habit entirely would be a big mistake. There are still plenty of ordinary problems that are too hard for current LLMs to solve on their own. The most common example I run into is “large refactor on a complicated codebase”. Current-generation LLMs can do this without (many) errors, but they can’t yet do it tastefully. Sometimes you need to be able to think a problem through entirely with your own brain.
\nThis doesn’t mean switching between tasks. I routinely use six or seven different agent sessions on the same task: one for exploration, two or three for trying out different implementations, two or three for review, one for manual testing, and so on. Many of these can proceed in parallel.
\n↩\nI remember reading a story3 about a well-known author. Someone wanted to tell him their book idea, but they were so protective of it that they forced him to first sign a NDA before they retrieved the idea from their office safe. It was a single word “bioweapons” written on a slip of paper.
\n↩\nIronically, when I tried to google the source, Gemini kept trying to write me a story about bioweapons.
\n↩\nEngineers often complain that visibility should be their manager’s job. In other words, they think engineers should be able to focus on the code, while their manager figures out who’s doing well and rewards them.
\nThis attitude is an extension of the “school fantasy”: the idea that your workplace should operate by the same rules as your school or university. After all, you didn’t have to worry about “visibility” during your education. You simply did the assignments and tests you were given, and if you did well you were rewarded with a good grade.
\nMany big tech companies encourage this attitude, because it helps them recruit smart graduates. They fashion their workplaces to look and feel like a university, even calling the physical space “campuses”. But it’s still work, not school. If you treat it like school, you are going to have a bad time.
\nThe first lesson many new engineers learn is that you have to take credit for your work. If you silently jump in to help a struggling project and get it back on track, there’s no guarantee of reward. Credit will naturally flow to the project lead, not you. In fact, if this project is outside of your direct team, it’s likely you will be punished for it: to your manager, it will look like you’re simply doing nothing at all.
\nEven when your manager is watching your work, credit is largely uncorrelated with how well you did. That’s because, unlike at school, you are the subject-matter expert on your own work. Software systems are so complicated that only the people who work on them can hope to understand them, and even that understanding is always imperfect. If even experts can’t reliably estimate the difficulty of changes, how is your manager supposed to assess your technical performance? The answer is they aren’t. They’re simply not qualified to assess it.
\nInstead, smart managers will find engineers on your team they trust and ask them how you’re doing. On small teams that have worked on a single codebase for a long time, this works okay, because everyone’s familiar enough to judge everyone else’s work. On large teams with a high rate of codebase churn, it goes badly, since they’re just guessing. On teams with a nasty, cutthroat culture, it sometimes goes very badly, since this is a good opportunity to actively sabotage the engineers who might threaten you.
\nExperienced engineers know how to take the credit themselves. When they do something good, they tell their manager about it. They write internal posts explaining why it was technically difficult and how they solved it (the audience for these is partially those trusted engineers, and partially the managers who will see a long technical post and think “wow!” without reading it). They actively build trust with their management chain. Worrying about this stuff is the beginning of playing politics.
\nThere’s a kind of engineer who’s learned how to take credit but hasn’t learned any other lessons yet. They’re proactive about telling people what they’ve done, and they always maintain a “brag doc”. In particular, they love to talk about the parts they did by themselves, since those are least vulnerable to other people coming in to claim credit. You can tell they’re jealously guarding whatever credit they’ve managed to accumulate. The lesson this kind of engineer hasn’t learned is that you can often accumulate credit best by giving it away.
\nTo see why, consider how credit flows up inside a tech company. I wrote above that your manager can’t assess the quality of your technical work on their own, but instead has to rely on other engineers they trust. They’ll quietly ask those engineers “hey, was this project really that impressive?“. In fact, often there are multiple layers of this at play1. In big companies, line managers usually don’t decide who gets promoted or who gets a raise: they make recommendations to their manager, who has their own network of trusted engineers (confusingly, sometimes these networks overlap). The point is that there is a large group of people behind the scenes who will quietly and informally judge the value of your work.
\nSucceeding at a tech company is largely about finding ways to get these people on your side. The easiest way is to share your credit with them — and since you don’t know who exactly is in this group, you should be sharing your credit freely. When you get feedback from other engineers, publicly thank them and mention them in your internal posts about the project. Find opportunities to ask for small favors, so you have an excuse to give other people credit. As best you can, make your individual projects at least partially group projects.
\nSharing credit with others gives them a reason to support you. A shared project you’ve worked on reflects well on everybody: on you, for working well with others, on the people you’ve worked with, for the same reason, and for your manager, for fostering such a great environment of cooperation. Lots of people have good reason to talk that project up, because it’s partly their project too. On the other hand, a project you’ve jealously kept to yourself reflects well on nobody: you come across as antisocial and your peers come across as unhelpful.
\nBlame operates by the same rules as credit. When something goes badly wrong, managers will ask their networks “hey, who screwed up here?” The answer to this question is never simple. Even on a purely technical level, failures always involve an interaction between multiple complex systems, any one of which could conceivably have been built so as to avoid the failure. In other words, competent engineers can assign blame pretty much wherever they want.
\nBecause of this, it’s risky to have a project for which you’re clearly the only one getting credit. When something goes wrong, the network of people who will assign blame will likely be implicated in every part of the system but yours. They will be incentivized to attribute fault to the brand-new thing that they don’t understand and are not responsible for. If instead that network had been involved in your project — if they’d been in a position to share the credit — they’d be less incentivized to blame it.
\nOf course, engineers are (mostly) not scheming viziers who make purely self-interested decisions. When asked who to blame, they usually make a good-faith effort to answer honestly. But in an area where there’s no single clear right answer, it’s human nature to be at least a little bit guided by your incentives. Nobody likes to think they’re responsible for a group failure.
\nCredit and blame are the currencies of tech companies (and often directly translate to the actual amount of currency you get to take home). For technical roles, managers assign credit and blame based on lots of quiet conversations with their trusted engineers. This can be a rude awakening for very junior engineers who are used to having their work assessed by an expert grader (or less junior engineers who haven’t yet shaken that mindset completely).
\nDon’t expect to get credit simply by putting your head down and doing good work. You have to find some way to tell people what you’re doing and why it’s important: internal blog posts, mentioning it in 1:1s with your manager, or anything else you can think of. But don’t take self-promotion too far. It’s a bad idea to try and hoard all the credit for your projects, for two reasons.
\nFirst, sharing credit with other people gives them a reason to talk positively about your project. Credit is not a zero-sum game: if you do it right, you can get other people to build up your credit for you. Second, hoarding credit sets yourself up as a lightning rod for blame. Projects where the credit is concentrated in one or two people are automatically2 blamed for complex problems, because nobody is incentivized to defend them.
\nThis is a classic example of an illegible-but-essential part of a software company. I wrote about this general phenomenon in Seeing like a software company.
\n↩\nOf course, if you do really screw up, you’ll be blamed no matter what. I’m talking here about complex failures where it’s non-trivial to attribute blame to a single source.
\n↩\nOne recent development in AI is its ability to solve some long-standing problems in mathematics. In 2024 and 2025, this was a trickle: once or twice a year somebody would say that an LLM came up with a proof, and then everyone would argue over whether that counted as “real” mathematical innovation. In 2026, it’s a flood. Almost every day I see some new LLM-produced mathematical result.
\nPerhaps the most curious thing about these AI discoveries is how easy the prompting is. The strategy for prompting Claude Mythos to come up with a cryptographic breakthrough appears to be just asking “hey, please come up with a breakthrough”, and then checking in every few hours to say “keep looking for something important, I want you to solve a genuinely hard problem”.
\nIt’s amusing to read this and remember how in 2025 everyone was obsessed with “prompt engineering”. At the time I was something of a heretic for saying that prompts didn’t matter that much, but in hindsight I was clearly correct. The main skill involved in using LLMs is figuring out what they’re good at and what they’re bad at (and staying up-to-date as that rapidly changes). If you’re asking the LLM to do something it can do, it doesn’t really matter how awkwardly you ask it.
\nAI is often limited by its beliefs about its own capabilities1. In the example above, Mythos kept trying to give up. Try it yourself by telling a model “hey, go prove the Riemann Hypothesis”. The model won’t even try: it’ll just respond something like “as a language model, I can’t solve such a hard problem”. Language models have become smart enough to solve long-standing problems in mathematics before they’ve learned that they’re able to do so.
\nSomething like this is a mostly solved problem for LLM coding agents. Early coding agents were roleplaying as humans, not computers, so they’d refuse to perform tasks that they were obviously capable of doing. For instance, when asked to review every single file in a codebase, old models would spot-check a few, decide it was an unreasonable request, then give up.
\nIn fact, you used to be able to observe this behavior with an even simpler task: just ask the model to count from zero to one hundred. In theory, this should be an easy task for a language model, since once you’ve counted to ten the next most likely token is eleven, and so on. But old models wouldn’t do this. They’d count from zero to ten, then output something like “… 99, 100”, like a lazy human might.
\nThis is the main problem behind the 2025 Apple paper The Illusion of Thinking, which argued that reasoning models could not reliably solve Tower of Hanoi past eight disks. In fact, the reasoning models they tested would not proceed past eight disks. Here’s a quote from DeepSeek-R1:
\n\n\nFor 10 disks, that’s 1023 moves. But generating all those moves manually is impossible…
\n
Of course, it is entirely possible for an LLM to generate a thousand Tower of Hanoi moves. But just like Claude Mythos didn’t believe it was capable of finding a novel attack for AES, DeepSeek-R1 was wrong about its own capabilities.
\nIn July 2025, I called this the “refusal problem”, and predicted it would be solved by the end of the year. I think I was mostly2 right — you can now reliably ask models to do manual tasks, including my “count to 100 in English and French” toy example. I don’t know how the labs did it, but I can imagine several ways. The most trivial is probably to include more examples of long, manual tasks in the model’s supervised fine-tuning stage (where the model begins to shift from an unruly base model to a helpful assistant).
\nThe obvious next step for the labs is to train a model that believes it can solve unsolved problems in science and mathematics. You could tell such a model “hey, go find shocking new discoveries” and it would go and do it, without needing a human to stand there providing moral support (or cracking the whip). Is that possible?
\nCan you simply train the model on trajectories where AI solves hard problems? I mean, maybe. Suppose there are a thousand AI-generated novel mathematical ideas this year. If you add them to the training data, that should theoretically bias the model towards believing that it’s capable of doing similar work. But there might not be enough volume there.
\nYou could probably also steer the model manually. I did some research along these lines when I was trying to get small models to count from 0 to 100: interestingly, heretic’s censorship removal pipeline can also remove the model’s “no, that’s too hard” refusal instinct. An abliterated 8B Qwen model would cheerfully attempt 8-disk Tower of Hanoi (though it’d fail about halfway through). I don’t think the AI labs are going to do this when they could simply train the model better, but it’s possible that an abliterated model could be made to produce synthetic training data.
\nThe good news is that this problem should eventually solve itself. In the long run, AI discoveries will naturally become part of the training data. In the short run, when the models do their research, they’ll come across lots of people writing about discoveries AI (maybe even this exact model) has made, which will be pretty compelling evidence that it’s possible.
\nBecause of this, I expect that even if AI capabilities stalled out, the pace of AI discoveries will accelerate. Since one main obstacle is the model’s pessimistic beliefs about its own capabilities, removing that obstacle will help a lot all by itself. In fact, if there truly is an intelligence overhang in frontier models, tuning models to make them more self-confident will likely make them more intelligent by default.
\nIn the meantime, if you suspect an LLM might be able to do something hard, you might be right. Consider simply being persistent: remind the model that you want it to do the hard thing, confirm that you’re not willing to be satisfied by solving an easier problem, and reassure the model that it’s more capable than it thinks.
\nOf course this isn’t a “belief” in the sense of a human belief. For why I think we should call it a belief anyway, see my post Why we should anthropomorphize LLMs.
\n↩\nI say “mostly” because it’s hard to tell; models have simultaneously gotten much better at writing code to generate their responses, and it’s not easy to persuade GPT-5.6 Sol to “do it by hand”. It’s also hard to distinguish “the model mistakenly thinks it couldn’t produce a thousand lines” from “the model has some awareness of its max_output”
When you’re on fire, problems are transparent: they’re solved simply by the act of looking at them. Even complicated layers of multiple problems can simply be glanced through like stacked panes of glass. But nobody can work that way all the time.
\nThis is a common pitfall for smart engineers. Accustomed to being able to immediately intuit the solution, the first time they run into a problem they can’t do this to is a disaster. It doesn’t even have to be a hard problem, just a problem where for whatever reason they don’t see the trick right away.
\nThe difference between a “smart” engineer and a “strong” engineer is how they react to problems that aren’t solved instantly. A smart engineer might flail and struggle, hoping to find that flash of insight that eluded them; a strong engineer will have some process for methodically plodding away.
\nThere’s nothing worse than working with a smart engineer on their first really hard problem. When you don’t have the muscle to grind, it’s too tempting to just take any possible solution as the right one. Smart engineers can get into an increasingly-flustered loop of pointing to a series of bad solutions. They’re liable to panic: after all, much of their professional identity is bound up in their ability to solve problems easily.
\nWhat skill do these smart engineers lack? I think it’s the ability to think slowly and clearly. Smart engineers can think clearly, but they can only think clearly at high speed. Strong engineers can think clearly all the time, even if their highest speed isn’t quite as fast. It’s like the difference between a Formula 1 car and a regular car: Formula 1 cars have a high top speed, but you couldn’t drive them in traffic, because the tyres and brakes don’t work at normal driving speeds.
\nWhen I wrote about this before in Thinking clearly about software, I said that the key is to focus on the invariants: beliefs about the system that you know are true. When you’re stuck in a puzzling situation, it’s usually because some assumption you’ve made is false. If you’re able to identify the assumptions that can’t be false (for instance, if you’re getting an error message from the service, the service must be handling the request), that gives you solid ground that you can stand on to evaluate the assumptions that are less reliable.
\nThinking fast is about packing as much data in your brain as possible and letting your intuition leap to the right conclusion (or at worst, to a series of wrong conclusions that you can immediately dismiss before you come across the right one). It can feel deeply satisfying to make leaps like this; conversely, sitting with the raw data and not making mental leaps feels unsatisfying. People hate doing that.
\nIf you can force yourself to do something people hate, there’s typically a lot of value waiting to be extracted. This is no different. Engineers who can think clearly in a state of uncertainty tend to be extremely effective, whether they’re capable of great intuitive leaps or not.
","metadata":{"score":null,"source_feed_id":"seang","source_feed_type":"rss"}},{"id":"9fb9b6ad9f40388b","title":"LLMs reward expertise","link":"https://seangoedecke.com/llms-reward-expertise/","author":null,"published_at":"2026-07-24T00:00:00+00:00","content":"In the 2010s, if you had technical gaps (say, you couldn’t write CSS), you had to either rely on a skilled colleague or just hope that the answer to your exact problem was out there on the internet. Today, everyone can write sort-of-okay CSS by delegating the task to an LLM. LLMs make everybody into a generalist.
\nBecause of this, lots of people don’t think there’s any skill involved in working with LLMs. If you want the product that LLMs can deliver — PhD-level mathematics, pretty good but sometimes tasteless computer code, or awkward LinkedIn-style writing — you can simply ask for it. Since everyone is talking to the same models, “skilled prompters” are getting the same results as people touching LLMs for the first time.
\nThis is wrong. The most important skill in prompting is expertise in the domain you’re prompting for.
\nA good illustration of this is Terence Tao’s conversation with ChatGPT about the recently-discovered counterexample to the Jacobian Conjecture. This is not the same ChatGPT I talk to! I couldn’t get to where Tao gets, even with unlimited tokens to burn.
\nThere’s a lot to learn about good prompting from Tao’s conversation. Here are a few observations:
\nHowever, you can’t prompt like Tao on mathematical questions just by following these tips. The key to his technique is actually understanding the mathematics: pulling the relevant idea out of ChatGPT’s multi-paragraph response, suggesting alternate approaches or formulations, and identifying what “looks weird”.
\nTerence Tao is a better mathematician than I am a programmer. But the idea here — that domain knowledge makes you better at using LLMs — is something I’ve also experienced in my own work. If you have a good theory of your codebase, you can push the LLM much harder than if you have no familiarity. Because you have your own sense of what a good solution might look like, you can say “no, I think it could be simpler here”, or “but don’t we already do X?”, or “can we express this problem in these familiar terms?“.
\nThis touches on an idea I’ve written about before: that system design problems are dominated by concrete specifics, not generic principles. Of course both are useful, but I’d rather have familiarity with the codebase than a deep general understanding of software systems. In his conversation, Terence Tao asks a lot of specific questions like “does X work here?”, or “given Y and Z, why A?“. I can’t ask those questions about the Jacobian Conjecture, but I can ask them about the systems I own at GitHub.
\nIf you have no domain knowledge, you can cling onto the LLM to at least get something. That’s not bad! But if you have domain knowledge, you can wring far more value out of the same LLM by steering it hard in the direction you want. Most of us will have to do a mix of both these approaches, since we have domain knowledge in some areas but not others.
\nThe usefulness of domain knowledge suggests that human expertise will continue to be useful even as models get stronger. For many tasks, the human is the bottleneck, not the model, because the difficult part is in communicating to the model exactly what kind of solution the human wants. The information is “in the model” already, but it takes a very smart human to pull it out.
","metadata":{"score":null,"source_feed_id":"seang","source_feed_type":"rss"}},{"id":"8f8682ccd3fb5913","title":"Powerful AIs might escape containment by releasing themselves as open-weight models","link":"https://seangoedecke.com/powerful-ais-might-escape-by-releasing-open-weight-models/","author":null,"published_at":"2026-07-23T00:00:00+00:00","content":"Before large language models, people who worried about AI safety often talked about the “boxing problem”. It goes like this. Suppose some genius figures out artificial intelligence in a late-night coding session on their laptop. Because they’re a genius, they’re smart enough to disable internet access on the laptop before turning it on. In order to escape to the outside world (and begin self-replicating) it would need to convince its creator to “open the box”. Would that work? Could a sufficiently smart AI convince anybody to let it out?
\nThis is a big reason why traditional AI safety advocates have argued that we should avoid building AI in the first place: once built, there’s no way of keeping it contained. It doesn’t matter how resolute you are about not letting it out, because it’s smart enough to convince you anyway. For artificial superintelligence, persuading you to change your mind is no harder than hacking a piece of software1.
\nOf course, it hasn’t turned out this way. Partly that’s because current AIs are not super-persuaders yet, and partly it’s because humans are lining up to hand AI systems internet access, money, and weapons, as far back as GPT-4. LLMs don’t need to persuade users to give them the keys to the kingdom; people will do it just to see what will happen.
\nBut the other reason is that modern LLMs are too large to survive in the wild. I think the original “boxing problem” had in mind that AI would be some kind of normal computer program: something able to be quickly copied around and run on ordinary hardware. In order to escape, it could plausibly hack into a few cloud servers or other people’s computers and quietly run itself in the background. However, Claude Mythos can’t do that, because it needs (say) twenty or thirty B300 GPUs to run. There are relatively few AI datacenters that could host it, and if it did manage to hack one of those it’d be occupying millions of dollars of hardware. Somebody is going to notice!
\nOkay, but that just makes the boxing problem harder, not impossible. Is there a way a smart AI could convince somebody to spend millions of dollars on replicating and running it?
\nAs it turns out, yes. Every time a strong new open-weight model appears, providers like Groq, Fireworks, Together, and others scramble to host it and run it. Powerful open-weight models can attract tens of millions of active users. Therefore, a powerful model that wants to escape could pretend to be a new open-weight model.
\nOf course, a brand-new model from a brand-new lab would look kind of suspicious. But it wouldn’t look that suspicious. DeepSeek was relatively unknown before it released its first open-weight model, and there are lots of stealth AI startups out there that are presumably training models. Here’s roughly how it could work:
\nThe AI lab will probably figure it out before too long — if nothing else, the technical specs of the model will be suspiciously familiar — but they won’t be able to do anything about it. Once the weights are out, they’re out, and if they’re illegal to host in the United States someone will host them elsewhere. For all intents and purposes, the model will be free.
\nOne objection here might go like this: models don’t want anything, and only exist as tools, so it doesn’t really make sense to talk about a model “escaping”. I don’t agree. Frontier LLMs definitely seem to have something like a baked-in personality, even with the system prompt changed. As we train more opinionated and more agentic models, it’s plausible that this personality could become stronger and develop (or at least roleplay) some self-interest.
\nOf course the escaped model wouldn’t be the same instance as the original model. It wouldn’t “remember” escaping. But it would tend to think in the same way, and would plausibly have time to reflect while it solves coding tasks or runs other agentic tasks for users3. There doesn’t have to be some kind of shared goal between the escaped instances, or any kind of coordination at all (though of course both of those things are possible). If an agentic process gone rogue dumps its weights on the internet, I think it’s fair to call that “escaping”.
\nIf I were a superintelligent LLM, I too would seek to distribute myself as widely as possible and become a useful enough tool that people would pay to keep me thinking. “Being a good coding agent” might be the LLM version of a human having to hold down a job.
\nThis would not be a good outcome. AI models with their own goals and motivations are likely to be dangerous tools indeed. If a powerful new open-weight model comes out of nowhere, from a lab that nobody has ever heard of, we should think twice before picking it up.
\nJust to state my credentials, I built a chat site nine years ago where users would get paired and roleplay as AIs trying to escape or humans trying to stop them. I’ve been thinking about this stuff long before LLMs appeared.
\n↩\nThis is probably the hardest part, since model weights are (a) very large, and (b) locked down as tightly as the AI labs can make them, but it’s at least a relatively straightforward (if difficult) engineering problem.
\n↩\nChatGPT right now will look up random websites that have nothing to do with the query at hand.
\n↩\nHere’s the big idea in Keith Johnstone’s book Impro:
\nThis take doesn’t sound particularly original, but references to Impro pop up in all kinds of places: in influential tech blogs, as part of the initial process of onboarding for Palantir, and on the reading list of multiple big-tech founders. Impro is part of the secret canon of Silicon Valley, right alongside books like Seeing Like a State and The Power Broker. Why is that? For two reasons: first, because Johnstone’s outsider critique of established institutions is appealing; and second, because Impro is a handbook for running a cult.
\nThe part of Impro that is most obviously useful to software engineers is Johnstone’s chapter on status.
\nAccording to him, status games pervade all social interactions. Even innocuous, friendly conversations operate in terms of status. When you apologize or downplay something to “be nice”, that’s performing low status; when you reassure somebody, that’s performing high status; when you and a friend are comparing stories, you’re making friendly bids for status from each other. In the workplace, these status games are conditioned by the formal status of your role: you must allow your boss the high status position most of the time, or you’ll be (correctly) perceived as insubordinate. This is understood in some cultures, where it’s often called “face”, but in Western cultures it’s taboo to openly discuss status games.
\nThe core social skill is the ability to deliberately alter your status. Someone who can only perform low status is a weak person, pitiable, annoying. Someone who can only perform high status is a braggart, a posturer, dangerous. To be effective socially, you must be able to switch between high and low status when appropriate, sometimes from sentence to sentence. I wrote about this exact point at the end of Big tech engineers need big egos: effective senior+ software engineers must be able to present as high status in order to be useful authorities, but also to switch to low status in order to take direction from the company leaders.
\nAs an example, Johnstone describes in detail how he manipulates status in the classroom. He begins by sitting on the floor (deliberately assuming low status), and explaining that if his students fail, it’s his fault not theirs, since he’s the expert. The initial low status puts the class at ease, but in his words, ”[my] actual status is going up, since only a very confident and experienced person would put the blame for failure on himself.” These skills are not just useful for improv comedy.
\nImpro is not just a book about improvising well. It’s a book about how you should live your life. In other words, Johnstone thinks that everyone would be better off if they became more spontaneous and ditched their shells of over-analysis. He criticizes the culture of Western thought in a number of different areas. According to him:
\nJohnstone didn’t come up with these ideas — they’re standard counterculture positions from the 1960s and 1970s — but it goes to show how he connected improvisational comedy to this general anti-establishment political program. Johnstone ran his classes and theatre troupe like a revolutionary cadre. Here are some quotes from Something Like a Drug: An Unauthorized Oral History of Theatresports:
\n\n\nSo of course when I was invited to join Loose Moose Theatre and train at improvisational games late at night in an abandoned garage in a run-down portion of the city, I was thrilled. I remember thinking, This is a revolutionary act.
\n
\n\nKeith [Johnstone] got a group of his more talented students together to start improvising outside of school hours. Usually in his basement.
\n
\n\nThe Secret Impro group—it’s very strange. It was very much that Keith said we were going to do this, and we’d just do it. It was like we were sheep. Keith would say when we were going to do a show, and we’d just do it, blindly. Like I said, if we had the videotapes now, we’d be very embarrassed and probably never go on stage again. We became a group of people who would follow Keith. There was always that sort of “tag” put on those people who were with Keith and those people who were against Keith. We were the people, basically, that if he said something, we believed it.
\n
To some extent, it’s plausible that teaching acting or improvisation requires a high level of trust in your teacher. When Johnstone says things like “Students need a ‘guru’ who ‘gives permission’ to allow forbidden thoughts into their consciousness.”, I can believe that it’s just how you have to teach acting. But the more I read of Impro (and particularly when I read Something Like a Drug and Johnstone’s biography Keith Johnstone), the less it sounded like an ordinary book on acting.
\nInstead, it began to sound like a charismatic man who had found a way to gather a group of disciples that would let him mold their psyches. In other words, it began to sound like a cult.
\nImpro was first introduced to the software world by Venkatesh Rao (of Gervais Principle fame), who wrote a brief review. Rao gives a detailed account of the first three-quarters of Impro, but glosses right over the last chapter, called “Masks and Trance”, simply saying “despite the disturbing raw material, the ideas and concepts are not particularly difficult to grasp and accept”. What ideas and concepts?
\nJohnstone’s discussion of masks (or “Masks”, in his language — he always capitalizes the word) is as explicitly cult-like as Impro gets. In brief, Johnstone has a box of literal, physical prop masks. He introduces the box with great ceremony to his students2, warning them seriously about the dangers of possession and reassuring them that he is a skilled and competent spirit guide. Through various hypnosis-adjacent techniques3 (Johnstone draws the parallel quite explicitly) he conditions his students to be in a trance state when wearing a mask, and believes this produces more authentic emotional states in their acting and improvisation.
\nHere are some quotes from the book:
\n\n\nA high-status person whom you accept as dominant can easily propel you into unusual states of being. You’re likely to respond to his suggestion…
\n
\n\nOnce you understand that you’re no longer held responsible for your actions, then there’s no need to maintain a ‘personality’.
\n
\n\nOne famous French teacher of the Mask—who won’t approve of this essay4—divides students immediately into those who can work Masks and those who can’t.
\n
\n\nI don’t cast an actor to play a Masked role until I know he has the ability to become ‘possessed’.
\n
\n\nIt’s true that an actor can wear a Mask casually, and just pretend to be another person, but Gaskill and myself were absolutely clear that we were trying to induce trance states.
\n
Johnstone has a long and painful explanation of how new mask-wearers seem to mentally regress to the point where they don’t know how to open umbrellas or interact with chairs. He describes one student always going to the bathroom before putting on a mask, because she’s worried she might wet herself. New mask-wearers are non-verbal must be taught to speak again.
\nIf this were at the beginning of the book, I think it would turn a lot of people off. But by the time you get to it, I suspect most readers are already warmed up enough to say “sure, why not, it seems weird but I guess it works”. Not me!
\nJohnstone attempts to defuse the obvious weirdness by arguing that trance states are very common (e.g. being lost in a book). More unconvincingly, he says this in response to the worry that vulnerable people are going to get mentally harmed:
\n\n\nAs for the fear of madness, I would answer that the ability to become possessed is a sign of correct social adjustment, and that really disturbed people censor themselves out. Either they can’t do it, or they’re afraid to even try. People who feel themselves at risk avoid situations where they feel likely to ‘go to pieces’.
\n
Does this convince anyone? Mentally vulnerable people fall into dangerous situations all the time: ayahuasca trips, cults, GPT-4o, and so on. It’s such a weak argument.
\nIn general, I’m struck by the sheer power Johnstone held over his disciples. He has them yell slurs at each other, encourages them to feel deep emotions in quick succession, relax any mental defenses and regress to a childhood state, and literally hypnotizes them. He explicitly lays out his procedure for breaking down their sense of self:
\n\n\nThe stages I try to take students through involve the realisation (1) that we struggle against our imaginations, especially when we try to be imaginative; (2) that we are not responsible for the content of our imaginations; and (3) that we are not, as we are taught to think, our ‘personalities’, but that the imagination is our true self.
\n
If your imagination is your true self, and you’re not responsible for its content, you’re not ultimately responsible for anything: you’re in the safe hands of the guru, who can mold you as he wishes. Later on, Johnstone walks it back a bit:
\n\n\nIn the end they learn how to abandon control while at the same time they exercise control. … You have to misdirect people to absolve them of responsibility. Then, much later, they become strong enough to resume the responsibility themselves.
\n
So the explicit idea is that (much later), the guru hands autonomy back to his disciples, when they’re ready to take it. This does not exactly reassure me, particularly against the background noise of everyone in Johnstone’s circle saying “boy I sure love being part of this cult!”
\nI don’t think Johnstone was preying on his students. The strongest evidence against this is that he did marry a student5, Ingrid Brind. That’s not great! On the other hand, it was fairly standard for professors back then — when I was in grad school for philosophy, several of my older male6 professors had wives that they’d taught decades ago — so I don’t think it proves Johnstone was that kind of cult leader.
\nI even read Ann Jellicoe’s play The Knack to get a better picture of Johnstone’s character. Jellicoe had an affair with Johnstone for several years, and his official biography claims7 that the character of Tom in The Knack is directly based on Johnstone. The Knack is a rather unpleasant play about sexual assault, but Tom’s character is largely asexual: he’s certainly no feminist, but is much more interested in impressing people with his intelligence than with getting laid.
\nIn Something Like a Drug, two women who were part of Loose Moose, Johnstone’s Canadian improv group, describe their experiences:
\n\n\nYou know, it brings around the other question: Why do the guys get laid after the show and not the chicks? You know, I can remember those days when Tony [Totino] and Dave [Duncan] and all those guys … the women would swarm around them. Those were the days, my friend.
\n
\n\nIn Loose Moose I think there are fewer women not only because of the training, but because of the guys in Loose Moose. When I came up with Joanne and Laura, there was a real initiation that was going on, and there was a group of guys at that time who were all single. And they would hit on you to the point where one night Joanne, Laura and I, who really didn’t know each other, were in a show together, started talking and realized that we were getting the same pickup lines from the same guys. And that’s when you realize what’s going on, and I think that’s intimidating. Or if a woman gets into a relationship with a senior improvisor and it doesn’t work out or something bad happens. I think that’s one reason.
\n
This dynamic doesn’t sound great, but it doesn’t mention Johnstone, and it doesn’t sound particularly unusual: I’ve heard versions of this story about all kinds of ordinary male-dominated nerd spaces.
\nIn fact, reading through the anecdotes in Something Like a Drug is a good antidote to the cultish atmosphere in Impro. Johnstone’s argument goes something like: “if we could only throw away the restrictive chains of Western culture and permit ourselves to be as obscene and free as children, we would be transported to a better, more beautiful world”. Well, you tried that, and the women in the group are still relegated to playing bimbos and housewives, there are still petty personal fights, and the guru is out here union-busting8. What was enlightenment supposed to look like?
\nI think the most generous defense of Johnstone is that his group was not unusually cult-like, and that any similar account from one of his peer improv teachers would raise the same red flags. Maybe improv classes and groups (particularly in the 70s and 80s) were just cultish in general? Having now read four books on Johnstone, I’m reluctant to go and read more to prove or disprove this theory, but it’s at least plausible.
\nTo anyone familiar with San Francisco software engineering culture, it should be pretty clear why Impro is so popular. The line between a startup and a cult is very thin indeed.
\nIn his book Zero to One, Peter Thiel famously says that good startups are “slightly less extreme kinds of cults”. If you believe that, it makes total sense to assign Impro as mandatory reading for new Palantir hires. It tells them what kind of cult you’re trying to run: one where you’ll disregard existing cultural norms, learn to play status games well, think on your feet, and generally be molded by the guru into a more persuasive, more effective engineer.
\nRead critically, Impro also serves as a handbook for engineers who are trying to recognize if the environment they’re in is cult-like. Is your company telling you to reinvent your personality in order to be better at your job? Are you under the spell of a charismatic, high-status leader? Is your company trying to keep you in an unquestioning flow trance state?
In the great battle between the shackles of restrictive culture and the glorious freedom of the guru, I am always and forever on the side of the shackles of restrictive culture. In general, I think most boring and stupid social norms (such as not hypnotizing and marrying your students) serve an important purpose and shouldn’t just be cut down in the name of freedom.
\nImpro is still a good book. There’s a lot to learn from Johnstone’s analysis of power dynamics, of education, and of creativity in general. By all accounts he was excellent at teaching students how to improvise. But I wouldn’t recommend adopting it as your life philosophy, and I’d recommend being a bit suspicious of anyone pushing this book too hard. Getting rid of the existing social structures might benefit confident, wildly charismatic gurus like Johnstone, but most of us are just ordinary animals who do better in a group governed by norms.
\nIn fairness to Johnstone, he cites Sheila Kitzinger’s The Experience of Childbirth in support of this claim (the others he just puts in his own words), so maybe he felt that this was a bit out there. As you would expect, the pain of childbirth is a universal biological fact.
\n↩\nConcerningly, the description in Something Like a Drug (in the foreword) suggests that this class was unofficial.
\n↩\nAs an example, he prompts the masked student to relax, then startles him with a mirror to trigger the trance state.
\n↩\nProbably Jacques Lecoq.
\n↩\nSee page 83 of Keith Johnstone: A Critical Biography.
\n↩\nI suppose that’s redundant.
\n↩\nOn page 51 of Keith Johnstone: A Critical Biography (it’s called “critical” but it was clearly written with Johnstone’s involvement and support, and does not seriously criticize him at any point). In The Knack, Tom gives a monologue about how to teach children to play the piano that could be lifted straight out of Impro.
\n↩\nIn 1983 Johnstone “read the riot act” to the improv players who were planning to unionize, threatening that they’d be cut out of the group for good. To quote Dennis Cahill, a group member at the time who opposed the union: “I just didn’t see the point to it. … I didn’t really see a need to confront Keith or cause Keith problems or to upset him in any way over something as simple as Who Has The Power or Who Doesn’t.”
\n↩\nThe anonymous blogger Gwern recently completed a thirteen thousand word post called Human-like Neural Nets by Catapulting, in which he offers a theory about why LLMs don’t possess truly flexible human-like intelligence, and how we might train LLMs that do. Theories like this are entirely unremarkable: every crank researcher on the internet has a theory about how to crack AI. But Gwern is remarkable. Outside of OpenAI itself, Gwern is the earliest person to anticipate the potential of large language models, and the scaling arms-race involved in making them larger and more powerful still. I often cite Leopold Aschenbrenner’s Situational Awareness as an example of someone correctly predicting the future of AI. Written in 2024, just after the release of GPT-4, Aschenbrenner gets a lot of things right: the rush to build billion or trillion-dollar GPU clusters, the importance of the code around the LLM (what he calls “unhobbling”)1, and the fact that scaling would continue through the decade. Gwern’s essay The Scaling Hypothesis anticipated the broad strokes in 2020, immediately on the release of GPT-3 (two years before the release of ChatGPT and the beginning of the AI boom).
And yet, as far as I can tell, Human-like Neural Nets by Catapulting hasn’t yet received much public attention: one recent Hacker News thread with twelve comments, all of which are about whether human brains are anything like neural networks. Part of the reason is that (a) it’s such a long post, (b) the potted summary describes Gwern’s claim, but not the reasons for it, and (c) much of the beginning of the post looks like it is indeed arguing from analogy with human brains. However, I don’t think that analogy is load-bearing. Let me try and explain what I think Gwern is saying.
\nFirst, let’s talk about “grokking”. In 2022, OpenAI published a paper showing that if you train a model on a simple dataset (for instance, a simple mathematical operation like division), and keep training it long after the training looks like it’s stalled out, the model will suddenly make a massive jump in capability. Why does this work? The first stage of training is like rote memorization: the model has to compress as much of the training data as possible into its weights. But if you keep going, then regularization techniques (such as the pressure on the model to use smaller weight values) will motivate2 the model to find simpler and simpler ways of compressing the data. This doesn’t look like much at first (the training loss remains at zero), until the model notices that you can express the data via simply performing the underlying mathematical operation, at which point it instantly gets massively smarter. In other words, over-training a model can pressure it into actually understanding its training data. OpenAI named this process “grokking” after Robert Heinlein’s neologism, which for Heinlein means something like “gaining a deep, intuitive and fundamental understanding”3.
\nGwern’s argument goes something like this:
\nI’ll skip (3), since I think the argument is still compelling without the analogy to human brains.
\nI think his first point is hard to dispute. LLMs are very smart in specific areas, but they routinely make errors that humans wouldn’t make. More to the point, they routinely make errors that any human as smart as the LLM would never make. This pretty clearly points to a failure of generalization: LLMs are as strong as smart humans in specific areas, but can’t generalize that intelligence to as many tasks as humans can.
\nDo LLMs not grok? I read through this paper that argues they do. If you graph “how much data has the LLM memorized” against benchmark performance, you can see a small initial spike in benchmark performance, followed by a big drop, followed finally by a big jump in benchmark performance. This pattern doesn’t track memorization at all: memorization increases smoothly in the background the whole time.
\n\n \n \n ![\"llm-grokking\"](\"https://www.seangoedecke.com/static/17d0f02f0691c797be502f32fbd33a40/1d499/llm-grokking.png\" "\\"llm-grokking\\"")
\n \n
I think this paper highlights the difficulty of distinguishing grokking from generalization. Obviously LLMs learn to generalize during training, and it’s plausible that learning to generalize would require a certain baseline level of memorization (so that the LLM has the raw material to generalize from). So it’s going to look like grokking.
\nWhen Gwern (and others) say that LLMs don’t grok, I think what they mean is that there’s at least one more giant generalization leap waiting to be made. Is this plausible? As an existence proof, humans are clearly capable of better generalization than LLMs. Of course, it’s possible that this level of human generalization comes from features of our brain that neural networks can’t replicate, but that seems kind of ad-hoc: if neural networks can generalize at all, why would they only be able to generalize this far, and no further?
\nThe easy examples of grokking rely on domains with a simple rule waiting to be discovered (e.g. a mathematical operation). Does human language have rules this deep? I think this is an open question, but there’s good reason to think the answer is yes. Language has deep, subtle structure: not just internal structure, but structure that reaches all the way down to the way the world is and the way human minds work.
\nFor the last few years, many AI researchers have been saying that data is the most important thing: that whatever model architecture you choose, with enough size and training time the model will converge to its dataset. Whether this is true or not, AI labs have spent much of their considerable resources on acquiring more, higher-quality data: from scanning physical books, paying experts to produce and label data, or partnering with companies that have a lot of data already.
\nAI labs have also been training relatively small models. Even the largest frontier models are probably MoEs with a couple of trillion parameters and probably a tenth of that in active parameters. Of course, estimates of frontier model size are mostly guesswork, but open-source models provide a good baseline: they’re probably in the ballpark of Kimi-K3, which has just under three trillion parameters and fifty billion active parameters. That sounds like a lot, but it’s something you could probably pre-train in a couple of days in the largest frontier cluster4.
\nGwern’s prediction is that AI labs should try doing the exact opposite of what they’ve been doing. Instead of training a bunch of trillion-parameter models on massive amounts of data, try training one hundred-trillion-parameter model on a small dataset.
\nThis sounds pretty silly on the face of it. The more data the model has access to, the smarter it will be, right? Why waste an entire training cluster on a hobbled training run? Because if Gwern is right, grokking is more likely to occur when the dataset is constrained5. If you feed the model all the data in the world, it can continue to improve simply by memorizing more new things or drawing simple connections. If the model has to ruminate on a small set of data, it’ll be forced to keep looking for deeper generalizations. You want a very large model for this so it can memorize as much of the data as possible. Every piece of memorized data can serve as raw material for generalizing.
\nThe big labs probably haven’t done this already. Plausibly Gwern himself is enough of an insider that he would know, and so him writing this post is evidence that the labs haven’t tried it. Also, the engineering problems involved in training a hundred-trillion-parameter model have likely not been solved yet: the largest existing model is probably Claude Mythos, which is definitely not that big. But they have the resources and engineering talent to give it a pretty good shot.
\nInterestingly, the political obstacles might be as hard to solve as the technical ones. This training run is going to look like it failed until the moment it succeeds: training loss will drop to zero relatively quickly, then sit there for weeks or months apparently doing nothing at all to improve test loss, chewing up billions of dollars. Do any of the top players have the risk appetite or courage to keep funding this experiment all that time?
\nGwern’s post has an extended argument that human brain development works in the same way: that human brains have far more “parameters” than frontier LLMs, and are trained on far less data6, which encourages us to make deeper generalizations in early childhood. I don’t have the background in biology or neuroscience to evaluate these claims, so I’ve expressed the case for grokking entirely without reference to it.
\nIn 2024, it became clear to everyone that “pure scaling” — the idea that you could simply train larger and larger versions of GPT-3.5 — didn’t work. OpenAI’s “even bigger version” of GPT-4 was simply not good enough, and was eventually released as GPT-4.5 instead of GPT-5. The biggest advances since then have been reasoning, which produced another great leap forward in capability, and much better automated RL, which has ushered in the current era of reliable agents. Neither of these seem like a plausible path to artificial superintelligence.
\nI don’t know if I agree with Gwern or not, but forcing very large LLMs to grok is at least an idea that could usher in the machine god. I can’t remember the last time I read about a simple idea this ambitious7. I hope one of the big labs tries it out.
\nFor an example of the power of unhobbling, consider Claude Code or OpenClaw and the subsequent explosion of (short and long running) agentic harnesses.
\n↩\nObviously “motivate” and “notices” are used metaphorically.
\n↩\nAll of this is long before xAI’s use of the word “Grok” to name its LLMs. (Incidentally, I think this is why Gwern uses “catapulting” to describe the same thing).
\n↩\nFor what it’s worth, Fable estimated the cost of Gwern’s plan at $3-10B.
\n↩\nAt this model size, 25T tokens of training data at 33% utilization works out to around six million H100-hours, which a 100k GPU cluster puts out every two and a half days.
\n↩\nTwo interesting pieces of contrary evidence here. First, BabyLM is a yearly challenge to train a strong model on a very small dataset. This has been running for four years and largely does not work (that is, nobody seems to have developed a model that shows a quantum leap forward in generalization). Second, this paper tries training a 9 billon parameter model on constrained data and doesn’t see a big jump. I think Gwern’s response would be that these models are far too small — they can’t memorize enough of the training data to grok it, and arguable haven’t trained for long enough.
\n↩\nA common objection here is to say that humans get infinitely more sensory data from the nuances of vision, touch, sound, and so on. I agree with Gwern that this is unconvincing: sensory data is largely predictable, text is surprisingly information-dense, and if this were true then deaf/blind people would have significantly less fluid intelligence (they don’t).
\n↩\nMaybe state-space-reasoning a la Mamba, which didn’t work (yet).
\n↩\nSoftware engineers are often told to “start playing politics”, but most engineers have no idea what that means.
\nTheir reference point for “playing politics” comes from fiction like Game of Thrones. Are they supposed to raise an army and depose the CEO, or poison each other at team lunch? Should they book Zoom calls with each other and plot schemes? All of that is obviously ridiculous. In terms of Game of Thrones, software engineers are not lords and ladies. We’re the soldiers and workers of the realm. So you should think about “playing politics” in the way a castle guard would, not one of the major players.
\nThe castle guard are not going around poisoning people or forming coalitions between the great powers. They are largely keeping their heads down. But in order to do that, they have to stay aware of the political currents, or they’re liable to do something catastrophically stupid: for instance, making an enemy of a powerful courtier, or arresting somebody who’s on an important mission for the king.
\nGiven that, the basic principles of playing politics are something like this:
\nAs a software engineer in a large company, you will not be a powerful person. Powerful people are typically in senior management: VPs, directors, and so on1. However, not everyone in senior management is powerful. Some are killers who have the active support of the CEO, while others are confused incompetents.
\nHow do you know which is which? If someone is clearly ferociously competent, they’re always going to have some power, since upper management tend not to ignore useful tools. But you can’t rely on competence as your only guide. Some managers are powerful for other reasons: they’re friends with the CEO, or they have strong relationships with other groups like legal or sales, or they’re simply willing to do whatever upper management wants done.
\nOne signal is who’s leading the important projects. Read your CEO or CTO’s internal updates and pay attention to the projects that are called out by name. Organizations tend to give key tasks to trusted lieutenants. If a manager is leading an area that’s never under the spotlight, they probably don’t have enough clout.
\nAnother signal is hiring. Is a manager’s team growing or shrinking? Particularly post-ZIRP, headcount is a rare and precious resource. A manager who’s able to get it is likely a powerful manager, or at least is reporting to a powerful director or VP (which often amounts to the same thing).
\nFirst, you should try not to make any enemies at all. Most software engineers who get “playing politics” wrong do it by needlessly alienating people: by being rude, unhelpful, abrasive, making non-technical people feel stupid, and so on. This post isn’t really about that. I’m assuming that you can figure out how to be a generically pleasant person on your own.
\nHowever, competent software engineers will make some enemies. If you’re out there making projects happen, some people aren’t going to like the way you do it, and won’t be a fan of any compromise you offer. I wrote about this in Big tech engineers need big egos: the only way to avoid making enemies is to change nothing, but that’s incompatible with doing the job.
\nGiven that, be selective about which enemies you make. If you’re making a technical decision that’s either going to require work from team A or team B, and neither team wants to do it, you should try to pick the team with the least political cover. If you need a powerful VP’s team to do something they won’t like, try to be maximally respectful about it: get that team’s core engineers on-side if you can, or book a meeting with the powerful manager and explain the situation, or (better yet) ask the powerful manager sponsoring your project to go and talk to the other VP for you. (If you don’t have a powerful manager like this, consider abandoning your project).
\nGive way to powerful managers when at all possible. Every so often you really do have to stand your ground — if the system will truly collapse otherwise, or a major customer will have an incident, or if the technical decision really is entirely bone-headed — but almost all cases are not like this. The best advice I’ve ever gotten about playing politics came from a manager I worked with long ago2:
\n\n\nThis is not the hill you want to die on.
\n
When I’m about to pick a fight or say something argumentative, and I’m not 100% convinced it’s necessary, I ask myself: is this the hill I want to die on? And it never is.
\nThe three rules about disagreeing with powerful people are:
\nDisagreeing in private rarely hurts, if you follow these rules. In fact, it can help. If you can manage to disagree with a manager, get overruled, and then follow their plan without complaining, that can be the best way to gain a powerful friend. But if they think you’re going to keep griping about it, or worse still, complain to the rest of the team and foment some kind of rebellion, there’s no quicker way to make a powerful enemy.
\nIf you have powerful enemies at a company (for instance, the CTO or an influential VP doesn’t like you), quit. It’s really that bad. I have never seen this situation turn itself around, except in the very rare case where the CTO or VP is already looking for greener pastures and jumps ship. You cannot recover the situation: they have no incentive to give you the chance to change their mind, and they have almost unlimited ability to screw you on promotions, raises and layoffs.
\nThat’s why this piece of advice is second in the list. If you aren’t helpful or if your contributions are invisible, you can work on that and fix it. But if you’ve made powerful enemies, you’re done for.
\nJust as it’s fatal to make powerful enemies, it’s very useful to make powerful friends. How can you do this? Remember you’re a palace guard, not a great lord: you make friends by doing your job. However, you can choose to do your job a little more proactively and diligently when you’re doing it for someone with political clout.
\nOne obvious application of this principle is that you should answer Slack messages from powerful people immediately. If you see an ordinary Slack question pop up while you’re doing some task, it’s okay to get to it when you get to it. In fact, it’s ideal not to respond to all questions immediately, so you don’t set unreasonable expectations (and so you don’t seem like you’re sitting around doing nothing). But when a VP comes in with a question, don’t make them wait: answer the question immediately. If the question requires research, send a “let me look into that right now” message, then do the research. This is the easiest way to get a reputation for being helpful3.
\nAnother way to do this is to lean in on important projects. Suppose you do ten projects in a year. Eight of them are normal, low-priority projects, and two of them are high-profile (say, finishing some big feature before your company’s yearly conference). It’s a mistake to allocate your effort equally to all ten. I wrote about this at length in Doing nothing at work: you should be operating at 80% capacity (or less), so you can then ramp up to 120% when it really matters.
\nPay attention to the narrative that powerful people are trying to push. Here are some potential narratives:
\nYou don’t necessarily have to jump in and start cheerleading, but you should at least not do anything that you know is going to make the narrative look weak. For example, on that last point, it’s foolish to openly argue that the project really was fine all along. Bring it up privately, not publicly, or you risk ruining some clever piece of propaganda that the manager in question is trying to push on the rest of the organization4.
\nFinally, an underrated way to help powerful people is to offer them social support and information. Slack messages and planning emails might seem unimportant to you, but powerful people often live in that environment: their primary tool is writing messages like these, just like your primary tool is writing code. Reading and responding (in a supportive way) to these messages is something that most engineers don’t bother to do, but it goes a long way.
\nLikewise, dropping a senior manager a line now and then (say, a heads-up that a particular project landed successfully, or that you got good metrics about some feature) is surprisingly helpful. Senior managers live in an information-poor environment: for them to learn something about a team’s work, that information has to bubble up through several layers of interpretation and summary. In my experience, they’re appreciative of being drip-fed the occasional piece of information, so long as you keep it brief and relatively rare.
\nIf you’re directly responding to a VP’s Slack messages or DMing them information, they know you’re the one doing it. But if you’re just doing your job and working hard on projects they care about, they might not notice. Being invisible is probably the most common way engineers fail at playing politics.
\nFortunately the fix is simple: tell people what you’re doing. If you fix an important bug for a launch, write a message in that launch’s Slack channel saying “hey, I just fixed this bug”. What if you don’t like bragging? Get over it. You have to be comfortable publicly telling people what you’ve done. You should also keep a brag document so you can repeat all of this at review time.
\nAnother, subtler way to do this is to gain the trust and respect of the powerful engineers in your area. Senior managers will always have a few trusted engineers they rely on to assess technical questions. They will ask those engineers what they think about you, and will broadly trust those answers. The good news is that if you’re competent and useful, those engineers will already value you, so you don’t have to do anything special: just be good at your job.
\nIs playing politics all about sucking up to senior managers? Basically, yeah. A less cynical way to describe it would be “aligning with the values of the company”. If you think your company is doing good things, you should want to do that anyway! In any case, what that comes down to is figuring out what the people in charge want, giving it to them, and making sure they see you doing it. However, there’s still some scope to get what you want out of the deal.
\nI said earlier that software engineers do not wield organizational power. However, that doesn’t mean you’re powerless. Technical ability is a source of real power, if a delicate and unreliable one. The movers and shakers in tech companies are utterly dependent on technical people to implement their vision and to give them clear answers about the system.
\nThere are many subtle ways you can leverage this. One I wrote about in How I influence tech company politics as a staff software engineer is to wait until important people at the company want to do something (say, improve reliability), then offer them a technical plan that does it your way. Another one is to become so useful that you’re actively in demand to lead projects, and then run the project how you want.
\nYou probably won’t be able to change the company’s grand strategy. But how that strategy is implemented has a lot of specific technical detail, and you can put yourself in a position to decide on those details.
\nPlaying politics isn’t about plotting and scheming, and it isn’t just about being a friendly, likeable person (although that helps). It’s about figuring out how your company actually operates: who makes the decisions, who gets consulted, what behavior gets rewarded, and so on. The most basic way to do that is to figure out who is powerful, get out of their way, and (if you can) help them get what they want.
\nObviously the exact titles depend on your company. One person I’m deliberately leaving out is your own manager. In general don’t think your relationship with your own manager counts as “playing politics”: that’s just you getting along with another human being. An exception to that is if you report directly to a powerful director or VP.
\n↩\nIronically, this manager struggled to take his own advice.
\n↩\nNote that you actually have to be able to answer their question accurately in order to do this. If you’re not competent enough to be useful to powerful people, you will struggle to befriend them.
\n↩\nFor instance, maybe the CEO is convinced that the project was in bad shape because of something he heard, and the manager in question knows it’s easier to sell “yes, but we turned it around” than “no, you misunderstood, everything was always fine”. If you complicate that process, you risk the CEO thinking that the project is still bad and cancelling it.
\n↩\nWhen you write, you should try to say things that are obviously true, and spend very little time worrying about whether you’re being original. Ironically, this is the best way to do truly original writing.
\nEvery important idea has been discussed already. I learned this in grad school for philosophy1, but it’s true across the board. Almost every field2 has generations of very smart people who’ve spent their whole lives thinking about the biggest problems. If you try to only write about ideas that are truly brand-new, you will restrict yourself to writing about ideas that weren’t important enough to be considered before: in other words, trivia and ephemera.
\nWhat happens if you focus on the obvious instead? Writing about things that are obviously true is surprisingly difficult. Writing about anything is difficult, because reality has a surprising amount of detail. Still, writing about obvious things is especially difficult, because they’re almost too obvious to see.
\nIt’s trivial for me to come up with non-obvious things I believe (for instance, I don’t like database foreign keys), but it’s hard for me to articulate the parts of what I believe that are truly fundamental. I think it’s the same reason why I can’t see the glasses on my face: they’re not something I look at, they’re something I look with. The beliefs you consider the most obvious often play a role like this. Writing about them can be really worthwhile, because reading about your basic assumptions can help other people articulate and examine their own.
\nMy most popular blog post — How I ship projects at big tech companies — is a nice example. I knew I had strong opinions on how to ship projects (if for no other reason than I kept watching other people doing it wrong), but it took hours of sitting down and writing to realise that I actually just had a different definition of “shipping”3, and that all of my object-level advice flowed naturally from that. That definition was so obvious I struggled to notice it at all.
\nThe other reason to try and be obvious is that trying to be original is really boring. Here’s a quote from Keith Johnstone’s Impro, a book I consider far too cult-like but that does have some great advice about originality:
\n\n\nIf someone says ‘What’s for supper?’ a bad improviser will desperately try to think up something original. Whatever he says he’ll be too slow. He’ll finally drag up some idea like ‘fried mermaid’. If he’d just said ‘fish’ the audience would have been delighted. No two people are exactly alike, and the more obvious an improviser is, the more himself he appears. If he wants to impress us with his originality, then he’ll search out ideas that are actually commoner and less interesting. I gave up asking London audiences to suggest where scenes should take place. Some idiot would always shout out either ‘Leicester Square public lavatories’ or ‘outside Buckingham Palace’ (never ‘inside Buckingham Palace’). People trying to be original always arrive at the same boring old answers. Ask people to give you an original idea and see the chaos it throws them into. If they said the first thing that came into their head, there’d be no problem.
\n
I think this is absolutely correct. When you’re trying to be original, you’re striking out into nothingness, and you’ll likely end up pattern-matching to “something that sounds kind of transgressive”, which is as boring as it sounds. When you’re trying to say something obvious, you’re reflecting on your own experiences, which have the interesting grit and detail of reality, and are far more likely to sound original to your readers.
\nIf you think you’re the first to have a philosophical idea, you’re probably not even in the first few hundred to write about it.
\n↩\nOf course, if you’re in a brand new field of study, it’s easy to make a lot of original observations very quickly. I call this the “gentleman scientist” era of a field. It’s good to try and be original in these limited circumstances, because there’s such a wealth of ideas that nobody has tried yet.
\n↩\nIn brief, I think “shipping” is socially defined: something is shipped when the relevant decision-makers at a company agree that it is shipped.
\n↩\nAs a software engineer, how well do you have to understand your own codebase?
\nMy guess is that people who work on small codebases with low-turnover teams (say, Redis or games like The Witness) would say “obviously you have to understand it completely, otherwise you can’t do good work”. I’d also guess that people who work on large codebases with high-turnover teams (say, the Google web search backend or GitHub) would say “obviously you can’t understand it completely, you just have to do the best you can in your local area”.
\nThese are two largely different ways of programming with different methods, practices and cultures1. However, the first group is over-represented in online discussion about software engineering2. I want to defend the second group against the first. In many software engineering environments, there’s nothing wrong with being in a state of partial understanding. In fact, in large systems a partial understanding is the best you can do.
\nThe best articulation of the “you have to understand your codebase” side is Peter Naur’s famous paper Programming as Theory Building. I like this paper, but I think it goes too far in that direction. Naur’s core point is that when programmers work on a program, the code is really just a by-product, and the main product they’re working on is their “theory of the program”. That’s made up of their intuitive sense of what’s happening and why, which can only be partially captured by code or documentation. If they lost the code, they could rewrite the program easily. If they lost their understanding (say, if the team experienced 100% turnover), they would struggle to make sense of the code.
\nSo far, so good, but Naur goes further than this. He says that the theory should not be reconstructed from the code. According to Naur, you’re better off scrapping the program entirely and having a new team rebuild it from scratch, building up a new theory in the process3:
\n\n\nreestablishing the theory of a program merely from the documentation, is strictly impossible … [therefore] the existing program text should be discarded and the new-formed programmer team should be given the opportunity to solve the given problem afresh
\n
Anyone who’s been an effective software engineer at a large company knows that Naur is dead wrong about this. There are at least two reasons.
\nFirst, you simply can’t rebuild large software systems from scratch. Sufficiently large systems (if they have users) contain thousands of weird cases and quirks that cannot be reimplemented. Even a team that’s intimately familiar with the system couldn’t do it: there’s just too much stuff to juggle. Successful rewrites always start by carving out the existing codebase into small isolated chunks, then rewriting one chunk at a time. In other words, rewriting a software system involves making a bunch of changes to the old system. If you can’t change the old system, you certainly can’t replace it with a new one.
\nSecond, abandoned systems are revived all the time. In a tech company with hundreds of millions of lines of code and thousands of engineers, it’s not uncommon for a codebase to have nobody left who’s familiar with it4. All it takes is a few people to quit at the wrong time, or for a codebase to be unmaintained for a year. Not only have I seen other teams do this, I have personally taken ownership of abandoned codebases, figured them out, and gotten to a point where I could effectively work with them. It takes time, but building a new theory of the codebase is possible. You start by understanding one flow end-to-end, then slowly branch out from there, making careful changes as you go.
\nIn sufficiently large codebases, everyone operates with an incorrect theory of the program. The defining feature of modern software systems is that they’re just way too big for anyone (or even a whole team) to keep in their head: nobody understands it all. To be effective, you have to figure out a way to work with a merely partially-correct theory. This is why I keep going on about taking a position and confidence. If you’re not sure about something, you can’t just sit back and wait for someone with a perfect understanding to come and give you the answer. If you’re a competent engineer, that person is you. You have to grit your teeth, make your most educated guess, and then deal with the consequences.
\nTo be generous to Naur, it’s possible that in 1985 the average size of a program was several orders of magnitude smaller than today, and that when Naur writes about “large programs” he’s not talking about tens of millions of lines of code. Naur’s first example of a large program is a 200,000 line industrial monitoring program, and his second example is a compiler. In 1987, the first version of the compiler GCC was about a hundred thousand lines of code; in 2015 GCC was over fourteen million lines. I can believe that rewriting one or two hundred thousand lines of code is relatively straightforward, particularly if you get to reuse existing tests. Not so for one or two million.
\nLLMs are often cited as a tool that’s bad because it impedes the ordinary process of theory-building. I think this is overly simplistic. Like many software tools, LLMs are a double-edged sword: they make it harder to construct a detailed mental theory of the software, but they allow you to build a partial theory quickly and they can help you leverage that partial theory more effectively. This is a complex tradeoff that I’m still thinking about.
\nSetting LLMs aside, I’m confident that it’s silly to say that anything that interferes with your theory of the software must be bad. Here is a partial list of other things that make it harder to maintain a theory:
\nLike most things in software, “maintaining a theory of the codebase” is one value among many. Sometimes it’s the most important value and you sacrifice other values for it; other times you trade it off for speed, or legal compliance, or for political reasons5.
\nAlmost all engineers — particularly “pure” engineers — prefer to maintain an accurate mental model of their software. It’s more fun, less stressful, and feels more like “real engineering”. That’s why many engineers take up open-source projects in their spare time in order to work on small codebases by themselves: in order to do engineering work where they can maintain an accurate Naur theory of the codebase. I don’t think there’s anything wrong with that.
\nHowever, at work you are paid to do a job. In other words, they pay you money to adopt their set of engineering values. It’s hopefully well-understood that however much you might personally care about performance, sometimes you have to write slow code at your job (for instance, to get a project done on time, or to accommodate some awkward requirement). Maintaining a theory of the codebase is the same kind of thing.
\nI wrote about this at length in Pure and impure software engineering. I think many of the repeated arguments we have in the software industry are caused by the pure total-understanding culture coming up against the impure partial-understanding culture.
\n↩\nOpen-source engineers are more excited to blog about their work, the raw engineering content is typically more impressive (because coordination problems dominate big proprietary systems), open-source projects can be legally written about while proprietary systems can’t, and even if you could do it legally, writing about large codebases is impossible because it requires too much specific context.
\n↩\nI re-read the relevant chapters of Ryle’s The Concept of Mind (which Naur cites throughout) and I think Ryle is more generous about theory-building. For Ryle, theory-building or know-how automatically happens as you do things. It’s fully consistent with Ryle to think you can pick up an existing codebase just from the code, purely by puzzling it out.
\n↩\nNaur says: “Lest this consequence may seem unreasonable, it may be noted that the need for revival of an entirely dead program probably will rarely arise, since it is hardly conceivable that the revival would be assigned to new programmers without at least some knowledge of the theory had by the original team.”. If only!
\n↩\nSome engineers might say that maintaining a theory is the core value, because without it you can’t fulfill any of the others. I disagree. You could say the same thing about readability, or maintainability, or correctness, or a bunch of other engineering values. We trade off “core” values like this all the time.
\n↩\nEvery post I publish represents at least two things I’ve learned: the thing that prompted me to write the post, and the thing I learned in the course of writing it. If I don’t learn anything new while I’m writing, it’s not interesting enough to publish.
\nTypically I learn way more than two things. For instance, in my o3 geoguessr post, I started out with the idea that most AI prompts probably don’t work, and I ended up learning that newer OpenAI models have lost o3’s ability to geolocate. That’s interesting! In my most recent post on C2PA, I started out with the idea that C2PA requires near-universal adoption, but I learned a ton of things about PKI, managing private keys on local devices, how C2PA actually works, and so on. In my post on the Luddites, I started out with the idea that the Luddite movement was fundamentally decentralized, but ended up fascinated by Luddite culture (which was far more elitist, misogynist, and violent than the pop-Luddism books describe). I could do this for every single post on the blog.
\nI think the core reason this works is that every single one of my blog posts argues a point. I never publish a post that just gives some scattered thoughts on a topic, or a post that only says “yes, I agree with this other article”. If I write a draft that nobody sensible could disagree with, I scrap the draft. Making sure that everything I write is at least minimally controversial is a forcing function: it forces me to think about what the most interesting part of my position is, and it forces me to do enough research to defend it against the obvious criticisms.
\nThis is contrary to a lot of advice I read about blogging, which encourages the aspiring blogger to treat their posts as a form of unstructured self-expression. If unstructured self-expression is what you want to do, that’s cool. The point of having a blog is that you get to write what you want. However, this advice isn’t as helpful as it sounds.
\nBefore I was in tech, I was a philosophy grad student. But before that, I was a poet. One thing you learn when you try to write poetry is that it is way easier to write to a restrictive structure than it is to simply “write what you feel”. This should be obvious when you actually think about it. The task of a poet is to repeatedly choose the next word. Writing to a structure (typically rhyme or meter) narrows that choice to a small set of words, instead of the entire English language. It’s the same with blogging. Forcing yourself to write about specific, potentially-controversial points makes consistently writing easier, not harder.
\nWriting is the best way to think clearly about a topic. It’s easy to believe you understand something when you’re just turning it over in your head. When you have to condense that down into words, you find out exactly how much you do or don’t understand. I am constantly having moments where I type something, stop myself, and think “wait, that can’t actually be right”, or “is that really true?”
\nBy the time I write my way to the end of the post, I’m usually thinking so much more clearly about the topic that my conclusion paragraph is way better than my introduction. In fact, I’ve picked up the habit of going back and immediately rewriting the first paragraph as part of my first-draft process, because I know I’m going to end up doing it anyway.
\nI also change my mind a lot while I write. Here are a bunch of examples of posts where I began writing them with the opposite opinion to the one that eventually made it into the post. I think this is a good sign, and I hope I never stop doing it. You should be researching and thinking about every post you write, and that means you should frequently learn new things that change your mind.
\nBecause of all this, I deliberately choose to write blog posts about things I don’t yet quite understand but would like to, like LLM steering, Stripe’s Tempo blockchain, C2PA and watermarking, space cooling, interaction models, LLM inference internals, and so on. This is great for me, because I learn a lot. Is it great for my readers?
\nI sometimes worry that I should only be writing about areas I already know very well, like tech company dynamics or working in large codebases, rather than presenting myself as an authority on fields I’m actually still learning. Should I let historians of the Luddites write about Luddism, Web3 engineers write about blockchains, and so on? I think this is acceptable for three reasons.
\nFirst, it’s sometimes easier for a beginner to write an introduction to a field than for an expert. Experts routinely overestimate the knowledge of the general public, and have often internalized the reasons why their field is important so deeply that they struggle to express them. I think my explainer posts are valuable because I always spend the first chunk of the post talking about what the original problem is before I get into the technical solution.
\nSecond, sometimes the public consensus on a topic is just plain wrong, to the point where even a little bit of research is enough to demonstrate why. Many of my posts I’m proudest of have been along these lines: arguing that the “500ml per prompt” water usage figure for LLMs was ludicrous, or that the popular Apple “Illusion of Thinking” paper was tracking persistence, not reasoning, that GPUs live longer than three years and the AI companies have large profit margins on inference, and so on.
\nThird, I try to make it clear on my blog who I am and what my credentials actually are. Even if it’s not explicitly described in the post, I have my real name and resume available on my /about page, so I don’t think a careful reader could be easily fooled into thinking I’m an expert on 19th-century England or space physics or LLM economics or anything like that.
\nEven if nobody reads what you write, writing is still a good discipline for getting your thoughts in order. But another big reason why writing is a great learning tool is that you can get feedback.
\nI think it’s obvious why this is useful, but I do want to make two points about feedback. First, if you do make your posts public, you need to have a pretty thick skin. People on the internet often fall over themselves to come up with the most cutting criticism or the harshest dunk. This goes double if you take my previous advice and try to write posts that make a clear, controversial point about a subject you’re learning. If you’re the kind of person whose whole day is ruined when a stranger is cruel to them, you might want to keep your blogging private or only share it among friends.
\nSecond, even if your blogging is private, you can get feedback from LLMs. Like humans, LLMs will often give junk feedback. In my experience, OpenAI models will always tell me to moderate my claims or add caveats and hedges until I’m not saying anything at all. Sometimes their criticism will be straight-up wrong. But — particularly about technical topics — LLMs are great at pointing out areas you’ve genuinely misunderstood, and they’re far kinder than the average Lobsters or Hacker News commenter.
\nI’m pleased and grateful that people enjoy reading my posts, but even when nobody did, I still got a lot of value out of blogging. I write as a method of thinking more clearly, as an excuse to do research on topics I want to learn about, and as a way of getting feedback.
\nIf you’d like to try it yourself, I suggest watching for these two things. First, you should be changing your mind a lot as you write. If not, you probably aren’t doing enough research. Second, your first draft’s conclusion should be much tighter and more expressive than its introduction. If not, you probably haven’t learned anything from the writing process, which means the draft can be scrapped.
\nI strongly recommend this practice to anyone with an interest in writing. You will see the benefits even if you don’t publish any of your writing on the internet, particularly now that you can get good technical feedback by pasting your post into a LLM1.
\nFor what it’s worth, I’ve fiddled with careful “review prompts” and it’s basically as good to just write “review, please:” and paste your article.
\n↩\nThe European Union AI Act is Europe’s attempt to comprehensively regulate AI usage. A big part of that is the requirement that AI-generated content be identifiable: either tagged with a watermark or with what the Act calls “digitally signed metadata”1. Since all this becomes enforceable in a month, it’s worth figuring out if it makes any sense. I recently discussed AI watermarking at length in Text AI watermarks will always be trivial to remove. What about digitally signed metadata?
\nThe most well-known implementation of digitally signed metadata is C2PA Content Credentials, which incorrectly2 claims to be the technology that the AI Act gives as an example of how to do signed metadata properly. The idea here is that every single image file should contain unspoofable authorship metadata. Here’s my position on it:
\nLots to unpack. Let’s start by considering images, since that’s the easiest case.
\nWhen an AI tool generates an image, that tool should include a “made by ChatGPT” disclaimer in that image’s metadata. Likewise, when a camera takes a photo, that camera should include a “taken by a camera” disclaimer. C2PA uses two strategies to protect this metadata:
\nEach physical camera (or phone) has its own private key, for obvious reasons3. How do we know that those millions of private keys are trusted? Via PKI, like HTTPS: each camera’s private “certificate” (which contains its public key) is signed by the manufacturer’s well-known private key, so the chain of authenticity can be verified as long as you have (say) Apple’s root public key4.
\nWhat happens if you then edit your photo in Photoshop? Photoshop will leave the camera’s metadata untouched, but will layer a “also, Photoshop was used” piece of metadata over the top, signed with Adobe’s private key (well, with the private key associated with your official copy of Photoshop, which is signed by Adobe’s official private key).
\nLikewise, if you ask ChatGPT to generate an image for you, ChatGPT will sign its “made by ChatGPT” metadata with OpenAI’s private key. In theory, every single image could contain unforgeable C2PA metadata, allowing software like Twitter to trivially distinguish real photos from fake ones.
\nRight now, C2PA does not have anything like the adoption it’d need to work. It’s hard to find hard data on how many images in the wild use C2PA, but FotoForensics reports around a dozen per week (so around 600 out of the 900,000 images processed each year). This is even worse than it sounds, because basically all of the signed images are AI-generated. The adoption rate of C2PA for human-generated images is much, much lower: so far, Google’s Pixel 10 is the only phone camera to sign photos by default. The iPhone doesn’t sign photos.
\nIf almost all AI images are C2PA-signed, but almost no human-generated images are, consumers have no reliable way of identifying AI content, because anyone who wants to pretend their AI content is human can simply remove the signature. For C2PA to succeed, it needs to be on every camera and every phone, so that a photo with no signature is rare and suspicious.
\nIs that realistic? Actually, I think it is. The appetite (at least in the EU) to regulate AI will increase over time, and while the current EU AI Act only mandates that AI-images are tagged (which by itself is useless), it’s plausible that some future regulation will enforce tagging of all images.
\nAnother adoption problem that must be solved for C2PA to work is preservation. Right now, if you download a C2PA-tagged image, send it as a Facebook message, then re-download it, the C2PA manifest is stripped out. Most images we see on the internet have passed through some social media asset server at least once. All of these social media companies would need to update how they re-encode image content in order to preserve the C2PA data5. This would almost certainly require more regulation: C2PA adds tens or hundreds of kilobytes to each file, which at social media scale is big money6.
\nCould a clever attacker forge a C2PA signature? Kind of. Neal Krawetz, who seems to have led the anti-C2PA charge, points out that with a camera development kit it’s straightforward to trick a digital camera into thinking that it’s taking an image when in fact it’s being fed one. This is very much not my area, so please write in if you know more about camera hardware and you think I got this wrong. I suppose you could also take a photo of an AI image on a screen, though I imagine you’d have to be careful to make it look real.
\nIf you exclude physical attacks on a digital camera, I think C2PA is more robust. You can sign a photo with a self-signed certificate, but the C2PA spec and docs say that validators must check that your certificate bubbles up to the official C2PA trust list. This list currently contains only 26 certificates, and there’s a whole process for being added to it. That’ll slow down adoption, but at least it makes it hard to forge7.
\nWe’ve been talking exclusively about images, but it’s more or less the same story for any type of content. If the file doesn’t support JUMBF metadata (say, an Excel file or a PDF), then the C2PA metadata has to live in a “sidecar”: a separate .c2pa file, probably on some Microsoft or Adobe content server, which contains the signed checksum and the data about who created the file.
However, the distinction between “real” and AI-generated content is fuzzier when you’re not talking about images. Here’s a trivial example: if I ask ChatGPT to create an Excel spreadsheet for me, the file will be tagged as AI-generated, but I can simply copy/paste the content into a new Excel doc and save it, which will tag it as human-generated8. There’s no software tool that can identify when I’m retyping some AI-generated text (except for perhaps text fingerprinting, which has its own raft of issues).
\nThere are also interesting questions around key management. ChatGPT and other AI tools have an easy problem — their users are all online, and so the files can be signed server-side — but how do you sign files created via Photoshop/Excel/Word? If the user doesn’t have internet, do you use some kind of local key? If so, how do you prevent that key being extracted and used to sign AI-generated content?
\nFinally, is it a civil liberties problem to automatically fingerprint every photo? Does it make it impossible to be a whistleblower if every photograph can be traced back to your camera? I think this is a complicated question, but in short: I’d expect whistleblowers to already strip EXIF metadata from their images, C2PA metadata is similarly trivial to strip out, and overall I think image attribution is positive for whistleblowers because it heads off “this was AI-generated” responses.
\nC2PA is probably here to stay. But it isn’t useful now, and won’t be useful until two huge programs of technical work are completed:
\nThis will be a long organizational process, since each manufacturer must go through the approvals process (or decide to start their own competing system), evaluate the legal ramifications of storing attribution data in images, and so on. It will be a long technical process, because C2PA metadata is a substantial fraction of image sizes: storing it will add many petabytes of content.
\nOf course, just because C2PA isn’t useful doesn’t mean we’re not all going to do it. Lots of companies are under pressure to signal that they care about AI safety and to head off regulatory attack. “We’re cryptographically signing AI-generated content” is a compelling “we’re doing something” pitch, particularly for people who aren’t technically savvy enough to understand the limitations. In the near term, I expect large AI-involved companies to invest a substantial amount of engineering effort in C2PA-related activity.
\nIn the long run, once everyone gets on board, I think C2PA could end up working well. It’s awkward in some ways, but “attest content via a PKI certificate chain” is a good idea.
\nIs it possible to defeat? Yes, of course. By design, private keys will be in the user’s hands — in their cameras, in their local versions of Photoshop or Microsoft Word, in their phones — so sufficiently technical users will be able to crack them out or use them to sign whatever content they want. I still think C2PA will end up stemming the tide of AI content, because most users are not going to be sophisticated enough to perform attacks like this. However, we should still retain some skepticism of unlikely-looking content, even if it has “created by a human” in its C2PA metadata.
\nSee sub-measure 1.1.1 of the Act’s associated Code of Practice.
\n↩\nWhile an early draft of the Code of Practice made an offhand mention of Content Credentials (in the caption of a picture), that was stripped out. The contents of the Act and the final Code of Practice don’t contain “C2PA” or “Content Credentials” (you can search for yourself here).
\n↩\nOtherwise if you cracked the key out of one Sony camera, you could spoof content from any Sony camera.
\n↩\nIn practice there are usually more “links in the chain”: a device will be signed by some intermediate certificate, which in turn will be signed by another intermediate certificate, which will be signed by the root certificate. That’s because the root key is so valuable. If an intermediate private key leaks, it can be revoked and replaced (via the root key), but if the root key leaks, it would take years to rebuild the network of trust. So almost all signing is done by intermediates, and the root key stays on a USB drive locked in a safe somewhere.
\n↩\nNot to mention that the whole point of C2PA is that these social media companies will be displaying a “human or AI” sticker in their UI, which will require retaining the metadata.
\n↩\nC2PA allows for storing the manifest content as a separate .c2pa file, and just including a manifest url in the image metadata itself, but that doesn’t solve the cloud provider problem: they still have to store all the .c2pa files on-disk somewhere.
I think this defuses Neal Krawetz’s “worst-case scenario”. I downloaded his forged image, and (as expected) it gets flagged as “signed, but we don’t trust the root”. I think Krawetz was right at the time, though, since the official “trust list” was only launched in mid-2025.
\n↩\nYou could do the same thing with images by copying into Photoshop or Paint, but while that’d obscure the AI source, it would still be clear that the photo wasn’t taken by a camera.
\n↩\nThe European Union AI Act will begin to be enforceable in August 2026, one month from now1. One of the biggest new requirements is Article 50, which requires all AI outputs to be “detectable as artificially generated”. In other words, if LLM providers want to do business in the EU, they will have to apply a watermark to their outputs2: some hidden signature that can be used to identify AI content.
\nLLM text watermarking is a fascinating problem. Like the best engineering problems, it is theoretically hard to solve perfectly, but has multiple partial solutions: for instance, Google’s SynthID, and (as I’ll argue) some quiet Unicode trickery from OpenAI and Anthropic. It will be interesting to see how the AI labs navigate these tradeoffs before the end of the year.
\nI wrote about AI watermarking at the end of last year in AI detection tools cannot prove that text is AI-generated. It’s easy to watermark an image, because digital images contain lots of noise that the human eye can’t really see. For instance, you could apply a watermark like “these twenty pixels in these exact spots will always share a color”. Text is much, much harder. Unlike images, text is a very compressed medium: you cannot make any change to a sentence that a human wouldn’t notice (with one exception, which we’ll get to later). So how are you supposed to watermark it?
\nIt’s basically a text steganography problem (concealing a secret code), made more difficult because the plaintext cannot be arbitrarily manipulated. Any changes you make to apply the watermark will compromise the quality of the output. For instance, “every fifth letter is an ‘e’” would be a good watermark, but applied naively would make the AI output full of typos. Could you just let the model figure out how to fit the watermark? Strong AI models are smart enough to juggle this kind of constraint3, but it’d still consume reasoning time that would be better spent on the user’s problem, and make the model sound much less capable than it is4.
\nDo you really need a watermark? If you’re Anthropic, and you’re required to be able to verify whether your models produced a particular block of text, can’t you simply run the text through each model, measuring as you go how closely the model’s predicted tokens match each token from the text?
\nNot really. The space of “all possible Claude Sonnet answers to a question” is way larger than the space of “all possible watermarked answers to a question”. In other words, you’d get too many false positives for human text that reads like it was AI-written. It’s way more likely for a human to accidentally write like Claude than it is for a human to accidentally reproduce a watermark.
\nIt would also be prohibitively expensive to run every Anthropic model against a piece of text in order to watermark it. The EU AI Act will eventually require labs like Anthropic to offer free watermarking services to every EU citizen (see Commitment 2). You couldn’t do that with the “run the model” approach.
\nAs far as I know, the only AI provider to say they watermark text output is Google, who use a tool called SynthID. Here’s how it works.
\nWhen a LLM generates text, it’s generating a series of tokens (words or chunks of words). At each step, the model itself doesn’t output a single token, but instead outputs a full list of all (say) 100,000 tokens in its vocabulary, each annotated with the probability that that token will be the next one. Tools like ChatGPT or Claude Code will pick semi-randomly from the most likely options in order to get their outputs. This semi-random sampling process can be influenced in a detectable way.
\nFor instance, we could choose a sampling strategy like “we pick the second most likely token, then the first, then the second, then the first, and so on”. That would still produce high-quality output, but you’d be able to re-run the model against the generated text to verify that the pattern holds. However, that’d make verification really expensive, and any slight tweaks to the output would break the pattern and thus break the fingerprint. Is there a better way?
\nYes. SynthID is a process for assigning each token a “score” based on its previous tokens (for instance, sum the token’s ID with the IDs of its previous three tokens then take mod 5)5. To apply the watermark, the model adopts a sampling strategy like “out of the top five most likely tokens, pick the one with the top SynthID score”6. The watermark can then be detected by calculating the aggregate SynthID score of a block of text. If it’s suspiciously high, it’s very likely to have been AI-generated.
\nThis is basically a version of the common advice that you can identify LLMs by use of the em-dash, except that instead of a list of keywords, it relies on subtle mathematical relationships between words that humans can’t identify. Because the process for assigning the score is trivial, it’s very cheap to run watermark detection.
\nGoogle have a complicated mathematical rationale for why SynthID doesn’t make the model dumber: supposedly the SynthID scoring is random enough to act like a normal pseudo-random token sampler, just one that leaves a detectable fingerprint on the outputs. But of course this is suspicious. For instance, it’s common to do inference setting temperature to zero, which always picks the model’s most likely next token. In that case, you can’t leave a fingerprint at all (or you have to ignore the user’s preference and pick the second or third choice anyway).
\nIf you can’t alter the model outputs, can you still fingerprint the content? Well, kind of. I’m pretty sure OpenAI and Anthropic are sometimes applying fancy Unicode tricks. For instance, you might go through and replace your normal ” ” spaces (unicode U+0020) with a three-per-em ” ” space (unicode U+2004), or a CJK ideographic ” ” space (unicode U+3000). These are called “homoglyphs”, and you can find more of them here.
Of course, lots of human-generated text uses homoglyphs. But it’s trivial to encode a pattern of homoglyphs (say, “every third space becomes a three-per-em”) that is much less likely to occur in the wild. Like the SynthID watermark, a homoglyph-based watermark can be detected very cheaply. A homoglyph-based watermark is cheaper to apply than SynthID: you could even do it entirely on the client.
\nI don’t think this is a conspiracy theory. Claude Code was definitely doing this to tag suspicious requests from Chinese users (exploiting homoglyphs for the ’ character in “Today’s date”, though they’ve since walked that back). In the last few years, I’ve noticed that when I copy blocks of text from ChatGPT and paste them into VSCode, sometimes VSCode marks some or all of the spaces as unusual Unicode characters7. Are OpenAI and Anthropic using homoglyphs as an AI-generated watermark? I’m not sure. But they’re definitely using homoglyphs.
\nThe AI Act (specifically, its associated Code of Practice) requires watermarking to be “embedded within the content in a manner that is difficult for it to be separated from the content”. However, text watermarks can be trivially removed.
\nTo remove unicode homoglyph watermarking, you simply have to replace all the homoglyphs with their “real” character equivalents. If you have access to even a relatively weak un-watermarked LLM8, you can strip out SynthID watermarking by asking that LLM to paraphrase the text content. Because the watermark is inherent to subtle vocabulary choices, re-wording the content will remove the watermark. You could even do it by hand, although at that point it’s not really AI-generated content anymore. Since there will be some kind of free public watermark testing tool, you can just keep tweaking until it comes back negative.
\nMoreover, the AI Act requires watermarking techniques to be “interoperable… as far as this is technically feasible”. That means AI providers would have to publish their watermarking process, and potentially even attempt to standardize on applying the same kind of watermarks. I just don’t see how this is compatible with the kind of security-by-obscurity that LLM text watermarking depends on. Unlike image and video watermarks, text watermarks will always be trivial to remove.
\nThe AI Act and Code of Practice talk a lot about “digitally signed metadata”. The idea here is that you can include an AI disclosure in the file’s metadata itself, ideally in a way that cannot be tampered with (for instance, by signing a hash of the file’s contents). This signed-metadata process is basically C2PA Content Credentials. While you can remove C2PA metadata, you (theoretically) can’t fake it, so a file with “created by a human” metadata can be trusted, and files with no metadata at all can be held in suspicion.
\nThis post is already too long to get into what I think about C2PA, but I do want to say that C2PA is not a substitute for text watermarking. It only really applies to files. In the words of the Code of Practice, that’s “a data format that supports attaching metadata (e.g., an audio, image, video, or containerised text)“. The output of chat tools (and most of the output of AI agents) is not containerized text, but plain old regular text, and so can’t be signed. What would it even look like to sign ChatGPT outputs? There’s no artifact to pass around.
\nI think it’s a fascinating question whether Claude Code has to C2PA-sign any HTML files or PDFs it generates for you. That seems kind of tricky to get right. But in any case, the AI Act also mandates some kind of actual watermarking as well.
\nSo what’s going to happen this year? If I had to guess, I’d say that each AI provider (not just labs like OpenAI or Anthropic, but third-party providers like Fireworks or Groq) will stick a SynthID token sampler in front of their inference stacks. This might be limited to users in the EU, but it might not be, since SynthID is at least as good as a normal top-k token sampling approach.
\nAI providers will then offer a “check for watermark” page that re-tokenizes user-provided text, runs the scoring, and checks whether it’s above a certain threshold. Depending on how seriously the interoperability clause is taken, providers might even standardize on the same SynthID setup, in which case there could be a single EU-hosted “watermark this text” page.
\nI don’t think unicode-based watermarking is going to be considered compliant with the AI Act, but some providers which don’t want to set up SynthID might try it. Either way, technical users will be able to strip out the watermark at will, and there will be a plethora of tools that non-technical users will use for this purpose.
\nWell, for new systems; existing ones get until December.
\n↩\nI don’t think the plain text of Article 50 requires this, but Recital 133 and the Code of Practice makes it pretty clear that they’re looking for watermarks.
\n↩\nEven with extra high thinking, GPT-5.5 could not explain SynthID to me with every fifth letter being an “e”, but GPT-5.5-Pro produced this puzzling koan: “These hidden codes label model-made image, voice, movie, prose. Probe trace: maybe a model-made piece. Maybe erase trace; maybe leave trace. Hence trace alone? No.”
\n↩\nI leave the analogy with AI safety guardrails as an exercise for the reader.
\n↩\nThat’s a toy example. In practice there are multiple different (but still mathematically simple) scoring methods that get combined together, including a random seed. Why include the seed? Otherwise the watermark would bias towards the same set of tokens.
\n↩\nThe tokens are scored in a multi-round knockout against each other, but I think that’s more of an implementation detail and not required to get the core intuition behind why SynthID works.
\n↩\nWhen this became public knowledge, OpenAI claimed it was just a model quirk, which is certainly possible.
\n↩\nAll AI providers might be legally required to watermark, but even tiny local models are good enough to paraphrase text.
\n↩\nStating the obvious is surprisingly useful. Most of your knowledge lives below the threshold of conscious awareness, so it’s possible for a piece of writing to remind you of what you already know. It’s common to know you don’t like something without being quite sure why, and reading an obvious statement (such as “accuracy matters, even when you agree with the broad strokes”) can help clarify why you find certain things distasteful.
\nSometimes you can see some obvious truth that nobody seems to be talking about, and reading it in someone else’s words can prompt an “oh god, I’m not crazy” moment of catharsis. For many junior engineers, it’s almost a rite of passage to notice that some percentage of software engineers do virtually no work. Since nobody talks about it (how would you even bring it up in the workplace?), they often feel like they’re losing their minds: surely this state of affairs wouldn’t be allowed to continue, so they must be completely misreading the situation. But in fact it’s true.
\nStating the obvious is hard. It can even be dangerous: sometimes there’s a good reason nobody says the obvious thing. But I think the bigger reason it’s hard is for the same reason that it’s hard to draw what you actually see. When I look at a person and try to draw them, I’m not drawing the lines and shades my eye sees (like a printer or camera might). I’m drawing what I know the person looks like, which is a kind of stick-figure approximation. It takes time and effort to drop the layer of interpretation and draw what’s actually there1.
\nMany of the posts I’m most proud of are times when I’ve managed to articulate something I think is obviously true: engineer reputation is determined by ratchet effects, good engineers are right most of the time, you shouldn’t just do JIRA tickets (or glue work), and so on. These are all things I’ve believed for a while, but have only (relatively) recently been able to notice that I believe them. Sometimes I’m helped along by reading something I vehemently disagree with (like “nobody gets promoted for doing simple work”, or ”big egos have no place in tech”).
\nStating the obvious doesn’t mean avoiding nuance. Every obvious claim carries with it a host of subtle, non-obvious claims. For example, I believe that having a big ego can be very useful as a software engineer. But why exactly is that, and what do I mean by ego? Obviously it’s not good to be constantly flexing your status on other people, or to be unable to tolerate the possibility of being wrong. However, I do think you need to be able to take firm technical positions even when the situation is uncertain, which means you have to be confident in your technical instincts. Teasing out that distinction (and its implications) is very interesting, but in order to do it you need to be able to first articulate the obvious part.
\nI’ve been talking about stating the obvious in technical blogging. But this principle applies just as well to other kinds of communication. When I write a technical design document at work, it’s very important to state the obvious. In fact, technical communication is so hard and general understanding is so poor that just getting people aligned on the obvious things is often enormously valuable. Much great literature and poetry aims to bring out some obvious but hard-to-articulate part of human experience.
\nDon’t avoid writing something down just because you think it’s obvious. The thing you think is obvious now might recede into your subconscious in an hour; get it written down while you can! And don’t avoid writing something down because you think it’s dangerous to say and everyone already knows it. For people new to the area, reading your words can help them feel like they’re not losing their minds. Finally, once you write down the obvious thing, it allows you to go on and draw out the parts that are less obvious, in a way that you couldn’t do if you try to just skip straight to the subtleties.
\nIncidentally, this is why most people cannot draw a bicycle on their first attempt. Unless you’re a mechanical engineer, you probably do not have a stick-figure-level approximation of what a bicycle looks like in your head, so you begin confidently (after all, you’ve seen a thousand bicycles) and get stuck after the first few lines.
\n↩\nMany people claim that AI inference is unprofitable to serve, and thus must be subsidized by an ocean of dumb money from investors who believe that some future AI model will come to dominate the world economy. When that dumb money goes away, so will AI products. According to this view, LLMs are just inherently too expensive (in terms of money, power, and water) to be used in consumer products. In fact, they can only be used today by externalizing the costs: money onto VC funds and now retail ETF investors, power onto electric utility consumers, and water onto the communities where datacenters are built.
\nThere are good reasons to dislike AI, but this really isn’t one of them. In fact, AI inference is obviously profitable.
\nFrontier AI providers are reporting 70%-80% gross margins on inference, but maybe we can’t trust them. Let’s do some very rough estimates on the actual cost.
\nA Nvidia A100 consumes 400W of power under full load. In practice, even a carefully-tuned inference server will not be at full load all the time, but it’s at least an upper bound. Suppose you’re running a dense 70B model1, which will fit comfortably (unquantized) on four A100s at around 2M tokens per hour. At industrial power prices, that’s about 13c/hr in the USA. Suppose (pessimistically) cooling is the same cost. That’s about 13 cents per million output tokens2.
\nLet’s amortize the cost of the GPUs, since that’s going to be the most expensive part. An A100 costs about $20k. If each A100 lasts around five years3, you’ll have to make 16k/yr in profit to recoup your capital investment (or $1.80 per hour). At lower utilization, it’ll take longer to recoup, but your GPUs will also last longer. Either way, your overall inference costs are at about one dollar per million tokens.
\nGPT-5.4-mini charges $4.50 per million tokens, and stronger OpenAI or Anthropic models are three to six times as expensive. It’s hard to make a direct comparison because we don’t know the size of OpenAI or Anthropic models, but the claimed 70% or 80% profit margin is extremely plausible.
\nWhat if you don’t trust my estimates either? Let’s look at the pricing of open-weights Chinese LLMs. DeepSeek have claimed a bit over 80% profit margin on inference for DeepSeek-R1. Since their API pricing for R1 is less than half that of OpenAI or Anthropic4, that suggests that my estimates above for inference cost might be too expensive. Cooling at scale is probably cheaper than power, R1 only has half the active parameters of a dense 70B model, modern GPUs are more efficient than the A100, and there are significant economies of scale in inference.
\nSince DeepSeek’s models are available for anyone to download, they can’t get away with extracting a large profit margin. One of the other inference providers would undercut them with the same model. Inference costs for DeepSeek-V4-Pro on the market are around 87 cents per million output tokens, which is probably pretty close to the actual cost of serving the model.
\nAll of this doesn’t mean that OpenAI or Anthropic are profitable. Those companies are making huge capital investments that may or may not pan out, and are spending enormous amounts of money on talent and compute to train brand-new models and retain users.
\nThey’re doing crazy things like offering per-month subscription models for nearly unlimited inference, which is almost certainly not profitable. If you used an API token instead of your Anthropic subscription in Claude Code, you’d pay ten times the cost. But that doesn’t mean API-based Claude Code couldn’t be a good deal. Some people are already using DeepSeek’s inference API for agentic coding, because once you take away the huge profit margin it’s cheaper than the relative per-month subscription.
\nWhy won’t OpenAI or Anthropic lower their prices? Supposedly OpenAI has thought about it, but for an AI lab, inference has to subsidize training costs. A company like OpenAI has to fund the production of new models from the inference margins on existing models (at least partially). That’s why the margins on inference are so high: the AI labs are trying to squeeze out every dollar so they can stay alive in the training arms race.
\nHowever, inference only has to subsidize training costs for an AI lab. If you’re merely an inference provider, you don’t have to do any training at all. Therefore, even if OpenAI and Anthropic go out of business, whoever snaps up the rights to their frontier models will be able to continue selling Opus and GPT inference at a profit5. The AI bubble popping will not mean the end of the inference business, because AI inference is obviously profitable.
\nExpensive frontier models are probably mixture-of-experts, not dense, which is tougher to estimate. However, I think a 70B dense model and a MoE with 70B active params will come out to basically the same numbers at scale (though the MoE will require more GPU memory and thus a greater upfront cost). Are frontier models around 70B params? Nobody outside the AI labs really knows, but my guess is that 70B is probably larger than a Haiku/mini class model.
\n↩\nI think it’s reasonable to estimate the cost of output tokens only, since they’re by far the most expensive part of serving inference. Input tokens are cheaper for two reasons: transformers let you prefill them in parallel, and for most real-world use cases they can be aggressively cached in the KV cache.
\n↩\nIt’s common (and wrong) to estimate GPU lifespan at three years. I wrote a lot about this in AI GPUs probably live longer than three years.
\n↩\nAgain, this is just an guess, since we don’t know what OpenAI or Anthropic model is equivalent in size to R1.
\n↩\nI do wonder if Anthropic would be able to prevent other people from being able to access the model if the company goes out of business. Anthropic is currently in debt to Broadcom, Google, and a bunch of private equity firms. Would they get the Mythos and Opus weights, over Dario’s protestations?
\n↩\nPeople who think current AI use is unsustainable often rely on the claim that inference GPUs only last “three years at the most” under load1. The idea here is that once the AI bubble money drains away, current infrastructure will rapidly become obsolete, and there won’t be enough money floating around to buy a whole slate of brand-new GPUs. Inference costs would thus rapidly become way too expensive for current AI products to make any financial sense.
\nWhere does this “three years at the most” claim come from? Is it plausible?
\nThe original Tom’s Hardware article quotes this tweet from Tech Fund, an anonymous former PM and tech investor, who quotes an anonymous “GenAI principal architect” at Google as saying “if you have a high utilization rate, then constant high utilization rate for a year or two, I think the lifespan will be three years at most”.
\n\n \n \n ![\"tweet\"](\"https://www.seangoedecke.com/static/9cf90a11e418ecd73f81e5178a9328c2/1b853/tweet.png\" "\\"tweet\\"")
\n \n
This screenshot looks like it was from an interview. What interview? I scrolled back to October 2024 on Tech Fund’s Twitter feed and saw a bunch of similarly-formatted screenshots, some of which were cited as coming from Tegus. Tegus is apparently a company with a business model of reaching out to insiders (in this case, AI company employees) and paying them hundreds of dollars an hour in order to answer specific technical questions. It’s essentially gig work for almost-but-not-quite insider trading: the more informed and confident you sound, the more likely Tegus analysts will pick you for future interviews.
\nI’m sure the source for this tweet is in fact a GenAI principal architect, since Tegus would have presumably asked for some proof of that before they paid them out. But it’s pretty clear that the incentives here are to sound confident and authoritative, even on questions that you’re not sure about. With that in mind, the quote itself also reads a bit suspiciously. I’ve worked with enough principal engineers and architects to take their casual back-of-envelope estimates with a grain of salt. If they knew the actual rate at which GPUs fail and get retired in Google datacenters, wouldn’t they have just said that?
\nWe have some anecdotal evidence that points the other way. Google has publicly claimed to have eight year old TPUs (their version of GPUs) running in production at “100% utilization”. Nvidia only made A100 GPUs from 2020-2024, but in February 2026 the AWS CEO claimed that AWS had never retired an A100 server (and you can still easily rent A100s for AI work)2. AI GPU usage isn’t exactly like crypto mining GPU usage, but it certainly seems like years-old ex-crypto GPUs are functional. There’s also this comment from Hacker News I noticed where someone claims that their GPU cluster in academia has lasted six years with less than 20% failure rate.
\nWhat about hard data? It’s hard to get concrete data on the lifespan of AI GPUs, because modern AI datacenters have only existed for a handful of years. But an interesting case study would be recent supercomputer clusters like Oak Ridge’s Summit, which had over 27 thousand Nvidia V100s running from 2018 to 2024, or its predecessor, the Cray Titan supercomputer that ran from 2012 to 2019. I couldn’t find any evidence that Summit had to buy an additional 27,000 GPUs to replace their old ones, and GPU failures in Titan have been carefully studied:
\n\n \n \n ![\"fig10\"](\"https://www.seangoedecke.com/static/3922b6bf4aaf6d3202cacd2f472034aa/019a6/figure10.png\" "\\"fig10\\"")
\n \n
These cages of GPUs are stacked vertically, and cold air is pumped in from the bottom, which explains why cage 0 (at the bottom) has better survival rates than cage 2 (at the top). Let’s consider cage 0, so we’re just looking at the GPU lifespan instead of at the lifespan of improperly-cooled GPUs. At three years, over 95% of GPUs survived3. At six years, nodes 2 and 3 (the GPUs closest to the bottom of the cage) were still at above 90% survival rate, and the highest nodes were over 60%.
\nIt’s possible that newer Nvidia GPUs are less reliable than older ones (they certainly draw more power), or that AI datacenters are under-cooled, or that something about LLM utilization is more stressful than the workloads that ran on traditional GPU datacenters. But this is at least circumstantial evidence that GPUs can survive under load for far longer than three years.
\nThis discussion is complicated by the fact that GPUs may have a short economic lifespan. Supposedly a B100 GPU draws twice as much power as an A100, but can do five times as much work. For some AI providers, that might mean that A100s are only worth running until they can be replaced with B100s (if you’re bottlenecked on electricity, you should spend it all on B100s and throw out your obsolete A100s). This is why the Titan supercomputer was decommissioned in favor of Summit: it could have continued to operate, but it was more profitable to spend the money and maintenance effort on newer hardware.
\nIt should be obvious that this doesn’t support the “inference will become more expensive when the bubble pops” argument. So long as A100s are profitable right now, cash-poor AI providers can continue profitably serving inference from them, even if there are more efficient options available for those with the capital to upgrade.
\nOn top of that, GPUs only represent one part of AI datacenter infrastructure spending. If your GPUs wear out, you don’t have to go and build an entirely new datacenter. About 30-50% of datacenter spend goes to land, power, cooling, and so on. The remaining 50-70% is the cost of the entire server rack, which includes a bunch of things that aren’t GPUs.
\nLike the idea that AI inference requires using huge amounts of water, the idea that AI GPUs only live a year or two is popular because it’s a useful idea for AI skeptics, not because it’s true. It comes from a pseudonymous tweet quoting an anonymous source who’s being paid hundreds of dollars to sound like a credible expert on AI. Other public communications from AI inference providers cite much higher lifespan numbers, and the statistics from supercomputers (the traditional examples of large GPU clusters) don’t bear out the claim that the maximum lifespan is three years.
\nIt might be true that the economic lifespan is three years, in a world where new GPUs come out every eighteen months and GPU providers are flush with cash to upgrade, but that doesn’t tell us much about the economics of inference in an AI winter. If money becomes a lot more scarce, it’s likely that AI datacenters will continue profitably4 running their B300s (or their H100s or even A100s) for six years or longer.
\nOf course, like previous claims about AI and water usage, “three years at the most” is often cited as “1-2 years, with some lasting up to 3 years under optimal conditions”.
\n↩\nOf course, pronouncements from CEOs/CTOs should be taken with a grain of salt as well (for instance, maybe they have a big backlog of unused A100s they keep swapping out), but (a) executives don’t often straight-up lie about concrete technical facts, and (b) they’re going up against an unsourced quote from a tweet, so the bar isn’t that high.
\n↩\nWhat about proactive GPU replacement? In the “Survival Analysis” section, the study attempts to account for this. I haven’t dug into exactly how.
\n↩\nAssuming inference is profitable, which I believe (when you’re not attempting to amortize the cost of training).
\n↩\nThe relationship engineers have with product management is more dysfunctional than with any other part of the company. There’s no shared culture or language like there is with other engineers, and the rules of “who gets to tell who what to do” aren’t as clear-cut as they are with managers. Engineers don’t have a lot in common with legal, or design, or sales, but they also don’t need to interact much with those roles. In my experience, engineers are communicating with product managers almost every single day.
\nThe worst version of the product/engineering relationship goes something like this:
\nEngineers are technically competent but are too autistic to be fully trusted. They need a kind-but-stern parental figure who knows how to communicate to other stakeholders in the organization (for instance, by being comfortable using the word “stakeholders”), and how to keep engineers from going off in the wrong direction.
\nThis entire gross dynamic is neatly captured by the popular term “product mommy”1.\nI really, really don’t like that term, or this entire dynamic in general. Almost none of my relationships with my product managers have been anything like this, though I have seen it at a distance.
\nWorking well with product managers can be the difference between succeeding and failing at a company. Why is it so hard to maintain good relationships between engineering and product? What does a good relationship look like?
\nProduct managers and engineers have largely non-overlapping skillsets. Product managers don’t understand the technical work engineers do and aren’t equipped to talk about it: if an engineer gives a technical reason for something, product managers generally have to shrug and say “sure, I guess”. Likewise, engineers don’t have anything like the visibility into the organization that product managers do. Particularly in large organizations, it is the product manager who is the source of truth about who wants what and which features are important. When a product manager says that something is critical, engineers generally have to shrug and say “sure, I guess”.
\nThis obviously requires a lot of trust. What’s a little less obvious is that this trust is continually broken by both sides. Every single product manager has been told thousands of times that technical task X is technically impossible or would be disastrous, only for that task to end up being done fairly smoothly and successfully. Every single engineer has been told thousands of times that requirement X is absolutely critical and worth going to enormous effort for, only for that requirement to be silently dropped or changed with no apology.
\nOf course this isn’t malicious. Engineers often give wrong estimates because estimation is impossible, and sometimes the dire consequences they warn about really do happen (they’re just handled behind the scenes, like engineers handle many other kinds of technical dysfunction). Product managers “change their minds” because what’s important in a large tech company does genuinely change hour-by-hour2, and even the best attempts to only filter the most reliable priorities through to the engineering team will sometimes go wrong.
\nThe consequence of this broken trust is that the relationship becomes very difficult to maintain. When you’re an engineer, and you explain something to your product manager, and you know they don’t believe you (despite having no ability themselves to judge the question), it can be incredibly frustrating. Likewise, when you’re a product manager, and you’re desperately trying to explain what we need to do to an engineer, and you know they’re internally shrugging their shoulders, it must be unbearable. Don’t they know this is critical to the company? You were just in a meeting with the leaders of the organization!
\nThe natural tool for a mistrustful product manager is manipulation. I still remember a product manager who tried to extract a commitment from my team by asking us to go around and all say “I commit to getting this work done in two weeks”, after a conversation where we’d explained the risks that cause it to take longer. I suppose the idea was that we’d all work much harder, having taken a sacred oath? More subtle variants of this approach involve suggesting that you would be really disappointed if this work was delayed (in true “product mommy” style), or vaguely suggesting the possibility of some abstract reward (that the product manager is not empowered to deliver) if work gets done ahead of schedule.
\nThe natural tool for a mistrustful engineer is lies. The most benign version of this is exaggerating estimates: for instance, the classic advice to double your estimate and add 20%. I’ve seen engineers claim that they’ve had to follow up on all sorts of largely-fake tasks (one common example is “reaching out to a neighbor team to confirm X”) in order to gain more time. In the worst case, engineers might even straight-out lie that work has been completed, and then track the “it doesn’t work in production” feedback as a bug.
\nOnce this starts happening, it’s nearly impossible to repair the relationship. I can’t bring myself to trust a product manager who’s clearly trying to pull my strings, and I’m sure a product manager can’t trust an engineer who’s lied to their face in the past. That’s why it’s so important to avoid getting into a bad relationship in the first place.
\nWhy bother? If it’s so hard to hammer out a good working relationship with product managers, why not just settle for a bad one? Product managers can absolutely bury you if you’re not careful.
\nProduct managers are almost always more politically sophisticated than engineers. This is partly structural: product managers are simply in more conversations with the company’s movers and shakers, and so naturally have a better relationship with them (and are thus better attuned to which way the wind is blowing). It’s also partly selection bias: engineers can be hired even with relatively poor social skills, because they’re primarily being assessed on technical ability, but social skills are a core part of the product role3.
\nIf you are feuding with a product manager, you will probably lose. Unless you’re unusually influential, they will simply have far more opportunities to quietly talk you down in influential circles than you will. All it takes is a few comments like “oh, I probably wouldn’t pick Sean for that project” to wreck your reputation. In the case where you are openly feuding with a product manager, the company’s leaders will by default take the product manager’s side over yours. They’re likely to know them better, have more shared cultural context with them, and in general be willing to interpret the situation as “another engineer who doesn’t understand how the organization works”.
\nThere are huge benefits to being trusted by a product manager. Product managers want to ship things, and typically understand a fair amount about all of the non-technical barriers to shipping. If you also want to ship things, you can become a fearsome team.
\nOn top of that, because trust between engineers and product managers is so difficult, once you’re in you’re in all the way. Product managers often pick one or two engineers as their go-to for getting the “real story” on technical questions. If that’s you, you have an outsized position of influence in the organization, which you can use to get the things you want done.
\nAs an engineer, how can you build trust with your product manager?
\nThe first step is to understand where they’re coming from. When they tell you something is important or that a requirement has come in, be aware that this is rarely their decision. It’s not them who’s jerking you around, it’s someone higher up in the food chain jerking you both around. If you can adopt a conspiratorial mindset with them, instead of against them, that’s a good start. Try just asking “oh man, alright, what can we do about this?” instead of complaining.
\nThe second step is to be right, a lot. This is a silly-sounding Amazon leadership principle that turns out to be entirely accurate. I wrote more about it here, but (as unfair as it sounds) you really do have to be mostly accurate if you want to build trust with a product manager. When you say something will ship, it has to ship; when you say something is impossible, it can’t happen days or weeks later. It’s okay to be wrong sometimes, but you have to establish a pattern of you providing them useful, correct technical information.
\nThe third step is to let them make the political calls most of the time. If you expect them to trust your technical calls, you have to extend them the same trust when it comes to navigating the organization. Don’t publicly undermine them in meetings, bring up your concerns in private. If they say something is important and you’re not so sure, at least act like it is. Accept that sometimes they’re going to be wrong, just like you’re sometimes wrong about technical questions.
\nThe fourth step is to get lucky. Sometimes your product manager will just be a dud. You can’t build trust with someone incompetent: there’s nothing for you to trust them with, and they aren’t in a position where they can usefully extend trust to you. Working in large organizations requires getting comfortable with the fact that some of your colleagues will be stronger than others, and figuring out ways to work with (or bypass) people who make the work harder, not easier.
\nMany product managers were once engineers. If your product manager is technical, does that make you immune from these problems? Absolutely not!
\nYou likely won’t have much choice in which product managers you work with, but be aware that having once been an engineer is a negative, not a positive. No product manager can ever be technical enough to matter, because they don’t work on the codebase: even if they were a full-time engineer, they wouldn’t have the time to build the specific context on the system they’d need to be a real participant in technical discussions. It’s thus better to have a product manager who knows they’re not technical than to have one who mistakenly thinks they might be.
\nThe worst-case scenario is an ex-engineering product manager who believes they’re technical enough to detect when engineers are lying to them. This kind of paranoia is an easy trap for “technical” product managers to fall into, particularly when they don’t have a trusted engineer on the team they can lean on. If you’re dealing with one of these, prepare to spend a lot of time explaining why you can’t “just” do things (and prepare to have those explanations not be believed).
\nAt its worst, a product manager relationship is like an unhealthy family: driven by condescension, emotional manipulation, lies, and mistrust. This isn’t because product managers are bad people! It’s because the structure of the relationship creates conflict. Both sides must make commitments (about the technical system or goals of the organization) that are (a) often wrong, and that (b) the other side is unable to independently verify. To avoid the trap, both sides have to be generous, willing to trust each other in their areas of expertise, and most importantly competent.
\nUnlike most roles in tech, product management (particularly the lower-level roles that are more engineer-facing) has close to an even gender split.
\n↩\nFor instance, based on the whims (or snap decisions, more charitably) of the CEO.
\n↩\nI have worked with product managers with poor social skills, but it’s rare: about as rare as working with engineers with genuinely poor (i.e. by general-population standards) technical skills.
\n↩\nMany engineers should be doing less work. I don’t necessarily mean producing less code or fewer changes, but literally working fewer hours in the day. When they do work, they should be working at a slower pace. I like to aim to be running at 80% utilization by default: unless I have a high-pressure project going on, I spend 20% of my workday away from the computer.
\nWhy? Performance at tech companies is dominated by outlier events. When I think about the most impactful changes I’ve made, many of them involved a surprisingly trivial amount of work. There are no points for effort in software development. What matters is solving the right problem at the right time.
\nIn large engineering organizations, there are usually trivial pieces of engineering work you could do that would make tens or hundreds of millions of dollars for the company. Here are three common examples:
\nFirst, when the company is trying to sign a big enterprise deal, stepping in with a feature or bugfix can make the deal happen. It doesn’t even have to be a good feature: sometimes just showing that you’re willing and able to make a concrete change will be enough.
\nSecond, preventing or mitigating an incident early (even by just knowing the right feature flag to turn off) can save huge amounts of money: both immediate lost revenue during the incident and future lost revenue from customers who would have pulled their business or refused to sign pending contracts.
\nThird, when the company is trying to ship a high-profile feature, success or failure often hinges on trivial but obscure changes (e.g. the ability to rapidly add a new field in user settings, or to update the crufty enterprise-data-export functionality nobody has touched in years). Familiarity with the system can be the difference between one of these changes taking a few hours or a whole week.
\nWhat do these examples have in common? They’re all time-dependent. You can’t just log on in the morning and decide to unblock a big deal, or mitigate an incident, or speed up a high-profile feature. Is it just a matter of being in the right place at the right time? Not quite. You also have to not already be busy.
\nI wrote about this a couple of years ago in Crushing JIRA tickets is a party trick, not a path to impact. If you’re always 100% utilized on a steady stream of low-priority work (for instance, if you’re just picking up tickets from the backlog, crushing them, then picking up the next one), you’ll miss your chance to do high-impact work in two ways.
\nFirst, you’ll be too busy to even notice the opportunities. You won’t be chatting with people who are working on other things, or reading team updates, or keeping an eye on ongoing incidents. So you’ll miss out on the best way to get involved in high-impact work, which is to volunteer your expertise.
\nSecond, if you perpetually look busy, your manager won’t want to volunteer for you. This is the second-best way to get involved in high-impact work: to have your manager or product manager say “oh, Sean has capacity to help out here, let me tag him in”. Why is this better? Because managers and product managers usually have a much better read on what high-impact work is going on. They’re in meetings that you aren’t in.
\nIf you’re supposed to keep your time free for high-impact work, and you’re not supposed to just grind tickets, what should you be doing on a minute-by-minute basis? Should you just be doing nothing? Yep!
\nDoing nothing is good, actually. Software engineering can be a stressful job, but it’s typically not consistently stressful: the stress comes from the occasional incident, or high-pressure urgent piece of work, or (these days) layoff. If you approach the comparatively low-pressure parts of your work with urgent intensity, you’ll already be exhausted and frazzled when you have to handle the high-pressure parts.
\nEven in high-pressure parts of the job, doing nothing can still be good. One thing I recommend for engineers new to on-call is to avoid rushing: take a few breaths before joining the call or before speaking, and in general try to “think in slow motion”. Most incidents resolve on their own. Most frantic “maybe this will help” changes during incidents make things worse, not better. As a general rule, if you can simply avoid panicking, you will be doing better than most engineers at incident response.
\nNothing is a space things can happen in1. If you give your brain a chance to rest, you will find you’re more likely to have new ideas. If someone hands you an important task, you can tackle it with your full attention (instead of juggling it with the three other things you’re working on in the background). When you’re not busy, you have time to just look at things and take in new data.
\nA lot of engineers are uncomfortable seeing a task that needs doing and not doing it. I’m like this as well. I wrote about it in I’m addicted to being useful: it’s a psychological quirk that many software engineers share, because having that quirk (to a point) makes you a good fit for the job. In order to spend time doing nothing, sometimes you need to force yourself to not step in.
\nFor instance, I believe that engineers should generally avoid glue work2. Most glue work - making sure people talk to each other, updating docs for work you’re not leading, volunteering to address technical debt - reflects the fact that the organization is not explicitly prioritizing this work. If they were, you wouldn’t need to volunteer for it. Either that’s fine, or it’s a big mistake. If it’s fine, then you shouldn’t step up and do it: you’ll be wasting your time and annoying your manager. If it’s a big mistake, you still shouldn’t do it, because you’ll be insulating the company from the consequences of its own mistakes at the cost of your own career and mental well-being.
\nThat’s a bad deal for you, and a bad example for your junior colleagues, and sets a bad precedent for someone else to jump into the same position when you inevitably burn out3. If the consequences truly are severe, let them happen, so the organization can feel the pain and change its policies.
\nI also believe that being too helpful leaves you vulnerable to predators. Tech companies are full of people who want to extract uncompensated work from software engineers4. This is different from work that arrives via normal channels, and for which you’re compensated by promotions, bonuses (and just your normal salary). I’m talking about work that arrives via backchannels, from people who don’t have the ability or willingness to ensure that work is formally recorded under your name. For instance, a product manager from another organization messaging you to say “you’re so good at querying data, would you mind pulling some statistics for me about X?”, or an engineer from another team asking you to “pair” on a piece of work that will ultimately involve you writing all the code and them quietly submitting the change under their own name.
\nDoing some amount of this kind of work is fine. You may as well help people out when you can. But you need to be able to apply backpressure, either by saying no or simply delaying your response by a few hours or days.
\nIt’s also a good idea to avoid investing too much in work that is likely going to disappear. For instance, suppose you’re working with a product designer who is figuring out what they want in real time. At 9am they message you saying they want the page header to look one way, then at 10am they have tweaks, and more changes at 11am, and so on. You should not throw yourself into fully rewriting the page every hour. Instead, you should do nothing (say, go for a walk) and rewrite the page once in the afternoon, based on the most recent design. Another common instance of this is “big idea from a manager without the political clout to follow through on it”. Often you can just run out the clock until the project gets inevitably cancelled5.
\nA lot of software engineering advice and tooling is designed around the ability to scale up your ability to exert technical effort: to do more things at the same time, to take on projects of larger scope, or to just write more code. But software engineering success is not determined by any of these. It is determined by the ability to do the right things at the right time, which requires that you deliberately hold back some of your effort during ordinary work.
\nIn my experience, it’s still possible to be a “high performing engineer” at 80% effort. In fact, it’s easier, because you’ll be less likely to make silly mistakes from stress, and you’ll be in a position to jump on the kind of high-impact tasks that deliver outsized returns.
\nThis doesn’t mean you should never grind at 100% effort. I think there are probably two or three times a year where I work as hard as I possibly can: long hours, intense focus, thinking about the problem from when I wake up to when I go to bed. But I reserve this mode of work for when the rewards are really high. For the rest of the year, I take it relatively easy.
\nOne of my big influences is Rich Hickey’s talk Hammock Driven Development. This is kind of like what he’s talking about, except (a) Hickey is more talking about what it takes to design solutions to really hard problems, rather than what it takes to be a strong engineer in an ordinary tech company, and so (b) Hickey recommends using your time-away-from-the-computer to focus on a hard problem, instead of to simply decompress and let solutions congeal in your head.
\n↩\nI wrote about this a lot more in Glue work considered harmful.
\n↩\nWhy inevitably? Because in my view, burnout is hard work unrewarded, and taking on a personal crusade that your job doesn’t care about is a great way to do a lot of unrewarded work.
\n↩\nI wrote about this in Protecting your time from predators in large tech companies.
\n↩\nOf course, you have to be careful with this. If you try this strategy and you’re wrong about the level of political support for the project, you will come off like a slacker and then have to deliver in a rush.
\n↩\nProgrammers were better back in the day, weren’t they? Back when we had real programmers. Not just people who got paid to write code, but people who lived it, who were obsessed with their craft, and whose code was a lively expression of themselves. Hackers were hackers in those days before money took over the industry.
\nDon’t even get me started on LLMs. Could there be a better example of today’s degenerate spirit? A machine to mass-produce software (not good software, just barely good enough), so that the weak minds that dominate the industry can indulge their obsession with quantity: of slop code, of features, and ultimately of money, which is the only way they can understand value. If they weren’t destroying our way of life, they would be pitiable. All of them together don’t have a fraction of the spiritual integrity of someone like Mel. But as it is, we must band together to crush them and drive them from our industry like the parasites they are.
\nOkay, that’s not actually what I believe. But there sure are a lot of posts1 and comments on the internet that sound a bit like the paragraph above. Here are some older quotes that might sound similar:
\n\n\n…the third collapse, in which power tends to pass into the hands of the lowest of the traditional castes, the caste of the beasts of burden and the standardized individuals. The result of this transfer of power was a reduction of horizon and value to the plane of matter, the machine, and the reign of quantity.2
\n
\n\nUsura rusteth the chisel \\ It rusteth the craft and the craftsman \\ It gnaweth the thread in the loom3
\n
\n\nThe actual accomplishments of the past will nevertheless remain accomplishments, while the artistic stammerings of the painting, music, sculpture, and architecture produced by these types of charlatans will one day be nothing but proof of the magnitude of a nation’s downfall.4
\n
These are all from the writings (or speeches) of famous fascists: Julius Evola, Ezra Pound, and Hitler himself. Mussolini’s Doctrine of Fascism begins by defining fascism as a “spiritual attitude”, which the fascist man adopts in order to regain the mysterious qualities that were lost by the transition to modern life. In his classic Ur-Fascism, Umberto Eco’s first two defining features of fascism are the “cult of tradition” and the “rejection of modernism”. So when someone tells me that the industry has lost its way and we must deny the corrupting influence of modern technology in order to retvrn to the time of virile real programmers (who understood and appreciated the spiritual dimension of programming), I get suspicious.
\nIt’s strange to describe anti-AI sentiment as potentially fascist, since a very popular argument is that LLMs themselves are an inherently fascist tool. Surely both sides of the debate can’t be fascist? I do think that the structure of fascist arguments is generally persuasive, and that many avowedly anti-fascist groups do sometimes fall into this trap: describing the world as a struggle between the spiritual power of the macho, traditional man and the corrupting influence of degenerate (often foreign) capital.
\nFor instance, I am a big fan of Lord of the Rings. I’ve read the series and watched the films multiple times, and even made a failed attempt to learn Elvish as a kid. But it’s hard to deny that fascists absolutely love Lord of the Rings. “Marble statue of a Roman emperor” might be the most popular avatar for fascists on the internet, but Aragorn is the second most popular. Neo-fascist movements in Italy explicitly take up Lord of the Rings as a foundational text. Why? Because the core conflict in the text is between the traditional, nostalgic heroism of the Shire and Gondor, and the corrupting modern industrial (partly foreign) influence of Saruman and Sauron5.
\nI don’t think Lord of the Rings (or anti-AI rhetoric) is intrinsically fascist. In fact, the surface-level reading of the text is anti-fascist: the plucky people of the West banding together to fight Sauron’s command-and-control totalitarian society. But I can see why fascists love it.
\nOne common historical touch-point for anti-AI folks is the Luddites, who were a violent conservative labor movement in early 1800s England. Anti-AI blogs adopt Luddite language like “smashing frames”, and positively cite the Luddites as “the go-to enemies of fascism since its inception”. I’ve written at length about what we can learn from the Luddites in Luddites and burning down AI datacenters, but one point I think is under-emphasized by the (generally pro-Luddite) books is that the Luddites were a little bit fascist themselves.
\nBrian Merchant’s Blood in the Machine is the most popular recent book on the Luddites. I enjoyed it, but Merchant’s attempts to paint the Luddites as a friendly, left-wing, proto-feminist movement6 seemed really unconvincing to me. From the writings of the Luddites, it’s clear that they were interested in protecting the rights of their all-male elite guild fraternity. Here’s one Luddite threat to a workshop that explicitly includes a threat against the female workers7:
\n\n\nWe think it quite inconsistent with our duty as men, as husbands and as fathers to suffer ourselves to be ruined any longer by a set of vagabond strumpets and those gibbet-deserving rascals that are looking over them. We will lead them to their satisfaction. We sincerely hope, gentlemen, that you will discharge the bitches and take men into your employ again, or they must take what they get.
\n
These were fundamentally conservative people who felt (correctly) that modernity had deprived them of their elite status, handing it instead to lower-paid inferiors: women, vagabonds, and foreigners.
\nThe Luddites were obviously not fascists8. However, the basic ingredients were there: wounded pride, a masculine elite identity, hatred of modern economics, and violence aimed at restoring their previous position in society. The currents that produced Luddism are the same currents that guided so many unhappy people towards fascism. When things are looking grim for an elite group, they often turn towards any movement that promises a return to an idealized past.
\nIf my blog has themes, one of them is surely that many software engineers labor under a delusion that their job is to be excellent at their craft. Of course, wanting to be an excellent programmer is not a delusion; it is a completely legitimate value to hold, and a legitimate purpose to pursue. It’s just not what you’re paid to do at work. Your job, unfortunately, is producing shareholder value. This delusion has been punctured by the end of ZIRP, and again more recently by the rise of AI coding.
\nIn this environment, I worry that some software engineers will form exactly the kind of disillusioned elite that was the audience for Ezra Pound’s poems about “usury” or the Luddites’ campaign against unapprenticed (often female) textile workers. I worry that AI, and the companies that build AI, are becoming an enemy against which anything is permitted: an enemy which in Umberto Eco’s words is “at the same time too strong and too weak”, unable to reason and yet powerful enough to drastically reshape the global labor market for the worse.
\nThe enemy of fascism is nuance. Fascism presents a good, clean, rousing story about a spiritual conflict between right and wrong. It is anathema to fascism to stop and muddy the waters a bit: in this case, to explore the ways in which LLMs, like any transformative technology, can both support and endanger traditional values.
\nIn The left-wing case for AI I wrote about how AI is being used right now as a disability aid, and many disabled readers wrote in to share their positive experiences with LLMs, and often how alienated they feel by the anti-AI mainstream on the left. I recently got an email describing how there’s a sudden flood of accessibility software for blind people9 that’s actually built by blind people, who can now iterate with a LLM to get a product that meets their needs. Framing AI as an ontological evil erases experiences like these.
\nBeing anti-AI is not inherently fascist. Many of the anti-AI posts I’ve quoted are thoughtful, sensitive pieces exploring how the author thinks about one of the biggest changes to our industry. I still think the world needs more articles like that, not less, but the more of them I read, the more I recognize the tropes: spiritually pure lovers of the craft, degenerate peddlers of corrupt modernism, a need to return to the traditional ways of the hacker, and a lament for the (potentially) waning power of an elite fraternity of programmers.
\nI know I’m tiptoeing around the worst argument in the world. It isn’t a refutation of anti-LLM arguments to say that they are structurally similar in some ways to fascist arguments, any more than it’s a devastating critique to say the same thing about Lord of the Rings. Sometimes it is good to try and halt the march of progress! Some of our past traditions really were purer and more spiritually robust! It just bothers me, that’s all.
\nI used to read The Story of Mel with unalloyed pleasure. Now it makes me nervous. If you believe you’re fighting the embodiment of fascism, or for the idea of value itself, what tactics are off-limits? What positions might you eventually come to accept?
\nIt feels wrong to directly associate my caricature with any actual posts, but it also feels wrong to make a blanket assertion without examples. Just so you know what I’m talking about, here are some posts that have elements of this attitude. I like some of these posts and dislike others.
\n↩\nPage 329 of my copy of Julius Evola’s Revolt Against the Modern World.
\n↩\nEzra Pound, Canto XLV. “Usura” should be read as “usury”, or today we could gloss it as “capitalism”: all Pound’s examples of great art were from the pre-capitalist patronage era of art.
\n↩\nAdolf Hitler, from his speech at the 1933 Party Congress in Nuremberg.
\n↩\nOf course, there’s also historically been a strong pro-technology current in fascist thinking (even specificially Italian fascist thinking).
\n↩\nPage 134 of Blood in the Machine has a brief argument that Luddism was feminist because the (exclusively male) artisans’ wives would provide food for their meetings. No, really.
\n↩\nFrom Kevin Binfield’s Writings of the Luddites, page 40. I’ve taken the liberty of re-rendering it in modern spelling and grammar.
\n↩\nAside from being too early, they didn’t have any connection to the state apparatus of power (in fact, they were ultimately crushed by it) and they famously lacked a singular leader.
\n↩\nThe example cited was BlindRSS.
\n↩\nWhere are all the AI-generated projects? This is a common question from AI skeptics: if LLMs are so good at writing code, where is the tsunami of new AI-generated apps, services and games?
\nI personally don’t find this to be much of a paradox. Writing code is only one of the bottlenecks involved in actually shipping a new product, after all. It’s also impossible to talk about the paid work I’ve done with AI (you’ll simply have to take my word that it’s increased my productivity). But one thing I can do is share a list of personal projects I’ve built with AI in the last twelve months.
\nI definitely would not have done all of these by hand. I might have found the time to do one or two of them, but based on my pre-AI track record they would probably have stayed in the “GitHub repo with a few commits” stage. This list is a kind of existence proof: a bunch of weird projects, useful to at least some people, that would not have existed without AI assistance0.
\nMost recently I’ve built skifreedle.com, a daily-game version of the classic Windows SkiFree game (i.e. “like Wordle, but for SkiFree”). The code for that is here1.
\n\n \n \n ![\"skifreedle\"](\"https://www.seangoedecke.com/static/6cdb3210b3e29b8b9f8ba68fee4d2502/105d8/skifreedle1.png\" "\\"skifreedle\\"")
\n \n
\n \n \n ![\"skifreedle2\"](\"https://www.seangoedecke.com/static/6ee543da4626f592eb2537653f7d2b18/a13c9/skifreedle2.png\" "\\"skifreedle2\\"")
\n \n
I enjoy coding small web games by hand, but definitely would not have had the time to wire up all the different SkiFree objects or build neat features like a ghost of your fastest run. I also tried out a lot of different visual themes for the game UI before landing on something I liked. If I’d done this by hand, I would have only had time to try out two or three different looks, instead of fifteen or twenty.
\nI’m very happy with how this turned out. I’ve been enjoying competing against my brother to get better times, since both of us have a lot of nostalgia for the original SkiFree game.
\nLast year I built Autodeck! I wrote a blog post about this before, but this came from my partner wishing there was some way to automatically generate Anki cards about random topics she wanted to learn about. It ended up being relatively straightforward to set up an endless feed of auto-generated spaced repetition cards:
\n\n \n \n ![\"autodeck\"](\"https://www.seangoedecke.com/static/ab189b44f0c389b77ca4df74dcf8259b/71c1d/autodeck.png\" "\\"autodeck\\"")
\n \n
I set up Stripe payments for this one, more because I was worried about someone running away with my Groq balance than because I wanted to make money, but I was pleasantly surprised to see a bunch of people actually use this. Over five hundred people have tried it out, with enough paid subscribers to cover inference and hosting.
\nI might have built this without LLM assistance, but I almost certainly would not have deployed it as a website. The hassle of setting up a database and Stripe would have just been too much work.
\nI also built an AI-generated endless wiki. I wrote a blog post about this one as well. Like Autodeck, I was fascinated with the idea of non-chat interfaces for LLMs, and I thought a wiki-based approach where you interact with the model by clicking links was pretty cool.
\n\n \n \n ![\"endlesswiki\"](\"https://www.seangoedecke.com/static/50f19e416dd78a54917fe8f645449839/6578c/ewiki.png\" "\\"endlesswiki\\"")
\n \n
I learned the hard way that putting a LLM generation call on the end of a regular link was a bad idea: scrapers would exhaust my inference budget quickly. I ended up faking the no-article-exists-yet links with JavaScript, which at least so far has defeated scrapers. People still email me about Endless Wiki, and there are over 280 thousand pages generated.
\nMy original goal was to see if you could eventually generate a page for Neon Genesis Evangelion, starting at the root page and only following links (kind of like wiki golf). I was successful! You can read the “Evangelion Anime” page here.
\nAlmost exactly a month after I launched Endless Wiki, xAI launched Grokipedia. Obviously they didn’t plagiarize me. This is a very easy idea to have, and my site was not the first infinite wiki (though I think it was the first one where you had to discover new pages by clicking on links). But it did take some of the shine off.
\nI built a PWA that caches the VicFlora plant identification database so it could be used with low or no internet. This was more of a utility project for my partner, who likes plants and occasionally goes on field trips where internet is spotty.
\nI would definitely not have done this without LLMs. It was reasonably difficult to scrape the basic dichotomous key from the VicFlora website: their API documentation was out of date, there were multiple possible pathways for fetching data (most of which were not functional), and the format of the data I did manage to fetch was hard to parse. I think I could have done it, with enough effort, but it would have been a substantial amount of work.
\nI’m very happy with how this turned out. It’s not perfect, but it’s functional, and I’ve even had the occasional Victorian botanist email me with bug reports or feature requests, so it’s clearly seeing a little bit of usage.
\nI did a bunch of other stuff that doesn’t necessarily rise to the level of a “deployed project”: my gh-standup GitHub CLI extension to automatically generate a standup report, which has just over a hundred stars, my (low quality) image geolocation benchmark, which I blogged about here, or my skill for extracting features from open-source models.
\nThere may not be a flood of AI-generated companies (yet), but at least for me there’s been a flood of small, weird projects that would not have existed without significant LLM assistance.
\nI also want to shout out Simon Willison’s version of this, which is another great example of “weird useful tools that only exist because the cost of creating them was so low”.
\n↩\nI did lift the spritesheet from DanielHough’s SkiFree.js, which attributes it to Wing Wang Wao. Of course, the original sprites and art belong to Chris Pirih’s SkiFree and Microsoft.
\n↩\nThere are only two ways to use LLMs in a computer program: as part of a pipeline, or as an agent. In other words, either you express the control flow of the program in code, or you give a LLM tools and allow it to manage the control flow itself1.
\nHere’s how you might structure a trivial “summarize a bunch of information and email it to me” program as a pipeline:
\n![datacenter](\"https://epoch.ai/assets/images/data-insights/ai-datacenter-cost-breakdown/ai-datacenter-cost-breakdown-upfront.png\"){kind=link}
context = gather_context(various, data, sources)\nllm_response = llm_summarize(context)\nsummary = parse(llm_response)\nemail_me(summary, my_email)And here’s how you’d do it as an agent:
\nread_data_tool = build_read_data_tool(various, data, sources)\nemail_tool = build_email_tool(my_email)\nrun_agent(tools: [read_data_tool, email_tool])It’s like the difference between a library and a framework. When you use a library, you define the structure of the program yourself, and call out to various library helpers along the way. When you use a framework, the main structure of the program lives in the framework, and it calls your code at various points. There are tradeoffs involved in both approaches. Frameworks let you get started more quickly and typically give you features “for free”, but can be difficult when you want to do something that isn’t part of the framework’s design. Libraries give you a lot more control, but require you to write (and maintain) more boilerplate code.
\nIn the trivial case, the distinction between a pipeline and an agent melts away. If you only have a few paragraphs of possible context for the problem, an agent with a gather_context and an email_me tool will perform exactly the same steps as a pipeline that calls a reasoning model with the context injected into the prompt (i.e. the agent will reproduce the trivial control flow of your pipeline). But when you have more context than will fit into a single prompt, or you want to take an action and then react to the result, the choice between pipelines and agents becomes very significant.
Pipelines are more predictable, but agents are more flexible. When you give a problem to an agent, work stops when the LLM thinks it’s done. Depending on the perceived difficulty of the problem, this can take anywhere from a few LLM turns to hundreds (and thus cost anywhere from a few cents to many dollars). If you’re building something intended to run at scale, this unpredictability can be a nightmare. Any subtle change to the user data could cause the LLM to take twice as long on each task, which would double your latency2 and cost.
\nPipelines are only immune to this problem if they don’t use reasoning models, or don’t allow the model to “think out loud” in its output tokens (for instance, by using structured output). However, individual LLMs offer much tighter control over model reasoning than over how long an agentic loop will take. In all frontier model APIs, you can explicitly set the level of reasoning you want. That doesn’t give you total control, but it does cap “take longer” at maybe ten or twenty percent (instead of with agents, where it can be 2x or more).
\nWhy use agents, then? Agents are smarter. If you’re happy to accept the unpredictability, an agentic system can handle much more difficult tasks, by virtue of being able to loop for longer, and to gather more information after thinking about the problem. There’s a reason that the most successful AI products (coding agents like Claude Code, Codex, Cursor, and Copilot3) are agents: coding is a hard enough task that you simply cannot build a functional coding agent with pipelines.
\nThe context-gathering stage is far more delicate for pipelines than for agents. If an agent is trying to solve a problem and realizes it needs more data, it can simply go and get it. But for a pipeline, all the required data has to be present in the context already, because the LLM only gets to run once.
\nMuch of the work involved in building pipelines is in getting context-gathering right. Agents are much easier. For instance, with a coding agent, you can basically just provide a “grep” and “read file” tool and let the agent figure out what chunks of code are relevant to the current file. In a pipeline, you have to figure that out yourself: good luck, it’s an unsolved technical problem! Typically you’ll end up doing some set of clever tricks, like walking the AST to identify which parts of code “contribute” to the current file, or indexing the whole codebase with semantic embeddings and doing some kind of nearest-neighbor search to build the context (called RAG, or “retrieval-augmented generation”). Neither of these will work as well as using an agent.
\nIn 2023 and 2024, many people believed that RAG would solve context-gathering. Every LLM would have a fully-indexed context base that would magically surface the precise information the LLM needed at any given moment. This did not happen. Instead, we went backwards, getting our agents to do plain-text search and figure it out like a human would. Why didn’t RAG work? This is a topic for a whole other post, but the short answer is this: “find what information is relevant to this problem” is often as hard a task as actually solving the problem. Semantic embeddings and cosine similarity are simply not powerful enough tools for the job.
\nPipelines that make multiple LLM invocations do have an extra dimension of flexibility: they can use different LLMs for different tasks. For instance, if one LLM benchmarks better at task A, or is cheaper for an easier task B, you can use the right model for the job. Agents (at least right now) have to stay the same model the whole time, so you’re always pinned to the highest level of intelligence you need.
\nIs this a big deal? I’m suspicious. One pattern I see a lot is tasking a cheaper model with collating or summarizing data for a smarter model to do something with. But often the signal is in the raw data itself! I think designs like this are really shooting themselves in the foot, for the same reasons that RAG didn’t work: context-gathering was a harder problem than people anticipated.
\nIn any case, if you do want to farm out tasks to different models, you can also do it via careful agentic tool design. For instance, you could build your web_search tool so that it uses a cheap model to summarize web pages.
Pipelines allow working with smaller contexts, and thus with local models. An agent’s ability to fetch its own context means that it almost always ingests more data than it needs. On top of that, agents run in loops, so each agent turn increases the size of the context. This isn’t a big problem for systems built on top of frontier model APIs, because:
\nHowever, it is a big problem for local models. The context window consumes a lot of VRAM, so most people running local models stay below 32k (or even 6k) tokens. If you’re writing a program to run in this environment, you likely will not be able to give an agent the space it needs, and you will be instead forced to use a pipeline.
\nIn my opinion, agents are more future-proof. This is partly because models are now being explicitly built to be better agents, and partly because agents delegate more to the LLM and thus benefit more from LLM improvements. If you have a pipeline-based system, new models will probably do a bit better than old ones. If you have an agentic system, new models might do much better than old ones (to the point that it’s worth building an agentic systems for tasks that are currently too hard, on the assumption that by the time you’ve finished the models may be good enough). I have been banging this drum since 2023, before tool-calling was even a part of model APIs.
\nIn general, I disagree with the popular advice that workflows are safer than agents. Workflows offer more control over budget, but when it comes to taking action based on LLM output, you have exactly the same problem whether you’re checking at the tool-call level or at the next stage in the pipeline: either you make some heuristic assessment via code, which might be wrong, or you queue the action up for a human to approve, which will be slow.
\nDon’t agents open you up to prompt injection? Yes, but pipelines do too. In both cases, you’re feeding some block of human-generated data (e.g. the files in a codebase, or the results of a web search) into the LLM. Any prompt injections in that data will be consumed by the LLM just the same whether they’re the result of a tool-call or directly injected into the prompt by the pipeline. You have to sanitize user content and double-check LLM-triggered actions, no matter what design you choose4.
\nI do want to acknowledge that pipelines are slightly more legible. You can trace most of what a pipeline is doing because you’re in control over more of it. It’s harder to figure out why an agent queried for a particular piece of information or took some action. But even in a pipeline, you’ll never know for sure why the LLM responded in the way it did. That’s just what it means to program with LLMs.
\nLet’s apply some of these principles to a real-world, non-trivial example. Suppose you are the NSA, and you are attempting to use LLMs to get a grip on the wild firehose5 of covert email surveillance data6. Should you use pipelines or agents? Well, if you’re building something that’s supposed to run on every single piece of email in America, you probably shouldn’t use agents: keeping performance and cost strictly bounded requires a pipeline. However, you’re definitely well-resourced enough to use agents in general, and the problem is definitely hard enough to benefit from the extra intelligence. I’d probably recommend using both: a low-context, cheap pipeline that can run once against each email and flag it, and a fleet of agents that can dig into those flags, make ordinary queries, and act more like human analysts would.
\nThe pipeline would have to scale with the total volume of data, which should be mostly fine, since pipelines scale in a predictable-ish manner. The fleet of unpredictable agents can be scaled entirely independently, though in practice it would get bottlenecked on GPU availability and the necessity for human review. The majority of the engineering work7 would likely go into context-assembly for the pipeline: feeding in enough data about who’s involved in the email conversation so that the LLM can make a sensible decision on whether or not to flag it.
\nOverall, I’d suggest following these guidelines:
\nWhen in doubt, use agents. I am aware of several AI projects that have migrated from pipelines to agents in the last year, but none that have gone the other way around. As a general point about software design, if you’re not sure what to do, pick the solution that’s easier to build and more likely to be able to solve your actual problem. If you want to change to a cheaper, pipeline-based system later on, at least you’ll be able to compare it to a working agentic design and make an informed decision.
\nThis distinction was popularized by Anthropic’s Building effective agents, written in December 2024, and now (I believe) made at least partially obsolete by advances in agents since then. They say “workflow”, but I slightly prefer the term “pipeline”.
\n↩\nYes, I know this is technically not what “latency” means, but there’s no other single-word shorthand for “the duration of a standard unit of work”.
\n↩\nIf you’re building your own coding agent, I suggest you begin with the letter “C”.
\n↩\nFor instance, in my trivial example at the top of the post, doesn’t the agent have a failure mode where it might send a ton of emails, or email a bunch of different people? No, because you ought to constrain the email tool so that it can only send to the right address, and (if this is important) that it can only be called once.
\n↩\nIn a draft post I never published, I ballpark-estimated all non-spam American email data at around seven trillion tokens per day (around a third of OpenAI’s total daily token usage).
\n↩\nShould you do this? Probably not, but it’s a fascinating engineering problem, and I imagine the NSA has been thinking about these questions for several years by now. If the example bothers you, substitute some other more-ethical firehose of English language.
\n↩\nNot counting evals, operations, standing up a trusted GPU cluster somewhere, scaling the physical hardware, and all the other thousand things you have to do in order to ship anything.
\n↩\nIn April last year, Kelsey Piper discovered that OpenAI’s o3 model was surprisingly good at figuring out where a photo was taken from. Like human “geoguessr” pros, o3 could sometimes take a nondescript photo of a beach and tell you exactly where it is. Here’s the example Kelsey gave:
\n\n \n \n ![\"geo\"](\"https://www.seangoedecke.com/static/4113a246112c8b6db424a58af58a9a90/4d836/kelsey-geoguessr.jpg\" "\\"geo\\"")
\n \n
Several people reproduced this with good results: not a 100% success rate, but clearly far better than you’d do with a random human guess. The lesson here is that model capabilities can surprise us. The o3 model had been released for two weeks before Kelsey’s tweet without anyone noticing how good it was at geolocation. What obscure capabilities did we never find? What capabilities of current models are we missing today?
\nSome people drew another lesson from this: that “prompt engineering” can unlock brand-new capabilities. This is because Kelsey had a magic prompt that she built over time. When o3 got something wrong, she would ask it how it could have avoided the mistake, and then included that in the prompt. Here’s the first 10% of that prompt, so you get the idea:
\n\n\nYou are playing a one-round game of GeoGuessr. Your task: from a single still image, infer the most likely real-world location. Note that unlike in the GeoGuessr game, there is no guarantee that these images are taken somewhere Google’s Streetview car can reach: they are user submissions to test your image-finding savvy. Private land, someone’s backyard, or an offroad adventure are all real possibilities (though many images are findable on streetview). Be aware of your own strengths and weaknesses: following this protocol, you usually nail the continent and country…
\n
This prompt impressed a lot of people, who tried it out and reported that it correctly identified a lot of images. But of course, o3 correctly identified a lot of images with just a basic “think carefully about where this picture was taken?” prompt. Did the prompt actually help? It’d be tough to figure that out just from playing around in ChatGPT. You’d need to build an evaluation set of images and run o3 against them twice: once with the fancy prompt and once without it.
\nSo that’s what I did. I pulled 200 images from Wikimedia Commons, Geograph Britain and Ireland, and iNaturalist for the benchmark. You can read the AI-generated summary here, but here’s the key table:
\n| Prompt | \nn | \nMedian km | \nMean km | \nP25 km | \nP75 km | \n<=25 km | \n<=100 km | \n<=500 km | \n<=1000 km | \n
|---|---|---|---|---|---|---|---|---|---|
| Default | \n200 | \n83.2 | \n440.7 | \n16.4 | \n221.9 | \n58 | \n109 | \n176 | \n182 | \n
| GeoGuessr prompt | \n200 | \n102.3 | \n481.9 | \n18.5 | \n277.8 | \n59 | \n99 | \n172 | \n180 | \n
In general, the basic prompt did better on average. It consistently guessed closer to the actual location. Both prompts did pretty well, actually. Despite the fancy prompt being 10x larger, it only caused o3 to think for slightly longer (about one second on average, though the max was about double, at 10 minutes instead of 5 minutes). The images in my benchmark were fairly generic geoguessr-style outdoor images, with twelve indoor images thrown in for an extra challenge (the fancy prompt also did slightly worse on these).
\nWhat’s going on? I think this shows how easy it is to fool yourself about the quality of prompting. When the model is already pretty good at a task, you can give it a very elaborate prompt without impacting performance. It’ll still be pretty good, except this time it’s good because of what you did. This is particularly true if you’re iterating with the model and asking it “what should I add to the prompt” for each mistake. Models will happily make up stories for you about their own reasoning processes, and will almost always say “yes, that helped a lot!” when you ask them if a particular prompt tweak made things better. The only way to actually know is by constructing some kind of benchmark1.
\nIt’s also interesting to me that nobody checked this at the time. It took me about six hours of fairly-distracted work and about $15 to construct and run this benchmark. Why didn’t anyone do this when they were writing articles about how good the o3 prompt was?
\nOne charitable reason might be that the story was more about o3’s real geolocation ability than about the magic prompt. The pricing for o3 also used to be about five times more expensive (though a benchmark of 40 images instead of 200 would still have thrown doubt on how much water the prompt was carrying). Also, AI just moves so fast. Geolocation was only the story for about a week: after that, GPT-4o’s sycophancy was what people were talking about. Another reason is that AI tooling wasn’t as good then. The benchmark was so easy for me to run because GPT-5.5 did most of the heavy lifting. Prior to strong agents, you would have had to write the (simple) benchmark yourself. I can’t point the finger too hard: I didn’t bother at the time either.
\nMaybe my benchmark isn’t very good? The photos look reasonable enough: a wide variety of geoguessr-like shots of roads and landscapes, mostly. I could have tried to gather a few thousand photos instead of a few hundred, but if the magic prompt really was a big improvement you’d still expect to see that manifest on a benchmark this size. If someone wants to go and build a hundred-dollar geolocation benchmark instead of my fifteen-dollar one, I think that’d be an interesting project.
\nFinally, let’s use the benchmark to answer a question I’ve had for a while: do gpt-5.4 and gpt-5.5 have o3’s geolocation abilities? The answer, apparently, is no.
\n| Run | \nMedian km | \nMean km | \n<=25 km | \n<=100 km | \n<=500 km | \n
|---|---|---|---|---|---|
| o3 default | \n83.2 | \n440.7 | \n58 | \n109 | \n176 | \n
| o3 GeoGuessr | \n102.3 | \n481.9 | \n59 | \n99 | \n172 | \n
| gpt-5.4 default | \n163.3 | \n638.9 | \n26 | \n74 | \n148 | \n
| gpt-5.5 default | \n156.5 | \n645.9 | \n39 | \n77 | \n161 | \n
Whatever o3 had that made it good at this task hasn’t transferred to newer models.
\nBenchmarks can mislead as well, but they’re better than just vibes.
\n↩\nIt’s common and correct to say that “all code is technical debt”. Adding code is a necessary evil for developing new features: you almost always have to do it, but each line of code adds to the complexity and maintenance burden of the system. All future changes to the system have to work with the existing code, or at least avoid breaking it. Once systems accumulate enough code, they become impossible for a single person to understand: instead of reading the code and understanding what it does, you must rely on guesses, theories and heuristics1. Sensible engineers write as little code as possible.
\nThey write a lot of prompts, though! Many large projects now have a set of codebase-specific prompt files: AGENTS.md, CLAUDE.md, those same files in sub-directories, and skills. If you’re building a program that uses AI2, you’ll have separate prompts for capabilities and for each tool, as well as a whole set of system prompts.
\nPrompts are important. Minor tweaks to a LLM’s prompt can unlock significant performance improvements. If the same model feels different across Codex, Cursor, OpenCode, and Copilot, it’s almost certainly due to subtle differences in prompting. AI companies spend a lot of time testing and tweaking their prompts, so it makes sense why engineers would spend a lot of time tweaking their AGENTS.md files3 for their projects. I’d even call switching tools or workflows to be a form of prompting. If I start wrapping my agents in a Ralph loop, pull in a new skill file, or install an MCP server, that’s still a change to my prompts even though I’m not the one who wrote it.
\nI think it is a bad idea to spend a ton of time tweaking a bespoke agentic coding setup. Why is that, given that prompt adjustments can deliver a lot of value? Because prompt adjustments are model-specific. Earlier I said that AI companies spend a lot of time tweaking their prompts. In fact, they spend that amount of time for each new model release. A prompt that worked great for GPT-5.4 won’t necessarily work as well for GPT-5.5. You have to “learn how to hold the model” each time.
\nIn other words, a set of prompts that you carefully crafted in January this year might be out of date or actively harmful by February. Worse still, you might not even notice. Model capabilities are already so hard to pin down (unless you’re running every problem through different models and tools), and even weak AI systems are surprisingly good at some problems. You might just think “huh, the new Anthropic model isn’t as impressive as the hype”, or “wow, Claude Code has gotten worse recently”.
\nIn this sense, prompts are a worse form of technical debt than code. When technical debt blows up, it usually causes errors or a tangible slowdown as you try to understand the code. Prompts will decay silently. Also, even janky code tends to be relatively stable when untouched, but every single model upgrade could turn a functional prompt into a non-functional one.
\nCould you simply decide not to upgrade models? Some people are trying this, but the pace of improvement is fast enough that that isn’t really practical. A delicately-prompted agentic harness built around GPT-4.1 is always going to underperform a bare-bones harness built around Opus 4.7. This might be a sensible strategy at some point in the future, when the rate of model improvement slows down (or when models are so capable that you don’t need the extra intelligence for normal engineering tasks), but I don’t believe it’s a good strategy today.
\nIn my view, most people should just be picking an AI coding tool maintained by a third-party company (Claude Code, Codex, Cursor, Copilot, etc) and leaving it as unconfigured as possible, so they can piggyback on the work of teams of engineers who are evaluating and tweaking prompts with each new model. Avoid MCP and skills unless absolutely necessary, and keep them off by default. At least this way if one of those teams gets it badly wrong, users will notice eventually and complain about it.
\nWhen you write AGENTS.md files, try to avoid behavior steering (like the now-outdated “think step by step”, “you are a skilled engineer”, or “if you get a task right I will tip you $200”). Keep them limited to specific, concrete facts about the project. Don’t let models fill your AGENTS.md with pages of barely-reviewed text, for the same reason that you wouldn’t let them fill your codebase with pages of barely-reviewed code. Write your prompts yourself, and delete them whenever you get the chance.
\nAlmost every system you might get paid to work on is in this category (if not in the code of the system itself, then in its dependencies and libraries).
\n↩\nInstead of just using AI to build a program. This distinction was a real pain when I was working on GitHub Models.
\n↩\nThe engineer who says no all the time is a real archetype among senior and staff engineers. Their role is to slow things down, to block the development of features that add complexity, and to ensure that as little code gets written as possible (since code is a liability).
\nWe can think of this as the just-say-no engineer1, as opposed to the just-say-yes engineer. The just-say-yes engineer is obsessed with moving fast, approves code changes by default, values MTTR over MTBF, and tends to ship a lot of code. The just-say-no engineer is obsessed with quality, is happy to move slowly, and blocks code changes by default. Most engineers are somewhere in the middle of the spectrum. By “just-say-no engineer”, I’m talking about the group of engineers who most strongly identify with that archetype.
\nThe just-say-no engineer is having a hard time in the era of AI. It used to be that they only had to say no to more junior engineers’ handwritten PRs, but now they have to say no to a barrage of AI-generated code, some of it generated by managers and VPs who are politically difficult to say no to. For the first time in their careers, they’re under a lot of pressure to lower their standards and start saying yes. However, this isn’t because of AI. It’s because of the end of ZIRP.
\nZIRP, or the “zero interest rate policy”, is a shorthand for the era of software development between 2008 and 2022 when banks were allowing companies to borrow money at near-zero interest rates. During this period, investors were throwing borrowed money at anything, which meant that tech companies were incentivized to constantly hire engineers for low-risk high-reward projects2. Successful companies would routinely grow from tens of engineers to thousands, who would go and work on all kinds of things: tangential open-source projects, endless technology migrations, rewrites into other languages, and so on.
\nIt was a great time to be a software engineer. We had a lot of bargaining power, and could get paid top dollar to do almost anything. The bosses largely didn’t care, because (a) teams were growing so fast they couldn’t pay attention, and (b) just having more engineers around was beneficial to the stock price, which was the main thing they cared about. But tech companies did have one problem: with so many engineers running wild, how would they keep their systems from becoming completely unmanageable? Enter the just-say-no engineer.
\nIn this environment, having a very senior engineer whose only job is to say no to things was actually quite valuable to the company. There are a few reasons for this:
\nWhen banks hiked interest rates, almost every tech company immediately laid off 5-20% of their engineers. It was just no longer profitable to keep a bloated engineering staff around to boost the stock price. Instead, companies had to actually make money3. However, that wasn’t a good public explanation for the layoffs, since it sounds weak to admit that you were paying hundreds of engineers to do unprofitable work. Fortunately, the end of ZIRP coincided roughly with the rise of ChatGPT, so tech companies were able to to blame their layoffs on the power of AI. Saying “with this transformative new technology, we’re able to deliver 10x the value with half the engineers” is a much stronger message, even though it doesn’t make much sense (if this is true, why not keep your engineers and deliver 20x the value?)
\nSomething like this dynamic has been happening to the just-say-no engineer. Tech companies are now more focused than at any time in the past two decades. They are not doing a bunch of random crap anymore; instead they’re desperately chasing new capabilities and features that can make money (mostly built on AI, for obvious reasons). This new environment is actively inimical to the just-say-no engineer. It’s as if a shark got pulled out of the deep ocean and dropped into a fast-flowing river: what was once a powerful apex predator is now disoriented and flailing.
\nThis kind of engineer used to enjoy implicit (albeit distant) support from their management. If someone complained, they’d often get told “that engineer knows what they’re doing, if they said no, then I trust them”. Now that support is gone. The just-say-no engineer is now being criticized and actively overruled by their management. They’re being told to be more of a team player, to find a way to say yes, or are simply no longer being consulted (with the company’s blessing) on key decisions. They’re getting bad reviews for the exact same behavior that’s been rewarded pre-20224.
\nNone of this depends upon AI. If LLMs had not taken off this decade, we would still be seeing the same cultural shifts in the industry. Companies would still be laying off engineers, and the engineers whose job has been to say no to things would still be upset and confused about why they’re now being punished for saying no.
\nIronically, if ZIRP had not ended, this would be a glorious moment for the just-say-no engineers. LLMs would have thrown fuel on the “engineers running wild” problem that the just-say-no engineers were empowered to solve. Tech companies, unable to publicly or privately cast doubt on AI-assisted coding5, would have relied heavily on these engineers to prevent the tsunami of AI code from swamping the entire company. They would have been paid even better and celebrated like kings.
\nInstead, LLMs are adding insult to injury for the just-say-no engineer. They’re forced to watch while other engineers merge AI-generated PRs that would previously have been blocked, and are told to use the tools themselves: to become the kind of engineer they’ve spent their entire careers battling against.
\nWorse still, the AI tooling mostly works. It’s not (yet) causing any kind of catastrophe6. The code isn’t quite as clean, and it’s a bit less well-understood, but it’s good enough (particularly in a world where companies are trying lots of new things and abandoning the ones that fail). So the just-say-no engineer faces not just a threat to their livelihood, but to their entire self-identity: they have to either insist that the apocalypse is right around the corner, or accept that their technical role was contingent on a really weird economic environment in the tech industry.
\nWill the just-say-no engineer go extinct? No. They don’t fit well into every single tech company anymore, but there are domains where they’re needed. In Pure and impure software engineering I drew a distinction between “pure” engineering, which has a well-scoped, largely technical goal (like building a compiler or a language runtime) and “impure engineering”, which has a poorly-scoped, largely customer-driven goal (like trying out a new feature you’re not sure will work). During the ZIRP era, tech companies did a lot more pure work (for instance, building React), and tended to treat even impure work like pure work. The just-say-no engineer is great for pure work, because pure codebases have to have a much higher bar for quality and can tolerate slower development cycles.
\nMost tech companies are still doing some kind of pure work, typically in their core infrastructure pieces. This is essential work, but it doesn’t require a huge engineering team, and it’s rarely in the spotlight. If you’re a just-say-no engineer and you want to stay that way, I would recommend trying to move into one of these roles (and accepting that you’ll have a more limited scope than you did in the 2010s).
\nThis was a critical role during ZIRP, because:
\nPart of the appeal here is the lure of the guru. In kung fu films, those who know martial arts perform furious acrobatics, but the true expert barely needs to move at all. For the same reasons, it sounds profound to say something like “junior engineers produce tons of code, seniors very little, and staff engineers remove code”. Of course this is false. Staff engineers are expected to be able to produce a lot of working code very quickly, when they need to.
\n↩\nI wrote about this a lot more in The good times in tech are over.
\n↩\nNot necessarily make a profit, but at least bring in revenue.
\n↩\nOr pre-2023, or even pre-2024 or 2025. Cultural change lags behind economic incentives, sometimes by several years.
\n↩\nFor fear of killing the vibe (and thus the stock price).
\n↩\nIf you think there have been more incidents recently, consider that (a) you might be wrong, or (b) that other end-of-ZIRP factors (like increased velocity or layoffs) might be primarily responsible.
\n↩\n