Note: If you don't like reading words, my feelings here are pretty accurately summed up by this absolute earworm of surprisingly high-quality AI slop.

You'll find countless internet commentators breathlessly declaring that nobody needs to read the code anymore. You'll find a similar number of folks arguing the exact opposite -- that untold atrocities will befall those who don't understand their code. Well, which is it?

I think it's too early to tell, and that the field of software engineering is undertaking the largest experiment in its history. The question it will answer is something like:

Can software systems continue to function long-term even when nobody knows how the code works?

We'll likely have something resembling an answer within the next year or two.

The Limits of LLMs

I think the usage of LLMs for programming by non-technical folk is an interesting case study, a sort of microcosm of what might happen to software engineering.

I have a bunch of highly creative and intelligent friends in business/product/etc that can now, for the first time, do the kinds of magic stuff that was formerly the exclusive domain of programmers^[1] -- things like making computer art, or prototyping an idea, or building a personal tool. I think there are likely to be positive knock-on effects to this kind of "democratization" of software.

But one repeating pattern I've seen is that, beyond a certain point, these projects tend to crumple in on themselves under some invisible weight. Prompting sessions start to take longer and cost more. Adding one new feature breaks two other unrelated ones. This "invisible weight" is, of course, tech debt. And it has the effect of pouring cold water over the entire endeavor, because what recourse does a non-programmer have at that point? Wait for stronger models to come out?

Newer and more powerful models may push that implosion point out, but probably never out to infinity.

The Slop Spectrum

I think the same thing is playing out with professional software engineering, on a longer, more drawn-out timeline. There's a spectrum of LLM usage: there are surely conservative enclaves either using none whatsoever, or using AI for small, well-scoped changes^[2] and otherwise doing the "normal" software dev lifecycle (including "normal" human code review). And on the other end, you have orgs where nearly 100% of code is AI-written and AI-reviewed, and perhaps no human has looked at the majority of the code.

I think this spectrum more or less gives us our control group, as well as cohorts getting various doses of the drug under investigation. Experimental success looks like nothing particularly exciting: businesses continue to function as usual, likely with much faster software development lifecycles.

Experimental failure is a lot harder to define, as it could take a bunch of different shapes:

• Slower development velocity
• Skyrocketing token costs
• Frequent production outages
• Increased security breaches

It'll likely look less like a sudden, unrecoverable catastrophic failure, and more like a death by a thousand cuts.^[3]

The Problems with LLMs (when you don't read the code)

TL;DR - you need to read the code to see if you gave the LLM enough information to correctly implement what you actually want.

You need go no further than the front page of Hacker News on any given day to get the sense that The Programmers Are Not Alright. Many of those posts cover the ennui experienced by developers who enjoyed the coding aspect of professional software dev.^[4] I'm going to ignore that whole angle, because businesses have never hired software engineers to write code, they hired them to solve problems and provide business value, whatever that may mean. Code was always incidental at best, and a liability at worst average.

The problems I want to talk about are the practical problems of using LLMs, even infinitely smart ones. The problems that plague my non-technical LLM-using friends, and I expect will plague SWEs over a long enough time horizon.

Problem #1: Literally Literal

I think the easiest way to describe it is that these models are "instruction-tuned", which tends to make them very literal in their interpretation of user prompts. By default, if you ask them to do <X>, they're going to do <X>, hell or high water. They aren't going to say "well, the current abstractions don't really allow for <X>, what if we refactored the data model to […]." They're going to find a ~sensible place to shove <X> into the codebase, and do it.

This isn't to say we can't just use LLMs differently. I'm sure even today that they're perfectly capable of recognizing when a refactor would be better for overall longevity + extensibility, but it'd require changing our prompts, our tooling, our attitudes, and our expectations of velocity.

And deeper than that, it'd require us to have a functioning mental model of how the code is structured, which is the first thing we lose when we aren't writing it and are barely (if at all) reviewing it. Again, even with a perfect LLM, you'd still need to decide if the refactor makes sense, because the LLM doesn't know how you expect the codebase to evolve over time.

…which conveniently segues us into problem number two.

Problem #2: Lacking Context

Not in the sense of a "context window", more in the sense of the context in which a change is being made.

Unless you explicitly tell an LLM specifics about your project + situation, it doesn't know them. It doesn't know about how urgent a task is, or if this codebase is a throwaway prototype, or that you intend to switch from MySQL to Postgres next month or that another service elsewhere depends on you emitting this one event in this really specific way.

Sometimes, egregious hacks are the most sensible approach. Other times, they'll cost you $10 million in additional AWS costs or reduced dev velocity because of extra cognitive overhead.^[5] An LLM can't evaluate those tradeoffs unless you tell them.

But a lot of times, these tradeoffs are things we've internalized to the point of not even realizing they're tradeoffs. The only way to find out that we've made them is, you guessed it, by reading the LLM output as a sanity check that it understood your intention.

Putting It All Together

The main thing that causes long-term instability isn't just that these issues exist, it's that they compound (exponentially?) over time. Without reading the code, you don't know how an LLM interpreted your request. It may have made some totally reasonable assumptions that don't apply to your situation at all. And future sessions will see that code, similarly lacking your context, and build on top of it, and so on. Small initial errors snowballing into larger errors down the line.

The piece I struggle with here is to what extent this is really an issue. Large codebases have always been Lovecraftian^[6] horrors, and most of the time it isn't actually an existential threat to the org running it in production. They might have to declare a "code red" or push out some roadmap or pay fines for some breach or sacrifice fire some token employees, but it usually isn't game over.

So why should this time be any different? In my mind the difference is the scale at which it's happening and the loss of institutional knowledge required to course-correct for these mistakes.

Pins and Needles Fans

A recent (small) example from my own experience: I've been building a personal hardware project,^[7] and was working with LLMs to design the hardware for the control board.^[8] I told it I was planning on using a particular 3-pin Noctua fan for cooling. It dutifully designed a variable-voltage power stage that converts PWM outputs from the microcontroller into a supply voltage for the fan. This required a whole host of random components,^[9] probably 2–3x what I actually needed.

I started the next session with:

In fw/design.md, we specced out a design for the firmware + hardware for the control board of our lamp. I'm realizing that 4-pin fans (i.e. with built-in PWM control) are the same price as 3-pin fans. Can we simplify our circuit if we use 4-pin fans?

The datasheet for the 4-pin fan is available in […]

And sure enough, it removed all those extra components and made a few minor tweaks to the firmware, saving me a bunch of complexity (in the fabricated board), money (for the extra parts), and time (I'm very bad at soldering).

If I hadn't read the LLM's output at all, I wouldn't have realized my mistake -- mainly that I didn't let it know I was flexible on the choice of fan components, and that 4-pin fans cost the same as 3-pin fans. It would have continued to build on top of invalid assumptions, to the detriment of the final result.

The Race Is On

To me, this whole thing feels like a race between how quickly a team can slop up a codebase, and how quickly the models and (perhaps more importantly) the tooling around them improve.

Trying to do 100% of your software development with GPT-3.5 (2022) would likely have been a short-lived disaster. Trying to do all your development with frontier^[10] models in 2026 might just be totally fine?

Right now, we're all still figuring out how to work with these tools,^[11] which is complicated by the fact that the tools themselves keep changing and improving. I think it'll be a hot minute (if ever) before we have anything we can confidently call "best practices", but that's part of what makes this experiment so fun to watch.