Posted by rzk 13 hours ago
Ask HN: How to deal with shipping AI-written code you don't understand?
https://www.linkedin.com/posts/ali-hamie_im-an-engineer-at-m...
> I'm an engineer at Meta and I never felt this uncomfortable doing my job. I'm writing code these days and it all feels like a black box. This new standard of shipping with AI is moving so fast that if we're going to spend our time checking each line of code, it's going to take forever. That means you can end up with systems you've built where you don't truly understand how all the code works. You understand the high level, but not every detail. And maybe this is becoming the new standard. It goes against every instinct of being a software engineer. I'm used to understanding what I'm building. Now I'm trying to figure out how much I need to understand before I'm comfortable moving forward. Are other engineers feeling this too?
How are you handling this?
To me, "It works" is pretty much the lowest possible quality bar for an acceptable creative work product. We should not aspire to "it works." We should not hold ourselves only to the "it works" standard. And, yes, I know that many traditional software companies fail to even meet "it works" for their shipping products.
I don't even know what to say anymore. This has become a deeply unserious industry putting out mountains of junk that barely even meets the "it works" bar, and LLMs are just accelerating this.
If the agent has written tests and crucially - witnessed them fail before starting the work to make them pass - you have a lot of confidence.
Review of your own work's easier too - test and production code tend to mirror each other, giving you a much more complete picture. Confused by the production code? Go read its tests for a bit, then come back.
It's also easier to understand the change if you were there for the build, watching the transcript, checking in every so often. Doesn't have to be your primary task - leave it on a second monitor, it'll catch your eye naturally.
And finally, on top of test-driven development, have the agent first (before the first test driven development cycle) build out an end-to-end test covering the work it'll do. Once that test is green (test driven word for "is passing") you're done with your work and can clean up at will.
Play with it before shipping, get it real nice. It's never been easier to clean the tech debt up and make the UI perfect.
Eventually you'll be ready to package it up into a PR!
Now's you can leverage agents and get creative. Make your colleagues froth to review your work. Consider making a Loom video to walk them through it. Consider reworking it into a commit series; it only takes a few second for your agent and you can tell a story for humans. Maybe you want to sequence it according to your struggles; maybe it's better to show the different layers emerge with each commit - it's up to you. Consider also that your agent can easily do one-off visualizations of the change for your colleagues if you think it'll help them, and you can attach those too - mermaid charts or even whimsical AI generated images. All this is because motivation is driven entirely by other humans' motivation and you want them frothing with excitement every time you drop a review. Make it fun, it takes seconds and gets your stuff merged fast while making your teammates smile.
This is a brand new job and we will work out how to do it together :)
There have been multiple times where my software dev team has gone back to prior implemented tests they had the model generate and found that after some time eventually the model got frustrated with the test no longer passing eventually modifying it to fit current needs around a feature push. You can probably anticipate how we came to discover this, as it ended up introducing a regression around multiple edge cases which the test was meant to control for.
The only answer is to inject actual engineering into the process as much as possible without giving it all up to the models to generate more slop in response to slop. tests are a great answer, but only if you treat them with serious intentionality.
There are multiple places in my post where I explicitly describe the process by which you intimately:
- make the agent do verified work (the e2e tests and test driven thing)
- watch the agent work (the second monitor thing)
- read your own code (the tests/prod mirror thing
- play with the work and perfect it (the tech debt and UI perfect thing)
Plus a ton of other places in my post I very clearly describe being intimately involved in building and selling the work.
That to me is basic god damn competency. What your engineers did is a fad, and it won't last, and I'm sorry they abdicated their professional duty.
If you wish to screen for that sort of shit in future/current hires, show them my post and see if they turn up their noses or if they get excited and creative. You want only the latter.
I then make sure they write a lot of tests and that if functionality is changed or added/removed then tests are updated alongside.
Generally I try to review changes myself and ask it to explain the changes.
It's not perfect but I feel like the planning step being iterative isn't all too different from my old research -> think about implementation -> implement loop.
At that point, reviewing any code changes goes hand in hand with how I coded before LLMs.
The major difference is I usually have it make any changes for things I don't like.
Just kidding, it's de same as with any other Open source plataform You install from their repo. It will be hard at the start but then You Will be very aware where it starts to fail and how to fix it.
It also took the industry quite a while to accept that server code needs to be secure even if that requires a lot of extra effort. AI will probably be the same.
What I do is ask Opus/Gemini to explain the parts I don't understand. And I do some of my own research to verify accuracy of the explanations.
In addition:
* Every new feature has a Feature Design document which I review, modify, and approve. And a corresponding Implementation Plan which I also review and approve.
* Every fix has a RCA document and fix Implementation Doc which I review, modify, and approve.
* There's a global high level Architecture and Design document with additional sub component design documents which are always kept in synch after a new feature or bug fix is done. Synching it is always a step in the Implementation plan.
This does adds some overhead but it allows me to have have check point gates. This has often allowed me to catch misunderstandings and align properly before any code gets edited by the AI agent.
Having the above docs also helps when it comes to referencing them as context to properly ground new features or bug fixes.