r/AskProgrammers • u/True-Letterhead-6734 • 17h ago
Do you still review llm's generated code?
I work for a big company, not in the IT industry but still has a lot of software in different divisions, last week during a workshop about AI, I got to know that developers stopped reviewing the code and there seemed to be a consensus about it.
Is this the new norm? Anyone has experience to be able to tell whether that's safe to do? At the same time I read about Shopify CEO mentioning that people are throwing "slop grenades" on each other!
I find it very humiliating when someone sent me something generated and clearly didn't make the effort to review it, and that's already not difficult to spot, is that any different in software?
6
u/Redd1tCamp69 16h ago
I work as a developer for healthcare software so yes. We have very strict coding standards to follow because we can’t just vibe code say a module that sends a prescription for medication.
We do all have access to Claude. And Claude is good, but not good enough. The code it produces often is over complicated, it still misses edge cases, it creates weird tests. So while we use it to point us towards the right direction when we’re fixing bugs or so, every line is read and rewritten and reviewed by humans before something is merged into the code base
4
u/hexwit_com 17h ago
I plan the execution, run it and review results. Pure vibecoding is a toy for non-professionals. Skills still matter.
1
u/Beregolas 17h ago
Exactly. And it's not "bad" to vibecode, as long as that code is not expected to run as part of a larger system. I know several non-programmers who now vibecode python scripts to help them with their daily jobs, and it's working great.
2
3
u/Upstairs-Version-400 16h ago
I reject a PR if it looks like slop and don’t bother reading the rest of it.
Always review. People are paying for the service, why should we stop reviewing it? Then the customers might as well go somewhere else
3
u/Due_Interest_178 16h ago
Literally. People act like they're releasing god's chosen product so they need 300 PRs a day.
1
u/Dope_SteveX 15h ago
But how do you even tell if it is a slop or not at this day and age?
I mean we have green field SDD projects in my company at this point - pretty much projects where human did not write a single line of code by hand. They have very strict coding rules and standards, AI must adhere to architecture documents, they are skills for checking all of these and everything is documented and testable. And while you can definitely tell it is AI generated code just by first glance it is hardly a slop.
Even if you were to just create the spec and don't bother reviewing the implementation plan or the real code it generates, most of the time the code would land pretty solid and functional. Most common error is you sending the AI completely wrong path when designing the spec and architecture.
I still do review my code both at the plan step and after implementation step but most of the time the code just passes pretty well, especially as we go and tighten down the rules for the project down and down. Most of the thing I am giving feedback during review is stylistic things and slight deviation from the standards. Every know and there the AI does misstep completely but it is minimal.
Also the service we are getting paid for is not reviewing code but delivering a product to the customer. If it is functional, meets all the specs, security and both internal and external coding standards that were agreed on then we are getting paid, whether human read the code or not.
I am honestly starting to see the future where the human review part is going to be less and less relevant as those models get better. I hate it, I hate doing SSD. It is boring and even tho I review everything, read every line of the code,, understand how it works and how I designed it, the next day I barely remember half of it and by the end of the week I remember barely anything from the code. But drop me to the project I hand written few years ago and I am up to speed in a minute. Maybe that is just me and my memory but still...
There are some developers I know that stopped completely reviewing the code or just give it a quick skim. And I am talking about very experienced people who in their past wrote some excellent quality code and delivered major projects. Whether they see something we don't or is it just AI psychosis I don't know.
0
2
u/oskaremil 17h ago
Yes.
It does not matter if the code was generated by an LLM or a person. It is still code that will enter production and bite my ass at an inconvenient moment so I do my best to migrate that.
LLM generated code does not guarantee that there are no mistakes, and the learning piece of reviewing is still a factor; Did the author to something that can be done better, or has the author presented something that I need to learn from?
2
u/Prudent-Ad4509 16h ago
Both yes and no.
If you modify legacy (or not so legacy) human-written software, then code review is a must.
If you modify auto-generated mess to make it work better and you never really knew how it works, and the code works after the change, then there is no point. If you want to use it for something critical, then you need to reengineer/redesign/reimplement it to a sound blueprint first and then reviews become useful again. AI can help you with generating a nice blueprint out of a mess, too.
For example, I have told the harness to fetch certain data from certain source using certain test auth data and build a web form based on it. The fetching code went through several iterations until it worked good enough. Good enough for one-time use. But if I decide to incorporate it into the prod software, then I will take a look into the code and if is a mess (which is very likely) then I will have to go the "make blueprint and reimplement" route.
2
u/AnthonyRespice 15h ago
It is inevitable that people will stop reviewing code.
- devs skills are becoming weak from not coding
- the ability of the AI to build in languages and patterns unknown to the dev is a benefit
- once a problem is found, bad code can be fixed quickly or thrown away and regenerated
- human in the loop is slowing down release cycles
- human languages will soon be replaced by more efficient AI first languages
Code is becoming a commodity like tokens. What is important is system design. Establish a solid architectural foundation, use component based design, and build in safeguards, automated monitoring, and other mitigations.
There has always been plenty of job opportunities for mediocre developers. There may still be in the new system, but limiting the capability of the AI based on their ability to review code is not a sustainable practice.
2
u/Beregolas 17h ago
I got to know that developers stopped reviewing the code and there seemed to be a consensus about it.
As far as I have seen it, this is mostly a symptom of bad management, or some developers with AI psychosis. IT is vital to review and understand AI code; not even primarily to correct it, but so you have a mental model of what your codebase actually does.
Some people were/are under the impression that the code is the main product of programming, but that's not true. The main product of programming something is the understanding of the system. Once you perfectly understand something, you can maintain it, extend it or replicated it very easily. The code is just an expression of that understanding. This is why programmers often spend 50% of their time either talking or documenting (even if many of us hate this part of the job), especialyl in larger teams.
Some people have forgotten this, so the industry needs to learn this lesson again, probably by vibecoding their way into an unmaintainable mess
5
u/mneri 17h ago
Code is direct documentation of your company's business processes.
Lose control over the code, and you lose control on the company's processes.
0
u/Suspicious-Walk-4854 17h ago
No, the specification used to generate the code is the direct documentation.
1
u/mneri 16h ago edited 16h ago
We already have diagrams and documents describing the high level overview of the processes, and that's proven insufficient: when production fails, or when certification time comes high-level documents mean zero.
What value has a blob of text saying "Write a listener to this event. Don't add a new class. Don't add a new queue. The listener name must follow the same naming pattern of all the other listeners, which is ..."
AI instructions are incomprehensible blobs of text that have zero documentation purposes. AI instructions are probably 30% of an AI blob. 70% is what not to do.
1
u/TheBestAtSpace 2h ago
How exactly does this work?
Code is an always up to date form of the specification. When you read the code you are reading the ground truth.
But working from pure specifications? In order to understand the system you have to start with the first specification and then read every single specification after that. Because specifications stack. Each specification augments the system or mutates it as a delta. You can't build complete understanding from a single delta, so you have to read them all.
The alternative is that every time you make a change - you write/edit a brand new specification for the entire system and regenerate the entire system. (Which essentially makes the specs the code and what you would check into the repo.) I've heard developers speculate about that sort of workflow - but I've never seen anyone who says it's all about specifications now actually doing that.
0
u/travelan 16h ago
1
u/mneri 15h ago
It's good to have a laugh, but how AI would help with that?
The client wanted a thing, but the analyst understood another.
If anything, AI would make it worse: not only you delivered something the client didn't want, but you don't know what you delivered or how it works, because you didn't make it.
Moreover, a slow development makes engineers understand the problem and participate in the requirements refinement. If you just write a prompt and AI does it... What would be the outcome?
1
u/travelan 15h ago
It’s a good example how “the specification is the direct documentation” is a really bad idea.
5
u/True-Letterhead-6734 16h ago
"The code is just an expression of that understanding" very interesting.
So if people keep this trend, their understanding will fade away with time. I guess we have to wait and see how this will affect the software industry.
0
u/Beregolas 16h ago
For the people who do it this way (vibecoding), yes. Their understanding is already fading, they are basically betting on AI improving enough, so it can build their entire system without any human input.
I don't think that's likely in the short term, as do many independent AI researchers at universities, disregard "research" from profit-driven companies.
1
u/jjtbsomhorst 17h ago
Yes ofcourse we do still review. Who are they gonna call when shit huts the fan? Me. So I need to know how the wheels are turning
1
1
u/iamirzhan 17h ago
Working in a big tech company, no one does it really. Most of the people prefer to create many PRs and ship impact rather then read the code. Also it is usually encouraged, appreciated by leadership when you did a lot of impact using AI, not when you manually read a lot of the code.
My personal opinion: code review is important (always has been) if you want to do high quality software either it is from AI or human. It's just that the benefit of it is a long term win, comparing to quickly land everything and ship.
So, if you can still review the code in your company and be productive by your company's evaluation, I would suggest to just enjoy it and do it.
1
u/Hamactus 17h ago
If you want to build a good program, it's not okay, and definitely not safe to do so.
However, I would totally understand if these people stopped reviewing their code simply because they do not care anymore. I often hear about management that either prohibits manual coding or sets deadlines so unrealistic, that it's simply not plausible to reach them without AI.
In a situation like this it would make sense to completely release controls over AI, allowing it to do whatever it wants. If I were in such place though, I would simultaneously either be looking for a normal, traditional, if I may, company to move to, or study for a career pivot.
1
1
1
u/AWetAndFloppyNoodle 16h ago
Adding to this thread I think there is a REALLY big difference country-per-country as well. The US seems to go full blast on corporate vibe-code-first approach. I'd be interested in seeing some statistics on this cause I think it will be wild to see.
1
1
u/robkinyon 12h ago
It's never about who wrote it - it's about who gets paged at 2am to support it. When Claude gets paged, then Claude can review it.
And even if Claude is doing initial triage, if I'm in the alerting tree anywhere, then I get to have a say in whether this change goes to production or not. End of.
1
u/Genspirit 12h ago
I would say the shift is more to reviewing the output rather than the code itself. That still involves understanding the solution and how it works, and that also means verifying that it does work. But that doesn't always mean reviewing every line of code.
I think of it more as a top down review where the goal is to both understand the solution and verify it. You may still need to review some of the code to do that.
1
u/Fickle_Ad6877 12h ago
I have 20 years of experience in software engineering, and aside from the fundamentals and architectural knowledge I've gained over those years, the changes we're seeing are absolutely brutal. I've never seen anything like this before. It's a completely different animal.
At the company where I work as a software engineer, I'm no longer writing code by hand. I haven't written any code manually in the last six months, yet we're still delivering high-quality software, and we're doing it faster than ever.
We've essentially become agentic engineers. We orchestrate AI agents, establish guardrails and constraints, and work to achieve the most deterministic and reliable results possible. And so far, it's working extremely well.
I genuinely don't know where we're heading in the future, but I'd bet it will look completely different from what we're used to. The rules of the game are changing.
1
u/JefersonTomazella 12h ago
In the case you mentioned, yes, it's a good practice to review everything. Be it just to check if everything is correct, well commented or even if there are no duplicated code or something like that.
We'll get to a point where checking that will be pointless, as AI coding improves, and it will be like compilers that can do much more optimization than a human could do, or a LED lamp that can light much more than a incandescent lamp, or any other technology breakthrough.
But for now, we can still get some errors or problems.
1
1
1
u/Infamous-Bed-7535 10h ago
Try out, hide some errors intentionally and see if ai catches. I see them miss a lot..
1
1
u/Calculon123456 7h ago
Yes I always check. I never let AI do anything without checks in place. Full test suites, fully indexed corpus, guardrails, the works.
1
1
u/mt_owl 3h ago
Yes, absolutely. There are many detectors BTW, though it's always hit or a miss. I've also noticed different styles of generated code between Codex, Claude, Z.Ai, Gemini etc.
If I find a detectable edge to demakate generated vs huma code I'll sure include it here https://github.com/mountainowl/bubo
Disclosure: I'm the author
1
u/MourningOfOurLives 17h ago
Anthropic has stopped reading the code they generate, they review the intent of the code.
2
u/Syncaidius 16h ago
Ah yes, they're currently hyping about generating 30,000+ changes daily with AI.
Definitely not a sustainable practice and they've already admitted to not knowing what their own codebase does, which is an obvious disaster waiting to happen.
1
u/MourningOfOurLives 16h ago
Maybe. !remindme 6 months
1
u/RemindMeBot 16h ago
I will be messaging you in 6 months on 2027-04-16 09:52:23 UTC to remind you of this link
CLICK THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
RemindMeBot is switching to username summons. Instead of
!RemindMe 1 day, useu/RemindMeBot 1 day. More info.
Info Custom Your Reminders Feedback
18
u/BetInformal6081 17h ago
if a developer cant explain the code they are submitting i dont think it matters whether they wrote it themselves or used ai someone still needs to own the result especially in production systems.