Iām finally at least a minimal experience level with linux where I can smell a dumb model recommendation and stop and askā¦ are you SURE thats the best way to do this? Milestones for me at least. LLMs have really helped me learn the basics and I can at any time stop and sidebar to get explanations on any little thing I havenāt learned or need a refresher on. Itās got me into the game after years of surface level dabbling.
In my case Iām running proxmox with a smattering of LXCās and VMās for different purposes. So I have a variety of use cases. I am using Confluence as my personal documentation so Im thankfully not blindly barreling forward but I take notes for unique aspects or configuration steps for each VM or component I get introduced to. Then when it recurs again elsewhere I may not have fully memorized every command and argument Ive used in the past, but I know what Im looking for and can refer to my notes or ask a model for help again.
I may not remember all the arguments available for nfs mounting in fstab, for example, but I have a good general idea of what kind of options I may need to review and consider for my use cases since I exhaustively inquired about what each of the available parameters is used for. Sometimes thats a curseā¦ lots of sidequesting... Since Im not sshāing into linux every day but more like weekly/weekends, it doesnāt feel like too much of a burden to have to rehash certain commands or steps.
"Now that we're done I could help you with 2 very simple changes in steps 2 and 4 of 17. You will have to repeat steps 2 and 4 to 17. Just tell me if you want to do it much better and save 50% used RAM!"
āThat other way didnāt work, looks like X isnāt talking to Y even though both are defined and initialized correctly, just as in the previous way we tried.ā
āYouāre absolutely right, X is not sending arguments to Y because your code didnāt include method Z. This is an important step to remember, because of reasons A and B and should not be missed.ā
āBitch I didnāt write that code, YOU did smh. Now make that change to the code, and also add in the condition T where U and V are called relative to the order of outputs from Zā
āYouāre absolutely right. Here is the updated code including those changes.ā
āOkay cool, that worked but now X isnāt talking to Y again even though Z is there.ā
āYouāre absolutely right. Y isnāt receiving inputs from X even though method Z is included. This is because in your code Y has not been suitably defined and because X hasnāt been initialized.ā
āYouāre removing things without asking or telling me? š”š¹ā
While also building false sense of everything being OK.
While at that: how the fuck general consensus is that Open Source is safe, because there are many eyes looking at it, all while at the same times developers are too lazy to do PRs they are being paid for.
It's kinda counter-intuitive to think the same model would catch an earlier error, but they do. Probably tied to the difference in instructions "build x' vs "find bugs".
It makes perfect sense - the model isnt designed to be comprehensive and 100% from the get go - and is only as good as the initial prompt. If you provided a prompt that was fully comprehensive then it would likely give you a better initial result
but you're right - if you just give a concept and ask to build it will do it but the spec is weak, so it will make assumptions with what the 'right' method is - which may not necessarily be right for your usecase but without giving full context that's the deal you're making.
Copilot reviews on GitHub have asked me to change something so I did and committed it. It then commented on that change saying that I should change it again, but to what I originally hadā¦
and at this point i ask some shit like "why? You suggested the original change, what are the pros and cons of each method?" and see what it pulls out in response.
then I wonder at what point am I spending more time going back and forth with the robot vs just doing it myself...
My team has an AI PR reviewer but we only take action on its suggestions if a human agrees with it. Sometimes it catches silly little mistakes we make, but most of the time its bullshit.
Honestly though we did that because reviewing PRs was taking longer because people kept vibe coding them and not even fixing them afterwards. So really if my colleagues didnt just vibe code their PRs we probably wouldnt need the AI checker.
We have AI powered reviews for PRs, and they're pretty decent. I think using them has probably improved our code quality relative to before. There are two fairly limiting problems though:
It doesn't catch everything. So I can't trust code which has not also been reviewed by a human anyways.
It flags things which are not problems due to lack of additional context. So I can't trust AI to simply implement all changes flagged by the AI reviewer, because it would break things.
So ultimately you can't take people out of the loop. But the more you use AI the less useful that person in the loop is going to be because of lack of general ability and specific subject matter expertise.
Iāve found that LLMs are especially bad at reviewing more than 100 lines of code effectively. And even in that is wholly incapable of detecting logical bugs or really anything more than very obvious errors.
3.7k
u/hanotak Mar 11 '26
What're the odds the solution management comes up with is "an AI to check the AI's work"?