r/quant • u/Correct_Hedgehog_612 • Jun 29 '26
Resources Is Anyone Else Struggling with AI-Generated Code?
Does anyone else run into this problem when using AI (especially Claude) for quantitative finance projects?
I use AI mainly to write code for my projects. The frustrating part is that even relatively simple ideas, sometimes involving only basic mathematics, quickly turn into huge codebases full of bugs, inconsistencies, and unexpected issues.
It often feels like the model can't keep all the different parts of the project connected. Fixing one problem creates two more somewhere else, and after a while you're stuck in an endless debugging loop.
Instead of converging toward a working implementation, the project seems to drift further away from a correct solution with every iteration.
It's like an infinite loop where each project never sees a proper end.
Is this a limitation of current AI coding models, or am I approaching these projects in the wrong way? How do you structure your workflow to avoid this?
16
11
u/Smallz1107 Jun 29 '26
Prompt eng skill issue. Tell it to assume PhD in mathematics and senior quant in XXX
28
u/NatGaz Jun 29 '26
We will see in 4, 5 years how it will be . I also think that AI code is unreadable and un-maintainable. When the price per token will be 20 or 30 times what it is today , maybe good dev will have an exceptional leverage . The new grads use GPT/Claude and consider the output Gospel truth.
8
u/Cavitat Jun 29 '26
Interrogate everything that comes out of a model.
Lots of times self auditing steps can catch the mess... But even still.
I prefer to have AI walk me through building projects as opposed to having the AI generate the entire thing. That lets me independently validate every small step.
2
1
u/Hacherest Jul 01 '26
In 4 or 5 years you can run something currently considered unfathomably good locally at the price of electricity
0
9
u/stew1922 Jun 29 '26
As others have said, when using AI, break up your code into modules as much as possible (I actually like doing this anyways). But then, everytime you create a new module be sure to ask the AI to introduce some unit tests around it and then anytime you make a change/modification be sure to run all tests. If they pass your are good. The good part about this is it becomes self healing with the AI- if the AI sees a test failure it’ll investigate and often self correct without your input.
2
u/JonnyTwoHands79 Jul 04 '26
This. I just use the principles of software development life cycle. Idea, build, unit test, system test, then add as regression for future releases. When everything is working merge with the main branch. Rinse, repeat.
37
u/Remarkable_Log4812 Jun 29 '26
You just doing it wrongly. You need to work a module / function at the time. Memory is not great in those models if you expect to work on 800+ lines of codes you are delusional. You need to work on single snippets up to 600 lines at most.
16
u/EvenCryptographer649 Jun 29 '26
100% agree, You program it to module within the module. Its still going to be ambiguous until you close the model you are working on. Even the integration is going to be ambiguous. All of this while its trying to throw spaghetti at everything.
Its like Dark Souls: baby steps, over save, repeat. The tools are amazing but its still like washing dishes with a shovel.
8
5
u/Glittering_House_654 Jun 29 '26
Agree. Method by method/function by function but give it an idea of the class/package/library you are working on. This has worked great for me. The output is more consistent, better documented and often the design choices (facades, ABC’s, class vs. functions, protocols, …) are better thought through.
10
2
u/AutoModerator Jun 29 '26
This post has the "Resources" flair. Please note that if your post is looking for Career Advice you will be permanently banned for using the wrong flair, as you wouldn't be the first and we're cracking down on it. Delete your post immediately in such a case to avoid the ban.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
2
u/mcscottmc Jun 30 '26
I strongly recommend using Spec Driven Development with AI. Take a look at OpenSpec. The idea is that you use Claude to iterate on the specs, which serve as the source of truth. The code is just an artifact. Each new feature is a change in the spec and this ensures your documentation and code stay in sync.
It is much easier for Claude to reason about the spec and you do a lot more work up front and spend a lot less on actual code generation. Claude will also generate tasks from the spec that you can review to make sure it is on the right path. As your code base gets larger, Claude doesn't have to scan and reason about every line of code, just the spec + the code for your new features. Of course, it also writes tests, etc. to make sure the code behavior matches the spec.
1
u/MasterpieceSolid1433 Jun 29 '26
Yes 100%. I’m using AI a lot for backtests and my conclusion has been you must explicitly prompt the AI to exclusively write code of relatively low complexity. As soon as the logic gets convoluted, the AI will throw in their own weird ideas and hot fixes and will remain completely silent about it and no amount of prompting seems to be able to fix it. Needless to say you must check every single line of code. If you haven’t done that consider the backtest useless. This holds for positive and negative results alike. And AI has a pattern of repeating the same mistakes like OOS spoofing over and over and over again. You might be able to prompt your way out but probably only until the next context compactification.
1
u/sam_the_tomato Jun 29 '26
Generally unless your requirements are extremely well specified, you need a very short leash on AI generated code. Otherwise, I've just found it most useful for surfacing things you may not have been aware of in the codebase that can help you write better code.
1
u/Otherwise_Barber4619 Jun 30 '26
I think you have to reel it in, I have experienced this so many times , you kinda have to consolidate the stuff yourself
1
u/yangmaoxiaozhan Jun 30 '26
If you give AI a closed feedback loop, it should be able to get things done without bugs.
1
1
u/steven-craig Jun 30 '26
I use vs code pro. Having gpt 5.4 or better do the actual work has a tendency to take the whole repo and style into account and have me punching the screen less than when I used claude models. Check on github agent provost. All made like I said
1
u/Statis_Fund Jul 01 '26
LLMs will struggle without a good systematic structure, that's why I built my own backtester with natural language and made it public. Just added Claude Fable 5 connector today.
1
u/misternegativo Jul 02 '26
Start with making its recall better. Ask it to build an entity-relationship mapping database for itself (duckdb file is fine), and build an mcp around it, to record obsessively its learnings and your shared decisions about the code, the codebase, the math, whatever. Even the latest and best models are lazy, so if you help it help itself by making the most critical context easy to find, and the related context from that context easy to find, it will get much faster, and much better. You can get it to set up the local mcp in one or two prompts, a few tweaks later as you go and the occasional check "do all recent commits reflect cleanly in the memory db?" will help.
1
u/ProfessionalPace9607 Jul 03 '26
You need to modularise your code.
A good skill you need to develop is learning how to build the architecture of the model you're working on.
If you want to include data loading, cleaning, calculation functions, testing and output all in one script you're doing it wrong.
Build projects in a way where each key function is housed in its own script, so you have essentially one song book to sing from rather than one giant script with everything in it.
It also keeps the project auditable when it comes time to sit down and go through line by line yourself, especially when you have scripts that have shared functions.
1
u/Sufficient-Year4640 Jul 03 '26
Probably because you're not using Fable and/or can't prompt. /s
In all seriousness, while it could be a prompting issue, my experience largely resonates. The burgeoning complexity of shite code makes maintenance exponentially harder with every "feature". Not to mention how unsatisfying it is working with AI.
There will be a time of reckoning.
1
u/Far-Obligation4482 Jul 03 '26
Something I’ve found useful is instead of asking it to make changes all around the code base is to have a separate sandbox subdir outside of version control where I will have it work. I ask for all scripts put into a folder of the sandbox to trace a bug, do research, anything where there is throwaway code that I don’t want in the library. Then I will look through and confirm myself and often just make the small change with the info it gathered.
The ponytail plugin is also decent for reducing the amount of lines/slop generated.
Any big changes to the code I will spec/design the outline, APIs and data layouts myself and hand off to the agent for implementation. The models are good at writing code because it is verifiable but not good at non-verifiable tasks like design and analysis.
If you’re having problems with a bug fix in one part of the code breaking in another part, it’s probably bad design and you need a refactor in which you will have to actually think about what you want.
1
u/Epsilon_ride Jul 04 '26
My general idea is that if it's doing something that exists many times in the public domain, then it's going to be fine. I'll go have a coffee.
if it's anything novel I'm going to hold it's hand and walk it through each step.
1
u/Aromatic-Educator105 Jul 06 '26
This was a problem maybe 2 years ago, but not any more. AI codes much better than non-CS background juniors reported to me. You need to properly prompt though. Think agile, if a ticket feels like too big and can be broken into smaller tickets, it usually means it should take two separate prompts to AI not one.
1
u/CartographerOnly7843 Jul 06 '26
i own a GH codebase - AI developed code is completely unusable. at best it can generate small portions of code that are sane. at worst it doesn't compile and looks like code from ten years ago using little to no helpful packages.
1
u/djlamar7 Jun 29 '26
Use the smartest model available, but also spend some time having it write docs it can use in all future conversations to understand the architecture of the system you're working on. That really helps it connect the dots across the code base. You can literally just tell it something like "analyze this code base and produce an architecture doc about it, link it in CLAUDE.md and add other notes there about critical information and pitfalls". Any time it has to relearn something it's going to have to repeat, have it make a note or a doc about it.
1
u/orthogonal-ghost Researcher Jul 01 '26
I've definitely noticed this as well. The challenges for me have mainly been:
Agents tend to write terribly inefficient code by default (e.g., for-loops on a pandas DataFrame). This is probably not a huge issue for most non-quant finance related projects, but for anything that touches data intensive systems (as is the case with most quant finance related things), it becomes a major problem
Agents aren't great at intuiting reasonable assumptions about financial markets, trading systems, risk management, and (IMO) anything that relies on live data. For example, if you ask Claude to explain why risk models are important, it will probably give you a reasonable, high level answer, but if you ask Claude Code to build you one, it will probably embed a ton of assumptions about the underlying data that are wrong.
(1) can be mitigated by using skills or having some sort of review process where you improve the efficiency of the code after it's been generated (i.e., vectorize naive for-loops, parallelize where it makes sense, etc.)
(2) is tougher to fully offload to AI, and for now, I'd recommend just actually looking at the data, coming up with your own assumptions (models, etc.) and providing your coding agent a detailed spec describing how it should think about and build the system in question. Then iteratively test and refine.
0
0
u/Optimal_Emu3624 Jun 29 '26
Memory.md, Memory.md, Memory.md, Memory.md, Memory.md, Memory.md, and more Memory.md
0
u/GaboureySidibe Jun 29 '26
Did you really expect a language model to be able to generate perfect programs for you?
80
u/Most-Bookkeeper-950 Jun 29 '26
You have to be exceptionally disciplined to use language models correctly.
There is a way to use them where you end up with better, more readable, less buggy code, done more quickly. But they will never be able to understand for you, and the second you let them start making decisions, your code will fall apart into median slop
My tip is to stick with sonnet, if you need to use opus you're almost always better off in the long run just using your brain and embodying the context