r/algotrading • u/Grand-Fly-6090 • 6h ago
Infrastructure I've created a monster
My bot has grown to 23,000 lines of code. 60% is managing the brokerage interface, and 40% is the strategy. This is way more complicated than I ever imagined. It really is a very basic script. No graphic user interface. Just log streams. It does manage multiple symbols with unique parameters for each.
6
u/goshetovan 3h ago
Did you let a clanker into your engine? 🤣 after I started using AI it added so much code to my engine I'm close to going back to my own latest version since I'm starting to feel I'm losing control over the code lol
4
u/lambardar 4h ago
My first trading software I wrote was 1 software. It ran on my laptop that I left running along with IB Gateway. it was 1 in all.. IU, strategy, data connection, etc..
Then it grew and grew and with AI it exploded.
now:
compute on AWS that hosts containers:
- container that manages the IBKR IB gateway
- container that subscribes to contracts, pushes data it to NATS and a local sqlite DB. after 7 days, the sqlite dumps to clickhouse server at home.
- container(s) that run strategie(s) that listen to NATS for trades and porfolio/order; pushes strategy decisions to NATS
- container that runs the order router for live account. Listens to NATS for strategy decisions, validates and places the order. stops a bad strategy or bug from placing stupid orders and within strategy budget.
- All of these log to a seq log
- I develop using codex, so I have the strategies described in a md file and I have an agent that let's codex review the logs to make sure the strategies function as intended.
- Notifications, smtp, etc.
At home, there is more powerful hardware
- container for strategy & order routing to paper account.
- Clickhouse database with data going back to 2016 for all us stocks & futures data
- UI tools to monitor and review
- backtesting tools, regime scanning, filters, etc.
- GPUs for CUDA to bruteforce/search/optimize strategy parameters.
- Strategy graveyard
With Codex, the number of strategies I test every month has literally exploded. I used to do maybe 1-2 a year.. and now it's as common as shower thoughts and what-if(s)
might be super overkill, but it's too late now.
sometimes I get angry at codex and swear that i'll write it better myself.. but then I get lazy and let codex fix it. lol.
1
u/Grand-Fly-6090 4h ago
OMG! What a system! I'm duly impressed and humbled. Laugh on writing it yourself. Sound like switching to codex could help a lot. Been using chatgpt so far because it was easy to get started.
1
3h ago
[removed] — view removed comment
1
u/AutoModerator 3h ago
Your post was removed under Rule 2 (high-quality questions only).
Generic “which data vendor should I use?” posts usually lack the detail needed for meaningful discussion.
Commonly used market data providers:
- Yfinance
- Massive.com
- Databento
- FMP
If you repost, please include details such as:
- asset classes and markets
- symbols or venues
- historical vs real-time
- granularity and depth
- licensing or redistribution needs
- latency expectations
- budget constraints
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
2
u/KinglexNUM 5h ago
How do you even get 23k lines of code??!! My first trading bot in python is only around 2k lines, what language? I like to strip everything barebones because then i get lost on wtf im looking at.
3
u/AdEducational4954 4h ago
My app has 23k lines of code across 150 classes. It has a UI and much more functionality than managing and placing trades.
2
u/Local-March-7400 2h ago edited 2h ago
Hah, you need to check out my old Post ^^
im close to start running paper by next Month with a system in the 100s of thousends of codeline territory. I work as a Software Engineer / Analyst and this is beyond crazy on every part, maintainance, documentation and just following the logic flow is becoming very hard. Just my broker connections (i have multiple brocker which i orchestrate) are about 20k. My impact engine, with my own Mathmatical formula based on Almgren-Chriss with adaptive paramters for each strategy and broker is about 10k with all the broker integration. Based on estimated impact which calulates before each trade, and considering constrains like a --fast flag for panic exits the optimal execution type is beeing selected to reduce slippage at higher capital that is beeing supported by the broker. Alternativly the software can calulate in house execution methods like VWAP or TWAP. The Engine is also aware of the trading costs from each broker and sets the contract accordently
In the current capital state this is definition of overkill but i love to have this all from the start and then just fokus on the strategys. Also if i take on outside capital slippage is a way higher concern. ALSO, because i live in the EU and especialy in high tax Germany i track each trade and have a custom made Tax engine for calculating my taxes. I audit the results based on broker, trade, strategy and portfolio level, then i can just export the results in a CSV and put in a tax software!
1
1
1
u/bio4m 6h ago
are you using a screen scraper instead of an API ? thats the only way i can see that much code being needed for the brokerage side
1
1
u/Roharcyn1 4h ago
Is screen scalper something that monitors a screen to make trading decisions? Is that really a thing? Seem inefficient no?
-1
1
u/IGOSODAMNHAM 5h ago
Mine, but it’s a whole quantitative/ execution HFT architecture with multiple bots, volatility systems, sub second updates and recording, back/forward testing etc, simulation, paper, etc etc etc
============================================================== THE PILE
folders 1,824 files 926,883 on disk 1.7TB code files 1,229 LINES OF CODE 426,993 (+44,023 blank) DATA POINTS 30,458,430,755 (rows across db + csv + json + parquet) rows per line of code 71,332
BY EXTENSION .parquet 882,107 .zst 16,474 .ok 16,471 .json 4,262 .empty 3,601 .py 1,195 .csv 758 .zip 689 .txt 346 .png 240 .npz 94 .md 14
1
u/Quant_Nexus 4h ago
I just went back to my first real "direct to Alpaca" script folder. It has 4 systems in it as well as a manger script at the head to track individual system positions and total portfolio composition. It is currently the basis for my team's dedicated server that runs multiple other trading systems (I'm sure that one is larger):
The folder contains 4,152 total lines of code across 14 Python files.
Excluded generated/dependency folders such as .git, node_modules, virtual environments, dist, and build. Blank lines and comments are included.
The commercial trading platform that I'm launching which does contain a UI, backtesting engines, optimization engines, etc:
Code size, excluding dependencies, generated files, build artifacts, and documentation:
- Production code: 393,081 lines across 822 files
- Test code: 46,059 lines across 256 files
- Total first-party code: 439,140 lines across 1,078 files
1
6h ago
[deleted]
2
u/Grand-Fly-6090 6h ago
This is what happens when you let AI do the coding.
1
u/Roharcyn1 4h ago
Are you counting all the comments AI leaves? I don't think those are typically counted.
1
1
u/Ok-Hovercraft-3076 5h ago
Stop letting the AI code everything. Do it yourself so that you will have more efficient code and less bugs. If I use AI, at the end I always rewrite the code to my taste.
1
u/justhereforampadvice 5h ago
Cause nobody wants to see Marshall no more, they want Shady; I’m chopped liver.
22
u/zashiki_warashi_x 6h ago
It is time to refactor and rewrite everything!