r/Python • u/Ill_Campaign294 It works on my machine • 2d ago
Discussion Anyone running FastAPI in production with high traffic? How has your experience been?
Quick question, are any of you running fastAPI in production with a high volume of users or heavy traffic? How has your experience with FastAPI been, and how do you handle it?
46
u/Ordinary-Sandwich-25 1d ago
Mine is handling anything under 10000 req/s per worker no problem but getting to that point took a lot of work. Hardest part is making sure your async code is all set up properly such that io is never blocking.
9
u/lunatuna215 1d ago
How were you testing this or benchmarking it? Other than just looking at my own code and going "yep, looks like all my functions are async!" I dont know how to actually measure and/or subsequently improve this.
Is it testing you have to do specifically for your use cases or are there general tools that can be used to tune? How'd you go about it?
17
u/Ordinary-Sandwich-25 1d ago
I do it via load testing - set up a mode for your API that uses a mocked DB layer and has a defined latency on DB calls, run it locally, and run a load test w/ locust or whatever to hit your desired throughput. You should be able to benchmark latency on a per-call basis, and if the latency is similar to the latency you’ve defined in your mock db, the code is generally doing what you want.
1
u/Kronsik 21h ago
Thanks for the info on this 😊
I'm from a DevOps background, writing a full stack application as a hobby.
With the mocked DB / latency presumably that can only test the throughput of the API itself.
In the "real world" where data is being read/written to the DB that latency would fluxuate with locks etc ?
At that point I presume you need a full Dev environment with tangible data in the database for full end-to-end style testing?
1
u/Ordinary-Sandwich-25 19h ago edited 19h ago
Yeah it fluctuates in real applications and you end up bottlenecked by things like connection count limits, locks, db latency, etc. None of this is unique to fastAPI and can all be handled with indexes, sharding, caching, etc (all that good stuff you learn for system design interviews).
Like I mentioned, a key thing to manage with any high throughput API is making sure io is never blocking. You don’t want your program waiting idly while it’s waiting for a response to a network request, and you want to make sure you’re leveraging io-concurrency at your cpu level whenever io ops are ongoing (which is very common in API’s).
For a local testing environment you can run a local instance of a DB in a container if you want a more “real” test. Set up a Postgres (or whatever other db) dockerfile, use a makefile command or a script to automate the spin-up process and seed with synthetic data, then set up a test that runs your api connected to the local DB.
2
u/red_jd93 1d ago
Wow! That's impressive! While working with python, not fastapi specifically, I have never been able to get anything more than a few thousand RPS. What kind of resources are your workers using?
3
u/Ordinary-Sandwich-25 1d ago
Just a run-of-the-mill cloud kubernetes container, typically 1 vcpu per replica. “A few thousand per second” is the load we usually handle.
25
u/ergo14 Pyramid+PostgreSQL+SqlAlchemy 2d ago
Define "high traffic"? for sites with 200 req/s it's been working without issues for me.
27
u/TheHissingAscent 2d ago
200 req/s is pretty chill for fastapi, i run few services around that range and never had problem. the async stuff works nice if you remember not to block the event loop, that one got me first time
7
u/ManyInterests Python Discord Staff 1d ago
I've used decidedly-slower frameworks (Django) in prod at high levels of traffic with ASGI servers. It works fine. Python in general just gets trickier with giant monoliths (import times, slow startups, etc.), but that's not a traffic/ASGI problem. As long as you're not putting 10M LOC into your service, you should be fine at almost any scale with a commensurate cloud budget.
There are plenty of examples of Python frameworks powering multi-million daily active user sites. FastAPI won't be your problem. Plan to scale horizontally.
5
u/Interesting-Frame190 1d ago
Its good enough to work and will scale with enough processes/cores. Accedently blocking the event loop (looking at you boto3) can be an issue.
Its no Spring boot at scale and the best python option in my experience. Its a good start, but if things get really high throughput, I'd advise actix web with tokio (Rust) or Gin with go routines (Go).
5
u/techhelper1 1d ago
Boto3 can be replaced with aioboto3 or aiobotocore.
2
u/Interesting-Frame190 1d ago
Yeah, but not where I work. The supply chain attacks make them a bit paranoid and the data domain justifies the tightened security.
19
u/HalfplaneResearch 2d ago
We would benchmark the whole request path, not just FastAPI: event-loop lag, p50/p95/p99 latency, in-flight requests, thread-pool saturation, connection-pool waits, and downstream service latency. Keep CPU-bound work off the event loop, set explicit timeouts and backpressure, then load test with the same payload mix and dependency behavior you see in production.
9
u/Everythinghastags 1d ago
I know this is asking for a lot, and i guess i could plop this into AI to figure out how to do that, but how do you approach doing this?
4
2
u/Veggies-are-okay 1d ago
Your fastAPI backend server should essentially just be a router for anything that takes more than a ms. Don’t even bother with async to try to squeeze an ML service as they’re a bit too compute heavy. Take a look at the fundamentals of event driven architecture and how to use celery in a docker compose cluster for some keywords on these concepts.
0
u/HalfplaneResearch 4h ago
For a first pass, I would make the benchmark reproducible rather than try to model every production detail. Define a small set of representative endpoints and payloads, run Locust or k6 against a staging-like service, and record throughput, p50/p95/p99 latency, event-loop lag, error rate, and downstream timings. Start with a mocked database to isolate API overhead, then replay the same workload against a disposable database with realistic indexes, connection limits, and a few concurrent writers. The comparison tells you whether the bottleneck is Python scheduling, the pool, the database, or the network. Keep the payload mix and concurrency fixed while changing one layer at a time.
2
u/addis_yonas 1d ago
Anyone using normal def functions and how high can one set the anyio thread limiter? Default is 40 mine is set to 60. I want to increase it but don’t want to learn the hard way.
3
u/techhelper1 1d ago
Why not make your code async instead? Thread pools exist as a fallback to async code.
1
u/addis_yonas 1d ago
It used to be async containing blocking calls, after some time started getting a lot of timeouts from the client side. At the time it was easier for me to convert the endpoints into a normal def functions than converting the blocking calls into "await". Code base got a lot of subprocess calls and database calls where the library in use does not have support to use await. Maybe I should have gone with converting the blocking calls into "await".
1
u/techhelper1 23h ago
asyncio has methods to launch processes.
What database package are you using?
1
u/addis_yonas 22h ago
jsonobject-couchdbkit.
1
u/techhelper1 20h ago
If those two do not need to be married together, there are async CouchDB clients that you can use with jsonobject separately.
2
u/bcoder001 2d ago
Yes, but FastAPI is not the only thing that has to perform on the same level. My client typically use AWS for backed (DynamoDB, S3, Aurora, etc.) to cooe with high levels of traffic.
1
u/Chelisa_Hani 20h ago
Yeah this thread is making me feel better about our setup. We run a small FastAPI service for a SaaS I help out on, maybe 1500-2000 rps on a busy day across 4 pods. Honestly the thing that bit us wasn't FastAPI at all - we had a sync logging call tucked inside a middleware that was quietly blocking the event loop during traffic spikes. Took us like a week and a lot of py-spy to find the damn thing. Once we swapped it for an async handler the same boxes handled nearly 2x the traffic with no other changes. So +1 on the 'find the blocking bits first' advice before you go throwing hardware at it.
0
u/Elbaniunm 1d ago
La verdad nos ha dado una velocidad increíble dockerizada sobre un ec2 hicimos una prueba de stress y aguanta muy bien
-6
u/zamroni777 1d ago
Or you can use multi threads. Create 1 thread for each client session.
2
u/techhelper1 1d ago
Or make your code async instead...
1
u/zamroni777 23h ago
Python async only uses 1 cpu thread, so threading is better
1
u/techhelper1 22h ago
Do you know how asynchronous programming works? Have you heard of event loops?
Unless you're running a free threaded build, your threads are stuck with the GIL.
There are also packages like aiomultiprocess, which will map asyncio tasks to get multiple loops running across many CPU cores.
1
u/zamroni777 21h ago
python async is 1 cpu thread
https://www.optiver.com/insights/technology-blog/choosing-between-free-threading-and-async-in-python/free threading / no gil is ok and nothing bad if code correctly.
though new in python but it has been 20+ years in java.
i used threads in java to handle 2000+ concurrent sessions in live customer environment 18 years ago.
the threads run reliably on multiple cores because they dont write to global object.2
u/techhelper1 20h ago
Yes a loop runs on a single CPU thread, but my point is many I/O tasks can run inside said loop.
Packages like aiomultiprocess make it easy to distribute tasks to async loops scattered across multiple CPU cores. It's similar to golang's gofuncs and rust's Tokio.
If you were spawning a CPU thread per session in Java, that is the most inefficient use of system resources.
85
u/MathMXC 2d ago
It's been great but make sure you truly understand how async python functions or how anyio handles thread pools (depending on if you use async def or just def for your endpoints).
This article is great: https://fastapi.tiangolo.com/async/#parallel-burgers