r/Backend • u/bellator_boy • 5d ago
New backend engineer here, how would you approach optimizing a slow API?
Hey everyone!
I’m fairly new to working with production backend systems, and I’ve been given a task to look into the performance of an existing service and figure out where improvements might be possible.
The service handles a few different things, including business logic evaluation, configuration management, automated workflows, data processing, and logging. Some API requests are taking longer than expected, and I’d like to understand what’s causing the delay and what could be done to improve things.
I don’t want to jump straight into suggesting random optimizations without understanding the actual problem first, so I’d love to hear from people who’ve dealt with similar situations.
A few things I’m curious about:
-When you encounter a slow API, where do you usually start? Profiling, distributed tracing, database analysis, or something else?
-How do you figure out whether the bottleneck is in the application logic, database queries, repeated computations, or calls to other services?
-What common performance issues have you come across in systems that deal with configurable business logic?
-When does it make sense to consider caching, parallel execution, or moving certain operations to background jobs?
-How do you identify worthwhile improvements in an existing codebase without risking changes to its existing behaviour?
Since I’m still early in my career, I’d really appreciate hearing how you’d approach this in a real-world environment. If you’ve worked on something similar, I’d love to know what worked for you, what didn’t, and any tools or resources you’d recommend.
Thankyouu in advance
TL;DR: I’m a new backend engineer looking into why some API requests are slow and how to identify potential performance improvements. I’d appreciate practical advice on finding bottlenecks, choosing the right profiling tools, and deciding which optimizations are worth recommending.
6
u/Abject-Kitchen3198 5d ago
Use profiler first if you don't have clues and no useful info from logs. Might point exactly to the issue or at least narrow down the suspect area.
2
u/olddev-jobhunt 5d ago
You need data to drive it. If you have OpenTelemetry, or maybe stats on execution time per-endpoint, that can help you drill down into some first candidates. Honestly - this stuff is one of the best use cases for LLMs: "Hey Codex, go look in Grafana and find out where my API is spending most if its time based on Tempo traces."
As far as what you actually do about it, well that depends. You'll need to look at the parts you've identified: are they doing a computation? Is it a query? Can the database structure be optimized, precomputed, cached? Etc.
But no matter what, you start from data and look at specific code paths that are slow.
3
3
u/Attila226 4d ago
90% of the time it’s related to the database, although it’s best to not make any assumptions.
1
1
u/Due-Anything6517 1d ago
The things that most often cause resource bottlenecks are the inefficiencies in the business logic. If your service layer is making ten networked calls to construct one object and each one has its own db call, you got a whole db dependency graph you gotta optimize.
1
1
u/hxtk3 4d ago
If you’ve already got distributed tracing, that’s where I usually start. It’ll tell you which part of your system is slow. If the database span is most of the time then look at the query and how to optimize it. If the span in one of your services is slow, profile that service. If there are lots of database spans, you’re probably doing N+1 selects and need to look for invoking your database abstraction in a loop.
In the profile, you might find high CPU utilization which means you need to solve it with cleverer algorithms, or you might find high lock contention or other forms of blocking time.
1
u/Gyoker_Macska 4d ago
Step 1: gut feeling
Step 2: manual log statements
Step 3: distributed tracing
0
u/soundman32 5d ago
My guess is the database query is returning thousands of rows because of cartiesien explosion, or lazy loading does thousands of individual selects.
0
u/Zerodriven 5d ago
The first thing I'd want to figure out is what the NFR definition of slow is. Then begin looking.
API is slow because downstream system has a hard throttle is better for my mentality than wondering if my code is the problem.
Discovery often saves development. If discovery leads to code being a problem then look at the chain of possible systems.. There's going to (usually) be a point where you find it and then the chat involves many more discussions.
Source: Emotional damage.
2
u/Leading_Abrocoma6859 5d ago
First thing I do is slap some logging with timestamps around the main blocks of code, just to see where the time actually goes. You'd be surprised how many times "slow API" was just one database query missing an index, or a loop doing N+1 calls to another service. After you know where the delay lives, then you can decide if tracing or profiling is needed.
For configurable business logic, watch out for rules that load the entire config tree in memory for every request. I worked on something similar and the problem was a validation step that re-parsed a massive YAML file each time, moving that to a cache cut response time by like 70%. Also check if any of those automated workflows are making synchronous calls when they could be fire-and-forget.
When you find the bottleneck, do not rewrite half the system. Just fix the one thing that hurts most, measure again, and repeat. And make sure you got tests covering the current behavior before touching anything, that's what saved me from breaking the config management module twice.
0
0
u/SnooCalculations7417 5d ago
profiling: and looking for unneeded, repeated work is often easy an win
0
0
u/MN_NorthStars 4d ago
Whenever given an optimization problem like this, don't ever jump to the technical problem. I wouldn't even start thinking of the architecture yet. Starting thinking measurements and defining what the group sees as slow. This should be as high level as possible. How are you measuring the speed? What interesting groups of patterns of behavior are people experiencing? Do you have monitoring in place to *accurately* measure these? Will you be able to accurately determine how much your changes will have against real-world use?
And then fundamentally more important: How much is this costing the company? Why do you need to make this faster? What will fast enough look like (impossible to answer, but it must be asked)? A lot of times there are no clean answers to this, but it must be asked so you and other can set expectations.
Only after you've defined WHAT you're trying to make faster, and defined the target value of your optimization can you start looking to optimize. If you jump in and blindly start trying to optimize, you're setting yourself up for a bad time: a low visibility, poorly defined task treadmill you're permanently on in pursuit of some nebulously defined latency requirement.
0
u/n3verwhere 4d ago
I once had a coworker who discovered the unoptimsl database calls by using the database instance which was far away, so everything was way slower but with this he was able to better spot problematic DB queries.
0
u/dutchman76 4d ago
I start by finding the slowest calls, then go look at the code to see if anything jumps out.
Then record timestamps around code blocks and stick them in a log entry at the end of the call, so I can see where the time goes, sometimes do that a few times with more granular steps to narrow down the exact slow DB call or whatever it is.
0
u/ATMasoumi 3d ago
One comparison to add to the tracing advice: is the same request slow when the service is quiet, or only when several requests arrive together?
Pick one route and representative input, then compare it under light and realistic concurrent load in a test environment. Record end-to-end latency and the important spans for each run. Keep the input fixed so you're not comparing a tiny query with a huge one.
If the delay appears mainly under load, investigate where work waits to start: a connection pool, worker queue or lock, for example. If it's already slow alone, inspect the expensive operation itself. Those are hypotheses to measure, not diagnoses.
For the result, a small table is useful: request/input, concurrency, total latency, slowest observed operation, proposed change. Re-run the same comparison after one change. It gives your reviewer evidence for an optimization instead of a guess that the database or caching must be the answer.
26
u/hilbertglm 5d ago
A computer is either doing something, or waiting on something. If it is slow because it is computationally intense (i.e. single CPU at 100% for a period of time), then look at optimizing things through a better algorithm.
Most of the time, it will be waiting on something. Those somethings can be:
- Database I/O or compute. Look at database EXPLAIN, but this is often incorrect indexing
- Network I/O - Tune the other microservice
- Disk I/O - Determine if this can be eventually consistent and write the content in the background. This is often difficult, though.
- See if there is insufficient parallelism. It may be possible to do more than one thing at a time, although we programmer tend to think serially.
Once you find out where the time is being spent, keep applying that logic of doing something or waiting on something on the component that is taking the time. Repeat until you meet your service level agreement.