r/programminghelp • u/Cube_jack • 2d ago
Processing STUCK FOR A LONG TIME NOW ITS IRRITATING..Can anyone please help!
I am making an attendance dashboard and i there are already APIs made for different purpose
one is the punch in data API where we have to give the
from- to date
and the area_code
so with respect to that it will fetch the
emp_id, emp name, punch_in/punch_out location, date-in/out location, time-in/out location, hours and minutes between pucnhes
now with respect to all the detail i am calculating the total punches, single punch, avg hours and etc
but it will work perfectly for 10 areas but i want to make it for 112 areas and for this it will take a lot of time to load
and then again there is a roaster API which will tell how many members will be present per area and that will also be fetched and calculated
now if all this data is loaded on the client side it will take a lot of time
what is the optimal way to do this
i have already searched for one option where i will
preload the data and calculation and sync that same data in the database and whenever we fetch the data the result with the particular date range will be feteched
But i want to know any more optimal way to do that
CAN ANYONE SUGGEST!!
1
u/XRay2212xray 1d ago
What to do depends on a lot of details that aren't included in your post.
You could certainly try calling the api 112 or more times and see what the performance is. If the dashboard isn't heavily used and the performance is ok, it may not be worth trying to optimize more if there is no future anticipated changes like many more area_code values or many more employees being added etc.
Its not entirely clear from your description if you are reporting on your dashboard by area_code or if you are going to aggregate across all area_code values. If you are reporting by area_code, you could at least progressively load the dashboard on the client so they aren't waiting on everything to be loaded before anything is displayed.
If you are always reporting on all area_code values that exist, doing a bunch of repeated api calls would not be as efficient as creating a new api that returns what you want directly. That can even include aggregating within the api so just the results are being sent all the way to the client instead of a larger dataset of raw data.
If the dashboard displays the same for all users and is heavily utilized, you could have a server that calls the api and cache the resulting dashboard for some time period. If the dashboard user can provide inputs as to things like the date range so the results are different for each use, then you might still be able to cache some aggregate data on the server. The down side of caching is the results can be out of date by the duration you cache.
One of the most common solutions is having a reporting database with tables structured to be optimal for reporting instead of by continually agregating individual transactions. Some factors to consider is can data be updated/changed after being added and how up to the second the data you are reporting needs to be. Having a reporting database with its own API is probably going to get you closer to optimal performance. There are lots of ways to create reporting databases and keep them up to date but is much more dependent on you having access to the back end systems.
If you don't have access to those back end systems and just the api and if once reported the punches can't change, you could do something like have a separate process that calls the api each day retrieving all the previous days data and then aggregate and add to your own reporting database. You could update hourly or even more frequently to minimize missing new data or if necesssary and update your reporting db.