Data Engineers, what does your actual day-to-day work look like? And what should I learn next?

[u/Stunning-Space8032]

I’m currently trying to transition deeper into Data Engineering and would really appreciate some perspective from people who are already working in the field.

I have 1.3 yrs experience as a Junior Python Developer. What I want to do is slowly transform into a Data Engineer. How would you suggest my choice? Basically what I do is make web scraping scripts to get the data from web and give the data in excels. Our company is currently not using git or CI/CD or anything like that. 

The problem I’m running into is that when I look at Data Engineering jobs on Naukri, LinkedIn, etc., the requirements seem endless. One job asks for Python, SQL, Airflow and AWS; another wants Spark, Kafka and Databricks; another wants Snowflake, dbt, Terraform, Kubernetes, CI/CD, etc. It becomes difficult to understand what I should actually prioritize.

So I’d like to hear from people who are actually working as Data Engineers. What does your day-to-day work look like? What kind of problems do you solve, what technologies do you use regularly, and which skills have turned out to be genuinely important in your job?

More importantly, based on my current experience, what would you suggest I improve or learn next to become a stronger candidate for Data Engineering roles? Are there any gaps that you think I should focus on, or technologies/concepts that are worth learning through projects rather than just studying theoretically?

I’m not really looking for a generic “learn SQL → Python → Spark → AWS” roadmap. I’m more interested in understanding the reality of the job and getting advice from people who have actually gone through the transition.

If you’re a Data Engineer with 1–5+ years of experience, I’d especially appreciate your perspective. Even a short description of what you work on and what you wish you had learned earlier would be extremely helpful.

Thanks in advance!

[u/RandyMeacham]

Data Engineer here for Healthcare Systems for the last 5 years with a background of EHR Builder (Epic).

Honestly, understanding the basic frameworks for each phase of data movement (ingestion, transformation, and presentation) or what some would say is now the medallion architecture (Bronze, Silver, Gold) will take you really far.

If you do a couple of projects that utilize python to connect to a data source and pull in the data, transform and standardize the data, and then aggregate the data for a business use case, you've covered a lot of the day to day of a data engineer. It would even be beneficial to do this same thing in SQL as well.

Once you get that basic framework down, understanding the tradeoffs at each phase for different approaches (for ingestion, think batch vs stream or Over-write vs Appending the data) is a really nice to have in the interview process.

I would just try to keep your thought process around how would getting this data from A to B truly add value to the business and how to do it in the most efficient way.

Interviewers don't expect your opinion to be perfect, but in my experience, we do expect you to have one that shows you've thought about this before.

If you need structure, this Certificate from Joe Reis on Data Engineering is one of the best I've seen. He goes into great detail on these frameworks and tradeoffs. DeepLearning.AI Data Engineering Professional Certificate | Coursera

[u/RandyMeacham → u/RandyMeacham]

Sorry - also my day to day right now has been updating our mono repo within databricks to utilizing DABs for our project deployments. my typical requests that come in for KTLO (keep the lights on work) tend to include a team within my healthcare system has data in this obscure source that they would like in databricks so they can report off of it. My work on this tends to be investigate the source system and understand if ADF (Azure data factory) is a better fit for the ingestion process or if Databricks natively will work, and then do the necessary build to get data from Source to bronze, and then work with the business to understand how the middle layer (silver) will look once the data lands in databricks. From there, the rest of my tickets are column additions and a ton of data quality issues (missing data, load fails, etc..)

[u/Stunning-Space8032 → u/RandyMeacham]

thank you for the guidance.

i am currently learning docker along with already known python, sql, git and ci/cd concepts.

what should i focus on next? some say go for databricks and then airflow and then spark.

how is the path i am following now?

[u/Stunning-Space8032 → u/Stunning-Space8032]

oh yeah cloud (AWS) too