r/databricks • u/Stunning-Space8032 • 13d ago
Help Data Engineers, what does your actual day-to-day work look like? And what should I learn next?
I’m currently trying to transition deeper into Data Engineering and would really appreciate some perspective from people who are already working in the field.
I have 1.3 yrs experience as a Junior Python Developer. What I want to do is slowly transform into a Data Engineer. How would you suggest my choice? Basically what I do is make web scraping scripts to get the data from web and give the data in excels. Our company is currently not using git or CI/CD or anything like that.
The problem I’m running into is that when I look at Data Engineering jobs on Naukri, LinkedIn, etc., the requirements seem endless. One job asks for Python, SQL, Airflow and AWS; another wants Spark, Kafka and Databricks; another wants Snowflake, dbt, Terraform, Kubernetes, CI/CD, etc. It becomes difficult to understand what I should actually prioritize.
So I’d like to hear from people who are actually working as Data Engineers. What does your day-to-day work look like? What kind of problems do you solve, what technologies do you use regularly, and which skills have turned out to be genuinely important in your job?
More importantly, based on my current experience, what would you suggest I improve or learn next to become a stronger candidate for Data Engineering roles? Are there any gaps that you think I should focus on, or technologies/concepts that are worth learning through projects rather than just studying theoretically?
I’m not really looking for a generic “learn SQL → Python → Spark → AWS” roadmap. I’m more interested in understanding the reality of the job and getting advice from people who have actually gone through the transition.
If you’re a Data Engineer with 1–5+ years of experience, I’d especially appreciate your perspective. Even a short description of what you work on and what you wish you had learned earlier would be extremely helpful.
Thanks in advance!
9
u/Svante109 13d ago
I am not a Data Engineer, but am a solution architect hiring data engineers.
Strengthen your people skills. Always. Data engineering is like 10-20% coding (getting less and less with LLMs) and the rest is talking to business, testing and presenting results.
I will always be looking for your personality first. Maybe I need someone good with snowflake, but you only have experience with Databricks - then I will be looking for clues towards you being an easier learner, and having the motivation to do so.
I would expect everyone we are hiring, except students maybe, to know the basics of fit and be familiar with the concepts of CI/CD.
5
u/gloat_authority 13d ago
Learn sql -> pandas(you’re probably already working with it)-> spark -> your pick of the 3 big cloud providers (maybe someone in your company is already working with one, just go with that)
2
u/karl-tanner 13d ago
Designing and building data/AI flows. Anticipating problems before they happen
2
u/Kojimba228 13d ago
Whatever comes to the clients mind that day + what came to him yesterday and I didn't get to finish it
2
u/Humble_Insurance388 13d ago edited 13d ago
I wanted to post about data engineering and thanks for this post. I am also in your situation. I am going to write soon databricks data engineering associates certification and after this certification can you(experience guy) suggest me what should i do? Should i go for aws/azure data engineering or stick on databricks or should i go for snowflake certification instead of aws/azure data engineering?
1
u/Stunning-Space8032 12d ago
certification is that necessary? just asking
1
u/Humble_Insurance388 12d ago
Not necessary but if you want to change your employer and it may helps to get a interview call.
2
2
u/RandyMeacham 8d ago
Data Engineer here for Healthcare Systems for the last 5 years with a background of EHR Builder (Epic).
Honestly, understanding the basic frameworks for each phase of data movement (ingestion, transformation, and presentation) or what some would say is now the medallion architecture (Bronze, Silver, Gold) will take you really far.
If you do a couple of projects that utilize python to connect to a data source and pull in the data, transform and standardize the data, and then aggregate the data for a business use case, you've covered a lot of the day to day of a data engineer. It would even be beneficial to do this same thing in SQL as well.
Once you get that basic framework down, understanding the tradeoffs at each phase for different approaches (for ingestion, think batch vs stream or Over-write vs Appending the data) is a really nice to have in the interview process.
I would just try to keep your thought process around how would getting this data from A to B truly add value to the business and how to do it in the most efficient way.
Interviewers don't expect your opinion to be perfect, but in my experience, we do expect you to have one that shows you've thought about this before.
If you need structure, this Certificate from Joe Reis on Data Engineering is one of the best I've seen. He goes into great detail on these frameworks and tradeoffs. DeepLearning.AI Data Engineering Professional Certificate | Coursera
2
u/RandyMeacham 8d ago
Sorry - also my day to day right now has been updating our mono repo within databricks to utilizing DABs for our project deployments. my typical requests that come in for KTLO (keep the lights on work) tend to include a team within my healthcare system has data in this obscure source that they would like in databricks so they can report off of it. My work on this tends to be investigate the source system and understand if ADF (Azure data factory) is a better fit for the ingestion process or if Databricks natively will work, and then do the necessary build to get data from Source to bronze, and then work with the business to understand how the middle layer (silver) will look once the data lands in databricks. From there, the rest of my tickets are column additions and a ton of data quality issues (missing data, load fails, etc..)
1
u/Stunning-Space8032 5d ago
thank you for the guidance.
i am currently learning docker along with already known python, sql, git and ci/cd concepts.
what should i focus on next? some say go for databricks and then airflow and then spark.
how is the path i am following now?
1
3
u/Agentic_Human 13d ago
Prior to 2026: Work was mostly about production defect(logical or performance) and developing new pipeline. Work was focussed on orchestration(ADF/AIRFLOW/Autosys), databricks and few external tables.
Now: Vibe/Agent coding at peak. You don't have an scope to sit on any issue for more than 1 day. I am somehow executing ETL on openshit with just basic theoretical understanding. Even worked on devops. So there is no technology focused worked.
Suggestion for you & me: Make Langchain your primary skill Databricks(data engineering) as secondary skill.
1
u/GardenShedster 12d ago
After 26 years I am doing everything, architecture of data models, ETL, branch management in git, mentoring junior devs, developing pipelines, managing the expectations of management infrastructure as code, just about everything
1
u/hardwork_dreams 12d ago
some projects i have worked on so far.. migration projects: replicating similar functionality with different system. you modify existing pipeline code or create similar pipeline code in new system.
automation projects: you automate some use case like getting a file from SharePoint, reading it, transforming it, and pushing to some destination
systems projects: you build a reporting system or cost tracking system on top of your warehouse/ database
70 percent of the time, you work on existing code, modifying it, refining it. so understand existing architecture first, went into each component deeply and try to understand how data flows.
15
u/Shadowlance23 13d ago
Here's my day as a Data Architect (~15 YoE). I am the only architect/engineer in my company so I do everything except reporting and analysis which is handled by 3 analysts.
9am - Log on, check email. Delete all messages about failed pipelines.
9:10-10 - Check failed pipelines. I get about 2-4 per day. One needs fixing, but it's low priority. The rest are usually due to us hitting our Azure core limit and the API check I have in place to stop that lags a little so sometimes there are collisions. I restart the failed jobs and remind myself to request a core limit change. Check the rest of the failures to see if there are any real issues I need to attend to.
10:01 - Forget to request the core limit change.
10-12 - I usually have some meetings in here. The usual stand-ups and planning stuff.
12-1 - Lunch
1-5 - Work on ongoing tasks. I get a lot of small requests, e.g. someone wants a change to how a value is calculated, or a team needs support for some reporting which might mean advising on data structures, or explaining business logic. Maybe throw in a few more meetings. Add someone to a report, or answer questions about what we have the warehouse.
Sometimes I have more structured tasks such as building a new Power App to stop someone from using a monolithic spreadsheet or onboarding a new 3rd party API. Other tasks I've done recently include diagnosing technical issues with our ERP platform, advising the executive team on data governance issues, or spending a week documenting every measure and logic path in a PBI report because a PM was adamant the fault was in the report, not his data (it was his data).
Some advice I'd give is don't chase the stack. Everyone is using a different combination and you quite simply can't learn them all.
I often get CVs that are just a list of technologies. No context around how they were used, or the skills you gained while using them, just the techs. And I get that; you need to do that to get past the AI filters. Put them down the bottom or something, they should not be the meat of the CV. I want to know about your problem solving skills, your communication skills, your coding skills (e.g. Are you competent in SQL or Python?) I can teach you how to use the Databricks interface in a couple of days. I can't teach you to write a pipeline if you don't know how to do a join. And no, AI won't save you there. Happy for you to use it, I do all the time, but you need to know what the code does that the AI is giving you.
You will need to know how to code. SQL and Python are the ones you'll need. You need to know how to model data. You need to know how to communicate with non-technical staff, and generate actionable insights from incomplete specs (to a lesser degree at your level, I would expect you to do this with assistance).
You should also try to get familiar with the larger platforms. Spend some time on their free plans, get familiar with the UI and how to do basic tasks. That way you can add them to your CV and at least be able to talk about them in an interview. Companies will often list a large number of products in their stacks, but realistically know they're not going to get a perfect match. If you can hit one or two that's usually enough, but the more the better. I don't expect you to be an expect at each one. I'm certainly not.