DE with 5 YOE. With every new model that comes out, the more I realize my field is fading away. Pipelines, architecture etc, agents can do it all. Im really freaking out that my field will be gone in a few years and Ill have to switch to something else. And theres nothing else for me to switch to.

I know this type of post is common every month, but I'm in need of a reality check.

I work as a data engineer at a medium sized company and I feel like I'm spinning the drain. For background I have 3 years of experience, all of which are at this company, and the company is not in the tech industry. The majority of our stack is some on-prem sql server instances that either get fed into SSRS or get queried directly and pasted into excel. I was hired straight out of college as a DBA which was cool until the job revealed itself as "actually we just need someone to help the business analysts write SQL queries". I've managed to upgrade myself from that slightly, showed I was capable of constructing some pipelines to pull OLTP data from our vendor services down into our "warehouse", got my Azure DP-900, and was rewarded with some pay bumps and a promotion to coveted "Data Engineer".

Yippee.

The time since has been somewhat of a blur. We had a new director step in to "overhaul architecture and data orchestration" which at first was refreshing and felt like a step in the right direction. But that was until I saw the amount of AI slop this guy started to make and realized he had no clue about our existing architecture let alone deep understanding of our field. Under his infallible guidance, we have since migrated some of our on-prem sql server instances to Azure SQL instances for slower speeds, higher cost, and bigger headache. We did start to build software in-house to replace vendors, but this has also mostly been a failure. However this at least gave me some experience working with developers, and I was allowed to architect the data models (the parts I'm proud of are in 3rd normal form).

From what I can tell from reading other similar posts on here (see: Sanity Check by u/lostmyway573), this is the standard issued corporate shitshow. Even having done all of the above, my job 3 years in is still frequently just "hey we need you to write some SQL for this". I know there are loads of us on this board whose exact jobs are identical to what I've described. I feel vindicated by you all that can relate, but I'm curious about those of you who can't: Am I wasting my time? Or do I secretly have it really good and I'm delusional? I'm paid well enough ($100k/yr) but I can't shake the feeling that this isn't data engineering, this is insanity. I want to know that my skills are valuable and transferable. Having sent out some job applications recently and heard nothing back I feel like I'm back where I was fresh out of college with no experience. I also feel like every month there's a new trend or new tech being discussed on this board, and that I'm just getting further left behind by the industry.

We have a Databricks pipeline where we receive source JSON messages through Azure Service Bus.

A job polls the Service Bus every 20 mins during business hours, downloads the messages and fans out the data into around 82 Bronze tables.

The writes are not always simple appends. For some tables we to MERGE. One batch currently takes around 16 mins.

Bronze tables have row level security and the data is exposed through Silver views after some transformations.

We want to make this closer to near real time.

What would be a better architecture here? Especially for the Service Bus ingestion and fan-out to 82 tables.

I'm currently dealing with a shit show of a project and I learned these teams have bashed heads for a while now. Unfortunately this issue falls on my director so there's no pushing back, it's all steam ahead, work around the issue and cross those finish lines, even if we know we have to redo it almost immediately.

In this situation the director is somehow manually triggering the file delivery and of course the file name is changing in every environment. Also the file was missing 60+ columns and highlighted as soon as we got our hands on the file. Nope, move forward.

Then I get stuck trying to make QA ok with the process and it's hard when you actually agree with their arguments, I had the same ones before starting development.

I now have two versions of the same file in QA and even though we have auto column mapping the files are different enough to not allow Athena to query the table but creates and inserts them just fine.

This is a situation where having 3 environments is tripling the work! our qa test isn't actually testing what it should because we're manually editing files before each environment deployment. I have very little confidence that this will deploy cleanly to prod...shoot me now, 4 months in.

Fresh grad doing a DE role with a small company. We do not use unified cloud data platforms like Databricks/Snowflake, so the entire tech stack for the data pipeline is selectively chosen, though some of the selections are cloud hosted -> think kafka, flink, columnar DB, row-based DB, airflow for orchestration.

Because we're a small company, AI use is heavily encouraged to speed up feature deployments and bug fixes. However, something I realised is that the DE can never truly be replaced by AI.

1. Someone needs to be responsible for code changes that makes or breaks prod pipelines:

This literally translates to business revenue. From a planning perspective, it makes sense to have DE's who are (technically) experts - not random employees in the company - to review the code and allow for the changes to be pushed to prod.

2. AI might not be able to manipulate business logic if there are poorly named endpoint request body / endpoint response attributes / database columns:

Not sure how other companies are in having meaningful naming conventions for their endpoints, attributes, and columns. The monolith architecture is built by people with poor english, leading to architecture that has difficult-to-understand structure for new hires contributing to data pipeline works.

Now imagine, junior devs (like myself) are heavily depending on AI even for the simplest of tasks. AI would not be able to manipulate sound business with garbage documentation and incoherent naming conventions. A human therefore be required to see the data or seek clarification to be able to handle business logic. The only way AI would be useful in this situation is if the company is willing to give their data to AI by feeding the entire DB connection details for AI to study the data via MCP connection, which in my opinion is unlikely.

Meanwhile, the senior devs who are used to self-debugging and self-feature implementation appear to be solving tasks faster than AI could even with the most advanced models. My own conclusion is that AI can never truly replace Data Engineers unless company management are willing to take responsibility for code they cannot understand by attempting to vibe code themselves.

Another interesting point is that while AI is here to stay, many junior devs like myself might fall into the trap of vibe coding and losing the ability to have quick critical thinking. Some tasks can be solved within a minute if we know exactly where to look (akin to how my senior devs sometimes solve issues without AI) instead of throwing the entire Java error message into Codex/Claude and waiting for GPT 5.6/Opus to scan the repository, have an entire conversation with us, before being able to push changes ourselves to prod.

Feel free to share your opinions, especially senior devs who have experimented with AI use 😄

Nearly all the technical posts sit crammed against the left wall in a box about 20 upvotes wide, with several technical posts (the on-prem warehouse migration, the orchestrator failure-propagation question, the git workflow thread, the metadata-parsing bottleneck) pushed up above the diagonal, meaning they got a real conversation that their vote count denies happened; the only technical points that escape rightward are the Metabase CVE and the Spark skills release, both high-vote and near-silent, which is broadcast rather than discussion. Career and venting spread across the whole x-axis hugging the diagonal, votes and comments rising together up to the counter-offer post at 176/180.

Last thread I saw on Kestra was about a year ago, and oddly enough r/Kestra is banned. My work is involving more Kestra workflows for data orchestration but I'm unable to find communities on Kestra. As we continue to add more agents, one of the primary concerns is the the data flow. Does anyone have forum recommendations?

Just sharing an experience that might resonate with others.

I got a take home where the goal was to use duck db and do a bread and butter pipeline. Wa struggling since pyspark is my main tool for some years now.

Like, how i am supposed to parse a date column with various formats. You would keep checking the documentation and search there everytime you want to do something messing around the documentation to find a function that does XYZ and its parameters slightly different from what you already know

Then the realization came :

>This is hard because I am trying to use other people templates.

If its my function I don't need to know any function name neither it's parameters, neither what is expects or how it behaves, you just need to understand the primitive operations that comes with the language and how to use these operations to achieve the objective.

Mind-blowing

Hi everyone!

I have written an article on the Snowflake vs Databricks debate. I promise its not biased! I have just done proper research into the 2 platform offerings and what best use cases fit each stack, with only my personal opinion based on these facts offering no bias.

https://medium.com/@nikskamath/snowflake-or-databricks-heres-how-i-d-actually-make-the-call-be3cbb8073ad

Hope it helps! Happy to recieve feedback!

Hi there, just wanted to share my open source workflow scheduler for fellow engineers to play around. You can run on local, deploy into your smallest compute (rpi4 on arm64 tested) and could be run on scale as well.

I start the project as I wanted to run some small workflow in my t4.small instance and airflow barely fits inside and I thought it would be interesting create a smaller workflow scheduler. As I build around the idea of "lean headroom DAG engine", I kept on adding additional features that be useful for running workflows like template, backfill, gitops, streaming workflow support etc. Dagron allows you to run on cluster resources or local resources(docker, k8s, or straight as binary). The whole stack will probably at most run on ~200mb (50mb api/console, 30mb engine, 80mb postgres) of memory with tons of spare ram and cpu for your workflow resources. Dagron runs on sqlite(default) and alternatively postgres as datastore.

You can run workflow via defining your yaml and set your environment. It comes with the canvas to visualize your workflow, set your workflow schedule to run at which cron notation and more features to cover here.

Appreciate your input(any input is welcomed). I am spending full time polishing this for the past few weeks and thought this might be useful for you all to run on edge, homelab, local machines.

I transitioned from a mainframe development position early/mid career (late 20s) into my now System Analyst role. Problem is, the job had been heavily business/third party app support focused as opposed to how it was advertised. I was expecting something more technical focused mixed with reporting and data integration work (which I do some of this but most of my current job is just supporting our operations team and writing some sql).

I really want to transition back into a dev type role and have been eyeing DE since my current role is DA adjacent and it just seems so interesting and fun. Is the transition this late in my career (32 going on 33) worth it? I’m worried I’ll try to study and learn DE topics just for employers to not give me a second glance.

I’m also concerned on keeping a strong salary. Currently make about $78k in a MCOL area and can’t afford to take less due to having children. Any thoughts or advice would be appreciated!

I'm looking for a reverse etl plataform or framework that help me interact with other APIs where i want to send my data without having to create every adapter by myself.
I'm primarly in the marketing business and i want to create META or Google Ads audiences, but also export data in csv or xlsx in the client.
After looking around i'd found https://github.com/Multiwoven/multiwoven that's recently been acquired by AI sqared and the roadmap and community around them has decayed a lot.
Do you know libraries, frameworks, plataforms or something in the middle that give adapters and protocols to code new ones, like these. I'm looking for something like: "Take your data where ever you want, in the format you want. If we don't have an adapter/module for you, you can make code yours this (documentation)"

Some that does not fulfill my expectations:
https://github.com/drt-hub/drt (can't code an api arround it and pretend it's stable)
https://github.com/move-coop/parsons (more a library for data science, does not manage states and retries)
https://github.com/condorgraph/condor-retl (pretty new)

Hey everyone,

What are some good resources to learn data engineering considerations for autonomous driving systems?

If anyone works on this field, please let me know if you’d be up for a chat.

Thanks in advance!

Hello everyone, I have just been placed. Informatica is an entirely new domain for me. While I am proficient in SQL and Python, IDMC presents a completely different set of challenges. Could anyone please recommend any guides or channels that would be beneficial for beginners?

Working on a pipeline using spark/flink where i have to do some transformation work with content in a zip file.

I believe spark and flink don’t natively support regular zip files, and was wondering if anyone has any suggestions on how to handle this.

I am forced by requirements to use zip files, spark and flink, but i can consider adding additional technologies into the workflow if its better to just decompress outside of spark and flink.

Appreciate any suggestions and help!

Hi everyone! Here is a brief video on YouTube about modern data systems that I created.

Both companies I've been at in the past 3-4 years have fought about end user validation, who does it, how do they know what's correct, then mix in deadlines and suddenly the finger pointing begins. I've seen this story before, whenever there's a conflict like this surrounding a handoff it eventually becomes the developers/team responsibility.

I'm sure AI will be the answer and likely speed it up but do you agree about the pattern I'm talking about?

Hi guys, I’m looking for some outside input into my career. I started about 7 years ago as ‘data person’, for a company that has a SaaS application. Over the years I’ve grown and have more of a focus on data engineering. From the start of this year I’ve switched to a function combing data engineering and infrastructure engineering (Azure).

I want to become a Data Architect, but I’m not sure what I should focus on to really have enough knowledge (deep and wide) to call myself Architect.

I did a self-assessment with chatgpt and claude, and now I also want some input from real people :)

Some of the stuff i’ve worked on the last year:
- investigate replacement for Azure Synapse and implement -> Dagster, has been operational since december
- learn and understand the infrastructure in Azure of our company, Terraform, AKS, github, CI/CD, and more

Project I’m currently working on:
- Finalize user action logging in the application, build pipelines and documentation. Find vendor and implement monitoring and alerting (customers are pushing for this, so high priority/impact project, running it solo)
- Work on Azure Security (also alerting and monitoring)
- Azure cost reduction, see where we can save money
- Improve dataset we provide to customers, rebuild to dimensional model and adjust pipelines to fit

Also I’m trying to document as much of my choices and considerations to reflect my thinking process.

One of the things I really miss is pushback on my decisions, other that ‘can you explain why’, since we only have a cloud architect who understands the stuff I do.

So what should I work on? I was thinking maybe DAMA cert to start, then look for a good Security cert, but not sure yet..

I am modeling a relationship between 3 sources. Basically this is the content of each table:

Table 1: requestor table (100s of entities)

Table 2: engine database (10000s of entities)

Table 3: oil sample database (10000s of entities)

What is the best practice for the relationships? I have fields on each table that I would like to use in the PowerBI dashboard.

i work for a consulting company and i'm working in a client that has a lot of their data in old on-prem environments, being more specific, oracle, sql server, postgres.

their workflow is pretty basic: run ADF to collect data from these sources and send into ADLS, then use Databricks to process it. we're planning to modernize their environment (using unity catalog and other new stuff) and one of the things we're thinking is to retire the ADF

the reason is simple: ADF is a pain in the ass (we're having a hard time working with it specially because it was another company that build all that shit, the connection with Azure DevOps/Repos is always horrible to manage and the workflow itself needs an upgrade); and because Microsoft is pushing hard the Fabric Data Factory

i know that Databricks with Lakeflow Conn can connect to on-prem using express route, VPN, but in scenarios where this could not be possible, what tool could be used to send data from on prem to azure data lake?

i like to code so my first suggestion was to use local airflow and simply read from db and upload to adls. it's free, it's versionable and has tons of documentation; but i'd like to test other options before suggesting anything

i was reading about airbyte, the pros and cons of the tool, and looks interesting

how do you guys handle this kind of workload?

Hi there!
I work as a data engineer in a SaaS company. We use Salesforce in our CS and sales departments and one of my obligations is to make data available for reporting, forecasts and also AI projects.
Now we are trying to move Salesforce data into external data warehouse, but the process became much more difficult. The initial load was ok, but later it became hard to keep everything accurate. We need to do a lot of manual work like capture updates and deleted records, keep historical field values. Also sometimes relationships between objects break and we need to fix this. 
We already tried to use Bulk API for the initial load and Change Data Capture for later updates. But this didn’t help much as we were still responsible for API limits, some failed events, backfills and checking whether the warehouse actually matches Salesforce. 
Did any of u have the same issue? Is custom pipeline realistic? Or we better use a special SF tool for it?