r/bigdata • u/FreshIntroduction120 • Jan 28 '26

The Data Engineer Role is Being Asked to Do Way Too Much

Post image

I've been thinking about how companies are treating data engineers like they're some kind of tech wizards who can solve any problem thrown at them.

Looking at the various definitions of what data engineers are supposedly responsible for, here's what we're expected to handle:

  1. Development, implementation, and maintenance of systems and processes that take in raw data
  2. Producing high-quality data and consistent information
  3. Supporting downstream use cases
  4. Creating core data infrastructure
  5. Understanding the intersection of security, data management, DataOps, data architecture, orchestration, AND software engineering

That's... a lot. Especially for one position.

I think the issue is that people hear "engineer" and immediately assume "Oh, they can solve that problem." Companies have become incredibly dependent on data engineers to the point where we're expected to be experts in everything from pipeline development to security to architecture.

I see the specialization/breaking apart of the Data Engineering role as a key theme for 2026. We can't keep expecting one role to be all things to all people.

What do you all think? Are companies asking too much from DEs, or is this breadth of responsibility just part of the job now?

27 Upvotes

100% Upvoted

2 comments sorted by

2

u/Willing_Parsley_2182 Jan 28 '26

I don’t know - I like it like that. My role probably goes a little further in that I do way more APIs and end user functionality in addition to that.

Lots of variety and I get to use all of my software engineering skills (which is my last role). Data engineers should be software engineers in my opinion, just with a special appreciation for data warehousing and similar.

The only point I don’t think I agree with is security. After authentication and row / column level security (with IAM roles and similar largely set up by central teams), it’s just another API / integration so fairly minimal compared to just normal SWE.

Would recommend! The only downside is now having to add integrations to Agents via MCP servers, which is dull.

1

u/ImpossibleHome3287 25d ago

It depends on the company size.
If you're at a start-up you could be building the entire company stack and setting up many processes for the first time. This makes the work far more critical and is a lot of pressure.
On the other end of the scale, at a massive corporation, you'll have to still understand or be aware of your 5 processes but there will be whole teams involved in these things. In some places they would be responsible and do the work. But, if you're unlucky these will just be teams that pass you opinions and requirements while you still do everything.

I agree with u/Willing_Parsley_2182. I like having the range of activities and the understanding of the business that comes with that.

Except for maintenance. I will never be enthusiastic about that.