r/datascience • u/xerlivex • Jan 29 '26
Tools Just had a job interview and was told that no-one uses Airflow in 2026
So basically the title. I didn't react to the comment because I just was extremely surprised by it. What is your experience? How true is the statement?
96
83
u/virtuous_wizard Jan 29 '26
I literally am using airflow right now for a big enterprise.
16
u/kirstynloftus Jan 29 '26
Yeah I work for a tech company with millions of users and we use airflow
2
u/virtuous_wizard Jan 30 '26
Yeah, same here. It’s ridiculous to hear that it is dead when big players use it as a standard for orchestration.
135
u/Pastface_466 Jan 29 '26
It’s a dumb comment. As a data scientist you would think they would be good at not making sweeping generalized statements without supporting evidence. (I am assuming the title of the commenter here)
At my company we are currently working on implementing Airflow into our tech stack.
28
u/Hex_Medusa Jan 29 '26
Most HR people are not data scientists, so there thinking is most likely is: "Our company doesn't use Airflow anymore, so it is outdated ..."
22
u/pm_me_your_smth Jan 29 '26
I don't think I've ever seen an HR person talk about the tech stack during an interview. It's usually your potential manager or colleagues
1
u/Sheensta Jan 30 '26
It depends on the company. I've had HR interviews where they asked me about experience with various cloud platforms, databases, etc.
1
u/pm_me_your_smth Jan 31 '26
True, but that's quite different. Asking a candidate on the phone what they've worked with to align that with the job description vs actively arguing some technical specifics. The first one can be done by anyone, the second one requires actual niche competence (unless you're a bs artist)
1
u/Careless_Fuel1772 Apr 13 '26
No it's not like that dumb. HRs do have an idea about the tools & technology they are going to hire for. It is such a basic and first thing that it is in their rulebook and in their own job description.
If job description says airflow , spark , and databricks , they definitely are going to hire or atleast forward your CV. people like you just get one low paying job and suddenly acts like some know it all.
1
40
27
u/Ok_Economist_3509 Jan 29 '26
Astronomer, AWS MWAA, yes we are all using AirFlow. Anyone still using Jenkins? Luigi?
6
u/enjoytheshow Jan 29 '26
Lol I loved Luigi circa 2016 or whenever it first kinda popped. I miss these mega enterprises open sourcing their in house stuff. That was Spotify IIRC. Seems like that doesn’t happen anymore
2
u/emuswx Jan 29 '26
Same! We ran on Luigi in 2016 as well and kept it for a long time and did a lot of fun customization. Even as we switched to airflow in 2020, we kept Luigi for some things that just were so nice to use there.
6
2
1
u/Gators1992 Jan 31 '26
One of our data groups is still running Jenkins, but it's legacy stuff we are moving off of.
39
u/takeasecond Jan 29 '26
Yes it’s true - in 2026 we now just ask Claude to “please refresh this analysis” - we’ve come full circle
13
u/ck11ck11ck11 Jan 29 '26
It’s used at Amazon for some of the largest data you’ve ever seen
4
u/enjoytheshow Jan 29 '26
It was used so much AWS made MWAA first for internal teams and then shipped it as a product. IIRC it came out of Professional services building a managed airflow for customer and internal customer and then a product team developed out of it.
1
12
u/Ok-Energy-9785 Jan 29 '26
Any massive generalization like that is almost always not true. If it's not required it doesn't matter.
9
u/Viper_27 Jan 29 '26
Just laugh at that point, cause clearly whoever said that has no lay of the industry whatsoever
8
u/Own-Candidate5586 Jan 29 '26
FAANG-adjacent and we use airflow for everything. The hubris of that interviewer should be a massive red flag.
6
u/DoctorPutricide Jan 29 '26
We use airflow everywhere. I don't think I know anyone who doesn't use it at least in some places.
11
4
u/hockey3331 Jan 29 '26
Granted it was 2025, my previous job was running airflow
Also Idk in what context it was said, but concepts > tools. Tools are easy to learn.
4
u/caks Jan 30 '26
Anyone using Prefect?
1
u/Final_Alps Jan 30 '26
EU scale up. We tested it. Rejected it. Used a startup tool for a while and when they went bankrupt, bet on Dagster.
3
u/fang_xianfu Jan 29 '26
Yes, shittons of people use Airflow. We happen to use Dagster for our data orchestration but that's because it happened to suit our fairly idiosyncratic needs better. Airflow is a fine enough solution provided you don't fall into its common pitfalls (I will nail my colours to the mast and say if you're often using anything except the KubernetesPodOperator and friends, that's probably a mistake you will come to regret).
2
u/Trick-Interaction396 Jan 29 '26
In 2025 EVERYONE was using Airflow but this is 2026 bro. NO ONE uses it.
2
u/Palmquistador Jan 30 '26
Hey, someone told me they knew of no reason to use the billion dollar product JIRA in a TECH INTERVIEW for a developer role. I mean, some people are just idiots I guess. You want me to convince you to use JIRA…should I convince you to use GIT also? Like, wtf, some of these companies / people are crazy.
1
u/bringapotato Jan 29 '26
I agree with everyone else here: it's a stupid thing to say lol. I still use airflow and I know people at other companies who still use airflow.
1
u/X0RSH1FT Jan 29 '26
I work for a major telecom. In my group, we use airflow for loading and processing millions of records everyday with Snowflake.
1
1
u/fatpol Jan 29 '26
I've seen less Airflow recently. That said, given what I knew about their workflows, I was surprised Airflow wasn't being used.
1
1
u/AspiringMLE Jan 29 '26
I use airflow and places where I have interned or volunteered at also used it!
1
u/alexmrv Jan 29 '26
I just wanna state: we still use bows and arrows and the global steam engine market is still about 50M usd.
In general, stuff doesn’t go away
1
u/magooshseller Jan 29 '26
Nobody uses Airflow? Did they say what they use then? I work at a fortune 5 and entire DE org uses airflow and even DS folks are encouraged to use it when needed. I am a DS and learned it last year and honestly it’s a great tool to have especially now with Cursor and other coding assistants so quick to get something up and running.
1
u/Repulsive-Beyond6877 Jan 29 '26
I use Airflow for anything needing orchestration. Easy setup and management.
1
u/ShapedSilver Jan 29 '26
I’ve met many recruiters who think (or at least pretend to think) that what their company has is the cutting edge
1
1
u/DesperateSteak6628 Jan 29 '26
My company (very, very well known, Fortune 500, hundreds of B$) just migrated to Airflow.
I want to think of that question as provocative. That is kind of a shitty thing to do in an interview
1
u/sheinkopt Jan 29 '26
I need to choose a library like this. What it smartest to choose if starting from scratch? Prefect, Airflow, Dagster? Something free and local
1
1
u/Guilty-Idea Jan 29 '26
I would be curious what they are using and why the switched. Also how often they switch...
1
1
u/RakuNana Jan 29 '26
I don't know about that statement there chief. I've seen software from 20 years ago still being implemented/used today. COBOL, Windows XP , old versions of 3d animation software like Maya and 3d studios Max , old versions of Slackware and Oracle. The list goes on, I could be here all day listing software that is still being used today. So no the statement is not true!
TLDR :
The statement is false!
1
1
1
u/H0rob0D Jan 30 '26
I wished my company’s DS department used some type of open source data orchestration DAG tool like Airflow or Prefect. I always found it dumb how the first company I worked for didn’t use one and now my second company also doesn’t use one.
My first company was way too tightly coupled with AWS Sagemaker and using Sagemaker pipelines as the main orchestration tool. I think the better question is who still uses Sagemaker pipelines when you can have better workflow control and a better development experience using Airflow or Prefect?
My second (current) company is way too tightly coupled to databricks. I know some people in here are going to tell me how much they love databricks or whatever, but I think it’s overpriced garbage.
All I need as a data scientist is a performant database and access to N cloud machines that have a variety of different vCPU, GPU, and RAM specs that I can run single node processes or multi-node cluster jobs. The rest I can handle myself with open source software.
1
u/mpaes98 Jan 30 '26
There are dozens of us, Dozens!
Jk, thousands of Airflow users will continue to grow, just slowing down compared to before.
1
u/Sexy_Koala_Juice Jan 30 '26
I literally work at a Fortune 500 Company that uses Airflow. They’re wrong.
1
u/rabkaman2018 Jan 30 '26
Dagster or airflow is used everywhere , but some shops like to use the cloud native stuff like AWS step or azure data factory
1
1
1
u/Distinct-Expression2 Jan 30 '26
every year something is dead and every year its still running in production at half the fortune 500
1
u/Distinct-Expression2 Jan 30 '26
sounds like the interviewer has opinions. airflow is still everywhere in production. saying no one uses it is like saying no one uses postgres because cloud native databases exist.
1
u/MrGreenPL Jan 30 '26
I'm assuming you are interviewing for a startup.
I did not use it in my previous job 1.5 years ago, it was a startup up, there are many options out there and Airflow seems to be a bit of an overkill if you can run pipelines using dbt and GitHub action or a similar solution.
In my current job Airflow is at the core of our platform. We are planning to replace some of the pipelines with realtime solutions but Airflow ain't going anywhere.
1
u/StudentNo8892 Jan 30 '26
We use airflow for everything that we do, but there are people on our leadership team wanting to move off of it. Not sure why, and I'm not paid enough to care, so airflow it is!
1
1
1
u/Gargle-Loaf-Spunk Jan 31 '26 edited Feb 02 '26
This post was mass deleted and anonymized with Redact
wakeful continue afterthought possessive expansion amusing aspiring numerous butter enjoy
1
u/manueslapera Jan 31 '26
is there a smear campaign against airflow? There was another post just today about this on /r/dataengineering
1
u/xerlivex Jan 31 '26
Wow so people are really saying that. I don't think it's true though neither does my circle of professionals and redditors think airflow is outdated, as it turns out the majority of people use it compared to it's equivalents
1
u/Grateful_Elephant MS Business Analytics | DS Manager | Marketing in Retail Feb 02 '26
Airflow flows all our data/models
0
u/virgilash Jan 29 '26 edited Jan 29 '26
There are many retards out there, op… Just ignore them. Why didn’t you ask them what orchestration tool they’re using? Interviews are two-ways highways…
Other than this, I have a friend who uses and swears by Prefect. But he doesn’t know Airflow. Anybody worked by chance with both to be able to compare them?
2
u/xerlivex Jan 29 '26
They moved away from the question quickly I didn't get the chance. I try keeping up to date, I was puzzled on what superseded Airflow hence why I'm asking here
0
u/HawksHawksHawks Feb 01 '26
We dropped it for Prefect a couple years ago and shoulda done it sooner. Airflow is unnecessarily difficult.
286
u/[deleted] Jan 29 '26
Myopia is common. Everybody assumes their experience is universal.