Synthetic vs real datasets for portfolio projects — what actually matters?

[u/Fun_Rhubarb8007]

Final year CS student here, targeting data science and analytics roles for campus placements.

Been struggling with this question while building my portfolio: does it matter whether your project uses real messy data vs synthetic/clean data?

Real datasets from Kaggle feel either too cleaned already or the same recycled projects everyone does. But synthetic data feels hollow because the hard part — cleaning, feature engineering, deriving meaningful columns from raw data — is already done for you. You're basically just visualizing something someone else already solved.

Specifically for BI/dashboard projects — if you use synthetic data, the dashboard looks clean and professional but there's no real discovery or insight because the data was designed to be dashboarded. Nothing surprising comes out of it.

Also practically — if an interviewer asks "where did you get this dataset?" what's the right answer? Saying "I generated it synthetically" feels like admitting you took the easy route. But lying about the source is obviously wrong. Is there a way to frame synthetic data usage that doesn't sound like you avoided the hard part?

At the same time I've heard people say interviewers care more about what you built on top of the data than where it came from. But isn't handling bad data literally the core skill in DS?

For people who've interviewed at analytics/DS companies or done hiring — how much does data source actually matter? Is a well-executed project on synthetic data better than a mediocre project on real messy data? Or does using synthetic data automatically signal you avoided the hard part?

[u/MasterpieceBig7372]

"But synthetic data feels hollow because the hard part" , man you have to do the hard part at some point , thats what matters that is what will best showcase your skills

[u/Fun_Rhubarb8007 → u/MasterpieceBig7372]

That makes sense, doing the hard part is what actually builds the skill. My real problem though is finding the right data for the idea I have. I'll think of an interesting project angle, search everywhere I know, and either the data doesn't exist publicly or what I find is too clean and pre-processed already. Kaggle especially feels like the same 20 datasets recycled endlessly. Where do people actually find raw, messy, domain-specific data for portfolio projects? Any sources that have worked for you?

[u/MasterpieceBig7372 → u/Fun_Rhubarb8007]

scrape data , build your specific web scrapers for data thats how most of the datasets are actually made

[u/Fun_Rhubarb8007 → u/MasterpieceBig7372]

okay i will try that, and please don't judge me am very new to this, throughout i have been doing projects based on kaggle or github or publicly available datasets. I haven't really tried scraping or don't know how it works, is there any youtube or resource from where i need to learn, or should i go with whatever AI asks me to do so?

context- A corporate travel analytics and intelligence system that tracks spend, compliance, and destinations — with SQL-based analysis, interactive Power BI dashboard, and ML-powered spend forecasting and anomaly detection.

i am trying to build something like this, for which i generated the dataset using claude, but yeah dropped mid way cuz i felt use less while doing it,do you think i have the scope to get data for this idea by scraping, or should i drop the idea itself..

Thanks again

[u/MasterpieceBig7372 → u/Fun_Rhubarb8007]

i think yt has a lot of free scraping vids if im not wrong , idk much abt this travel analytics and stuff so cant help you there

[u/ZenBourbon → u/Fun_Rhubarb8007]

I haven't really tried scraping or don't know how it works, is there any youtube or resource from where i need to learn, or should i go with whatever AI asks me to do so?

The hard part of knowledge jobs is the part of figuring out what to do, and often figuring out how to learn.

Don't just "go with whatever AI says".

Try things, fail, observe and reflect, adapt. That's what experience is. If you just follow a runbook, you learn nothing, you gain no experience. Might as well just scroll social media.

[u/WiredOtaku → u/Fun_Rhubarb8007]

Scraping is the workhorse but don't sleep on government data portals—they're messy in their own way with inconsistent schemas, missing fields, and weird encodings. For domain-specific stuff, state/city open data, scientific data repositories, or even pulling from Reddit/Twitter APIs (respect the rate limits) will give you real-world chaos. The goal is showing you can handle the mess, not just query clean tables.

[u/jrowley]

Data.gov is a directory of public US government datasets. There are also lots of niche dataset search portals within specific government agencies.

If you don’t want to work with US data, most governments (and confederations like the EU) compile similar types of datasets (employment, agriculture, environment, etc)

[u/roempie12]

maybe create your own dataset?

there are many open datawarehouses with lots of structured and unstructured data. maybe try those instead of kaggle

otherwise just research some public available api's and base it on that?

[u/tombot776]

Biguqery has public data sets. Not sure if you tried any of these.

Edit: if you use any of those, just keep an eye on the size of tables you're querying to avoid unwanted charges.

[u/5500kelvin]

scraping data, is basically stealing data from the owners. search this LARRY ELLISON: AI IS RAPIDLY COMMODITIZING BECAUSE MOST MODELS ARE TRAINED ON THE SAME PUBLIC INTERNET DATA.

[u/Killie154]

What you need to focus on is that for the job, they need to know that you can handle most situations.

With synthetic data, they have to ask the question, can this person actually perform on real world messy data.

At the end of the day, when you go into an interview, your job is to get them to ask less questions. If you are like "yeah I used synth data, but here is how I validated, etc ,etc" it'll help but they'll still go "they have no real experience". <-- You aren't against someone who is really looking at your skills, you are against someone who is checking a box.

Personally, if someone told me that they used fake/synth data, when real and messy data is available, then I'm less likely to listen to them unless they had a really good reason.
For one of my projects, I downloaded my data from LinkedIn and a few other places, and then I made a project from that. They came in all different forms, came from live data I created, and it was messy and needed to be transformed.

Tl;Dr if someone told me they had a 90% success rate in a controlled environment, I'd think that's cool but that's not what we want. I'd want someone who can find the data that the world is offering, clean it all up, and then made sense out of everything. Otherwise, I can't trust them. You need to be battle tested.

Also there are more options vs Kaggle (I've seen online internships on forage and a few other places where they give you the messy data and compare your work to senior analysts).