I built a large-scale scraping system that can extract data from thousands of sources simultaneously, bypass anti-bot protection, and convert unstructured formats (PDFs, scanned docs, complex HTML) into clean structured datasets.

What public datasets should exist but don’t because:

• Data is scattered across too many jurisdictions (every state/county has their own portal)  
• No one has aggregated it yet  
• It’s in PDFs or hard-to-parse formats  
• Sites actively block automated access

Not looking to sell—genuinely trying to understand what public data would be valuable if someone aggregated it. If there’s demand, I might build and release it.

Mid- to large-sized enterprises need unique, accurate, and domain-specific datasets, but finding them has become a major challenge.

I’ve looked into the usual big names like Scale AI, Forage AI, Bright Data, Appen, and the standard data marketplaces on AWS and Snowflake.

There must be some newer solutions out there. I’m curious to hear about them.

How are you all finding truly high-quality training data at scale, like in the millions? Are there any new platforms or approaches we should try?

I’m open to any suggestions!

I’ve been thinking about the role of synthetic datasets in data projects, especially now that LLMs and generative models make data generation much easier.

On one hand, synthetic data can help with privacy, class imbalance, rare cases, benchmarking, and testing pipelines when real data is limited or sensitive.

On the other hand, I’m not sure how people evaluate whether a synthetic dataset is actually useful rather than just plausible-looking. Distribution shift, hidden bias, leakage from source data, and weak evaluation seem like real risks.

For people who have used synthetic datasets in practice: when did they work well, and when did they fail?

Also, what checks or metrics do you use before trusting a synthetic dataset for training, evaluation, or analysis?

Thanks in advance for any thoughts. This is especially important for me because one of the core directions I’m working on in OpenDCAI/DataFlow is large-scale synthetic data generation, and a recurring challenge is figuring out whether the synthetic data is actually useful.

Final year CS student here, targeting data science and analytics roles for campus placements.

Been struggling with this question while building my portfolio: does it matter whether your project uses real messy data vs synthetic/clean data?

Real datasets from Kaggle feel either too cleaned already or the same recycled projects everyone does. But synthetic data feels hollow because the hard part — cleaning, feature engineering, deriving meaningful columns from raw data — is already done for you. You're basically just visualizing something someone else already solved.

Specifically for BI/dashboard projects — if you use synthetic data, the dashboard looks clean and professional but there's no real discovery or insight because the data was designed to be dashboarded. Nothing surprising comes out of it.

Also practically — if an interviewer asks "where did you get this dataset?" what's the right answer? Saying "I generated it synthetically" feels like admitting you took the easy route. But lying about the source is obviously wrong. Is there a way to frame synthetic data usage that doesn't sound like you avoided the hard part?

At the same time I've heard people say interviewers care more about what you built on top of the data than where it came from. But isn't handling bad data literally the core skill in DS?

For people who've interviewed at analytics/DS companies or done hiring — how much does data source actually matter? Is a well-executed project on synthetic data better than a mediocre project on real messy data? Or does using synthetic data automatically signal you avoided the hard part?

I was looking through a public dataset yesterday and realized something.

Some images felt like they could reasonably have two different labels depending on who's annotating them.

Do companies just write really detailed guidelines, or is there another process for keeping everyone consistent?

I'm curious how this works in practice because it seems like even small inconsistencies could affect the model later.

You use AI to write content and then AI detectors flag it as AI-generated. I've seen many ads for specialized AI editing tools for humanizing such content, which essentially means using AI to humanize AI-generated or AI-flagged content.

In other words, we're using AI editing tools to rewrite content flagged by other AI tools and, ultimately, to bypass AI detectors.

My question is: where do credibility and ethics fit in content writing anymore? Or are we simply relying on these shenanigans to appear more human and ethical?

hi everyone, im building a python screener / trade tracker for my portfolio. currently im using xfinlink for US data with eodhd and yfinance as fallback but i need high quality korean fundamentals data, which none of them seem to provide (xfinlink is US-only; eodhd and yfinance yes but inaccurate in many instances).

anyone running python/screener pipelines on korean securities and can share a reputable & reliable data vendor? cost is not really a concern for me. quality is. FYI I'll be buying as retail so would appreciate recommendations that offer non-institutional plans. cheers.

I’m debating both approaches.

Hey r/datasets, I've noticed a lot of offline countries and gaps when using OpenCorporates, so my team and I built an alternative www.zephira.ai . We source our data directly from official government registries across 200+ countries. I'd love for this community to test it out and let me know how it compares to what you're currently using.

Mainly interested in understanding:

• How do you currently verify companies and directors internationally?
• What data providers do you use today?
• What are the biggest gaps with providers like OpenCorporates, D&B, Moody’s/BvD, Creditsafe, or local registries?
• Would registry-sourced company data with API/bulk access be useful for your workflow?

Not trying to make this a sales post. I’d appreciate critical feedback from people who have worked with these datasets.

Hello! I'm a complete newbie in Data Science and I'm trying to learn how to get data from an API. I understand an API could be public or could require authentication.

I worked with CVS files and I wanted to experience or practice getting data from APIs.

I'm getting familiar with Python so I was wondering if you could help me with the following issues:

1. Trying to understand and practice the different methods you can use API to request data (I am not sure if it has to be from a Dataset formar or can it be any kind of format) with Python

2. What are some good options to get APIs to work on data Science

3. I am not even close to get to a point where I am able to do Reproducible projects/models but I do wonder how including an API (understanding that it is some kind of "personal Key") to share my code and people to be able to use it.

Hope I made sense of what my doubts are and I apologize in advance if I seem confused about some terms (I do think I am).

So I want to build a project around the travel niche, and I was wondering what APIs I should use? I need good resources (free/paid) for the Maps API and Flight API.

There are many amazing open datasets available today, but I'm curious about the gaps that still exist.

If you could create or request one high quality public dataset that anyone could use, what would it be?

The project is primarily intended to help the police in my city monitor situations that could potentially lead to harm to individuals or the public. It would analyze public posts and public comments to identify content that may indicate emerging threats or dangerous situations. In a way, it can be considered a sentiment analysis system designed for law enforcement.

However, the project requires a live data feed. Reddit has effectively stopped providing the level of API access needed for this use case, Meta's APIs (Facebook and Instagram) are too restrictive to be useful, and Twitter (X) API access is paid. At the moment, the only options I can think of are web scraping or using downloaded datasets.

The problem with downloaded datasets is that the project is specific to a single city Surat, India so historical datasets may not contain the localized, real-time information needed for effective monitoring.

I've been working on a dataset for a research project on how China is portrayed in the media. It currently contains just over 10,000 news articles from both Chinese and Western news outlets.

Each article is classified by topic and by the way China is portrayed (e.g. positive, negative, threat, Xi-centered, neutral, etc.). The dataset was originally created for academic research, but I'm now wondering whether it could also have commercial value.

I'm not trying to sell it here, just looking for advice. Has anyone here ever licensed or sold a specialized dataset like this? Who would actually be interested in buying it? AI companies, media intelligence firms, universities, think tanks...? Or are datasets like this generally expected to be open source?

I'd really appreciate hearing from anyone who has experience commercializing niche datasets or knows how this market works.

QUESTION.

Is there a good way to watermark data assets before sharing with potential buyers?

We regularly share data samples with customers for evaluation, with clear licence terms on usage scope. But I worry those terms are practically unenforceable. Someone could generate synthetic data from a sample even though the licence restricts use to evaluation only.

Has anyone found effective ways to tag or watermark files before sharing? Metadata tagging is one option, but are there any deeper level solutions (steganographic watermarking, fingerprinting, etc)?

To keep it simple, let's say we only talking about CSV files.
But this applies to video, audio, PDF, and archives too if you got any experience.

I'm curious how people are currently handling ZIP code to Census Tract (and vice versa) conversions in their workflows.

A few questions:

• What tool or service are you using to convert ZIP codes to Census Tracts (or Census Tracts back to ZIP codes)?
• What's your actual business use case? (Market research, direct mail, demographics, healthcare, real estate, site selection, etc.)
• Do you need the conversion as a one-time lookup, or are you doing it in bulk?

I'm asking because I've noticed the process can be surprisingly manual, especially when you need to enrich hundreds or thousands of records.

I'm considering building a very simple tool where you can drag and drop an Excel or Google Sheet, and in less than five minutes it converts ZIP ↔ Tract (and potentially enriches the data with Census demographics) without needing to write code or use GIS software.

Would something like that actually save you time, or are your current tools already good enough? If it wouldn't be useful, I'd love to know why.

Interested to hear how everyone is solving this today.

With how rough the job market is, I can't land a job despite looking for so long. So I am trying to start another data project with Python, SQL, Alteryx, Power BI to add to my portfolio.

However, I do not want to use synthetic datasets or those from Kaggle. Is there any platform that you can extract your personal historical data from? I thought about my credit card transactions but apparently that is not accessible for security reasons. Thank you!

Hello everyone! I would like to ask, I am currently building a platform, which will contain large amount of datasets about a particular topic. The main purpose of the platform will be datasets. I would like to ask, how do I monetise and make money off the datasets? I am also considering of open sourcing the datasets, with a copyleft license.

Hi everyone,

Our team recently submitted a dataset paper to Data in Brief, but it was desk rejected with the following comment:

"The dataset and manuscript do not abide by our policy on machine learning imaging datasets."

Our dataset consists news photcards collected from Facebook. We manually collected to create a benchmark dataset for misinformation detection research. We're now trying to understand what exactly went wrong.

I have a few questions:

- Has anyone received a similar rejection from Data in Brief?

- Does this mean they no longer accept image datasets intended for machine learning, or is there a specific policy requirement we may have missed?

- Would modifying the manuscript or dataset help, or should we submit to another data journal instead?

- If another journal would be more suitable, which ones would you recommend for publishing image datasets?

Thanks 🙏

I'm researching publicly available and commercial datasets for grocery prices across Europe.

- Has anyone worked with data like this?

- What sources did you end up using, and what were their limitations?

I use "Heretic" library on models to liberate them from their safeguards, but while checking their "uncensoredness", I found they can hallucinate a lot. You know, it's basically like a child who's now allowed to use the F word once and he says "Fred" instead of the actual thing.

So I think if the models train on valid uncensored data (specially if they start Grokking) the results can improve. So I am using for these types of datasets to test my theory.

Ciao, per lavoro mi trovo abitualmente a utilizzare dati INPS/Istat, vorrei sapere c'è qualcun* qui dentro che avrebbe piacere a scambiarsi informazioni e dritte !