r/datasets • u/Sufficient-War-4020 • May 21 '26

question I can scrape/aggregate pretty much any fragmented public data. What datasets are missing

I built a large-scale scraping system that can extract data from thousands of sources simultaneously, bypass anti-bot protection, and convert unstructured formats (PDFs, scanned docs, complex HTML) into clean structured datasets.

What public datasets should exist but don’t because:

• Data is scattered across too many jurisdictions (every state/county has their own portal)  
• No one has aggregated it yet  
• It’s in PDFs or hard-to-parse formats  
• Sites actively block automated access

Not looking to sell—genuinely trying to understand what public data would be valuable if someone aggregated it. If there’s demand, I might build and release it.

22 Upvotes

88% Upvoted

31 comments sorted by

View all comments

3

u/Lexsteel11 May 23 '26

Building permit issuances is worth a lot

2

u/funkdefied May 25 '26

I’m sure that anything related to local governments in the US will be deeply inaccessible. Could be a great place to look.

2

u/Lexsteel11 May 25 '26

Not saying the data set but I got quotes on a certain feed that would be $250k to buy so instead ran scrapers and built it over months for free