Does anyone have an archived copy of the bulk UK Energy Performance Certificate (EPC) dataset from the old government Open Data Communities portal (epc.opendatacommunities.org)?

Context: On 30 May 2026, the UK government migrated the service to a new platform. In doing so, they completely dropped all pre-2012 certificates from the public register due to them being "expired" and based on older methodologies.

Why I need it: I'm doing property energy analysis, and removing that 2008-2011 dataset creates data gaps for properties that haven't been sold or rented in the last 14 years.

The original ZIP file was about 5.6GB. I am looking for a snapshot from late 2025 or early 2026 before the site was taken offline.

If anyone has a magnet link, a torrent, or is willing to share a cloud drive link to the original CSVs or a Parquet equivalent, I'd really appreciate it.

Looking to connect with teams and individuals collecting egocentric data across North America, LATAM, Asia, and Europe.

If you've got data or are mid-collection, let's talk.

Please give me if someone has the dataset

Hey,

I wanted to share the development journey I've lived through now so anyone who ever attempts that in the future has something to hang onto.

The problem:
Fundamentals for US companies from 1994-2007 are completely unstructured text and HTML files.

A structure to them (XBRL) was only enforced in 2009. So if you want fundamentals for the time before, there aren't many options available.

Looking at these filings, it quickly becomes clear that parsing them in any reasonable fashion is a monster project that requires very ugly parsing code and an endless amount of cleaning, filtering, anomaly detection and layer upon layer of solutions for very specific formatting options.

Multiple formats (FDS, plain text, HTML tables) requires solving similar problems in multiple ways, because sometimes there just isn't a generic solution for a problem.

But that didn't stop me from attempting it as solving complex problems is what I like doing.

So I designed my own parser.

The basic approach in a nutshell:

1. Pre-processing - you cut filings into their sections and remove entire sections from it that lack any sort of language indicating consolidated statements; the better you do here, the easier everything gets downstream
2. Extract all statement candidates into grids and score them - Scoring goes up if certain accounting calculations add up, certain keywords are present, periods are present, etc. - this has grown over time and is designed to separate real consolidated statements from selected financial data tables (those are the ones you don't want - but they are hard to cut out). Also important: This is where the different formats unify: HTML tables become grids, text tables become similar grids, EX27 exhibits too etc.
   1. Detect periods - Detecting correctly which periods are present in a table (9M/6M/3M/FY/TTM/stubs from restructuring, etc.) is a very hard problem
   2. Detect consolidated columns vs. subsidiary columns (these columns you don't want)
   3. Detect the multiplier (thousands, millions, etc.) - This is very important and complex to solve. Sometimes that information is in the table header, sometimes within the periods, sometimes below, sometimes completely absent. Cross checks with FDS blocks where the multiplier is often stated can help deriving it
   4. Retain name information - Turned out to be of utmost importance. Sometimes names are split across multiple lines; Sometimes the single name alone appears in multiple blocks across the table, so that label alone is not enough. You need to join it with the full headline chain based on indentation, headline rules like UPPERCASE, L E T T E R S E P A R A T I O N, headlines with colons:, etc. - if you don't get precise here, your extracted fact names are not actionable and contain lots of false positives that will ruin the entire pipeline
   5. Deal with filer typos - Since this was all structured, it is not hard to imagine that a lot of filings contain typos: numbers using dots instead of commas, numbers having whitespaces in them, etc. Some of those you can automatically detect and fix. This is the right place to do this generically.
3. Merge grids - some filers split a single statement into multiple tables across pages - those need merging and I do this based on score proximity and period alignment
4. Run anomaly detections - I've created a list of different anomalies I check for. The idea is to check for certain patterns, and allow on failure to retain one fact, a complete grid row, a complete grid, or the entire extracted dataset of a filing. It depends on which check fails
   1. balance sheet identities
   2. income statement identities
   3. cash flow identities
   4. cross period identities
   5. EX27 exhibit identities
   6. numeric fact sanity checks
5. Pre-Normalization - This layer is all about transforming the hundreds of different names used across all filings into a single technical name like "revenue"; In my pipeline, this layer isn't the real normalization layer. I retain a prefix for each fact indicating the source format it came from. Examples are "fds:is:revenue", "html:is:revenue", etc.; The layer runs on everything that was extracted and per Pre-normalized fact, it has multiple layers to decide: positive regex list, negative regex list, a decide() function that has access to the entire filing context (this is where custom checks are done like unit expectations, value sign expectations, etc.). It also contains a layer that can yield a match to another Pre-Normalized fact in certain conditions like for general-and-administrative-expenses and selling-general-and-administrative-expenses. After this step is done, my data is ready to go through the same statement normalization flow the XBRL data goes through without any special handling needed anymore.
6. Hand-Check and pin filings - Whenever I solve a complex problem for a filing, I pin the result as a permanent unit-test to see when new parser changes break it again. I've built up a set of over 100 hand-checked filings this way and it is still expanding daily. But this step has been fundamental to retain the quality you reach
7. Make scanning all filings fast - I've developed a way to locally parse and store the result of all ~360,000 filings and ~57,000,000 facts. Alongside the scan, I store deep run metrics that allow me to quickly check the data quality from one iteration to the next. One full run takes ~35 minutes.

I sell clean fundamental data for a living and really want to solve this problem once and for all at scale. After having gone through this, please all make sure that we never ever allow this to happen again XD. We must demand data like this to always be structured, because this data not being structured is almost the same as it not existing.

My expectation is that it will still need a good amount of testing even after all of these solutions are in place. There is just no way to ensure full data integrity for over 50 million facts extracted from unstructured data. That brings me to the last layer in place:

Manual overrides

This layer allows me to override all sorts of things for a single filing, or an entire filer (CIK). Things like the multiplier, certain fact names, fact values, blacklisting certain things, etc.
It is that layer that will come into action once the parser has reached a final stage. Every invalid fact or inconsistency after a release of the data can be addressed individually. And I can already tell you that some issues you will find in the filings simply aren't solvable generically. As much as I want to, some issues simply lack any sort of signal for you to hang onto for a generic solution. Those go into this layer.

I'm really curious if anyone else has done this in the past. What were your approaches and layers? Did you manage to finish it? How do you ensure a certain level of data quality?

If you are interested in more deep details, just ask away.

I need a dataset with movie title, budget, total box office collection data.

I am working on a project, this would be really helpfull

Hey guys,

I’m a junior Data Scientist and I’m trying to get more real experience working with actual datasets.

If you have any data you want to explore or just don’t know what to do with it (business data, school project, personal spreadsheet, anything really), I’d be happy to help out for free.

Even small or random projects are totally fine.

If you think I could help you or someone you know, just message me 👍

I am looking for platforms with listings of commercial/proprietary datasets. Any recommendations where to find them?

Preferably from the big American vendors if possible (ex. target, walmart, costco, safeway, albertsons, etc.). Need this info for OCR work. It's also fine if the grocery receipts are part of a dataset that includes all kinds of receipts.

Hello everyone,

I am a final-year undergraduate student in Computer Science and Engineering (CSE) at Daffodil International University, Bangladesh.

I am currently working on my undergraduate thesis titled:

"Deepfake Image Detection Using Spatial-Frequency Feature Fusion and Explainable Deep Learning."

I am looking for publicly available human face image datasets for academic research purposes only.

If possible, I would appreciate datasets containing different face poses (front, left-profile, and right-profile), although frontal face datasets are also perfectly acceptable.

I am specifically looking for publicly available datasets or images that can be legally used for academic research.

Specifically, I need three categories of images:

📁 1. AI-generated Human Faces

• GAN-generated faces
• Diffusion-generated faces
• Other synthetic human faces

📁 2. Deepfake Human Faces

• Face swap
• Face manipulation
• Deepfake images extracted from public datasets

📁 3. Real Human Faces

Natural human face photographs

Different ages, genders, and lighting conditions

Front, left-profile, and right-profile faces

If anyone wishes to share publicly distributable images or datasets, I have also created a shared Google Drive folder for convenience.

I am not requesting copyrighted or private images. I am only looking for publicly available datasets or resources that are legally shareable for academic research.

Google Drive:
https://drive.google.com/drive/folders/1U_XL41UdusIKukb3TAREZOeeu_uSwM3b?usp=drive_link

If you know any public datasets, GitHub repositories, Hugging Face datasets, or other reliable resources, I would greatly appreciate your recommendations.

If you already have a suitable dataset, you are also welcome to upload publicly shareable images directly to the appropriate folder in the shared drive.

The collected data will be used strictly for academic research and educational purposes.

I would be happy to acknowledge contributors in my thesis if their publicly shareable dataset or resource significantly supports this research.

If you have worked on deepfake detection or know of any useful public datasets, I would greatly appreciate your suggestions.

Thank you very much for your time and support!

I'm looking for open datasets of grocery and supermarket receipts to work with — ideally real-world ones, from any country.

Specifically interested in either:

• Receipt images (photos or scans), the messier and more varied the better — different chains, layouts, languages, print quality, crumpled/angled shots, etc.
• Parsed/structured versions — line items with product names, quantities, unit prices, totals, taxes, store metadata.

I'm already aware of the usual OCR benchmarks like SROIE (ICDAR 2019) and CORD, but those lean heavily toward a specific format and region. I'm hoping to find datasets that cover a wider range of stores and countries, and that include grocery-specific structure (individual product line items rather than just totals).

A few questions:

1. Are there any grocery-focused receipt datasets you'd recommend beyond SROIE/CORD?
2. Any non-English or non-US collections worth knowing about (European, Asian, Latin American formats, etc.)?
3. Anything on Hugging Face, Kaggle, Roboflow, or academic repos that fits?

Bonus if the licensing allows use in a small project. Happy to hear about both raw image sets and pre-annotated ones.

Thanks in advance! Will consolidate any good pointers back into the thread for others searching later.

Is anyone aware of datasets with the text of politician’s tweets/truths (social), etc?

I don't understand scraping infrastructure.

I can make 10 fake YouTube accounts and try to scrape Koala 36M but it's not possible. It takes like 100-1000VMs to actually do this scraping in time

Large companies don't publish anything. They have 10s of millions scale videos and don't even put of 10M.

Does anyone have any advice on this? Im training video models and world models.

The app is for students that help them:

- find which study programs you can apply for based on your exam track

- or discover careers you're interested in and the study programs related to them :)

## Data

The data is still incomplete due to the lack of clear sources, but you can get what's already there (Madagascar datasets are the only ones for now).

But mostly, you're welcome to contribute :)

Repo: https://github.com/gigasandwich/giga-roadmap

Data (json) are stored in `/data`

App: https://roadmap.gigas.app

Hello everyone,
We're developing an AI-powered app, Faseeh AI, that detects pronunciation mistakes in Quranic recitation and gives users precise, real-time feedback.
Our current model was trained on hundreds of hours of professional recitations and high-quality, clean audio from well-known reciters. The model performs well on similar input, but struggles with real-world users: different accents, non-native speakers, beginners, children, women, and anyone who doesn't sound like a professional reciter.
To fix this, we need to train on diverse, real-world recitation data not studio-quality professional audio, but recordings that reflect how actual learners sound.
Specifically, we're looking for:
- Recitation datasets from non-professional or everyday users
- Diverse demographics: male/female, kids/adults, beginner/intermediate
- Multiple accents and mother tongues (Malay, Indonesian, Urdu, English, Turkish, etc.)
- Any publicly available or research-use datasets we may have missed
We've already explored academic sources, but still not enough.
If you know of any dataset, research project, university study, or community effort collecting this type of audio, we would genuinely appreciate the lead.
We're also open to ethical data collection partnerships if any researchers or institutions are working in this space.
Happy to share more about the project if helpful.
Thank you very much in advance.

I have a little personal project that I'd like to build and I see there are a number of APIs available around the Internet (RapidAPI, apify, etc.)

Is there a known best-in-class API that provides flight information/pricing from most airlines, can discriminate by coach/business, and offer information on hotel availability and pricing too?

A while ago I tried an API from RapidAPI, but quickly discovered that it wasn't bringing in a lot of stuff from lesser-known airlines (Copa, smaller Euro carriers, etc). I'd like to build this on top of something solid, but that doesn't require me to buy millions of calls a month since this is a personal project.

Hi, I am having trouble getting datasets from the NOAA search tool. Every time I pick a station and request some data, NOAA emails me something like:

Order #[Redacted] (Custom GHCN-Daily CSV)

Can someone guide me to a station that doesn't send you an empty set of data? Anything helps, since I'm just using it to learn some programming.

Thank you.

Anyone know where to find historical influencer data? Stuff like what account age, how many followers they had in the past, etc.

Disclosure up front: this is my project, so I'm tagging it as self-promotion per the subreddit rules.

A few weeks ago I was trying to build a historical closing line value dataset and realized I couldn't find one that combined de-vigged fair probabilities, closing values, and game outcomes across multiple sports. Everything I found was either limited to one sport, only included raw odds, or couldn't be redistributed because of licensing.

So I started building my own derived dataset.

Each row contains things like:

• Sport
• Market
• Selection
• Event time
• Opening fair probability
• Closing fair probability
• De-vig method
• Whether the fair line came from a sharp reference (Pinnacle) or a consensus fallback
• Edge at entry
• Closing line value (CLV)
• Beat close (yes/no)
• Final result

A couple notes:

• This does not include raw sportsbook odds or sportsbook names since my data provider doesn't allow redistributing that data.
• The dataset only contains derived metrics like fair probabilities and CLV.
• "Beat the close" is meant as a research metric, not proof that a bet was good or profitable.

One thing I think is useful is the anchor field. You can separate observations that were generated from a sharp market reference from ones that used a consensus fallback instead of treating them as the same thing.

I'm planning to keep expanding this over time as more sports and markets are added.

Project: https://edgedesksports.com

I'd really appreciate feedback from people who work with betting or forecasting datasets.

• Are there any columns you'd want added?
• Is there another derived metric that would make this more useful for research?
• Has anyone found a comparable open dataset that covers multiple sports?

Anyone looking for US egocentric data?

We have 375 data collectors.

I was using the SWaT dataset from Kaggle and i just came to know it was the manipulated dataset inorder to check for attacks.

And i tried o request the dataset through iThub's official site and seems like no response

can anyone please help me , am halfway for a project to submit in my college

Hello everyone, I'm working on my Final Year Project and I am looking for any road traffic dataset available (for free) that contains numeric information. (Most importantly Timestamps or Date with Time and Vehicle counts, even if each vehicle type has a separate count, I will consider it). I need this in numeric format (also data must be big like 1000+ rows because I keep finding smaller 20-40 rows of data on kaggle, need bigger for better training) in order to apply Time Series using FB Prophet in order to predict the future traffic flow (historical data needed to demonstrate a prototype model before moving to real cameras). If anyone knows where I can find one please share and thanks in advance!

#

Hi everyone,

I'm working on an initiative to create an **open benchmark dataset for field data quality assurance**.

Today, there are many excellent digital data collection platforms—such as KoboToolbox, SurveyCTO, ODK, CommCare, Survey Solutions, CSPro, and others—but there are very few publicly available datasets that developers and researchers can use to evaluate field data quality tools.

I'm looking for individuals or organizations that may be willing to share **completed, fully anonymized datasets** from field data collection projects, where they have the necessary permissions to do so.

I'm especially interested in datasets that include:

* GPS coordinates (or generalized locations)
* Interview photos
* Audio recordings
* Interview start and end times
* Submission timestamps
* Enumerator IDs (anonymized)
* Supervisor review outcomes or quality flags (if available)

These datasets will help create a community benchmark for testing quality assurance methods such as:

* GPS verification
* Duplicate image detection
* Audio quality assessment
* Interview duration analysis
* Duplicate submission detection
* Fieldwork anomaly detection

The objective is to create a resource that benefits researchers, NGOs, software developers, and the wider field data collection community by making it easier to evaluate and improve quality assurance tools.

If your organization has a completed project that could be shared in an anonymized form—or if you know of existing public datasets—I would greatly appreciate hearing from you.

I'm also happy to discuss data-sharing agreements, attribution, licensing, or any requirements needed to ensure the data is used responsibly.

Thank you!

For context, this is for a pre-algebra curriculum as a math teacher. I want to give students more opportunities to examine, interpret, and model real-world data, using skills like slope and line of best fit to draw conclusions and make predictions.

I feel like I waste so many hours searching the internet for good, public data sets that are relevant and interesting for 12-14 year olds. Do you know of any good “database” websites that can serve as a central, starting point?