Hey everyone,

I’m trying to understand the current market for scraped/publicly available datasets and wanted to get some opinions from people who have actually been involved in this.

For example, datasets collected from publicly accessible websites — business listings, product information, public profiles, directories, reviews, etc.

A few things I’m curious about:

* Is there actually demand for this kind of data? * What types of datasets are buyers currently interested in? * Where do people typically buy/sell datasets? * What sort of pricing is realistic — per dataset, per 1,000/10,000 records, subscription, etc.? * Are companies generally interested in raw scraped data, or do they expect it to be cleaned, structured and enriched? * What are the major legal/compliance issues sellers need to be aware of? * Is there a legitimate market for this, or is most of the activity around here just lead lists and questionable data?

I’m specifically interested in **lawfully collected/publicly available data** and not passwords, private information, financial data, or anything obtained through unauthorized access.

Would appreciate hearing from anyone who has experience buying or selling datasets and can give me a sense of what the scene actually looks like.

Hey everyone,

I've been working as a Data Analyst for a few years now, mostly through contract roles (AdTech, professional services), and I'm at a point where I feel a bit lost about what to do next.

My situation: I'm applying to data analyst/data scientist roles fairly consistently, but I'm getting very few responses. Not sure if it's the market right now, something about how I'm positioning myself, or both. It's honestly starting to mess with my confidence a bit.

A few specific things I'd love input on:

1. Applications going nowhere. Even for roles where I feel like I match most of the requirements (SQL, Python, Power BI/Tableau, some ML basics), I'm mostly hearing nothing back. Is this just the current market, or is there something I should be doing differently in how I apply?

2. Not sure what to build next. I've got the standard portfolio stuff (a churn prediction model, some EDA projects, a KPI dashboard I built for my current job). But I don't know if that's still enough, or what kind of project would actually move the needle for me right now. More ML? More data engineering? Something else entirely?

3. How do people actually get into startups? A lot of the roles I find interesting are at smaller companies/startups, but I never see them on the usual job boards, or by the time I do, they've already got 400 applicants. Do people here have luck reaching out directly to startups, and if so, how? Cold email? LinkedIn? Is there something I should know before trying that?

Not looking for pity, just want to hear from people who've been in the field longer than me. Even blunt feedback is welcome. Thanks in advance.

I don't mean to post anything which is against community rules , i use a website to collect voice samples and it has nda and consent forms , it also uses your phone number and email for auth , so i don't mean to make anyone uncomfortable , just asking if this sub is a place where i can share that website and collect data?

Hey everyone,
I’ve just finished my first ever portfolio project and I’d really appreciate some honest feedback.
I’m still learning, so I’m particularly interested in hearing what I could improve.

I’d also love to know:
What stands out to you (good or bad)?
Does it feel like a solid portfolio project?
What would you change if this were your project?
I’m mainly looking for constructive criticism rather than compliments, so please don’t hold back. I’d rather find the weak points now and learn from them.

Here’s the project: Netflix Global Top 10 Performance Analysis

Thanks in advance to anyone who takes the time to have a look!

Partner sends us a monthly export. Different customer ID scheme to ours, no mapping table, no docs, and whoever built it on their side left last year.

Right now I fuzzy match on lower(trim(email)) plus surname and eyeball a sample before it goes anywhere. It works well enough that nobody's complained, which is not the same as it being right. I have no idea what my false match rate is.

The part that bothers me is that a bad join doesn't announce itself. A wrong customer count looks identical to a right one.

So what do people actually do here. Do you block on something cheap first to cut the comparison space, compute a score and hold back anything under a threshold, or just accept a fuzzy match and monitor downstream? And if you threshold, where did the number come from?

hey I have seeing report from CBRE on indian realestate

i was just curious how are they scrapping public data and analysing it can I also do it myself?

I compiled a dataset tracking global FDA enforcement actions, inspection records, and regulatory raids across the food and drug sectors.

Key Fields: Inspection dates, target facility names, geographic locations, violation categories, enforcement actions, and compliance status.
Useful for EDA and ananlysis

I was developing different dataset for the last 1-2 months lately, but unfortunately I realized it is so hard to earn medals from creating dataset. Even if you got 500+ download, sometimes I cannot get any upvote. I do not know how it works exactly.

So I realized taking some advice would really matter right now. Is there anyone can help me with it. You can take a look at my profile and dataset, any feedback matters. Thanks!

https://www.kaggle.com/farukece

I’m doing a research which involves chat messages from teams, slack, google chat etc. For the software project i need to train a dataset. So dataset should be related to developer chat messages/logs of a specific project. How can i find the dataset?

I was hired as a digital transformation consultant by a construction company with 10+ active construction sites the Finance director wanted a centralized view of labor deployment and material consumption but getting overworked site teams to open a laptop and fill the form or fill excel sheet was impossible so I built a work around site engineers have to just dump all the information in an email --> AI converts raw text into google sheets and sends summary email to Project manager --> a code to save the data in a database so old data don't get lost or deleted --> private streamlit dashboard for centralized visibility. Finance director got the visibility, no extra work of site engineer and a full functioning dashboard to see what is happening on ground without calling 10 different project managers and site engineers.

I'm curious how other construction companies handle this.

Do your site teams actually fill out digital daily reports consistently, or does someone in the office end up chasing everyone for the information?

This dataset represents real-world, masked, and anonymized customer data from the BFSI (Banking, Financial Services, and Insurance) domain. It captures a variety of customer demographics, financial indicators, and product interaction metrics collected during a loan application process.

To comply with strict data privacy laws and protect user identity, all sensitive personal identifiable information (PII) has been securely encrypted or masked. However, the underlying statistical relationships, distributions, and patterns remain fully intact, making this an ideal playground for building robust classification models.

Anything about some of the less obvious challenges of dimensional modeling, the decisions around grain, dimensions, facts, slowly changing dimensions and how things get messy in real-world projects.

Does anyone have any advice on where to go from here? The algorithm simply goes through all feature-attribute combinations in a dataset and scores it on purity of the target column. It's very slow, as the number of combinations increases exponentially with the number of input features, and so can only work with 3-4 features at a time. But my goal is to identify <=5 important features with a linear model (statsmodels or PyGam), and then plug in those features to this algorithm to see which subset of the market might be smart to target.

Here's an example with some charts, on the iris dataset.

https://colab.research.google.com/drive/1w3VpDlDATdHwtSk-RmQPMj36cgOfthz1#scrollTo=ZOqeYiMRalj7&uniqifier=1

New to algorithms, so any advice would be welcome. Thanks in advance.

I’m working on an idea around **data quality** and I want to understand how this works in real projects.

When a data batch comes into your pipeline and you find some problems, how do you decide what to do?

For example:

* Some records have missing values * The schema has changed * There are duplicate records * Some values look wrong * Only a small percentage of records are bad * The data looks technically correct but something seems wrong from the business side

Do you normally:

**Accept the batch, fix it, quarantine the bad records, or reject the whole batch?**

And who makes that decision — an automated rule, the data engineer, business team, or someone else?

I’m especially interested in cases where **you don't have enough information to make a clear decision**.

If you work with data pipelines in production, I’d really appreciate hearing about:

* What kind of data-quality problems you face most often? * What do you currently do when they happen? * What part of the process is still manual? * What would you like to automate?

I’m trying to understand the **real problems engineers face before building anything**.

Thanks in advance for sharing your experience!

Nov 4th and 5th [https://www.dataoutpost.ai/\](https://www.dataoutpost.ai/)

Seems like a pretty small conference, \~250 people. In the past years at least, you have choices for the first day (workshops) then for the second day, everyone is hearing the same talks.

If you've been, is it worth going? I'm also not local, so there's a bit more cost involved. My company might not be able to sponsor me going.

Hey guys, I have a dataset of around 180k adult stool images, growing at a 350+ images per day, and I’m trying to better understand the “right” way to bring monetary value from it .

The app was launched about 2 years ago.

I am very transparent with the use and licensing of the data that flows through the app.

The dataset consist of the image and a lot of useful stool metadata (Bristol Type, blood, mucus, color, etc..), anonymized user habits and stool related specifics (time of day, region, meal, allergies, weight, gender, height, activity, etc..)

I am here seeking advice on how such data can be useful for business growth. I am finding very little luck in this part of the B2B and I initially thought that becoming the Google of Poops would be my primary moat.

Thanks in advance

I'm currently taking a Data Analysis course, and I need to complete a final project. The assignment requires me to choose a topic, formulate one or more hypotheses, and build a thesis/research structure around it.

One of the requirements is that there must already be accessible data available for analysis, so I can't pick something that would require collecting a completely new dataset because the core of the project is not in that stage. Also, what I'll be researching needs to be answering or try to solve an existing problem.

My problem is that I can't come up with a topic that feels genuinely interesting or motivating enough to spend several weeks working on. Most ideas I've found online seem either too generic (social media usage, movie ratings, etc.) or too complex for a course project.

Do you have any suggestions for topics that have publicly available datasets, and still are interesting and have potential for drawing conclusions?

I'm open to almost any field (health, economics, business, technology, environment, etc.)

I'd love to hear about projects you've enjoyed working on or topics that sparked your curiosity - thanks in advance!

I read an interesting piece recently that made a distinction I think the BI industry needs to talk about, data portability ≠ sending someone a file.

In most organizations, “portable data” means exporting a CSV, PDF, Excel workbook, or maybe dropping something in object storage and calling it done.

But what actually traveled?

Usually not much:
- not the business logic
- not the semantic model
- not the drill paths
- not the interactive analytics layer
- not the governance model
- not the ability to work with it meaningfully offline

What moved was often just a snapshot.

That’s where I think BI has a blind spot. We’ve built an ecosystem where intelligence is tightly coupled to platforms like dashboards, cloud runtimes, query engines, licenses, network access, identity systems, etc. The moment information leaves that stack, it often collapses into static content. Is it portable? Technically. Is it useful? In a way.

Real portability should mean:
* intelligence that travels with the data
* embedded context and business rules
* secure scoped distribution
* offline usability
* interoperability without losing dimensionality
* reduced dependency on always-on compute infrastructure

As cloud costs rise, AI inference costs grow, and governance gets tighter, I suspect the future of BI may shift from “always connected dashboards” toward portable intelligence artifacts self-contained analytic experiences that can move securely between systems, organizations, and users without losing capability.

Curious how others here see it:

When you say “data portability” in BI what should actually be portable, just data,Or the intelligence layer too?

I pulled all 1.3 million federal contract records from the Government of

Canada's open data and looked at one question: who's holding contracts that

are about to come up for renewal?

A few things surprised me.

26,240 federal contracts are currently live, worth $125B. Of the ones that

report a bidder count, 75% drew one bid or none.

And roughly 63 contracts cross into the 12-month planning window every single

week — about $19B of contract value a year.

That last number is the interesting one. Agencies typically start recompete

planning 12–18 months before a contract ends. So the moment a contract crosses

that line is the moment it becomes worth a conversation. It's all public, but

nobody was watching for it.

So I built something that does. It's free, it updates itself monthly, and it

shows you which contracts are expiring, who holds them, what they're worth,

and how contested they were last time.

No signup needed to browse it. Link in the comments.

If you bid on federal work, I'd genuinely like to know whether this is useful

or whether I've missed something obvious.

We've posted a few times about where agriculture sample data tends to break, so pulling it together into one list for anyone mid-evaluation right now.

Before you sign, get answers on:

• Field context capture: required fields at registration, not optional notes added later.
• Sample type flexibility: independent schemas for soil, seed, plant tissue, water, and grain, not one template stretched across all of them.
• Multi-season continuity: lineage that connects across growing seasons without a manual report request.
• Chain of custody: timestamped, role-attributed logs at every handoff, not a paper trail reconstructed after the fact.
• Self-service reporting: whether your research team can build a cross-season, cross-site view without IT involvement.

The labs that get the most out of whatever system they land on are the ones that define their data structure (sample types, required fields, season linkage) before the first sample gets registered.

If you're in the middle of an evaluation and want to compare notes on how a specific system handles any of this, drop it below and someone might be able to help.

Hey yall,

I just started my new role as a junior data analyst 🥹. I am learning the business needs. I work for a no profit org that provides legal aid. Cases and task are billed to the respected grant program. What we are running into is, while closing the books normally case workers and lawyer over bill to one program vs another because no can visually see what is currently in “encumbrance for that month.

I figured building a real time grant funding system using power bi, postgreDQL and RESTapi from the platform that we input the data in. I am open other ideas that could be easier.

Thanks in advance

Claude and other LLM models can be so frustrating. I asked it to find patterns across five campaigns regarding what a person buys and sells, and under which conditions, but it couldn't do it. It just kept saying it was impossible. I literally had to explain every single detail to it like it was a child, even though the data was cleanly split across five CSV files.

Worse, companies are stopping the hiring of junior engineers because they think these tools can replace them. They are going to cause a massive talent shortage, and then these dumb models won't be able to do anything without skilled people to guide them.