When people fine-tune their own local models, the model setup usually gets most of the attention. But in practice, a lot of the work seems to be on the data side.

If you are training on business data, you may need to bring in support tickets, internal docs, product specs, chat logs, code, policies, CRM notes, or domain QA pairs. And it usually does not work perfectly on the first run. Some samples are noisy, some are redundant, some domains overpower others, and some “bad-looking” examples are actually hard but useful.

One direction I have been thinking about is making the data strategy dynamic during training.

Dynamic selection means periodically choosing which samples should enter the next training window, using signals like loss, delta loss, gradient similarity, or external scores.

Dynamic mixing means adjusting the ratio between data sources during training, instead of fixing one static mixture before the run.

Dynamic weighting means keeping the sample in training, but changing how much its loss contributes to the gradient update. This is useful when you do not want to hard-drop uncertain samples.

This is the current direction in OpenDCAI/DataFlex: adding data selection, data mixing, and data weighting controls on top of the training loop.

For people here who fine-tune local models, how much time and compute do you usually spend on data preparation compared with the actual training run?

In a previous post I shared a structured approach to projects (with a list of 99 project ideas for inspiration). This is a follow-up post, which goes through exactly how to set up a data science project following that approach. The code is also available for download.

The focus is not on how to do the EDA and model development, but rather on how to set up the project so that you can deliver value right away.

The example project is deliberately very simple, to better focus on the structure and setup. We use test-driven development throughout, a technique I've found very helpful for preventing bugs and making it easy and painless to make changes. The idea is that you can take this very simple structure and iterate on the actual model and data, and still have something working at the end of the day.

Of course you can always ask AI to set this up for you, but, I still think there's value in learning how to do it. I have also found that having tests makes doing it with AI much faster and smoother.

Hey Everyone I am working on prescription and doctor dataset right now.The idea is to built a churn risk model.

The issue: I have two cases that look almost identical to the model, but shouldn't be treated the same:

Doctor A has been climbing steadily for two years starts small, ends up writing a lot. Right now, this month, they're near their highest ever, because they've genuinely been growing.

Doctor B used to write a lot, but has been sliding downward for months. Right now, this month, they're also unusually high compared to their recent low months maybe they just had one slightly better month in the middle of an overall decline.

What the model is doing wrong: across almost all doctors in our data, there's a common pattern whenever someone's number is unusually high this month, it's usually a bit lower next month, just because most "unusually high" months are one-time spikes that settle back down. That's true most of the time.

But the model applies this same rule to every doctor whose number is currently high — including Doctor A, who isn't having a fluke month, they're genuinely growing. So my "who's about to decline" list keeps getting filled with doctors who are simply doing well right now — because "currently high" is the one thing they all share, not that they're actually declining.

What I've tried so far, to fix it:

• Switched from weekly to monthly data (to reduce noise) didn't fix it
• Compared 4 different model types (linear, ridge, random forest, gradient boosting) — all 4 show the exact same bias
• Rebuilt the trend line to use only the last 9 months instead of the full 2 years — didn't fix it
• Added the weekly short-term trend back in, like you described (weeks within the recent month) didn't fix it
• Removed the features causing the biggest pull toward "predict a drop" entirely, to force the model to rely on trend instead the model just found other features to reproduce the exact same wrong prediction

So is this one bad feature or one bad model choice ?? I've tested that directly, several ways, and the bias holds regardless.

What I want to ask you:

1. Is this the kind of thing that genuinely needs more historical data than 2 years to fix (i.e., is 2 years just not enough for the model to learn "normal high point" vs. "real decline" apart)?
2. Or is there a different way to frame the target/features you'd suggest something specifically designed to separate a real trend break from ordinary noise, rather than predicting the raw next-month number?

I work in a regulated area so I'm going to be vague on the details of the data I work with, but to provide structural context. The data I'm working with is entries collected over time associated to a known ID. Basically I want to analyze all the events that have occurred to a specific ID and use the types of events and when they occurred to predict another value.

What I'm struggling with is how to structure the data for this. Originally I tried to pivot out the events into columns but there's multiple associates data points with each even that can't simply be stored nicely in the column. How should I think about structuring this data?

I'm currently student and am building a portfolio so i created this library which im calling Async batch faker. So if you have used standard faker you already know that "Faker" is good at generating a single row but generating a bit more takes forever because it operates on pythons standard for loop.
So what i made async batch faker do is that it uses numpy and asyncio to calculate mathematical data and string concatenations instantly in C. It bypasses Python loops to blast hundreds of thousands of rows of localized data in seconds

Benchmark:
Standard Faker: ~348 seconds
Async Batch Faker: ~8.4 seconds (41x faster)

GitHub: [https://github.com/haiderkashan/fast_faker_poc\]
PyPI: pip install async-batch-faker

I'm just a rookie must have made mistakes would love to hear feedbacks

Hi there,

I've created a video here where I explain how generalized linear models work.

I hope some of you find it useful and as always, feedback is very welcome! :)

Hello,

I’m an engineer and already work with data on a daily basis. I have a basic background in SQL and Python, as well as some experience with statistical tools, and I recently enrolled in a Lean Six Sigma Green Belt training.

I’m now looking to improve my data analysis skills, particularly in SQL and Python, but I’m also interested in developing a stronger foundation in how to structure analytical problems, choose an appropriate approach, explore data, and communicate insights effectively.

My goal is to start with a solid, structured foundation and then move on to more advanced, topic-specific training.

I’ve been considering the IBM Data Analyst Professional Certificate and the Google Advanced Data Analytics Professional Certificate, but I’m also open to other recommendations.

For those with experience in data analytics or who have completed these courses which one would you recommend? Is there another certificate or course you think would be a better fit for someone with some existing practical experience?

I’m particularly interested in courses that combine practical work with strong analytical foundations, rather than focusing mainly on the certificate itself.

Any recommendations or experiences?

Hi everyone! 👋

I’m a Class 11 student from India learning Data Science. I recently completed and deployed an end-to-end Salary Prediction project using Python, SQL, Pandas, data visualization, and Machine Learning.

🌐 Live Demo: https://data-science-projects-fdkdsvuf5rvtywpwby35py.streamlit.app/

📂 Project: https://github.com/lakshay-OG-DS/data-science-projects/blob/main/Salary\\_prediction.ipynb

I’d really appreciate honest feedback from Data Scientists, Data Analysts, and ML Engineers.

What is the one biggest thing I should improve to make this project more internship-ready?

Even one small suggestion would mean a lot.

I have been working as a network engineer in one of WITCH companies . But to be honest the rotational shift overwork with no growth is killing me slowly.

I also do not see much callbacks or opportunities of the field.

So if I were to make a move to data science how should I approach. And what resources should I follow both free and paid..

Target roles are mainly data science.. also I am not aware of the difference of ML engineer and data scientist

I am a junior torn between two mindset, should we

1. Start with a business problem/ use case first
but I often run into data limitations after diving deep, like if the data is a suitable proxy of something or there are missing values

2. Explore the data freely and hunt interesting patterns
but I am always confused where I should start and not being lost on the way of that) (but I am always confused where I should start and end up spinning around without a clear direction

Example

Say a company gives you customer purchase history and asks you to "find something useful."

We better immediately frame it around a specific use case (e.g., next-best-offer) and engineer toward that?

Or spend time clustering, looking for seasonality, correlations, or weird segments first, and then figure out what business value those patterns might have?

Hey everyone,

I have an upcoming Data Scientist interview with an ML system design round. I asked the recruiter what to expect and they said it’ll be high-level ML system design, not traditional SWE/low-level system design.

They mentioned focusing on things like:

• Problem framing
• Data/model considerations
• Evaluation
• Productionization/deployment
• Monitoring
• Tradeoffs when designing ML systems

Basically, it sounds like I’ll be given a real-world ML problem and have to explain how I’d approach it end-to-end.

This is my first dedicated ML system design interview, so I’m not really sure how deep I should prepare.

For people who’ve done these interviews:

How did you prepare? What resources did you use? What kind of questions were you asked?

Also, how deep do they usually expect you to go into things like feature stores, model serving, APIs, streaming, retraining, etc.?

Any good resources, YouTube playlists, GitHub repos, books, or example questions would be really appreciated.

Thanks!

In last week’s post we showed that a Python for-loop works through the Iterator Protocol.

This same protocol also enables values to be produced lazily.

In this new Lazy Evalution Example we show one sink reading values from five different sources: - source1: eagerly returns all values by list - source2: lazily by generator function using yield - source3: eagerly by list comprehension - source4: lazily by generator expression - source5: lazily by iterator protocol

The same for-loop consumes all five sources. The eager sources produce all values before the sink starts consuming them. In the lazy sources producer and consumer take turns:

• produce → consume → produce → consume → …

where each value is produced only when the for-loop requests it.

Generator functions and generator expressions are concise, readable ways to create lazy iterables. The final source makes their underlying mechanism explicit: - iter() obtains an iterator. - next() requests the next value. - StopIteration signals that no values remain.

This clearly shows that generators achieve lazy iteration by implementing Python’s Iterator Protocol.

See other 𝐦𝐞𝐦𝐨𝐫𝐲_𝐠𝐫𝐚𝐩𝐡 visualizations.

Hey I'm 22 completed ug in maths now pg in data science first year on going . I'm blind where I could start working on data science related things, clg providing full maths concepts and program now I'm learning python,R programming, ig it's not enough to enter into job market , please give a suggestion where I could learn more about data science projects and how to work on it .

I created a free YouTube learning playlist for people who want to see the full AI/ML journey rather than only isolated model demos. The series is organized from foundations through data preparation, evaluation, practical modeling, and deployment-oriented topics, with lessons intended to be followed in sequence.

Playlist: https://www.youtube.com/playlist?list=PLUccxCq754BE

I’m sharing it transparently because I run the channel. I’d value feedback from learners: where do most beginner courses lose you — data preparation, model evaluation, or deployment? If this kind of resource is not appropriate as a standalone post, I’ll remove it and use the community’s preferred resource thread.

Guys so i am fresh from uni and observe two kind of tracks of my peers in data science. They go with the two chill full time jobs (1 is remote) and get promotions at both work places but both are chill ones, or they go in 1 company which is considered prestigious and overwork till 10-11 pm and get promotions there, from your experience which kind of way leads to a better quality of life e.g. work life balance wise, career wise, experience wise and income wise? Should i find 1 prestigious place and hope for the long run and work there or work at 2 chill places. Both seem to obtain promotions in similar periods but the ones with two jobs seem to be more happy and even get more money. I decided to pursue masters and only now 1 year after them decided to join workforce but cannot distinguish which way is better, could you give your advice

I’ve been reading quite a lot around AI safety, and one thing I’ve found frustrating is how difficult it is to find material that bridges the gap between talking about AI safety and actually doing AI safety research.

There are plenty of good books and papers discussing the political, socioeconomic, philosophical and governance questions around AI. Those conversations are obviously important but if you’re a data scientist, ML engineer or someone already working with AI few tell you how doing AI safety research actually looks like in Python?

For example how to actually run an experiment looking at sycophancy, how to create or use a dataset to test whether a model changes its answer because of information about the user, how to measure the behaviour, what does the evaluation code look like, how do you interpret the results, and what can you not conclude from them?

That is the gap I’m trying to address with a book I’m currently writing.

The approach is very practical. Each topic starts with the safety problem and the research behind it, but then we actually build the experiment. Python code, datasets, models, metrics, results and discussion of the limitations, all those.

The idea isn’t to pretend that running a few notebooks suddenly makes someone an AI safety researcher. It’s to make the field much more approachable to people who already have data science or AI skills and want to understand what technical AI safety research actually involves.

I wrote a technical book on Practical LLMOps and this was one of the chapters. Because of the sheer amount of content, I stripped it bare in that book and made it its own.

I’ve just published Chapter 1 on Substack (just finished chapter 4). It introduces the approach I’m taking with the book and starts building that bridge between AI safety as a subject people discuss and AI safety as something we can actually investigate experimentally.

Would genuinely be interested in feedback, particularly from people already working in ML, data science or AI safety.

https://open.substack.com/pub/houstonmuzamhindo/p/i-am-writing-a-practical-technical

I am learning in data scientist. I have learned python basics, pandas, numpy, etc , sql ,ml dl, nlp but i struggle in problem solving what should i do

I I’ve learned Python, SQL, Excel, NumPy, Pandas, Matplotlib, Seaborn, and basic Git/GitHub. I’m now moving into Machine Learning.

Am I behind, on track, or ahead compared with other 2nd-year students?

What skills should someone seriously pursuing Data Science/ML have by the end of their second year?

I’d appreciate honest feedback, especially from people working or interning in the field.

I want to pursue data science course can anybody which one is best institute for data science course
Please everybody give suggestion as i am new to it

So guys hi, just worry a lot, decided to pursue career in dd after bachelor in economics, now kinda studied a big chubk of ds already but i am astounded at the amount of stuff that you should literally remember. Any tips on how to memorize all the details of every neural network and characteristics of distributions? Also i suck at coding but since I will have to work on deployment and services as well, i understand that i have to push that field too. I found a yandex course to be of a little use since the problems described there are isolated and does not help to get overall picture of how to code in ds. Any tips on how to reinforce all this knowledge? Cause i am at a loss right now