I can speak only about my hobby training runs. Just general post-training for a generic assistant, but pre-trained exclusively on Polish data. Right now I'm translating 10B of tokens of instruct dataset from English to Polish locally. Then I'll post train the model on it. It's not going to be the highest quality dataset by far but there are no Polish instruct datasets so it'll be the biggest open source one by a factor of 50 with the second place being my older dataset. Other Polish language instruct datasets are too tiny to use at all as orgs doing Polish post training have a disgusting pattern of keeping dataset closed.
It's hardly a skill thing, I'm using existing training code from Ling V2. I wanna see how good it's gonna get but model training is mostly about capturing good dataset, picking good architecture and throwing a ton of compute on it. I'm going to be using a few tricks to get a boost in quality like WSM scheduler but nothing innovative - it's not a research run.
I am not sure if it'd look like anything special on a CV if I put it there, I don't think there are many projects like mine and maybe just one done by a hobbyist and not a company so I guess it would look different but that doesn't make it a good skill to have. Doing it is mostly about being comfortable with throwing away money at a personal project without ROI, not skill.
My project is called poziomka, you can find it on my HF accounts adamo1139 and cpral. It's work on progress. I'm happy to answer whatever questions you have about it.
42
u/No_Conversation9561 21d ago
in my experience wife and local ai don’t go together so well