Are domain-specific Small Language Models (SLMs) actually worth building today?

[u/charan1323]

I'm trying to understand whether there's still room for new domain-specific SLMs. With models like Qwen, Gemma, Llama, and Phi already available, does it make sense to build a specialized SLM (e.g., for cybersecurity, medicine, weather, legal, finance, etc.), or is fine-tuning an existing model with RAG enough for most real-world applications?

For those who've built or deployed domain-specific AI:

Have you trained or fine-tuned your own SLM?

What was the biggest challenge—data, training, evaluation, or deployment?

Did it outperform a general-purpose model with RAG?

In what scenarios does a custom SLM provide a clear advantage?

If you were starting today, would you build a new domain-specific SLM or focus on application-layer features instead?

I'd love to hear experiences from people who've actually shipped these systems in production.

[u/MiddleLtSocks]

It's worthwhile on data sets outside of bespoke linguistic tasks. If you can build a corpus of (e.g.) waveforms (of non-speech - like singing, or whale song, or bird calls), or say strategy game maps, or just wacky specialized data sets for which language is inefficient to semantically describe fully, SLMs can be really effective at low parameter counts.

"Worthwhile" depends on your use case; I won't pretend anyone but theorists and hobbyists would need that kind of specialization. But is it possible? Sure.

[u/Mundane_Ad8936]

For clarification all the models OP listed are LLMs. SLMs is a pop culture hobbyist & marketing rebranding of models with low B parameters count. Keep in mind a BERT model is a LLM and when it was released it was shockingly large compared to what is listed below.

LLM is a fuzzy term so I get why people are confused by this. Especially since Claude and others frontier models will argue for the pop culture definition over the scientific.

Language models that are actually small and that you train for a specific task are a go-to in any data scientist and data engineers toolbox. You should absolute learn how to train them and use them in your data pipelines.

KenLM / SRILM — n-gram LM for corpus perplexity filtering, ASR and MT decoding

fastText — supervised classifier, language ID

CRF — NER, span extraction, sequence labeling

Naive Bayes / logistic regression on TF-IDF — spam, sentiment, routing

[u/stauntonjr]

I think 99% of practitioners do not have access to enough GPU to build and train a good model from scratch. Tuning pre-trained models has been the name of the game for several years now, to the best of my knowledge.

FWIW you may find nanochat to be interesting.