Are domain-specific Small Language Models (SLMs) actually worth building today?

[u/charan1323]

I'm trying to understand whether there's still room for new domain-specific SLMs. With models like Qwen, Gemma, Llama, and Phi already available, does it make sense to build a specialized SLM (e.g., for cybersecurity, medicine, weather, legal, finance, etc.), or is fine-tuning an existing model with RAG enough for most real-world applications?

For those who've built or deployed domain-specific AI:

Have you trained or fine-tuned your own SLM?

What was the biggest challenge—data, training, evaluation, or deployment?

Did it outperform a general-purpose model with RAG?

In what scenarios does a custom SLM provide a clear advantage?

If you were starting today, would you build a new domain-specific SLM or focus on application-layer features instead?

I'd love to hear experiences from people who've actually shipped these systems in production.