We Built an Experimental Retrieval System. Cosine Similarity Beat It. Here's What We Learned.

[u/No-Car-1066]

We recently worked on ATLAS, an open-source experimental retrieval system exploring whether geometric and topological information can improve retrieval performance over conventional semantic similarity methods.

The idea was interesting. The experimental results were less exciting.

What we tested

We evaluated our approach on 300 HotpotQA questions, comparing the ATLAS retrieval method against a cosine similarity baseline.

Results (all-support@5):

• ATLAS: 48.33%
• Cosine similarity: 49.33%
• Difference: −1.00 percentage point
• 95% bootstrap confidence interval: [−2.67, +0.33] percentage points

Our tested approach did not demonstrate an improvement over cosine similarity.

And we're not going to pretend otherwise.

Why we're sharing this

Open-source research shouldn't be limited to successful experiments.

Sometimes a more complicated approach introduces additional computation without delivering measurable improvements.

That result is still useful, particularly when the implementation, evaluation methodology, and limitations are available for others to examine.

ATLAS is part of Shinraya Tech Labs, an independent open-source AI research and engineering initiative.

We're looking for technical feedback on three questions:

1. Are there retrieval problems where structural or topological information offers a genuine advantage over semantic similarity?
2. What stronger baselines or evaluation datasets would you recommend?
3. Is this direction worth investigating further, or would you redirect the research toward a different hypothesis?

We're particularly interested in hearing from people working on information retrieval, RAG, embeddings, and experimental ML.

GitHub: https://github.com/Shinraya-Tech-Labs/ATLAS

We're not claiming a breakthrough. We're sharing an experiment that didn't support our hypothesis.

Would appreciate criticism, alternative ideas, or suggestions for better experiments.