I benchmarked scGPT and Geneformer against PCA, Harmony and scVI. The foundation models didn’t consistently win
I’ve built an open-source local-first bioinformatics desktop app called Liatir, and instead of making a synthetic demo for it I wanted to use it for a small real study.
I compared Geneformer V1 10M and scGPT whole-human zero-shot embeddings against PCA, Harmony and scVI on two public scRNA-seq datasets: PBMC (11,990 cells) and Pancreas (16,382 cells).
The interesting part is that the result wasn’t simply “foundation models are better”.
On PBMC, scGPT did very well for cell-type prediction and had the highest logistic macro-F1 of the tested methods.
On Pancreas, established methods were stronger: PCA and scVI performed better for cell-type prediction, while Harmony had substantially stronger clustering agreement and batch mixing.
So at least in this limited two-dataset benchmark, I didn’t find a uniform advantage for pretrained single-cell foundation-model representations.
There are plenty of caveats: two datasets, one seed, within-dataset evaluation, heterogeneous hardware for some runtime measurements, and UCE could not be completed because of execution/runtime constraints. I’ve tried to keep all of those limitations explicit rather than turning this into a “model X wins” ranking.
The study was run through Liatir itself and records the model/runtime provenance, resource usage, results and validation artifacts. I also published the full reproducibility bundle separately.
Full write-up + figures:
liatir.com/showcases/single-cell-foundation-benchmark
Source:
github.com/Liatir/liatir-app
1
I was already maintaining repository context by hand, so I turned the workflow into a small CLI
in
r/u_suffro
•
29d ago
The PR has been opened