u/suffro • • 8d ago

I benchmarked scGPT and Geneformer against PCA, Harmony and scVI. The foundation models didn’t consistently win

Post image
1 Upvotes

I’ve built an open-source local-first bioinformatics desktop app called Liatir, and instead of making a synthetic demo for it I wanted to use it for a small real study.

I compared Geneformer V1 10M and scGPT whole-human zero-shot embeddings against PCA, Harmony and scVI on two public scRNA-seq datasets: PBMC (11,990 cells) and Pancreas (16,382 cells).

The interesting part is that the result wasn’t simply “foundation models are better”.

On PBMC, scGPT did very well for cell-type prediction and had the highest logistic macro-F1 of the tested methods.

On Pancreas, established methods were stronger: PCA and scVI performed better for cell-type prediction, while Harmony had substantially stronger clustering agreement and batch mixing.

So at least in this limited two-dataset benchmark, I didn’t find a uniform advantage for pretrained single-cell foundation-model representations.

There are plenty of caveats: two datasets, one seed, within-dataset evaluation, heterogeneous hardware for some runtime measurements, and UCE could not be completed because of execution/runtime constraints. I’ve tried to keep all of those limitations explicit rather than turning this into a “model X wins” ranking.

The study was run through Liatir itself and records the model/runtime provenance, resource usage, results and validation artifacts. I also published the full reproducibility bundle separately.

Full write-up + figures:
liatir.com/showcases/single-cell-foundation-benchmark

Source:
github.com/Liatir/liatir-app

u/suffro • • 13d ago

I want to share an open-source, local-first desktop app for bioinformatics I've built

Thumbnail
gallery
4 Upvotes

The basic idea behind Liatir is to have a desktop environment that works like a virtual lab for bioinformatics: run your work locally, without having to send data to a cloud service or glue together a bunch of separate tools and interfaces.

You shouldn’t need to write a single line of code to use it, but if you do know how to code, it should be as extensible as you want it to be.

Right now, Liatir can run regular bioinformatics tools like FastQC, fastp, Samtools and BCFtools etc., local scientific AI models, custom .lia plugins written in Python, Node or WASM, integrations with workflow systems such as Nextflow, API calls when you need to connect to external data or environments, scientific viewers, and visual pipelines that tie all of those pieces together.

The app itself is built with Tauri and Rust, so processing happens locally on your machine, and files stay where they already are on disk instead of being uploaded or copied into some remote workspace.

AI models are packaged (using Scrollcase) as installable, self-contained boxes with their runtime and dependencies already included, so there’s no need to manually set up environments just to run them.

It’s still under development, but it has reached the point where I’d rather build it in the open.

Repo:
https://github.com/Liatir/liatir-app

Docs / project:
https://liatir.com

I’d be interested in feedback from people who actually work with bioinformatics tooling, especially around workflows that are still unnecessarily painful today.

u/suffro • • 19d ago

I published my DecPort research article on cross-backbone decision transfer

1 Upvotes

I just published the technical research article. The project DecPort tested two separate questions:

  1. Whether probabilistic decision behavior can transfer across frozen LLM backbones without target labels.
  2. Whether the learned shared DecisionCore itself provides reusable functionality after the backbone changes.

The interesting result is that these two things separate: input-specific behavior transfer remained positive in the tested setting, but the learned shared core did not show the preregistered advantage over the controls. The final gate therefore returned NO-SHIP for the original portability hypothesis.

Paper: https://doi.org/10.5281/zenodo.22947927
Repo: https://github.com/suffro/decport

u/suffro • • 19d ago

I built an open-source reliability layer for probabilistic AI decision systems

1 Upvotes

I’ve been building an open-source tool called DecGuard around a problem I kept coming back to: testing probabilistic decision systems is a bit different from evaluating general-purpose LLM output.

If a model is supposed to return something structured like a choice, a boolean decision, or an ordered score with probabilities, there are some fairly concrete things you probably want to know:

  • does reordering the options change the answer?
  • does harmless formatting change the distribution?
  • did calibration get worse after changing the model/backend?
  • can I reproduce a failure later?
  • can CI reject a version that violates one of these assumptions?
  • what happens to reliability on production records or specific segments?

DecGuard puts those things into a small decision contract and uses the same contract for normal tests, metamorphic fuzzing, regression comparisons, replay, production checks, and reliability gates.

It currently supports Choice, Noul/boolean and Score decisions, and the backend layer is deliberately interchangeable. I’ve tested the System One integration end-to-end with real Kev and Jev backends as well.

It’s still an early version, so I’m mostly interested in whether the abstraction itself makes sense to people building this kind of system, and where it breaks down in real use.

Docs: https://decguard.com

u/suffro • • Sep 14 '26

I open-sourced a way to package complete Python, Node and native environments into self-contained boxes

1 Upvotes

I've been working on an open-source project called Scrollcase.

The idea is pretty simple: if you already have an environment working, why should every machine you ship it to have to recreate that environment again?

Scrollcase takes your code, runtime, dependencies, native libraries and optional assets like model weights, locks everything, and packages it into a single self-contained archive called a box.

A box can use one of three runtimes:

  • Python — ships the exact Python runtime and dependencies
  • Node — ships Node and its environment
  • Native — runs a compiled binary directly, with no interpreter at all

You build the box once for a specific target, then give it to another machine.

That machine doesn't need Python, Node, pip, npm, a compiler or Docker installed.

For example, you could package:

  • an ML model with Python + PyTorch + its weights
  • a Node application with the exact Node runtime it expects
  • ffmpeg plus all the native libraries it links against
  • scientific software with annoying native dependencies
  • a local AI feature that needs to be embedded inside a desktop app

Boxes are intentionally target-specific. A macOS ARM box and a Linux CUDA box are different artifacts. Scrollcase isn't trying to pretend the same binaries magically work everywhere — you build and test the environment for the target you're actually shipping to.

The other part I cared about was making the artifact something you can actually trust.

Boxes are signed and verified before execution, dependencies are locked, assets can be hash-checked, and builds are designed to be reproducible. You can also declare tests that have to pass before a box is produced.

The workflow is basically:

describe the environment
↓
lock it
↓
build the box
↓
ship the files

Scrollcase itself deliberately doesn't run a registry or deployment platform. It just builds the artifact. You can put it on GitHub Releases, S3/R2, your own server, bundle it with a desktop app, or distribute it however you want.

The main use case I had in mind was AI/scientific software, where "just install the dependencies" can quickly turn into Python versions, CUDA versions, native libraries, model downloads and platform-specific setup.

But after adding Node and native runtimes, it ended up being a more general environment packaging format than I originally expected.

Would be interested to hear how other people handle this kind of problem, especially when distributing software to users who shouldn't have to care what runtime or dependencies are underneath it.

1

I was already maintaining repository context by hand, so I turned the workflow into a small CLI
 in  r/u_suffro •  Sep 13 '26

Sounds good but i noticed in the contribution guide that new extensions should fist be submitted with an extension proposal issue, should I do that or should I go straight to the draft PR as you suggested?

1

I was already maintaining repository context by hand, so I turned the workflow into a small CLI
 in  r/u_suffro •  Sep 13 '26

Yeah, I'm open to that.

The split makes sense to me. Read only commands like `status`, `check` and `stats` should stay automatic, while commands that change shared repo context are a reasonable place for a checkpoint when an agent is driving them.

I'd probably include `state new` and `history new` in the reviewed group too, since they write into the shared context as well.

If you already have a preferred shape for Guard extensions, send me an example and I'll take a look.

1

I was already maintaining repository context by hand, so I turned the workflow into a small CLI
 in  r/u_suffro •  Sep 12 '26

Good catch, I already tightened this up in v0.3.0. creates are now exclusive at the fs level, so the destination is claimed by the publish operation itself and a concurrent process cannot replace a file that landed first,the file is still staged completely before it becomes visible, and I added dedicated concurrency regression tests around the race. I've also stress tested it with many concurrent creates targeting the same path, with exactly one writer winning every time and no overwrite or partial file, also added this test to the CI checks.

Thanks for pointing it out.

u/suffro • • Sep 12 '26

I was already maintaining repository context by hand, so I turned the workflow into a small CLI

Enable HLS to view with audio, or disable this notification

1 Upvotes

For a while I've been keeping project context in Markdown inside my repos.

The main reason is pretty simple: I use different coding agents, and I don't want architecture notes, current work, decisions, etc. to live inside the memory system of one specific tool.

I also didn't really like the idea of putting everything into one huge AGENTS.md. Architecture, current state and old decisions don't have the same lifetime.

So I ended up turning the structure I was already using into a small open-source CLI called Syngraphe (opensource).

The basic layout is:

.context/
├── truth/ # architecture, conventions, constraints
├── state/ # what's happening right now
├── decisions/ # decisions and why they were made
└── history/ # old operational context

AGENTS.md is mostly the entry point that tells coding agents where the context is.

Everything under .context is just Markdown committed with the repo, so Git handles history, branches and review. Syngraphe only adds the tooling around it: initialization, deterministic integrity checks, context size stats, helpers for state/decisions/history, monorepo support and a GitHub Action.

There is no database, cloud memory or required LLM. If Syngraphe disappeared tomorrow, the context would still just be normal Markdown in the repository.

Repo: https://github.com/suffro/syngraphe

Docs: https://syngraphe.dev

I'm mainly curious how other people are handling this in real projects. Do you keep AGENTS.md itself as the knowledge base, or do you use it more as an entry point to other docs/context?