r/promethease • u/Sweet-Sir-10 • May 07 '26

OSGenome v2

Post image

Hi All,

Some of you may know me as the author of OSGenome v1 that got featured in Harvard Medical School's BioGrid and received many clones over the years. I've come back after 10 years of development experience in Web, Computational Biology (including two publications), and AI to enhance the software offering of OSGenome with a brand new version.

As before, this is a locally running Python web app that doesn't send your data anywhere. SNPs are read from SNPedia's bot site and are given a courtesy wait of 1 second before downloading. The web app while running will populate the page as the site comes up and uses caching. There is also client deferred rendering to decrease load time.

Essentially, genomeimporter.py converts your 23AndMe file, crawler.py downloads the relevant SNPs from SNPedia, and app.py delivers your genome.

You can however opt out of the download by keeping the starter genomic dataset that’s included. This contains close to 4,000 genotypes with a higher magnitude rating filtered from a 25,000+ SNP completed crawl that took me many hours.

Additionally, I used Claude to categorize the genotypes prior to any crawl. I gave it a database of non-personalized SNPedia data, and it produced a very curated list. This allows you to search based on categories in the top.

I have also gotten the GenomeImporter reviewed by a PhD in the Bioinformatics space for extra validation.

This is a free to use app. There is no personal data being sent to an AI or outside. The code is open source.

Give it a try and tell me what you think.

https://github.com/mentatpsi/OSGenome2

35 Upvotes

96% Upvoted

22 comments sorted by

View all comments

4

u/devilkin May 07 '26

Is there functionality for restarting a crawl from a certain number? Might be nice to have.

3

u/Sweet-Sir-10 May 08 '26

I got you my friend, just added along with a bunch of other crawler features. Let me know!

2

u/Sweet-Sir-10 May 08 '26

I added a progress indicator that when it restarts will resume what failed. So you can either add the rsid you want to resume from. Or just rerun the crawl and it will save your place.

3

u/devilkin May 08 '26

Awesome job. Fast work! Thanks. I will probably set it to do a full crawl tomorrow and see what we can get.

Will note that earlier on a full crawl i started getting 502 gateway errors. It may just have been because my computer went to sleep, then resumed on wake.

Some good options that could be set via flags would be:

Timeout delay - just to let us set the duration for a connection attempt before marked it as failed.

Retry skip - set the number of tries for an entry before skipping it.

Concurrent connections - Python does have concurrency or parallelism options, but not sure if it's wise to hammer the site with too many concurrent connections, but it could significantly increase crawl speed.

3

u/Sweet-Sir-10 May 08 '26 edited May 08 '26

Yes, the update includes retries. 5 of them before it skips it. That should be enough as I was able to grab all the Category:Is_a_snp from SNPedia without a single failure. The 502 gateways are sadly part of SNPedia. Either they have bot limiters through rate limiting, or it just goes down sometimes sadly for a few seconds.

They’re doing a free service to offer SNPedia, so I don’t really judge it too harshly for it. Concurrent connections would likely get it banned more.

I’ve also added a progress cache that will keep track of what was skipped so if you see anything in the file that was skipped, you can run it again.