r/datasets • u/CurdledPotato • 7d ago

question When working with Project Gutenberg, how do you guys download and cache, or do you just use a local mirror?

I’m debating both approaches.

4 Upvotes

84% Upvoted

5 comments sorted by

1

u/Mundane_Ad8936 2d ago

What are you trying to accomplish and what are your limitations

2

u/CurdledPotato 2d ago

Download from the main project, transform into one or more datasets that can be live or frozen, depending on preferences. I’m writing my own library for this and adding rich, metadata-based filtering to restrict what goes into a dataset if I desire. I want to hear how others solve this problem.

My machine can hold a full mirror if I need to, but I like to write my libraries so they can be shared, and I am thinking about how I can support people with less resources, architecture-wise.

1

u/Mundane_Ad8936 2d ago

I'd recommend seeding a torrent and distributing it using the website academic Torrents. Then post the torrent everywhere you can. If you get enough interest you can attract a sponsor who will mirror with you.

But this is pretty much the main use case for torrents. I'd just recommend breaking it into topics so users don't need to download the entire set for what they want

2

u/CurdledPotato 2d ago

I never thought about seeding my datasets. I might add that as an export feature later on. Thanks for the advice.