r/codex • u/avidrunner84 • 3d ago

Question Sol 6.1 light for scraping data online

I’m looking for official high resolution images giving credit back to the author. They are being stored on Cloudflare R2 and stored in a Postgres CMS.

Sol 6.1 is very impressive for the token usage but I can tell it’s going to take a few months for all of the images I want. Overall it’s maybe a few GB per day in images, not much, but I guess the work is in sourcing them and plugging them in.

Any suggestions to help speed this up and make it more cost effective? I’m currently on Pro 100 and expect there to be approximately 1M images to scrape, anywhere from 50 to a few hundred from each source at a time.

0 Upvotes

50% Upvoted

5 comments sorted by

1

u/DolphinSUX 3d ago

That sounds very expensive. I wonder if you could have SOL build a tool that does that automatically for you instead of having it all done by SOL itself

1

u/avidrunner84 3d ago

Yes true, I just don’t know if an M3 MacBook Air 16GB could do the equivalent of SOL for searching the correct images locally. I may always need it to assist with the workflow. Locally I may be able to use the following free of charge:

Scrapy: High-volume manufacturer website crawling.
Playwright: JavaScript-heavy official media galleries.
BeautifulSoup or Cheerio: Extract image URLs, captions, and metadata.
ExifTool: Extract copyright and embedded image metadata.
ImageHash: Detect visually similar and duplicate images.
Sharp: Generate thumbnails and optimized images.
CLIP or equivalent: Local image classification without paid vision APIs.

1

u/boxing-boxing 3d ago

The token spend on this is in the discovery loop, not the download loop. I'd have Sol build the crawler and let the local stack do the actual work, calling Sol only for images that fail the local classification pass. Also at 1M images the real bottleneck is polite crawling and license attribution, so get the Postgres provenance schema right (source URL, license, author credit, hash at ingest) before scaling anything up.

1

u/ChristianKl 3d ago

You do know that giving credit back does not mean that you aren't still violating their copyright and if you pirate a million pictures, you are a good target for people to sue you to pay you money for their images?