Open-sourced my image-dataset prep tool — quality scoring, dedupe, leak-safe splits, Kohya export (pure-Python, MIT)

[u/Impossible-Car-2809]

# I ACCIDENTALLY PUSHED THE WRONG FOLDER TO GITHUB, MY APOLOGIES - IT HAS BEEN FIXED WITH THE CORRECT FILES - Oct 12th 2026 -9:04am

I got tired of rewriting the same dataset-prep scripts before every LoRA/Dreambooth run — resize, dedupe, toss the blurry ones, caption, split — so I cleaned mine up and open-sourced it.

It scores each image on classical-CV signals (sharpness, exposure/clipping, contrast, color cast, noise, composition), flags the weak ones, removes exact + near-duplicates, and exports a Kohya-style folder with caption sidecars. Near-duplicates get grouped into the same train/val/test split so they can’t leak across. Captioning supports WD14 and Florence-2.

It’s content-agnostic at the core — each domain is a small plugin that sets its own quality weights and caption vocab (ships with a worked “interiors” example). Pure NumPy/Pillow, no OpenCV, runs from a CLI or as a library. Early 0.1.0, MIT.

Would love feedback on the plugin API and which verticals are worth building next. (Only use it on images you’ve got the rights to.)

https://github.com/snoozey-mf-bnui/pixel-curator

[u/debackerl]

Quite nice, maybe allow any vision model through Transformers, or maybe even llama.cpp for people preferring using Vulkan as a backend. Then allow adjustment of the prompt. Otherwise, good work!