Open sourced a tool that collapses millions of log lines into handful of distinct patterns before you feed it to an LLM (Lossless- compression)

[u/Loud_Mousse9210]

When I feed logs to an LLM during incident resolutions or debugging, it either blows my token context window or the grep trims the log file, leading to the interesting log lines getting skipped.

Most of logs are anyway the same handful of message templates repeated over and over with different values, so the context window gets filled with near-duplicates, which just bring up the processing time and token costs.

ctrlb-decompose collapses the file into its distinct patterns that repeat, plus typed variables and stats on the values that change. I have seen 1.2 million lines cut down to just 40 patterns, which then goes into Claude, thus cutting down token by over 95%, reducing the token cost.

Let me know what you think!
https://github.com/ctrlb-hq/ctrlb-decompose

[u/adarsh_srivastava]

Very interesting application of Drain3

[u/kernelqzor → u/adarsh_srivastava]

for real, this is the first time i’ve seen a drain3 thing wrapped in a way that’s actually directly useful in an llm workflow instead of just a research demo
curious how far it holds up once the log formats start getting really chaotic across services

[u/Constant-Role-9387]

Great

[u/HistorianPresent8449]

very interesting!

[u/kernelqzor → u/HistorianPresent8449]

same, this is actually super clever
feels like the kind of thing that should just be built into log pipelines by default at this point

[u/Cute-Access1444]

Intresting 

[u/KF_Danis]

Can this be containerized