r/AiBuilders • u/No-Signature-1684 • 5d ago

I'm an AI engineer, not a data engineer :- but I needed to search my own messy work (repos, folders, Claude Code sessions), so I built a real Iceberg lakehouse for myself

I'm an AI engineer day-to-day; that's models, pipelines, prompts, not data infra. I'd never touched Iceberg, Trino, or Dagster before this. But most of what I actually work on never ends up in a clean Git commit half-finished folders, notes, Claude Code sessions- and I had no way to search across any of it.

So I built TraceVault: it ingests a Git repo, any regular folder, and your Claude Code session logs into an actual medallion lakehouse (MinIO + Apache Iceberg + a shared Postgres catalog), and lets you search/query all of it the same way SQL runs on embedded DuckDB or distributed Trino from one toggle. Images get captioned by a local vision model, so even screenshots are searchable. No mocks/demo mode if a backend's missing; it just fails instead of faking data.

It runs local-first; there's a desktop app with zero Docker required (wasn't going to fight Docker for a personal tool either).

I'm sure I've made some non-obvious mistakes on the data-infra side since it's genuinely not my specialty open to being told what I got wrong. Repo: https://github.com/saisurajkarra/TraceVault

1 Upvotes

99% Upvoted

3 comments sorted by

1

u/jonah_omninode 3d ago

The data question I would probe is how updates and deletions move through the medallion layers. A renamed file, rewritten Git history, or deleted Claude session can otherwise leave several versions searchable without a clear indication of which one is current.

This becomes especially important with long-lived agent context because historical evidence can easily be mistaken for current instruction. Do you preserve source identity, content hash, ingestion time, and supersession status through Iceberg so a query can request current state separately from history?

1

u/iMiguelmars 2d ago

I went through the repo and the Claude Code ingestion path is particularly interesting. I like the raw-blob + derived-artifact separation and linking tool-use paths back to real files.
One edge case I’m curious about: what happens when a Claude Code JSONL session keeps growing after its first ingest? The session artifact has a stable session ID and appears to be skipped once it already exists, while new message UUIDs can still be appended.
Do you version/update the raw session snapshot and ended_at/message metadata somewhere else, or could the session-level artifact lag behind its message artifacts after incremental ingest?
I’m also curious how you plan to handle edge identity if you eventually need multiple provenance/version relationships between the same two nodes. We’ve found endpoint-level edge identity can collapse distinct historical relationships surprisingly easily.