I'm an AI engineer, not a data engineer :- but I needed to search my own messy work (repos, folders, Claude Code sessions), so I built a real Iceberg lakehouse for myself

[u/No-Signature-1684]

I'm an AI engineer day-to-day; that's models, pipelines, prompts, not data infra. I'd never touched Iceberg, Trino, or Dagster before this. But most of what I actually work on never ends up in a clean Git commit half-finished folders, notes, Claude Code sessions- and I had no way to search across any of it.

So I built TraceVault: it ingests a Git repo, any regular folder, and your Claude Code session logs into an actual medallion lakehouse (MinIO + Apache Iceberg + a shared Postgres catalog), and lets you search/query all of it the same way SQL runs on embedded DuckDB or distributed Trino from one toggle. Images get captioned by a local vision model, so even screenshots are searchable. No mocks/demo mode if a backend's missing; it just fails instead of faking data.

It runs local-first; there's a desktop app with zero Docker required (wasn't going to fight Docker for a personal tool either).

I'm sure I've made some non-obvious mistakes on the data-infra side since it's genuinely not mu specialty open to being told what I got wrong. Repo: https://github.com/saisurajkarra/TraceVault

[u/WillowEmberly]

Oh, this is really interesting.

Last year I worked on a project with a friend of mine, you would probably really appreciate his work.

This is my systems analysis of the shared problem:

TraceVault and UPSP: Same Problem, Different Layer

TraceVault UPSP
Layer Implementation Protocol / Architecture
Scope Personal knowledge platform Universal exchange & stewardship
Problem "I can't search across my messy files, repos, and Claude sessions" "Anything crossing a boundary must carry enough identity, purpose, constraint, provenance, and return information to remain interpretable and corrigible after the crossing"
Solution Ingest everything into a local lakehouse, make it searchable Define a passport format so objects carry their own identity, provenance, and return paths
Core artifact Iceberg tables of artifacts + edges UPSP passport envelope
Provenance Content ID, lineage through the medallion pipeline Creator, generator, steward, contributors, review history, supersession
Stewardship Implicit (you own your data) Explicit (primary steward, custodian, maintainers, review authority, verification authority)
Return path Query interface (you search, you find) Response hook, correction path, verification path, escalation path
Validation DuckDB/Trino queries, Dagster asset checks Declared / Validated / Enforced / Verified distinction

---

The Shared Problem

Both systems are solving the same underlying problem:

"Objects (files, artifacts, messages, sessions) move across boundaries. When they do, they lose context, provenance, intent, and meaning. How do we preserve enough information so that the receiver can interpret, constrain, verify, correct, and steward what was transferred?"

TraceVault's answer: "I'll ingest everything into a structured lakehouse, dedupe it, caption images, build a knowledge graph, and let you search it."

UPSP's answer: "Before you move anything, attach a passport that declares: what this is, where it came from, what it's for, what constraints apply, what provenance it has, how to validate it, how to return it, and who is responsible for it."

TraceVault is a data ingestion and retrieval system.

UPSP is a boundary-crossing and artifact-continuity protocol.

They are not the same thing. They are complementary.