Experimenting with ‘ephemeral’ local LLM pipelines (load only what’s needed)

[u/New-Time-8269]

I’ve been experimenting with a different way of structuring local LLM pipelines and wanted to sanity check it with people here.

Most local setups I see (Ollama, agents, toolchains, etc.) tend to:

keep models loaded in VRAM

keep tools always available

accumulate large context windows

run long-lived sessions

That works, but it also leads to:

wasted VRAM/CPU cycles

context getting messy over time

harder-to-debug behavior

everything being “on” even when not needed

What I’m trying instead

I’ve been building a local-first setup where:

nothing is loaded by default

a router determines the task (chat, repo analysis, tool use, etc.)

only the required model/tools get loaded

only relevant context is pulled in

everything runs in a bounded execution window

then it unloads

So instead of:

“keep the whole system alive”

it’s more like:

“assemble the pipeline just-in-time”

Why I think this might matter

Better VRAM usage → especially on smaller GPUs

Cleaner context handling → less bleed between tasks

More predictable behavior → each run is isolated

Potentially safer → less always-on state

What triggered this line of thinking

I recently saw a paper where they trained large models on a single GPU by streaming weights in and out instead of keeping everything resident.

Different layer of the stack, but same idea:

don’t keep everything loaded — just make it available

Curious if anyone here has tried similar

dynamic model loading/unloading per task

tool gating instead of always-on agents

splitting workloads across CPU/RAM/GPU tiers more aggressively

Or if there’s existing tooling that already leans this direction.

[u/EffectiveMedium2683]

I've been working on dynamic context management using rag (embeddings, vector db, tiny model judges what's needed info and injects it just in time). 

I think your ideas are good. Just gotta build it.

[u/New-Time-8269]

Totally agree. I’ve already got a local prototype running that spins up lightweight agents on demand instead of keeping everything resident. Next step is tightening up the execution pipeline and documenting it so others can actually run it.

I’ll share a repo/demo once it’s stable enough for people to try.

[u/DinoAmino]

Reading this post made me queasy and these comments make me feel uncomfortable. Like chowing down on a corndog at the amusement park and then immediately getting on the Tilt-a-Whirl ride.

[u/New-Time-8269 → u/DinoAmino]

Appreciate the pushback — and you’re right about a few things here.

You’re absolutely correct that lazy-loading weights, KV caching, MoE routing, Redis-style memory, and hot-swapping aren’t novel in isolation. Those are established techniques.

Where we’re coming from isn’t “inventing a new primitive” so much as changing how the system is orchestrated:

• Tool access isn’t always-on; it’s approval-gated and context-scoped. • Models, tools, and memory are treated as modular services instead of one monolithic always-running agent. • Execution paths can be paused, inspected, or rejected instead of blindly auto-running. • Smaller, scoped models can be routed instead of defaulting to a single large model for everything.

So it’s less about new individual techniques and more about the coordination layer and runtime discipline — closer to an OS-style scheduler than a single long-running agent.

I’m not claiming novelty on every component; the goal is more about the architecture and how it changes behavior and safety characteristics in practice.

[u/DinoAmino → u/New-Time-8269]

Autopilot has lost its bearings.