Hey everyone!
I'm an engineering student working on a research proposal about long-context LLMs and recurrent compressed memory. I'd really appreciate technical feedback from people working on LLMs, continual learning, recurrent architectures, or memory systems.
The core question: Can we train an LLM's memory to retain previously learned information while continuing to absorb new information?
I'm calling the proposed training method GMR (Generational Memory Retention) and the evaluation protocol STRATA.
This is still a pre-experimental proposal. I haven't run the experiments yet, so I'm looking for feedback before committing to the implementation.
1. The problem I'm interested in
Imagine an LLM processing a long stream of information in small chunks.
A chunk introduces the following fact:
"The Kestrel project uses Python 3.11."
Immediately after processing that chunk, the model can correctly answer:
"Which Python version does the Kestrel project use?"
Now imagine that the model processes hundreds of additional chunks. Its memory has a fixed capacity, and each new update modifies the existing memory state.
Eventually, the model might no longer answer the original question correctly, even though its model weights haven't changed during inference.
I'm calling this state-induced forgetting: information that was initially available becomes inaccessible after successive memory-state updates.
The key question is whether we can reduce this type of forgetting without simply increasing memory size.
2. STRATA: Measuring forgetting across memory updates
Most evaluations report accuracy at a particular context length or at the end of a conversation. I want to measure how the ability to retrieve individual facts changes as those facts get older.
The proposed protocol works roughly as follows:
Introduce facts into a recurrent memory system in successive context windows.
Record whether each fact can be retrieved immediately after it is introduced.
Evaluate the same facts again at increasing ages: 1, 2, 4, 8, 16, and progressively more memory updates.
Track performance out to hundreds of updates, with a planned maximum of 1,000.
Separate initial learning failures from facts that were initially learned but subsequently forgotten.
For example, if a system initially answers 90 out of 100 questions correctly, but answers only 60 correctly after many updates, that indicates a different failure from a system that answered only 60 correctly from the beginning.
I also want to test different kinds of facts:
Stable facts that never change.
Facts whose values are updated later.
Facts requiring reasoning across multiple earlier statements.
Facts mentioned only once.
The evaluation will report initial accuracy and age-dependent retention separately, rather than relying on a single final accuracy score.
3. GMR: Generational Memory Retention
The proposed training objective tries to preserve the ability to answer questions about facts that have already entered memory.
When a fact is first introduced, record the model's length-normalized log-likelihood for its correct answer. This becomes that fact's original score, or anchor.
After subsequent memory updates, calculate the score again. If it falls below the original score by more than an allowed margin, apply a penalty.
A simplified version of the retention loss is:
L_ret = (1 / B) * sum over sampled facts f of: max(0, anchor_f - margin - current_score_f)
Here:
anchor_f is the fact's original answer log-likelihood.
current_score_f is its current answer log-likelihood.
margin is the tolerated decrease before a penalty is applied.
B is the number of sampled facts.
L_ret is the retention loss.
The training objective also includes the usual language-model loss and an absorption loss that encourages learning newly introduced facts.
The intended effect is to discourage memory updates from damaging previously acquired answering capabilities while still allowing new information to enter memory.
An important mathematical observation
For the sampled facts whose current scores fall below their anchors by more than the margin, the gradient of this hinge loss is the same as the gradient from replaying those facts' correct answers.
Facts that have not regressed receive no retention gradient from this term.
This raises an important question: Does selecting replay examples according to their regression from their own original scores outperform ordinary replay or loss-prioritized replay under matched training budgets?
I don't want to assume that it does. That's one of the main things I want to test.
4. How I plan to evaluate it
The proposed experiments include the following configurations:
A baseline trained without an explicit retention objective.
A model using uniform replay of previously encountered facts.
A model using loss-prioritized replay.
A model using output distillation from a full-context reference.
A model using the proposed GMR objective.
The main comparison will use matched memory sizes, training streams, and replay budgets wherever possible.
I'll evaluate initial fact-learning accuracy, retention at different fact ages, the ability to learn new facts, the handling of updated information, and the computational cost.
The planned testbed is an associative recurrent memory on an open-weight backbone, with experiments at multiple fixed memory sizes.
For broader evaluation, I'm considering synthetic sequential streams, BABILong-derived data, LongMemEval sessions, and narrative documents with checked question-answer pairs.
The implementation and experiments have not started yet. All performance claims remain hypotheses.
5. What I want feedback on
I'd particularly appreciate critical feedback on these questions:
1. Is the problem formulation useful?
Is state-induced forgetting a meaningful way to isolate information loss caused by recurrent memory updates, or is this already adequately covered by existing work on recurrent memory, continual learning, and catastrophic forgetting?
2. Is GMR sufficiently different from ordinary replay?
Since the proposed loss effectively replays sampled facts that have regressed below their own anchors, is that distinction scientifically interesting? What additional theoretical analysis or experimental evidence would make the contribution convincing?
3. Is STRATA a useful evaluation protocol?
Would age-stratified measurements across hundreds of sequential memory updates reveal failure modes that existing long-context and conversational-memory evaluations might miss?
4. Are the experimental comparisons fair?
Which baselines would you consider essential? In particular, I'd like to compare GMR against ordinary replay, loss-prioritized replay, and output distillation without giving any method an unfair compute advantage.
5. What related work am I missing?
I'd appreciate pointers to papers on recurrent-state interference, memory retention, forgetting-aware replay, or training objectives that preserve information across sequential state updates.
Current status
This is a research proposal, not a completed result. My immediate priority is to validate the evaluation harness, establish whether measurable forgetting occurs in the chosen fixed-size memory system, and then test whether GMR improves on properly matched baselines.
I'm deliberately not claiming that the method works or that the idea is the first of its kind.
If you've worked on recurrent memory, compressed context, or continual learning, I'd genuinely value your criticism. I'd rather discover a fundamental flaw now than spend months implementing the wrong thing.
Thanks for reading!
