Proposal: Context File Growth: Consolidation + Smart Retrieval

[v0lkan]

Problem

Files that grow every session — LEARNINGS.md, DECISIONS.md,
CONVENTIONS.md — accumulate entries without bound. After weeks of active
use, these files become expensive to load into agent context and dilute
signal with entries that aren't relevant to the current task.

The problem isn't storage (flat files are fine on disk). The problem is
retrieval: ctx agent --budget 4000 must decide what to include, and
today it truncates by token count rather than by relevance.

Constraints

• ctx is intentionally file-based. SQLite or any database layer violates
  the core design principle that context is human-readable, git-tracked
  Markdown.
• Archiving buries signal. Moving old entries to an archive folder makes
  them harder to find. A learning from 3 months ago might still be the most
  critical thing an agent needs today.
• Pagination fragments. Splitting into LEARNINGS-01.md,
  LEARNINGS-02.md loses the single-file simplicity that makes ctx work.
  Agents would need to know which file to read, and humans lose the ability
  to grep one file.
• Old entries aren't necessarily stale. Unlike tasks (which complete),
  learnings and decisions are often permanent. "Don't use go install in
  hooks" is as true on day 100 as day 1.

Proposal: Three-Layer Approach

1. Periodic Consolidation (reduces without burying)

A /ctx-consolidate skill that:

• Groups related entries by topic/tag similarity
• Merges redundant or overlapping entries into denser combined entries
• Moves originals to .context/archive/learnings-YYYY-QN.md for reference
• Is human-triggered, not automatic (preserves "you control the context")

Example: 5 separate learnings about hook edge cases become 1 consolidated
entry covering all the gotchas, with the 5 originals archived.

Key distinction: consolidation ≠ archival. Archival moves entries out.
Consolidation replaces verbose entries with tighter ones — the file stays
useful, just denser.

2. Relevance-Aware ctx agent Retrieval

Instead of truncating by token count, ctx agent scores entries:

• Recency boost: entries from the last N sessions rank higher
• Task relevance: keyword/tag overlap with active TASKS.md entries
• Entry type weighting: conventions and active decisions rank higher
  than old learnings (conventions are always relevant; learnings are
  situational)
• Summarize the rest: entries that don't make the budget cut get a
  one-line summary rather than full inclusion

The files stay flat and append-only. The presentation layer gets smart.

This is the highest-impact change: it solves the problem without touching
the files at all.

3. Soft Caps with Nudges

ctx drift warns when a file exceeds a threshold:

⚠ LEARNINGS.md has 47 entries (recommended: ≤30)
  Run /ctx-consolidate to review and merge related entries

Not enforcement — just a nudge in the existing maintenance workflow. The
threshold is configurable via .contextrc.

Alternatives Considered

Approach	Why not
SQLite	Violates file-based design; not human-readable; not git-diffable
Folder pagination	Fragments the file; agents don't know which page to read; humans can't grep one file
Pure archival	Buries signal; a 6-month-old learning might be the most relevant today
Automatic pruning	Dangerous — who decides what's stale? Only the human should make that call
Tagging + filtering	Helps retrieval but doesn't reduce file size; complementary, not sufficient

Implementation Phases

Phase 1: Smart Retrieval (highest impact, no file changes)

Enhance ctx agent to score entries against current tasks before including
them. This alone solves the "too much context" problem without any file
format changes.

• Modify internal/cli/agent/ to score entries
• Add keyword extraction from TASKS.md for relevance matching
• Entries below the relevance threshold get one-line summaries
• Budget allocation: constitution (fixed) → tasks (fixed) → conventions
  (high weight) → decisions (medium) → learnings (scored)

Phase 2: Drift Nudges (low effort)

Add entry count checks to ctx drift:

• Warn when LEARNINGS.md exceeds 30 entries
• Warn when DECISIONS.md exceeds 20 entries
• Configurable thresholds via .contextrc

Phase 3: Consolidation Skill (full solution)

Build /ctx-consolidate skill:

• Analyze entries for topic similarity
• Present consolidation suggestions to the user
• Merge approved groups into denser entries
• Archive originals to .context/archive/
• Update cross-references if any entries link to consolidated ones

Open Questions

• Should consolidated entries carry a consolidated-from: metadata field
  linking back to the originals?
• Should the archive be gitignored (like sessions) or tracked (like context
  files)?
• Should ctx agent budget allocation weights be configurable, or are
  sensible defaults sufficient?
• Is 30 the right soft cap for learnings? Should it vary by file type?

Labels

enhancement, context-management, agent-ux

[josealekhine]

Thanks @v0lkan 🙏 .

This directly affects agent effectiveness.

And looks like you've started to feel the pain.

Pain is good; it drives structure.

"As projects mature, the context files will bloat and the agent gets worse at picking what's relevant": %100 agree with that.

imho, Phase 1 alone ( relevance-aware retrieval) would be a significant improvement with no file format changes.

It's a pragmatic take.

Let me dig into the codebase and see what comes out.

[bilersan]

thanks @v0lkan for coming up an idea with this and addressing the issue. i do suffer from the same problem when it comes to use ctx long sessions and/or large codebases. i was thinking what could be done and my reasoning is as follows.

if we reason from first principles about how human knowledge scales versus how LLMs process data, the root cause isn't actually retrieval, and the solution isn't batch consolidation.

The problem is that we are treating our context files as append-only ledgers.

Ledgers scale linearly to infinity; human technical knowledge does not. When you learn three things about React hooks, your brain doesn't store three isolated log entries—it updates your relational mental model of React.

Furthermore, trying to solve context limits by moving knowledge into Vector DBs (Embeddings) or fine-tuned weights (LoRAs) is a trap. Embeddings are opaque and still eat context tokens upon injection. LoRAs are probabilistic—they are great for implicit coding style, but they will confidently hallucinate strict rules.

Here is a first-principles alternative that honors your core constraint (human-readable, flat, git-tracked Markdown) while permanently solving the context bloat, leveraging bleeding-edge architectures like Test-Time Training (TTT-LM) and Semantic Ontologies.

---

The Root Problem

Files like LEARNINGS.md grow without bound because the agent is "appending." This forces expensive token-truncation and requires brittle batch-consolidation. We need to shift the data structure from a Chronological Ledger to a State-Based Wiki, and upgrade our inference engine to handle infinite sequences natively.

Proposal: The "Living Ontology" & TTT Architecture

1. Write-Time Compression (The Semantic Markdown AST)

Instead of the agent appending "Learning 48" to the bottom of the file, we rigidly structure context files as an Abstract Syntax Tree (AST) using Markdown headers (## Domains -> ### Topics).

When an agent learns a new edge case about WebSockets, it does not append. It uses a new /ctx-update skill to patch the existing ## WebSockets node. Knowledge naturally compresses at write-time. The git diff shows the evolution of a concept, not just a growing list.

To make this robust without databases, we introduce machine-readable (but human-friendly) tags:

• Time-To-Live (TTL): > Expires: 2026-12-01. Knowledge decays. ctx drift no longer counts entries; it flags expired TTLs for revalidation.

• Validity Tags: ``. If the agent's environment doesn't match, the AST parser drops the node entirely before the prompt is built.

• Semantic Edges: *(Derived from: ../LEARNINGS.md#auth-bug)*. We use standard markdown anchors to link a rule in DECISIONS.md directly to the "why" in LEARNINGS.md.

2. Context-as-a-Dependency (CaaD)

Because our context is now a structured tree rather than a flat log, it can be shared across repositories just like an npm package.

• Repositories define dependencies in context.yaml (e.g., @acme-corp/react-conventions).

• Running ctx install performs an AST Tree Merge, pulling collective knowledge into local files under read-only namespaced headers (## @global/React).

• Lexical Shadowing: If a rule under ## 🏠 Local/React contradicts the global registry, local always overrides global.

3. The Engine Layer: Test-Time Training (TTT) & Native Caching

We stop trying to carefully budget 4,000 tokens.
Standard Transformers use a KV Cache, which grows linearly and crashes VRAM. But by adopting models based on Test-Time Training (TTT-LM) (or leveraging advanced native KV caching), the model replaces the KV cache with a hidden state trained via gradient descent during inference.

• The Result: The memory footprint remains constant regardless of sequence length.

• Instead of guessing which 4,000 tokens to push, ctx agent streams the entire parsed Markdown Ontology into the model's test-time weights. The agent processes your entire repository's rules with effectively zero ongoing VRAM bloat per turn.

Alternatives Considered

Approach | Why it's a Trap in this Paradigm -- | -- Embeddings / Vector DBs | Violates the flat-file constraint. RAG is a "guess and push" model that still injects tokens into the context window, blowing up budgets anyway. LoRA / Fine-Tuning | Weights are for implicit style, context is for explicit facts. LoRAs are probabilistic and will hallucinate strict rules. Periodic Consolidation | Requires expensive LLM batch processing. Highly prone to losing technical nuance. Treats the symptom (ledger bloat) instead of the disease.

Implementation Phases

Phase 1: Data Structure Shift (Highest ROI, No File Changes)

• Restructure existing .md files into strict Header/Topic trees.

• Deprecate "append to bottom". Build the /ctx-update skill to patch existing headers.

• Modify ctx drift to enforce AST structure and scan for expired TTL tags (> Expires: YYYY-MM-DD).

Phase 2: Context Registries (The CaaD Fleet Update)

• Introduce context.yaml for defining organizational dependencies.

• Build ctx install to fetch remote Markdown files and perform an AST merge into local files.

• Implement Contextual Validity parsing to drop AST nodes based on the local package.json dynamically.

Phase 3: Infinite Engine Migration (TTT / Advanced Caching)

• Transition the ctx agent backend to utilize native Prompt Caching or a TTT-LM runner.

• Remove the --budget 4000 constraint. Feed the entire parsed AST into the model prefix, relying on the model's fixed memory state to retain the rules indefinitely.

Open Questions

1. TTT Exact Recall vs. Lossy Compression: While TTT elegantly compresses infinite context into a fixed state, it acts as lossy compression. Can it reliably recall exact API strings (e.g., sk_test_...) buried deep in the ontology, or do we need a hybrid attention/TTT approach for hard constants?

2. TTL Enforcement: When ctx drift flags an expired learning, should we allow the agent to automatically write a test script to attempt to re-validate it, or must TTL renewals strictly be human-approved?

3. Governance: Should agents be allowed to automatically propose /ctx-publish PRs to the global company registry when they patch a novel bug into the local ontology?