feat: add multi-layer memory protection to consolidation engine #16

danieliser · 2025-12-03T08:59:03Z

Summary

Implements "Hybrid #6" memory protection strategy in the consolidation engine to prevent accidental deletion of important memories.

Protection Layers

Explicit Protection Flag - protected: true on Memory nodes
Importance Threshold - Skip memories above configurable threshold (default: 0.5)
Grace Period - New memories protected for N days (default: 90)
Type Protection - Configurable protected types (default: Decision, Insight)

Environment Variables

CONSOLIDATION_DELETE_THRESHOLD (default: 0.05)
CONSOLIDATION_ARCHIVE_THRESHOLD (default: 0.2)
CONSOLIDATION_GRACE_PERIOD_DAYS (default: 90)
CONSOLIDATION_IMPORTANCE_PROTECTION_THRESHOLD (default: 0.5)
CONSOLIDATION_PROTECTED_TYPES (default: Decision,Insight)

Test plan

Verify protected memories are never deleted
Verify high-importance memories are preserved
Verify new memories within grace period are protected
Verify protected types are skipped

🤖 Generated with Claude Code

… engine Implements multi-layer protection to prevent accidental data loss during consolidation, particularly for restored memories. Protection criteria: - Explicit 'protected' flag on memory node - High importance (>= 0.5 by default) - Grace period (90 days for new/restored memories) - Protected memory types (Decision, Insight by default) New environment variables: - CONSOLIDATION_DELETE_THRESHOLD (default: 0.05) - CONSOLIDATION_ARCHIVE_THRESHOLD (default: 0.2) - CONSOLIDATION_GRACE_PERIOD_DAYS (default: 90) - CONSOLIDATION_IMPORTANCE_PROTECTION_THRESHOLD (default: 0.5) - CONSOLIDATION_PROTECTED_TYPES (default: "Decision,Insight") Changes: - consolidation.py: Add _should_protect_memory() method with multi-criteria checks - consolidation.py: Update apply_controlled_forgetting() to respect protection - consolidation.py: Track protected memories in stats alongside deleted/archived - automem/config.py: Add configurable protection threshold variables - app.py: Wire protection config to both MemoryConsolidator instantiations

coderabbitai · 2025-12-03T08:59:12Z

📝 Walkthrough

Walkthrough

The pull request introduces memory protection configuration constants and extends MemoryConsolidator to enforce protection rules during controlled forgetting. Five new configurable parameters (delete_threshold, archive_threshold, grace_period_days, importance_protection_threshold, protected_types) are added to configuration, wired through the application layer, and used to prevent deletion of memories meeting specified protection criteria.

Changes

Cohort / File(s)	Summary
Configuration constants `automem/config.py`	Adds five memory protection configuration constants: `CONSOLIDATION_DELETE_THRESHOLD`, `CONSOLIDATION_ARCHIVE_THRESHOLD`, `CONSOLIDATION_GRACE_PERIOD_DAYS`, `CONSOLIDATION_IMPORTANCE_PROTECTION_THRESHOLD`, and `CONSOLIDATION_PROTECTED_TYPES`, all sourced from environment variables with sensible defaults.
Memory consolidator protection logic `consolidation.py`	Extends `MemoryConsolidator` with protection-aware deletion: adds `__init__` parameters for thresholds and protected types, introduces `_should_protect_memory()` method checking explicit protection flag, importance level, grace period, and memory type, and integrates protection checks into `apply_controlled_forgetting()` to skip deletion and record protected memories in stats.
Application layer wiring `app.py`	Updates scheduler and consolidation construction paths to accept and propagate protection-related parameters (`delete_threshold`, `archive_threshold`, `grace_period_days`, `importance_protection_threshold`, `protected_types`) from config into `MemoryConsolidator` initialization.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

consolidation.py requires careful review of the new protection logic and its integration into the forgetting workflow, particularly the _should_protect_memory() method logic and the updated apply_controlled_forgetting() stats structure.
app.py parameter propagation paths should be verified for consistency across both scheduler and consolidation routes.
Configuration constants in config.py should be validated against expected environment variable formats and default value appropriateness.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 77.78% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main feature being introduced: multi-layer memory protection in the consolidation engine, which aligns with the primary changes across all modified files.
Description check	✅ Passed	The description clearly relates to the changeset, detailing the protection layers, environment variables, and test plan that are implemented in the code changes.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 0

🧹 Nitpick comments (2)

automem/config.py (1)
32-37: Missing error handling for invalid environment variable values.

Unlike _DECAY_THRESHOLD_RAW (lines 25-28) which strips and checks for empty values before calling float(), these new config values will raise ValueError if the environment variable is set to an empty or non-numeric string, causing application startup to fail.

Consider adding defensive parsing:
-CONSOLIDATION_DELETE_THRESHOLD = float(os.getenv("CONSOLIDATION_DELETE_THRESHOLD", "0.05"))
-CONSOLIDATION_ARCHIVE_THRESHOLD = float(os.getenv("CONSOLIDATION_ARCHIVE_THRESHOLD", "0.2"))
-CONSOLIDATION_GRACE_PERIOD_DAYS = int(os.getenv("CONSOLIDATION_GRACE_PERIOD_DAYS", "90"))
-CONSOLIDATION_IMPORTANCE_PROTECTION_THRESHOLD = float(
-    os.getenv("CONSOLIDATION_IMPORTANCE_PROTECTION_THRESHOLD", "0.5")
-)
+_DELETE_THRESHOLD_RAW = os.getenv("CONSOLIDATION_DELETE_THRESHOLD", "0.05").strip()
+CONSOLIDATION_DELETE_THRESHOLD = float(_DELETE_THRESHOLD_RAW) if _DELETE_THRESHOLD_RAW else 0.05
+
+_ARCHIVE_THRESHOLD_RAW = os.getenv("CONSOLIDATION_ARCHIVE_THRESHOLD", "0.2").strip()
+CONSOLIDATION_ARCHIVE_THRESHOLD = float(_ARCHIVE_THRESHOLD_RAW) if _ARCHIVE_THRESHOLD_RAW else 0.2
+
+_GRACE_PERIOD_RAW = os.getenv("CONSOLIDATION_GRACE_PERIOD_DAYS", "90").strip()
+CONSOLIDATION_GRACE_PERIOD_DAYS = int(_GRACE_PERIOD_RAW) if _GRACE_PERIOD_RAW else 90
+
+_IMPORTANCE_THRESHOLD_RAW = os.getenv("CONSOLIDATION_IMPORTANCE_PROTECTION_THRESHOLD", "0.5").strip()
+CONSOLIDATION_IMPORTANCE_PROTECTION_THRESHOLD = (
+    float(_IMPORTANCE_THRESHOLD_RAW) if _IMPORTANCE_THRESHOLD_RAW else 0.5
+)
consolidation.py (1)
141-148: Consider edge case: empty protected_types set.

The expression protected_types or self.PROTECTED_TYPES will fall back to the class default when protected_types is an empty set (since set() is falsy). If an empty set is passed intentionally to disable type-based protection, it would be ignored.

If disabling type protection via an empty set should be supported:
-        self.protected_types = protected_types or self.PROTECTED_TYPES
+        self.protected_types = protected_types if protected_types is not None else self.PROTECTED_TYPES
If the current behavior (always having at least the default protected types) is intentional, consider documenting this.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between abc689d and f3d4326.

📒 Files selected for processing (3)

app.py (3 hunks)
automem/config.py (1 hunks)
consolidation.py (6 hunks)

🧰 Additional context used

📓 Path-based instructions (3)

app.py

📄 CodeRabbit inference engine (CLAUDE.md)

app.py: Use Flask for the main API server (port 8001) with request validation and orchestration
Implement 13 API endpoints covering core memory operations (POST /memory, GET /recall, PATCH /memory/, DELETE /memory/, GET /memory/by-tag), relationship management (POST /associate), consolidation (POST /consolidate, GET /consolidate/status), health checks (GET /health), enrichment (GET /enrichment/status, POST /enrichment/reprocess), analysis (GET /analyze), and startup recall (GET /startup-recall)
Expose enrichment metrics at GET /enrichment/status including processed counts, last success/error, queue depth, and inflight jobs
Combine vector similarity, keyword match, tag overlap, and recency in recall scoring
Support authentication via Bearer token, X-API-Key header, or query parameter

Files:

app.py

**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.py: Use black for Python code formatting
Use flake8 for Python linting

**/*.py: Use Python with type hints. Indent 4 spaces; line length 100 (Black).
Use snake_case for module and function names in Python.
Use PascalCase for class names in Python.
Use UPPER_SNAKE_CASE for constants in Python.
Format code with Black (line length 100) and organize imports with Isort (profile=black).
Lint with Flake8 and ensure CI passes before committing.
Add or adjust tests for new endpoints, stores, or utils; maintain comprehensive test coverage.

Files:

app.py
automem/config.py
consolidation.py

consolidation.py

📄 CodeRabbit inference engine (CLAUDE.md)

consolidation.py: Implement consolidation engine (consolidation.py) with four biological memory patterns: Decay (hourly exponential relevance updates), Creative (REM-like processing hourly), Clustering (semantic grouping every 6 hours), and Forgetting (archive low-importance daily)
Use ConsolidationScheduler to manage consolidation intervals with configurable values via environment variables
Run consolidation in background threads without blocking API requests
Use exponential decay with fractional-day intervals for memory relevance updates in Decay consolidation task
Use CONSOLIDATION_DECAY_IMPORTANCE_THRESHOLD to skip truly low-importance items during decay consolidation

Files:

consolidation.py

🧠 Learnings (7)

📓 Common learnings

Learnt from: CR
Repo: verygoodplugins/automem PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-02T05:34:12.678Z
Learning: Applies to consolidation.py : Implement consolidation engine (consolidation.py) with four biological memory patterns: Decay (hourly exponential relevance updates), Creative (REM-like processing hourly), Clustering (semantic grouping every 6 hours), and Forgetting (archive low-importance daily)