Add grammar module system and dynamic loading capabilities

[steveluc]

Summary

Implements static TypeScript-based module system for grammar symbols and dynamic loading capability for runtime grammar extension. Enables learning from user request/action pairs and immediate cache updates.

Depends on: #1856 (NFA infrastructure)

Module System

Static TypeScript Linking

• Symbol matchers (for cache - matching only)
• Symbol converters (for agents - matching + conversion)
• Module namespacing: Global.Ordinal, Calendar.CalendarDate
• Unqualified names within modules
• Normal TypeScript import/export for dependencies
• Tree-shaking compatible

Built-in Symbols

✅ Global.Ordinal: first → 1, second → 2, twenty-third → 23
✅ Global.Cardinal: one → 1, two → 2, "42" → 42
✅ Calendar.CalendarDate: today, tomorrow, 2026-01-23 → Date objects

Dual Client Support

Cache clients (matching only):

const result = matchNFA(nfa, ["play", "first", "track"]);
// Just needs to know it matched

Agent clients (matching + conversion):

const result = matchNFA(nfa, ["play", "first", "track"]);
const trackNum = result.captures.get("n") as number; // 1
player.playTrack(trackNum);

Dynamic Loading

Runtime Grammar Extension

Enables learning from user examples and immediate cache updates:

// User makes request
const testCase = {
    request: "play Shake It Off by Taylor Swift",
    action: { actionName: "playTrack", parameters: {...} }
};

// Generate grammar from example
const analysis = await generator.generateGrammar(testCase, schemaInfo);
const rule = generator.formatAsGrammarRule(testCase, analysis);

// Add to cache immediately
const result = cache.addRules(rule);

// Similar requests now match from cache
const similar = matchNFA(cache.getNFA(), ["play", "Bad Blood", "by", "Taylor Swift"]);
// matched: true, no LLM needed

Key Features

✅ Symbol validation (checks all symbols resolved)
✅ Automatic NFA recompilation
✅ Merge into existing grammar as alternatives
✅ Error handling (cache unchanged on failure)
✅ Statistics tracking
✅ Works with grammarGenerator (agentSdkWrapper)

Components

Module System

• symbolModule.ts: Core symbol interfaces and registry
• symbols/: Built-in symbol implementations
  • ordinal.ts, cardinal.ts, calendarDate.ts
  • index.ts: Registration and exports

Dynamic Loading

• dynamicGrammarLoader.ts: Loading, validation, and merging
• grammarMerger.ts: Grammar merging utilities
• nfaInterpreter.ts: Updated with symbol resolution

Documentation

• MODULE_SYSTEM.md: Complete symbol system guide with examples
• DYNAMIC_LOADING.md: Dynamic loading guide with usage patterns
• examples/: Sample grammar modules (globalModule.agr, calendarModule.agr)

Integration Points

Works seamlessly with:

• grammarGenerator (agentSdkWrapper): Generates rules from examples
• NFA compiler: Automatically recompiles when rules added
• Cache matching: New patterns immediately available
• Agent execution: Type-safe value conversion

Testing

All 257 tests passing ✅

New tests:

• symbolModule.spec.ts: 15 tests

  • Symbol registration and lookup
  • Matcher functionality for cache clients
  • Converter functionality for agents
  • NFA integration

• dynamicGrammarLoader.spec.ts: 12 tests

  • Dynamic loading and validation
  • Symbol resolution checking
  • Grammar merging
  • DynamicGrammarCache operations
  • Integration with grammarGenerator format

Example Usage

import { DynamicGrammarCache } from "./dynamicGrammarLoader.js";
import { registerBuiltInSymbols } from "./symbols/index.js";

// Initialize
registerBuiltInSymbols();
const cache = new DynamicGrammarCache(baseGrammar, baseNFA);

// Generated rule from grammarGenerator
const generatedRule = `@ <Start> = <playTrack>
@ <playTrack> = play $(trackName:string) by $(artist:string) -> {
    actionName: "playTrack",
    parameters: {
        trackName: $(trackName),
        artist: $(artist)
    }
}`;

// Add to cache
const result = cache.addRules(generatedRule);
if (result.success) {
    console.log(`Grammar now has ${cache.getStats().ruleCount} rules`);
}

// Match immediately
const matchResult = matchNFA(cache.getNFA(), ["play", "Song", "by", "Artist"]);
console.log(matchResult.matched); // true

Documentation

See comprehensive guides:

• MODULE_SYSTEM.md
• DYNAMIC_LOADING.md

Test Plan

cd packages/actionGrammar
npm test -- symbolModule.spec
npm test -- dynamicGrammarLoader.spec
npm test  # All tests

All tests passing ✓

🤖 Generated with Claude Code