Merge pull request #2562 from redis/DOC-6122-toc-metadata

DOC-6122 page table-of-contents metadata

Author: andy-stark-redis

build/metadata_docs/IMPLEMENTATION_NOTES.md

@@ -0,0 +1,119 @@
+# Implementation Notes: Table of Contents Metadata
+
+## Overview
+
+This document captures lessons learned from implementing auto-generated table of contents (TOC) metadata for Redis documentation pages. These insights should help guide future metadata feature implementations.
+
+## Key Lessons
+
+### 1. Start with Hugo's Built-in Functions
+
+**Lesson**: Always check what Hugo provides before building custom solutions.
+
+**Context**: Initial attempts tried to manually extract headers from page content using custom partials. This was complex, error-prone, and required parsing HTML/Markdown.
+
+**Solution**: Hugo's `.TableOfContents` method already generates HTML TOC from page headings. Using this as the source was much simpler and more reliable.
+
+**Takeaway**: For future metadata features, audit Hugo's built-in methods first. They often solve 80% of the problem with minimal code.
+
+### 2. Regex Substitution for Format Conversion
+
+**Lesson**: Simple regex transformations can convert between formats more reliably than complex parsing.
+
+**Context**: Converting HTML to JSON seemed like it would require a full HTML parser or complex state machine.
+
+**Solution**: Breaking the conversion into small, sequential regex steps:
+1. Remove wrapper elements (`<nav>`, `</nav>`)
+2. Replace structural tags (`<ul>` → `[`, `</ul>` → `]`)
+3. Replace content tags (`<li><a href="#ID">TITLE</a>` → `{"id":"ID","title":"TITLE"`)
+4. Add structural elements (commas, nested arrays)
+
+**Takeaway**: For format conversions, think in terms of sequential substitution patterns rather than parsing. This is often simpler and more maintainable.
+
+### 3. Hugo Template Whitespace Matters
+
+**Lesson**: Hugo template whitespace and comments generate output that affects final formatting.
+
+**Context**: Generated JSON had many blank lines, making it less readable.
+
+**Solution**: Use Hugo's whitespace trimming markers (`{{-` and `-}}`) to prevent unwanted newlines.
+
+**Takeaway**: When generating structured output (JSON, YAML), always consider whitespace. Test the final output, not just the template logic.
+
+### 4. Markdown Templates Have Different Processing Rules
+
+**Lesson**: Hugo's markdown template processor (`.md` files) behaves differently from HTML templates.
+
+**Context**: Initial attempts to include metadata in markdown output failed because the template processor treated code blocks as boundaries.
+
+**Solution**: Place metadata generation in the template itself, not in content blocks. Use `safeHTML` filter to prevent HTML entity escaping.
+
+**Takeaway**: When targeting multiple output formats, test each format separately. Markdown templates have unique constraints that HTML templates don't have.
+
+### 5. Validate Against Schema Early
+
+**Lesson**: Create the schema before or immediately after implementation, not after.
+
+**Context**: Schema was created last, after implementation was complete.
+
+**Better approach**: Define the schema first, then implement to match it. This:
+- Clarifies the target structure
+- Enables validation during development
+- Provides documentation for implementers
+- Helps catch structural issues early
+
+**Takeaway**: For future metadata features, write the schema first as a specification.
+
+### 6. Test Multiple Page Types
+
+**Lesson**: Metadata features must work across different page types with different content.
+
+**Context**: Implementation was tested on data types pages and command pages, which have different metadata fields.
+
+**Takeaway**: Always test on at least 2-3 different page types to ensure the feature is robust and handles optional fields correctly.
+
+## Implementation Checklist for Future Metadata Features
+
+When implementing new metadata features, follow this order:
+
+1. **Define the schema** (`static/schemas/feature-name.json`)
+   - Specify required and optional fields
+   - Use JSON Schema Draft 7
+   - Include examples
+
+2. **Create documentation** (`build/metadata_docs/FEATURE_NAME_FORMAT.md`)
+   - Explain the purpose and structure
+   - Show examples
+   - Document embedding locations (HTML, Markdown)
+
+3. **Implement the feature**
+   - Create/modify Hugo partials
+   - Test on multiple page types
+   - Verify output in both HTML and Markdown formats
+
+4. **Validate the output**
+   - Write validation scripts
+   - Test against the schema
+   - Check whitespace and formatting
+
+5. **Document implementation notes**
+   - Capture lessons learned
+   - Note any workarounds or gotchas
+   - Provide guidance for future similar features
+
+## Common Gotchas
+
+- **HTML entity escaping**: Use `safeHTML` filter when outputting HTML/JSON in markdown templates
+- **Whitespace in templates**: Use `{{-` and `-}}` to trim whitespace
+- **Nested structures**: Test deeply nested content to ensure regex patterns handle all cases
+- **Optional fields**: Remember that not all pages have all metadata fields
+- **Markdown vs HTML**: Always test both output formats
+
+## Tools and Techniques
+
+- **Hugo filters**: `replaceRE`, `jsonify`, `safeHTML`
+- **Validation**: Python's `jsonschema` library for schema validation
+- **Testing**: Extract metadata from generated files and validate against schema
+- **Debugging**: Use `grep` and `head` to inspect generated output
+
+

build/metadata_docs/PAGE_METADATA_FORMAT.md

@@ -0,0 +1,108 @@
+# Page Metadata Format
+
+## Overview
+
+Redis documentation pages include AI-friendly metadata that helps AI agents understand page structure, content, and navigation. This metadata is automatically generated during the Hugo build process and embedded in both HTML and Markdown output formats.
+
+## Metadata Structure
+
+### Core Fields (Required)
+
+- **`title`** (string, required): The page title
+- **`description`** (string, required): A brief description of the page content
+
+### Navigation Fields
+
+- **`tableOfContents`** (object): Hierarchical structure of page sections
+  - **`sections`** (array): Array of top-level sections
+    - **`id`** (string): Unique identifier matching the heading anchor ID
+    - **`title`** (string): Display title of the section
+    - **`children`** (array, optional): Nested subsections with the same structure
+
+### Categorization Fields
+
+- **`categories`** (array): Category tags for the page (e.g., `["docs", "develop", "stack"]`)
+- **`scope`** (string): Scope or domain of the page content
+- **`topics`** (array): Related topics
+- **`relatedPages`** (array): Links to related documentation pages
+
+### Command Reference Fields (for `/commands/` pages)
+
+- **`arguments`** (array): Command arguments
+- **`syntax_fmt`** (string): Command syntax format
+- **`complexity`** (string): Time complexity of the command
+- **`group`** (string): Command group
+- **`command_flags`** (array): Flags associated with the command
+- **`acl_categories`** (array): ACL categories for the command
+- **`since`** (string): Redis version when the command was introduced
+- **`arity`** (integer): Number of arguments the command accepts
+- **`key_specs`** (array): Key specifications for the command
+
+## Example
+
+```json
+{
+  "title": "Redis data types",
+  "description": "Overview of data types supported by Redis",
+  "categories": ["docs", "develop", "stack", "oss"],
+  "tableOfContents": {
+    "sections": [
+      {
+        "id": "data-types",
+        "title": "Data types",
+        "children": [
+          {"id": "strings", "title": "Strings"},
+          {"id": "lists", "title": "Lists"},
+          {"id": "sets", "title": "Sets"}
+        ]
+      },
+      {
+        "id": "time-series",
+        "title": "Time series"
+      }
+    ]
+  }
+}
+```
+
+## Embedding
+
+### HTML Output
+
+Metadata is embedded in a `<script>` tag in the page header:
+
+```html
+<script type="application/json" data-ai-metadata>
+{...metadata...}
+</script>
+```
+
+### Markdown Output (`.html.md`)
+
+Metadata is embedded in a JSON code block at the top of the page:
+
+````markdown
+```json metadata
+{...metadata...}
+```
+````
+
+## Auto-Generation
+
+The `tableOfContents` is automatically generated from page headings using Hugo's built-in `.TableOfContents` method. The HTML structure is converted to JSON using regex substitutions in the `layouts/partials/toc-json-regex.html` partial.
+
+## Schema
+
+The complete JSON schema is available at: `https://redis.io/schemas/page-metadata.json`
+
+This schema enables:
+- Validation of metadata structure
+- IDE autocomplete and type checking
+- AI agent understanding of page structure
+- Consistent metadata across all pages
+
+## Notes
+
+- The in-page JSON metadata does **not** include a `$schema` reference. The schema is available separately for validation and documentation purposes.
+- The metadata is auto-generated during the Hugo build process and does not require manual maintenance.
+

layouts/_default/section.md

@@ -16,7 +16,8 @@
   "key_specs": {{ .Params.key_specs | jsonify }}{{ end }}{{ if .Params.topics }},
   "topics": {{ .Params.topics | jsonify }}{{ end }}{{ if .Params.relatedPages }},
   "relatedPages": {{ .Params.relatedPages | jsonify }}{{ end }}{{ if .Params.scope }},
-  "scope": {{ .Params.scope | jsonify }}{{ end }}
+  "scope": {{ .Params.scope | jsonify }}{{ end }},
+  "tableOfContents": {{ partial "toc-json-regex.html" . }}
 }
 ```
 

layouts/_default/single.md

@@ -16,7 +16,8 @@
   "key_specs": {{ .Params.key_specs | jsonify }}{{ end }}{{ if .Params.topics }},
   "topics": {{ .Params.topics | jsonify }}{{ end }}{{ if .Params.relatedPages }},
   "relatedPages": {{ .Params.relatedPages | jsonify }}{{ end }}{{ if .Params.scope }},
-  "scope": {{ .Params.scope | jsonify }}{{ end }}
+  "scope": {{ .Params.scope | jsonify }}{{ end }},
+  "tableOfContents": {{ partial "toc-json-regex.html" . }}
 }
 ```
 

layouts/partials/ai-metadata-body.html

@@ -37,5 +37,8 @@
   {{- $metadata = merge $metadata (dict "scope" .Params.scope) -}}
 {{- end -}}
 {{- $json := $metadata | jsonify -}}
+{{- $toc := partial "toc-json-regex.html" . -}}
+{{- /* Manually insert tableOfContents into JSON string */ -}}
+{{- $json = $json | replaceRE `}$` (printf `,"tableOfContents":%s}` $toc) -}}
 {{- printf `<div hidden data-redis-metadata="page">%s</div>` $json | safeHTML -}}
 

layouts/partials/ai-metadata.html

@@ -37,5 +37,8 @@
   {{- $metadata = merge $metadata (dict "scope" .Params.scope) -}}
 {{- end -}}
 {{- $json := $metadata | jsonify -}}
+{{- $toc := partial "toc-json-regex.html" . -}}
+{{- /* Manually insert tableOfContents into JSON string */ -}}
+{{- $json = $json | replaceRE `}$` (printf `,"tableOfContents":%s}` $toc) -}}
 {{- printf `<script type="application/json" data-ai-metadata>%s</script>` $json | safeHTML -}}
 

layouts/partials/toc-json-regex.html

@@ -0,0 +1,28 @@
+{{- /* Convert Hugo's HTML table of contents to JSON using regex substitutions */ -}}
+{{- /* Input: page object */ -}}
+{{- /* Output: JSON structure representing the table of contents */ -}}
+{{- $toc := .TableOfContents -}}
+{{- /* Remove the nav wrapper and all newlines/extra whitespace */ -}}
+{{- $toc = $toc | replaceRE "<nav[^>]*>" "" -}}
+{{- $toc = $toc | replaceRE "</nav>" "" -}}
+{{- $toc = $toc | replaceRE "\\n\\s*" "" -}}
+{{- /* Step 1: Replace <ul> with opening bracket for children array */ -}}
+{{- $toc = $toc | replaceRE "<ul>" "[" -}}
+{{- $toc = $toc | replaceRE "</ul>" "]" -}}
+{{- /* Step 2: Replace <li><a href="#ID">TITLE</a> with {"id":"ID","title":"TITLE" */ -}}
+{{- $toc = $toc | replaceRE "<li><a href=\"#([^\"]+)\">([^<]+)</a>" "{\"id\":\"$1\",\"title\":\"$2\"" -}}
+{{- /* Step 3: Replace </li> with } */ -}}
+{{- $toc = $toc | replaceRE "</li>" "}" -}}
+{{- /* Step 4: Handle nested structure - replace ][ with ],[ (sibling arrays) */ -}}
+{{- $toc = $toc | replaceRE "\\]\\[" "],[" -}}
+{{- /* Step 4b: Handle nested structure - replace [  with ,"children":[ (child arrays) */ -}}
+{{- $toc = $toc | replaceRE "\"\\[" "\",\"children\":[" -}}
+{{- /* Step 5: Add commas between sibling objects - replace }{ with },{ */ -}}
+{{- $toc = $toc | replaceRE "\\}\\{" "},{" -}}
+{{- /* Step 6: Wrap the entire structure */ -}}
+{{- $toc = print "{\"sections\":" $toc "}" -}}
+{{- /* Step 7: Remove all newlines and extra whitespace for clean output */ -}}
+{{- $toc = $toc | replaceRE "\\n\\s*" "" -}}
+{{- /* Output the JSON - use safeHTML to prevent quote escaping */ -}}
+{{- $toc | safeHTML -}}
+