feat: Add token output format and refactor code architecture

[AVGVSTVS96]

Adds a new tokens output format that returns raw Shiki token data for custom rendering, plus refactors the codebase architecture for better maintainability.

Token Output Format

The hook now supports three output formats:

Format	Return Type	Use Case
'react'	ReactNode	Default, safe rendering
'html'	string	Faster, requires dangerouslySetInnerHTML
'tokens'	TokensResult	Custom rendering, full control

Usage

import { useShikiHighlighter, type TokensResult } from "react-shiki";

const result = useShikiHighlighter(code, "typescript", "github-dark", {
  outputFormat: 'tokens'
});

// result: TokensResult | null
// {
//   tokens: ThemedToken[][],  // Token array per line
//   fg: string,               // Foreground color
//   bg: string,               // Background color  
//   themeName: string,        // Theme identifier
//   rootStyle: string         // CSS style string
// }

Custom Rendering Example

function CustomCodeBlock({ code, language }) {
  const result = useShikiHighlighter(code, language, "nord", {
    outputFormat: 'tokens'
  });

  if (!result) return <pre>{code}</pre>;

  return (
    <pre style={{ background: result.bg, color: result.fg }}>
      <code>
        {result.tokens.map((line, i) => (
          <div key={i} className="line">
            {line.map((token, j) => (
              <span key={j} style={{ color: token.color }}>
                {token.content}
              </span>
            ))}
          </div>
        ))}
      </code>
    </pre>
  );
}

When to Use Tokens

• Building custom code editors
• Implementing line-by-line animations
• Adding interactive token features (hover, click)
• Integrating with virtual scrolling libraries

Note: Token output is only available via the hook. The component supports 'react' and 'html' formats only.

Architecture Refactor

• lib/output.ts - Output transformation logic (React nodes, HTML, tokens)
• lib/options.ts - Shiki options builder
• Generic type narrowing - Return type inferred from outputFormat option

Test plan

• Added tests for token output format
• All existing tests pass
• Type inference works correctly for all output formats

[changeset-bot]

🦋 Changeset detected

Latest commit: 04bd967

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
react-shiki	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
react-shiki	Ready	Preview	Comment	Dec 5, 2025 10:20am

[claude]

Claude finished @AVGVSTVS96's task —— View job

---

Code Review Complete ✅

I've completed a comprehensive review of the token output format feature and architecture refactoring. This is an excellent implementation with clean architecture and solid engineering practices.

Review Checklist

• Read repository guidelines (CLAUDE.md) - None found
• Review architecture changes and refactoring - Excellent separation of concerns
• Check code quality and best practices - High quality implementation
• Analyze potential bugs and performance issues - No major concerns identified
• Verify test coverage - Comprehensive tests including tokens format
• Review documentation updates - Well documented
• Provide comprehensive feedback

---

✅ Positive Highlights

Excellent Architecture Refactoring

• Clean separation of concerns: Splitting resolvers.ts into domain-focused theme.ts and language.ts modules significantly improves maintainability
• Type-safe generics: The OutputFormatMap and generic type narrowing for return values is excellent - provides compile-time safety
• Strategic module extraction: lib/output.ts and lib/options.ts create clear responsibilities

Robust Implementation

• Comprehensive test coverage: The new token output tests in package/tests/hook.test.tsx:417-443 properly validate the TokensResult structure
• Smart fallback handling: Multi-theme vs single-theme logic in transformToTokens (package/src/lib/output.ts:55-79) handles edge cases well
• Type safety: Component restricts outputFormat to 'react' | 'html' at compile-time, preventing tokens usage where not appropriate

Documentation & Developer Experience

• Clear API documentation: README updates properly explain the three output formats with usage examples
• Helpful code comments: JSDoc comments explain the "why" behind design decisions (like the cast justification in output.ts:100)

---

🔍 Technical Analysis

Performance Considerations

✅ Efficient transformers: The transformer registry pattern in output.ts:81-85 avoids function lookups
✅ Smart caching: Maintaining the existing memoization patterns with revision tracking

Type Safety

✅ Excellent generic usage: UseShikiHighlighter<F extends OutputFormat = 'react'> provides perfect type narrowing
✅ Proper type casts: The cast in output.ts:101 is well-justified with a comment explaining why it's safe

Architecture Quality

✅ Single Responsibility: Each module has a clear, focused purpose
✅ Dependency Direction: Clean dependency flow without circular references
✅ Extensibility: Easy to add new output formats by extending the registry

---

📋 Requirements Verification

• ✅ Changeset included: .changeset/curvy-baboons-play.md properly describes the minor version changes
• ✅ Documentation updated: README includes comprehensive token output documentation
• ✅ Test coverage: New tests specifically validate token output format
• ✅ Clean architecture: Excellent refactoring with clear module boundaries

---

🚀 Minor Suggestions for Future Consideration

1. Performance optimization opportunity: Consider memoizing the transformer selection in transformOutput if called frequently with the same format

2. Type refinement: The LooseHighlighter type in output.ts:18-24 could potentially be more precise about which methods are being widened

3. Error handling: Consider adding validation for invalid outputFormat values, though TypeScript mostly prevents this

---

🎯 Conclusion

This is a high-quality implementation that demonstrates excellent software engineering practices. The architecture refactoring significantly improves code organization, and the token output feature is implemented with proper type safety and comprehensive testing. The code is clean, well-documented, and follows React/TypeScript best practices.

Recommendation: ✅ Ready to merge - this PR enhances the library's capabilities while maintaining excellent code quality.

[AVGVSTVS96]

• perf: add deferRender prop for viewport-based lazy highlighting #113
• perf: improve throttling with useThrottledDebounce hook #112
• refactor: simplify useStableOptions hook to improve dependency tracking #109
• feat: Add token output format and refactor code architecture #108 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.