Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

## Project Overview

MelonAccessibilityLib is a C# library that adds screen reader accessibility to Unity games via MelonLoader mods. It provides speech management, text cleaning utilities, and P/Invoke wrappers for UniversalSpeech with SAPI fallback.
MelonAccessibilityLib is a C# library that adds screen reader accessibility to Unity games via MelonLoader mods. It provides speech and braille output, text cleaning utilities, and P/Invoke wrappers for UniversalSpeech with SAPI fallback.

## Build Commands

Expand Down Expand Up @@ -34,9 +34,9 @@ No test framework is currently configured. If adding tests, use standard `dotnet

### Component Overview

- **SpeechManager** (`SpeechManager.cs`): High-level static API for speech output with duplicate prevention, repeat functionality, and text formatting
- **UniversalSpeechWrapper** (`UniversalSpeechWrapper.cs`): Low-level P/Invoke wrapper for UniversalSpeech.dll with SAPI fallback
- **TextCleaner** (`TextCleaner.cs`): Removes Unity rich text tags and normalizes text for screen reader output
- **SpeechManager** (`SpeechManager.cs`): High-level static API for speech and braille output with duplicate prevention, repeat functionality, and text formatting
- **UniversalSpeechWrapper** (`UniversalSpeechWrapper.cs`): Low-level P/Invoke wrapper for UniversalSpeech.dll with SAPI fallback; provides `Speak()` and `DisplayBraille()` methods
- **TextCleaner** (`TextCleaner.cs`): Removes Unity rich text tags and normalizes text; supports custom string and regex replacements via `AddReplacement()` and `AddRegexReplacement()`
- **AccessibilityLog/IAccessibilityLogger** (`IAccessibilityLogger.cs`): Logging facade with pluggable logger interface
- **Net35Extensions** (`Net35Extensions.cs`): Polyfills for .NET 3.5 compatibility (e.g., `IsNullOrWhiteSpace`)

Expand All @@ -50,8 +50,9 @@ Consumer (MelonMod)
└─ Calls SpeechManager.Output()
├─ Duplicate suppression (time-based)
├─ TextCleaner.Clean() (strips rich text)
└─ UniversalSpeechWrapper.Speak() (P/Invoke)
├─ TextCleaner.Clean() (strips rich text, applies custom replacements)
├─ UniversalSpeechWrapper.Speak() (P/Invoke)
└─ UniversalSpeechWrapper.DisplayBraille() (if EnableBraille)
```

### Extensibility Points
Expand All @@ -60,6 +61,8 @@ Consumer (MelonMod)
- **Text Formatting**: Set `SpeechManager.FormatTextOverride` delegate
- **Repeat Logic**: Set `SpeechManager.ShouldStoreForRepeatPredicate` delegate
- **Custom Text Types**: Use constants starting from `TextType.CustomBase` (100)
- **Text Cleaning**: Use `TextCleaner.AddReplacement()` and `TextCleaner.AddRegexReplacement()` for custom text transformations
- **Braille Control**: Set `SpeechManager.EnableBraille` to toggle braille output (default: true)

## Key Conventions

Expand Down
95 changes: 88 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@ A reusable library for adding screen reader accessibility to Unity games via Mel
## Features

- **UniversalSpeech integration** - P/Invoke wrapper for the UniversalSpeech library with SAPI fallback
- **Braille display support** - Automatic output to braille displays via screen reader
- **High-level speech manager** - Duplicate prevention, repeat functionality, speaker formatting
- **Text cleaning** - Strips Unity rich text tags (`<color>`, `<size>`, `<b>`, etc.)
- **Text cleaning** - Strips Unity rich text tags (`<color>`, `<size>`, `<b>`, etc.) with extensible custom replacements
- **Logging abstraction** - Integrate with MelonLoader or any logging system
- **Multi-target support** - Builds for net6.0, net472, and net35 for broad compatibility with various games

Expand Down Expand Up @@ -123,13 +124,16 @@ SpeechManager.RepeatLast();
| `Stop()` | Stop current speech. |
| `ClearRepeatBuffer()` | Clear stored repeat text. |

| Property | Description |
| ------------------------------- | ----------------------------------------------------- |
| `DuplicateWindowSeconds` | Time window for duplicate suppression (default: 0.5s) |
| `EnableLogging` | Whether to log speech output (default: true) |
| `ShouldStoreForRepeatPredicate` | Custom predicate for repeat storage |
| Property | Description |
| ------------------------------- | -------------------------------------------------------------- |
| `DuplicateWindowSeconds` | Time window for duplicate suppression (default: 0.5s) |
| `EnableLogging` | Whether to log speech output (default: true) |
| `EnableBraille` | Whether to output to braille displays (default: true) |
| `FormatTextOverride` | Custom delegate for text formatting (see Extensibility) |
| `ShouldStoreForRepeatPredicate` | Custom predicate for repeat storage (see Extensibility) |
| `TextTypeNames` | Dictionary mapping text type IDs to names for logging |

### TextType Enum
### TextType Constants

| Value | Description |
| ------------ | ------------------------------------------------- |
Expand All @@ -138,6 +142,7 @@ SpeechManager.RepeatLast();
| `Menu` | Menu item text |
| `MenuChoice` | Menu selection |
| `System` | System messages |
| `CustomBase` | Base value (100) for defining custom text types |

### UniversalSpeechWrapper

Expand All @@ -146,12 +151,15 @@ Low-level access to UniversalSpeech:
```csharp
UniversalSpeechWrapper.Initialize(); // Initialize (called by SpeechManager)
UniversalSpeechWrapper.Speak("text", true); // Speak with interrupt
UniversalSpeechWrapper.DisplayBraille("text"); // Output to braille display
UniversalSpeechWrapper.Stop(); // Stop speech
UniversalSpeechWrapper.IsScreenReaderActive(); // Check if screen reader is running
```

### TextCleaner

Basic usage:

```csharp
string clean = TextCleaner.Clean("<color=#ff0000>Red text</color>");
// Result: "Red text"
Expand All @@ -160,6 +168,22 @@ string combined = TextCleaner.CombineLines("Line 1", "<b>Line 2</b>", "Line 3");
// Result: "Line 1 Line 2 Line 3"
```

Custom text replacements (applied after tag removal):

```csharp
// Simple string replacement
TextCleaner.AddReplacement("♥", "heart");
TextCleaner.AddReplacement("→", "arrow");

// Regex replacement
TextCleaner.AddRegexReplacement(@"\[(\d+)\]", "footnote $1");

// Clear custom replacements
TextCleaner.ClearReplacements(); // Clear string replacements only
TextCleaner.ClearRegexReplacements(); // Clear regex replacements only
TextCleaner.ClearAllCustomReplacements(); // Clear all
```

### AccessibilityLog

```csharp
Expand All @@ -183,6 +207,63 @@ The library includes `Net35Extensions` with polyfills for methods not available

- `Net35Extensions.IsNullOrWhiteSpace(string)` - Use instead of `string.IsNullOrWhiteSpace`

## Extensibility

### Custom Text Types

Define custom text types for game-specific content:

```csharp
public static class MyTextTypes
{
public const int Tutorial = TextType.CustomBase + 1; // 101
public const int Combat = TextType.CustomBase + 2; // 102
public const int Inventory = TextType.CustomBase + 3; // 103
}

// Register names for logging
SpeechManager.TextTypeNames = new Dictionary<int, string>
{
{ TextType.Dialogue, "Dialogue" },
{ TextType.Narrator, "Narrator" },
{ MyTextTypes.Tutorial, "Tutorial" },
{ MyTextTypes.Combat, "Combat" },
};

// Use custom types
SpeechManager.Announce("Press A to jump", MyTextTypes.Tutorial);
```

### Custom Text Formatting

Override how text is formatted before output:

```csharp
SpeechManager.FormatTextOverride = (speaker, text, textType) =>
{
// Custom formatting logic
if (textType == MyTextTypes.Combat)
return $"Combat: {text}";
if (!string.IsNullOrEmpty(speaker))
return $"{speaker} says: {text}";
return text;
};
```

### Custom Repeat Storage

Control which text types are stored for repeat functionality:

```csharp
SpeechManager.ShouldStoreForRepeatPredicate = (textType) =>
{
// Store dialogue, narrator, and tutorial text for repeat
return textType == TextType.Dialogue
|| textType == TextType.Narrator
|| textType == MyTextTypes.Tutorial;
};
```

## UniversalSpeech Setup

1. Download UniversalSpeech from [GitHub](https://github.com/qtnc/UniversalSpeech)
Expand Down
18 changes: 18 additions & 0 deletions SpeechManager.cs
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,13 @@ public static class SpeechManager
/// </summary>
public static bool EnableLogging { get; set; } = true;

/// <summary>
/// Whether to send output to braille displays. Default is true.
/// When enabled, all speech output is also sent to the braille display
/// via the screen reader's braille support.
/// </summary>
public static bool EnableBraille { get; set; } = true;

/// <summary>
/// Initialize the speech system.
/// </summary>
Expand Down Expand Up @@ -74,6 +81,12 @@ public static void Output(string speaker, string text, int textType = TextType.D
// Output via speech
UniversalSpeechWrapper.Speak(formattedText);

// Output via braille if enabled
if (EnableBraille)
{
UniversalSpeechWrapper.DisplayBraille(formattedText);
}

if (EnableLogging)
{
string typeName =
Expand Down Expand Up @@ -104,6 +117,11 @@ public static void RepeatLast()
string formattedText = FormatText(_currentSpeaker, _currentText, _currentType);
UniversalSpeechWrapper.Speak(formattedText);

if (EnableBraille)
{
UniversalSpeechWrapper.DisplayBraille(formattedText);
}

if (EnableLogging)
{
AccessibilityLog.Msg($"Repeating: '{formattedText}'");
Expand Down
32 changes: 32 additions & 0 deletions UniversalSpeechWrapper.cs
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,15 @@ int interrupt
[DllImport(DLL_NAME, CallingConvention = CallingConvention.Cdecl)]
private static extern int speechStop();

[DllImport(
DLL_NAME,
CallingConvention = CallingConvention.Cdecl,
CharSet = CharSet.Unicode
)]
private static extern int brailleDisplay(
[MarshalAs(UnmanagedType.LPWStr)] string str
);

[DllImport(DLL_NAME, CallingConvention = CallingConvention.Cdecl)]
private static extern int speechSetValue(int what, int value);

Expand Down Expand Up @@ -104,6 +113,29 @@ public static void Speak(string text, bool interrupt = false)
}
}

/// <summary>
/// Display the given text on a braille display via the current screen reader.
/// </summary>
/// <param name="text">The text to display on braille</param>
public static void DisplayBraille(string text)
{
if (!_initialized || !_dllAvailable || Net35Extensions.IsNullOrWhiteSpace(text))
return;

try
{
brailleDisplay(text);
}
catch (DllNotFoundException)
{
_dllAvailable = false;
}
catch (Exception ex)
{
AccessibilityLog.Error($"Braille display error: {ex.Message}");
}
}

/// <summary>
/// Stop any currently playing speech.
/// </summary>
Expand Down