test(llm): add model initialization scenario tests

[Pouyanpi]

Description

Adds comprehensive test coverage for the LangChain model initialization flow in nemoguardrails/llm/models/langchain_initializer.py. This test suite validates:

• Success scenarios: all 4 initialization methods (special cases, chat completion, community chat, text completion)
• Error handling: single and multiple error scenarios with proper error preservation
• Exception priority: ImportError prioritization and last-exception-wins behavior
• Error recovery: successful fallback when early initializers fail
• Special case handling: gpt-3.5-turbo-instruct and nvidia provider edge cases
• Mode filtering: correct initializer selection for chat vs text mode
• Edge cases: None providers, empty model names, invalid modes
• E2E integration: full flow through RailsConfig -> LLMRails -> model initialization

Test Coverage

• 27 test scenarios organized into 8 test classes
• Tests verify both PR fix: Surface relevant exception when initializing langchain model #1516 (exception surfacing) and the TypeError fix (f54cd20)
• Uses MockProvider factory pattern for clean, reusable test setup
• Includes detailed documentation of initialization order and exception priority rules

Validation: Tests Fail Before Fix, Pass After Fix

8 tests that fail without PR #1516 (proving the fix works):

These tests verify that meaningful errors are preserved instead of being masked by "Could not find LLM provider" RuntimeErrors:

1. TestSingleErrorScenarios::test_single_error_preserved[chat_error_preserved]

   • Before fix: "Could not find LLM provider '_test_err'"
   • After fix: "Invalid API key"

2. TestSingleErrorScenarios::test_single_error_preserved[community_error_preserved]

   • Before fix: "Could not find LLM provider '_test_err'"
   • After fix: "Rate limit exceeded"

3. TestMultipleErrorPriority::test_exception_priority[valueerror_last_wins]

   • Before fix: "Could not find LLM provider '_test_priority'"
   • After fix: "Error B" (last ValueError wins)

4. TestMultipleErrorPriority::test_exception_priority[different_types_last_wins]

   • Before fix: "Could not find LLM provider '_test_priority'"
   • After fix: "Error B" (last exception wins)

5. TestE2EIntegration::test_e2e_meaningful_error_from_config

   • Before fix: "Could not find LLM provider '_e2e_test'"
   • After fix: "Invalid API key"

6. TestMultipleErrorScenarios::test_all_initializers_raise_valueerror_last_one_wins

   • Before fix: "Could not find LLM provider 'fake_provider'"
   • After fix: "Community chat error"

7. TestMultipleErrorScenarios::test_chat_and_community_both_fail_community_wins

   • Before fix: "Could not find LLM provider '_test_chat_community_fail'"
   • After fix: "rate limit exceeded"

8. TestMultipleErrorScenarios::test_runtimeerror_vs_valueerror_last_wins

   • Before fix: "Could not find LLM provider '_test_runtime_vs_value'"
   • After fix: "ValueError from community"

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Overview

Greptile Summary

This PR adds comprehensive test coverage for the LangChain model initialization flow and reorganizes the test directory structure.

Key Changes:

• Added test_langchain_init_scenarios.py with 27 test scenarios covering success paths, error handling, exception priority (ImportError prioritization), error recovery, and E2E integration through RailsConfig → LLMRails
• Reorganized tests from tests/llm_providers/ to a more logical structure: tests/llm/models/ for initializer tests and tests/llm/providers/ for provider-specific tests

Test Coverage Validates:

• PR fix: Surface relevant exception when initializing langchain model #1516 fix: Meaningful exceptions are now surfaced instead of being masked by "Could not find LLM provider" RuntimeErrors
• The TypeError fix (f54cd20) for _handle_model_special_cases handling None returns correctly
• All 4 initialization paths: special cases, chat completion, community chat, and text completion
• Exception priority rules: first ImportError wins, otherwise last exception wins

Test Organization:

• tests/llm/models/ - Model initialization logic tests
• tests/llm/providers/ - Provider registration and discovery tests
• tests/llm/ - Integration and compatibility tests

Confidence Score: 5/5

• This PR is safe to merge - it only adds and reorganizes test files with no changes to production code
• Score of 5 reflects that this is a test-only PR with no production code changes. The tests are well-structured with proper cleanup fixtures, mocking patterns, and clear documentation. The reorganization follows logical grouping conventions.
• No files require special attention - all changes are test files with proper isolation and cleanup

Important Files Changed

File Analysis

Filename	Score	Overview
tests/llm/models/test_langchain_init_scenarios.py	5/5	New comprehensive test file (990 lines) with 27 test scenarios covering model initialization paths, exception priority, error recovery, and E2E integration for PR #1516
tests/llm/models/test_langchain_initialization_methods.py	5/5	Renamed from tests/llm_providers/; tests individual initialization methods (_init_chat_completion_model, _init_community_chat_models, _init_text_completion_model)
tests/llm/models/test_langchain_initializer.py	5/5	Renamed from tests/llm_providers/; unit tests for init_langchain_model with mocked initializers to verify call order and exception handling
tests/llm/models/test_langchain_special_cases.py	5/5	Renamed from tests/llm_providers/; tests special case handlers for gpt-3.5-turbo-instruct and NVIDIA provider initialization
tests/llm/providers/test_langchain_nvidia_ai_endpoints_patch.py	5/5	Renamed from tests/llm_providers/; tests ChatNVIDIA streaming decorator, async generation, and LLMRails integration
tests/llm/test_langchain_integration.py	5/5	Renamed from tests/llm_providers/; integration tests for LangChain model initialization with OpenAI

Sequence Diagram

sequenceDiagram
    participant Test as Test Suite
    participant init as init_langchain_model()
    participant special as _handle_model_special_cases
    participant chat as _init_chat_completion_model
    participant community as _init_community_chat_models
    participant text as _init_text_completion_model

    Test->>init: init_langchain_model(model, provider, mode, kwargs)
    
    alt Chat Mode
        init->>special: Try special cases first
        special-->>init: None (no match) or Model
        
        alt Special returns None
            init->>chat: Try chat completion
            chat-->>init: Model or raises ValueError
            
            alt Chat returns None/error
                init->>community: Try community chat
                community-->>init: Model or raises ValueError
                
                alt Community returns None/error
                    init->>text: Try text completion (fallback)
                    text-->>init: Model or None
                end
            end
        end
    else Text Mode
        init->>special: Try special cases
        special-->>init: None (no match) or Model
        
        alt Special returns None
            Note over init,chat: Skip chat-only initializers
            init->>text: Try text completion
            text-->>init: Model or None
        end
    end

    alt All initializers fail
        init-->>Test: ModelInitializationError<br/>(ImportError prioritized, else last exception)
    else Success
        init-->>Test: Return initialized model
    end

Loading

[greptile-apps]

_{10 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}