Eval bug: [Router] Claude Code - model not found - invalid_request_error

[isgallagher]

Name and Version

$ llama-server --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090 Ti, compute capability 8.6, VMM: yes
version: 7360 (53ecd4f)
built with GNU 12.2.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

Ryzen 5950x + nVidia 3090 Ti

Models

Devstral-Small-2-24B-Instruct-2512-UD-Q4_K_XL.gguf

Problem description & steps to reproduce

When I run the llama-server in router mode and connect Claude Code to it, the llama-server works, but Claude Code returns an immediate error:

⎿  API Error: 400 {"error":{"code":400,"message":"model not found","type":"invalid_request_error"}}

When running llama-server in normal single-model serve fashion (prior to router mode), Claude Code works fine.

First Bad Commit

This is an issue with the new router mode feature. This is not a bug with an existing feature.

Relevant log output

⎿  API Error: 400 {"error":{"code":400,"message":"model not found","type":"invalid_request_error"}}


Dec 12 16:42:18 llm-api-engine-rig2 llama-server[729483]: [56401] srv    operator(): all results received, terminating stream
Dec 12 16:42:18 llm-api-engine-rig2 llama-server[729483]: [56401] srv    operator(): http: stream ended
Dec 12 16:42:18 llm-api-engine-rig2 llama-server[729483]: [56401] srv  log_server_r: request: POST /v1/messages 127.0.0.1 200
Dec 12 16:42:18 llm-api-engine-rig2 llama-server[729483]: [56401] srv  log_server_r: request:  {"model":"Devstral","messages":[{"role":"user","content":[{"type":"text","text":"Hi"}]},{"role":"assistant","content":[{"type":"text","text":"{"}]}],"system":[{"type":"text","text":"You are Claude Code, Anthropic's official CLI for Claude."},{"type":"text","text":"Analyze if this message indicates a new conversation topic. If it does, extract a 2-3 word title that captures the new topic. Format your response as a JSON object with two fields: 'isNewTopic' (boolean) and 'title' (string, or null if isNewTopic is false). Only include these fields, no other text. ONLY generate the JSON object, no other text (eg. no markdown)."}],"tools":[],"metadata":{"user_id":"user_2b56a1ab9d1f7f3d80c12ffb75a944a2f012828d1d29c5d54c7cd4a28169ad91_account__session_edae2a8c-5630-4bb6-98fc-7386afc79814"},"max_tokens":32000,"stream":true}
Dec 12 16:42:18 llm-api-engine-rig2 llama-server[729483]: [56401] srv  log_server_r: response:
Dec 12 16:42:18 llm-api-engine-rig2 llama-server[729483]: [56401] res  remove_waiti: remove task 0 from waiting list. current waiting = 1 (before remove)
Dec 12 16:42:18 llm-api-engine-rig2 llama-server[729483]: [56401] srv          stop: all tasks already finished, no need to cancel
Dec 12 16:42:18 llm-api-engine-rig2 llama-server[729483]: srv  log_server_r: request: POST /v1/messages [removed] 200

[ServeurpersoCom]

This is expected behavior. Claude Code uses the Anthropic API format (/v1/messages), not OpenAI format (/v1/chat/completions).
llama-server, even in router mode, exposes an OpenAI-compatible API, not Anthropic-compatible.
You need to either:
Use an OpenAI-compatible CLI tool instead
Build a translation proxy that converts Anthropic API calls to OpenAI format

[isgallagher]

@ServeurpersoCom the Anthropic Messages API was added in a couple weeks ago. #17570

Again, this works when running llama-server in regular mode (not router mode).

[ServeurpersoCom]

Really interesting! I'll have to set up a proxy and test that too! So it's just the router that needs fixing.

[ServeurpersoCom]

Router work for me :

(root|~/llama.cpp.pascal) curl http://127.0.0.1:8082/v1/messages -H "Content-Type: application/json" -d '{"model":"Dense-Devstral-Small-2-24B-Instruct-2512","max_tokens":50,"stream":true,"messages":[{"role":"user","content":"Hi"}]}'
event: message_start
data: {"type":"message_start","message":{"id":"chatcmpl-MXKQFu8mNrm8uLy9rsV1dPZQSxXBP53I","type":"message","role":"assistant","content":[],"model":"Dense-Devstral-Small-2-24B-Instruct-2512","stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":556,"output_tokens":0}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"!"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" How"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" can"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" I"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" assist"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" you"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" today"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"?"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":10}}

event: message_stop
data: {"type":"message_stop"}

A problem with one of my proxies in anthropic mode misled me. So, with the correct model name in the query from Claude Code, what happens?

[ServeurpersoCom]

Extracted and beautified the Claude Code binary

(root|~/claude) cp /usr/bin/claude .
(root|~/claude) npm install js-beautify

added 53 packages in 242ms

14 packages are looking for funding
  run `npm fund` for details
(root|~/claude) npx js-beautify claude > claude-beautified.js
(root|~/claude) l
total 26184
-rwxr-xr-x  1 root root 10335889 13 déc.  10:39 claude
-rw-r--r--  1 root root 16439902 13 déc.  10:40 claude-beautified.js
drwxr-xr-x 46 root root     4096 13 déc.  10:39 node_modules
-rw-r--r--  1 root root       57 13 déc.  10:39 package.json
-rw-r--r--  1 root root    22560 13 déc.  10:39 package-lock.json
(root|~/claude)

Found an anthropic's internal dev mode (out of topic but interesting!)

(root|~/claude) sed -n '48250,48290p' claude-beautified.js
        case "prod":
            return Ug0
    }
}
var E9A = "user:inference",
    qR9 = "org:create_api_key",
    tu = "oauth-2025-04-20",
    LR9, EK1, Ng0, Ug0, MR9 = void 0,
    OR9;
var eJ = L(() => {
    rQ();
    LR9 = [qR9, "user:profile"], EK1 = ["user:profile", E9A, "user:sessions:claude_code"], Ng0 = Array.from(new Set([...LR9, ...EK1])), Ug0 = {
        BASE_API_URL: "https://api.anthropic.com",
        CONSOLE_AUTHORIZE_URL: "https://console.anthropic.com/oauth/authorize",
        CLAUDE_AI_AUTHORIZE_URL: "https://claude.ai/oauth/authorize",
        TOKEN_URL: "https://console.anthropic.com/v1/oauth/token",
        API_KEY_URL: "https://api.anthropic.com/api/oauth/claude_cli/create_api_key",
        ROLES_URL: "https://api.anthropic.com/api/oauth/claude_cli/roles",
        CONSOLE_SUCCESS_URL: "https://console.anthropic.com/buy_credits?returnUrl=/oauth/code/success%3Fapp%3Dclaude-code",
        CLAUDEAI_SUCCESS_URL: "https://console.anthropic.com/oauth/code/success?app=claude-code",
        MANUAL_REDIRECT_URL: "https://console.anthropic.com/oauth/code/callback",
        CLIENT_ID: "9d1c250a-e61b-44d9-88ed-5944d1962f5e",
        OAUTH_FILE_SUFFIX: ""
    }, OR9 = {
        BASE_API_URL: "http://localhost:3000",
        CONSOLE_AUTHORIZE_URL: "http://localhost:3000/oauth/authorize",
        CLAUDE_AI_AUTHORIZE_URL: "http://localhost:4000/oauth/authorize",
        TOKEN_URL: "http://localhost:3000/v1/oauth/token",
        API_KEY_URL: "http://localhost:3000/api/oauth/claude_cli/create_api_key",
        ROLES_URL: "http://localhost:3000/api/oauth/claude_cli/roles",
        CONSOLE_SUCCESS_URL: "http://localhost:3000/buy_credits?returnUrl=/oauth/code/success%3Fapp%3Dclaude-code",
        CLAUDEAI_SUCCESS_URL: "http://localhost:3000/oauth/code/success?app=claude-code",
        MANUAL_REDIRECT_URL: "https://console.staging.ant.dev/oauth/code/callback",
        CLIENT_ID: "22422756-60c9-4084-8eb7-27705fd5cf9a",
        OAUTH_FILE_SUFFIX: "-local-oauth"
    }
});
import {
    accessSync as RR9
} from "fs";
import {

Added debug logging to llama-server

(root|~/llama.cpp.pascal) git diff
diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cpp
index 6c618a673..8b9ce4d98 100644
--- a/tools/server/server-models.cpp
+++ b/tools/server/server-models.cpp
@@ -842,6 +842,7 @@ void server_models_routes::init_routes() {
         std::string method = "POST";
         json body = json::parse(req.body);
         std::string name = json_value(body, "model", std::string());
+        SRV_INF("client requested model: %s\n", name.c_str());
         bool autoload = is_autoload(params, req);
         auto error_res = std::make_unique<server_http_res>();
         if (!router_validate_model(name, models, autoload, error_res)) {
(root|~/llama.cpp.pascal)

Discovered Claude Code's actual model requests

srv  log_server_r: request: POST /v1/messages 127.0.0.1 400
srv    operator(): client requested model: claude-opus-4-5-20251101
srv  log_server_r: request: POST /v1/messages/count_tokens 127.0.0.1 400
srv    operator(): client requested model: claude-haiku-4-5-20251001
srv  log_server_r: request: POST /v1/messages 127.0.0.1 400
srv    operator(): client requested model: claude-opus-4-5-20251101
srv  log_server_r: request: POST /v1/messages 127.0.0.1 400

Adding a proper router block on my server :

[claude-opus-4-5-20251101]
m = unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/Devstral-Small-2-24B-Instruct-2512-UD-Q6_K_XL.gguf
c = 131072

Work !

[ServeurpersoCom]

This is cool! we can reuse Claude CLI in local mode with any model we want. All you need to do is declare a section with the model name in the router's "presets.ini". Having a small server-side variable that returns "API Error: 400 {"error":{"code":400,"message":"model XXXXXXXXXX not found","type":"invalid_request_error"}}" would be great because the error message would expose what model the client is requesting, without needing any other hacking/modification.

[ggerganov]

I thought you can already do that without any changes and just running:

ANTHROPIC_BASE_URL=http://localhost:8080 claude

ref: #17570

[ServeurpersoCom]

Yes Claude Code provides the ANTHROPIC_BASE_URL environment variable for API endpoint spoofing, but does not provide a model override variable model names remain hardcoded in the client (?) to use with router

[claude-opus-4-5-20251101] ; Spoof Claude Code default model (set ANTHROPIC_BASE_URL to redirect)
m = unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/Devstral-Small-2-24B-Instruct-2512-UD-Q6_K_XL.gguf
c = 131072

[isgallagher]

That is interesting. That is why it works when not in router mode because llama-server will respond to any model request. I should have anticipated this since I also had to add that into my old litellm config. Thanks all, not a bug and working as intended.

@ServeurpersoCom btw you can override the models (but it still has those hard coded ones it tries to access from the API endpoint):

export LOCAL_BASE_URL=http://llm-api-engine-rig2:8080
export LOCAL_AUTH_TOKEN="asdf"
unset ANTHROPIC_API_KEY
export ANTHROPIC_BASE_URL=${LOCAL_BASE_URL}
export ANTHROPIC_AUTH_TOKEN=${LOCAL_AUTH_TOKEN}

export API_TIMEOUT_MS=6000000
export ANTHROPIC_MODEL=Devstral
export ANTHROPIC_SMALL_FAST_MODEL=Devstral
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
export CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000

It didn't like my Devstral model alias, hence the above error, but ones I changed it to devstral-small-2-24b-instruct-2512 it suddenly quit having those errors.

; ------------------------------------------------------------
; devstral-small-2-24b-instruct-2512
; ------------------------------------------------------------
[devstral-small-2-24b-instruct-2512]
; alias = claude-opus-4-5-20251101
m = /ai_models/LLMs/mistralai/unsloth.Devstral-Small-2-24B-Instruct-2512-UD-Q4_K_XL.gguf
; mmproj = /ai_models/LLMs/mistralai/unsloth.Devstral-Small-2-24B-Instruct-2512-mmproj-F16.gguf
; ctx-size = 262144
temp = 0.15
min-p = 0.01
cache-type-k = q8_0
cache-type-v = q8_0
flash-attn = on
jinja = true
kv-unified = true
cont-batching = true
threads = 12
threads-batch = 24
batch-size = 8192
ubatch-size = 4096

; 131k context size
;ctx-size = 131072
;n-gpu-layers = 30

; 70k context size
ctx-size = 70000
n-gpu-layers = 99
;v = true