[Bug] speculative report error when PD agg #14368
Description
Checklist
- I searched related issues but found no solution.
- The bug persists in the latest version.
- Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
- If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
- Please use English. Otherwise, it will be closed.
Describe the bug
lack of concern when prefill with speculative decoding
log report length of new_indices: 5 != length of topk_p: 7, this should not happen
[2025-12-03 15:11:42] INFO: 61.173.159.58:33314 - "POST /flush_cache HTTP/1.1" 200 OK
[2025-12-03 15:11:42 TP0] Cache flushed successfully!
[2025-12-03 15:11:42 TP2] Cache flushed successfully!
[2025-12-03 15:11:42 TP1] Cache flushed successfully!
[2025-12-03 15:11:42 TP6] Cache flushed successfully!
[2025-12-03 15:11:42 TP4] Cache flushed successfully!
[2025-12-03 15:11:42 TP5] Cache flushed successfully!
[2025-12-03 15:11:42 TP3] Cache flushed successfully!
[2025-12-03 15:11:42 TP7] Cache flushed successfully!
[2025-12-03 15:11:42] INFO: 61.173.159.58:33315 - "POST /flush_cache HTTP/1.1" 200 OK
[2025-12-03 15:11:42] INFO: 61.173.159.58:33318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:42 TP0] Prefill batch, #new-seq: 1, #new-token: 640, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-12-03 15:11:42] INFO: 61.173.159.58:33316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:42] INFO: 61.173.159.58:33319 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:42] INFO: 61.173.159.58:33317 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:42] INFO: 61.173.159.58:33322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:42] INFO: 61.173.159.58:33323 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:42] INFO: 61.173.159.58:33320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:42] INFO: 61.173.159.58:33321 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:42 TP0] Prefill batch, #new-seq: 7, #new-token: 6016, #cached-token: 0, token usage: 0.00, #running-req: 1, #queue-req: 0,
[2025-12-03 15:11:42 TP0] Req Time Stats(rid=d8b4aad44a664515986c3e542929245d, input len=192, output len=1, type=unified): queue_duration=1.72ms, forward_duration=213.59ms, start_time=1929576.255
[2025-12-03 15:11:42 TP0] Req Time Stats(rid=4d442f11f6bf4cedbaf4d8fdd69d7250, input len=2028, output len=1, type=unified): queue_duration=1.64ms, forward_duration=214.51ms, start_time=1929576.255
[2025-12-03 15:11:42 TP0] length of new_indices: 5 != length of topk_p: 7, this should not happen
[2025-12-03 15:11:42 TP1] length of new_indices: 5 != length of topk_p: 7, this should not happen
[2025-12-03 15:11:42 TP2] length of new_indices: 5 != length of topk_p: 7, this should not happen
[2025-12-03 15:11:42 TP4] length of new_indices: 5 != length of topk_p: 7, this should not happen
[2025-12-03 15:11:42 TP6] length of new_indices: 5 != length of topk_p: 7, this should not happen
[2025-12-03 15:11:42 TP5] length of new_indices: 5 != length of topk_p: 7, this should not happen
[2025-12-03 15:11:42 TP3] length of new_indices: 5 != length of topk_p: 7, this should not happen
[2025-12-03 15:11:42 TP7] length of new_indices: 5 != length of topk_p: 7, this should not happen
[2025-12-03 15:11:42] INFO: 61.173.159.58:33325 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:42] INFO: 61.173.159.58:33324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:42 TP0] Prefill batch, #new-seq: 2, #new-token: 2816, #cached-token: 0, token usage: 0.01, #running-req: 6, #queue-req: 0,
[2025-12-03 15:11:43] INFO: 61.173.159.58:33326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:43] INFO: 61.173.159.58:33331 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:43] INFO: 61.173.159.58:33327 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:43] INFO: 61.173.159.58:33332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:43] INFO: 61.173.159.58:33329 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:43 TP0] Prefill batch, #new-seq: 5, #new-token: 4288, #cached-token: 0, token usage: 0.02, #running-req: 8, #queue-req: 0,
[2025-12-03 15:11:43] INFO: 61.173.159.58:33333 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:43] INFO: 61.173.159.58:33330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:43] INFO: 61.173.159.58:33328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-03 15:11:43 TP0] Prefill batch, #new-seq: 2, #new-token: 14080, #cached-token: 0, token usage: 0.03, #running-req: 13, #queue-req: 1,
[2025-12-03 15:11:44 TP0] Prefill batch, #new-seq: 1, #new-token: 4672, #cached-token: 0, token usage: 0.06, #running-req: 15, #queue-req: 0,
Reproduction
SGLANG_ENABLE_JIT_DEEPGEMM=1 python -m sglang.launch_server --model-path /data/DeepSeek-V3.2 --served-model-name deepseekv32 --trust-remote-code --watchdog-timeout 1000000 --page-size 128 --reasoning-parser deepseek-v3 --host 0.0.0.0 --port 8003 --mem-fraction-static 0.8 --max-running-requests 32 --context-length 163840 --max-prefill-tokens 16384 --chunked-prefill-size 65536 --attention-backend fa3 --tp-size 8 --tool-call-parser deepseekv32 --enable-nan-detection --speculative-algorithm EAGLE --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2
Environment
Python: 3.12.12 (main, Oct 10 2025, 08:52:57) [GCC 11.4.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H200
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.9, V12.9.86
CUDA Driver Version: 580.95.05
PyTorch: 2.9.1+cu129
sglang: 0.5.6
sgl_kernel: 0.3.18.post2
flashinfer_python: 0.5.3
flashinfer_cubin: 0.5.3
flashinfer_jit_cache: Module Not Found
triton: 3.5.1
transformers: 4.57.1
torchao: 0.9.0
numpy: 2.3.5
aiohttp: 3.13.2
fastapi: 0.123.5
hf_transfer: 0.1.9
huggingface_hub: 0.36.0
interegular: 0.3.3
modelscope: 1.32.0
orjson: 3.11.4
outlines: 0.1.11
packaging: 25.0
psutil: 7.1.3
pydantic: 2.12.5
python-multipart: 0.0.20
pyzmq: 27.1.0
uvicorn: 0.38.0
uvloop: 0.22.1
vllm: Module Not Found
xgrammar: 0.1.27
openai: 2.6.1
tiktoken: 0.12.0
anthropic: 0.75.0
litellm: Module Not Found
decord2: 2.0.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX PIX NODE NODE NODE SYS SYS SYS SYS SYS 0,2,4,6,8,10 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE PIX NODE NODE SYS SYS SYS SYS SYS 0,2,4,6,8,10 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE NODE PIX NODE SYS SYS SYS SYS SYS 0,2,4,6,8,10 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE PIX SYS SYS SYS SYS SYS 0,2,4,6,8,10 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS PIX NODE NODE NODE PIX 1,3,5,7,9,11 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS NODE PIX NODE NODE NODE 1,3,5,7,9,11 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS NODE NODE PIX NODE NODE 1,3,5,7,9,11 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS NODE NODE NODE PIX NODE 1,3,5,7,9,11 1 N/A
NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X PIX NODE NODE NODE SYS SYS SYS SYS SYS
NIC1 PIX NODE NODE NODE SYS SYS SYS SYS PIX X NODE NODE NODE SYS SYS SYS SYS SYS
NIC2 NODE PIX NODE NODE SYS SYS SYS SYS NODE NODE X NODE NODE SYS SYS SYS SYS SYS
NIC3 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE NODE X NODE SYS SYS SYS SYS SYS
NIC4 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE NODE X SYS SYS SYS SYS SYS
NIC5 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS SYS X NODE NODE NODE PIX
NIC6 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS SYS NODE X NODE NODE NODE
NIC7 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS SYS NODE NODE X NODE NODE
NIC8 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS SYS NODE NODE NODE X NODE
NIC9 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS SYS PIX NODE NODE NODE X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: ib6s200p0
NIC1: ib7s400p0
NIC2: ib7s400p1
NIC3: ib7s400p2
NIC4: ib7s400p3
NIC5: ib7s400p4
NIC6: ib7s400p5
NIC7: ib7s400p6
NIC8: ib7s400p7
NIC9: mlx5_bond_0
ulimit soft: 1048576
Metadata
Metadata
Assignees
Labels
No labels