[Bug] crash when sleep but not wakeup in version 0.11.0

[torinchen]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

i start a server and try sleep wakeup api, see crash

Reproduction

step1
run_server.py
step2
test_time.py
then wait crash

Environment

lmdeploy==0.11.0

Error traceback

HINT:    Please open http://127.0.0.1:23333 in a browser for detailed api usage!!!                                                                                                                                      
HINT:    Please open http://127.0.0.1:23333 in a browser for detailed api usage!!!
HINT:    Please open http://127.0.0.1:23333 in a browser for detailed api usage!!!
INFO:     Started server process [226438]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:23333 (Press CTRL+C to quit)
INFO:     127.0.0.1:60974 - "POST /sleep?level=1 HTTP/1.1" 200 OK
INFO:     127.0.0.1:32806 - "POST /wakeup HTTP/1.1" 200 OK
INFO:     127.0.0.1:33350 - "POST /sleep?level=1 HTTP/1.1" 200 OK
INFO:     127.0.0.1:33362 - "POST /wakeup HTTP/1.1" 200 OK
INFO:     127.0.0.1:33970 - "POST /sleep?level=1 HTTP/1.1" 200 OK
Segmentation fault

[torinchen]

whl from https://files.pythonhosted.org/packages/cc/f3/4912d5ba3d1ea650aefcd405afef99adc2e4b2d67c9cc68ac5d84b0b4442/lmdeploy-0.11.0-cp310-cp310-manylinux2014_x86_64.whl