qwen3-vl fails on photos sporadically #1182
Description
Doing some testing on unicorn to see if object-detection-llm should be made the standard on pegasus. Brief answer, not yet. It still seems to crash or time out on some photo inputs, and I haven't tried that many. In this case, it is the street scene photo directly from the IMAGE website:
https://image.a11y.mcgill.ca/tutorial/
First time I tried it, it took a very very long time, then returned an error. Second time it worked ok. Logs from first attempt below.
I don't think this is reliable enough yet for production deploy.
image-unicorn-cim-mcgill-ca-1 | 2025-12-03T18:48:59.540611445Z 174.88.167.45 - - [03/Dec/2025:18:48:59 +0000] "POST /render HTTP/1.1" 504 569 "-" "Mozilla/5.0
(X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36"
object-detection-llm-1 | 2025-12-03T18:49:17.557237037Z [2025-12-03 18:49:17 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:8)
object-detection-llm-1 | 2025-12-03T18:49:17.565002993Z [2025-12-03 18:49:17 +0000] [8] [ERROR] Error handling request /preprocessor
object-detection-llm-1 | 2025-12-03T18:49:17.565029372Z Traceback (most recent call last):
object-detection-llm-1 | 2025-12-03T18:49:17.565038059Z File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/sync.py", line 134, in handle
object-detection-llm-1 | 2025-12-03T18:49:17.565046184Z self.handle_request(listener, req, client, addr)
object-detection-llm-1 | 2025-12-03T18:49:17.565053918Z File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/sync.py", line 177, in handle_r
equest
object-detection-llm-1 | 2025-12-03T18:49:17.565062264Z respiter = self.wsgi(environ, resp.start_response)
object-detection-llm-1 | 2025-12-03T18:49:17.565070700Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565078444Z File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1536, in __call__
object-detection-llm-1 | 2025-12-03T18:49:17.565086339Z return self.wsgi_app(environ, start_response)
object-detection-llm-1 | 2025-12-03T18:49:17.565093873Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565101247Z File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1511, in wsgi_app
object-detection-llm-1 | 2025-12-03T18:49:17.565109362Z response = self.full_dispatch_request()
object-detection-llm-1 | 2025-12-03T18:49:17.565117197Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565124932Z File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 917, in full_dispatch_reques
t
object-detection-llm-1 | 2025-12-03T18:49:17.565132957Z rv = self.dispatch_request()
object-detection-llm-1 | 2025-12-03T18:49:17.565140962Z ^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565148736Z File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 902, in dispatch_request
object-detection-llm-1 | 2025-12-03T18:49:17.565156791Z return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-
any-return]
object-detection-llm-1 | 2025-12-03T18:49:17.565164997Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565172801Z File "/app/object-detection-llm.py", line 171, in detect_objects
object-detection-llm-1 | 2025-12-03T18:49:17.565180576Z qwen_output = llm_client.chat_completion(
object-detection-llm-1 | 2025-12-03T18:49:17.565188481Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565213287Z File "/app/utils/llm/client.py", line 160, in chat_completion
object-detection-llm-1 | 2025-12-03T18:49:17.565225931Z response = self.client.chat.completions.create(**params)
object-detection-llm-1 | 2025-12-03T18:49:17.565250567Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565273300Z File "/usr/local/lib/python3.11/site-packages/openai/_utils/_utils.py", line 287, in wrapper
object-detection-llm-1 | 2025-12-03T18:49:17.565289871Z return func(*args, **kwargs)
object-detection-llm-1 | 2025-12-03T18:49:17.565302064Z ^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565311852Z File "/usr/local/lib/python3.11/site-packages/openai/resources/chat/completions/completions.py
", line 1131, in create
object-detection-llm-1 | 2025-12-03T18:49:17.565316911Z return self._post(
object-detection-llm-1 | 2025-12-03T18:49:17.565321169Z ^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565325307Z File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1256, in post
object-detection-llm-1 | 2025-12-03T18:49:17.565338091Z return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
object-detection-llm-1 | 2025-12-03T18:49:17.565343020Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565347589Z File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 979, in request
object-detection-llm-1 | 2025-12-03T18:49:17.565352127Z response = self._client.send(
object-detection-llm-1 | 2025-12-03T18:49:17.565356435Z ^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565360643Z File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 914, in send
object-detection-llm-1 | 2025-12-03T18:49:17.565365072Z response = self._send_handling_auth(
object-detection-llm-1 | 2025-12-03T18:49:17.565369269Z ^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565373608Z File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_a
uth
object-detection-llm-1 | 2025-12-03T18:49:17.565379098Z response = self._send_handling_redirects(
object-detection-llm-1 | 2025-12-03T18:49:17.565383226Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565387604Z File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 979, in _send_handling_r
edirects
object-detection-llm-1 | 2025-12-03T18:49:17.565392172Z response = self._send_single_request(request)
object-detection-llm-1 | 2025-12-03T18:49:17.565396110Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565399586Z File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1014, in _send_single_re
quest
object-detection-llm-1 | 2025-12-03T18:49:17.565403524Z response = transport.handle_request(request)
object-detection-llm-1 | 2025-12-03T18:49:17.565407812Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565411889Z File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 250, in hand
le_request
object-detection-llm-1 | 2025-12-03T18:49:17.565416478Z resp = self._pool.handle_request(req)
object-detection-llm-1 | 2025-12-03T18:49:17.565420686Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565435854Z File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 256, in
handle_request
object-detection-llm-1 | 2025-12-03T18:49:17.565441024Z raise exc from None
object-detection-llm-1 | 2025-12-03T18:49:17.565445152Z File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 236, in
handle_request
object-detection-llm-1 | 2025-12-03T18:49:17.565449590Z response = connection.handle_request(
object-detection-llm-1 | 2025-12-03T18:49:17.565453948Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565458146Z File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 103, in hand
le_request
object-detection-llm-1 | 2025-12-03T18:49:17.565462604Z return self._connection.handle_request(request)
object-detection-llm-1 | 2025-12-03T18:49:17.565466993Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565471140Z File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 136, in handle_r
equest
object-detection-llm-1 | 2025-12-03T18:49:17.565475729Z raise exc
object-detection-llm-1 | 2025-12-03T18:49:17.565481881Z File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 106, in handle_r
equest
object-detection-llm-1 | 2025-12-03T18:49:17.565588140Z ) = self._receive_response_headers(**kwargs)
object-detection-llm-1 | 2025-12-03T18:49:17.565604691Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565619208Z File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 177, in _receive
_response_headers
object-detection-llm-1 | 2025-12-03T18:49:17.565625049Z event = self._receive_event(timeout=timeout)
object-detection-llm-1 | 2025-12-03T18:49:17.565635839Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565648914Z File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 217, in _receive
_event
object-detection-llm-1 | 2025-12-03T18:49:17.565655135Z data = self._network_stream.read(
object-detection-llm-1 | 2025-12-03T18:49:17.565660706Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565666286Z File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 128, in read
object-detection-llm-1 | 2025-12-03T18:49:17.565672227Z return self._sock.recv(max_bytes)
object-detection-llm-1 | 2025-12-03T18:49:17.565677748Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565683058Z File "/usr/local/lib/python3.11/ssl.py", line 1295, in recv
object-detection-llm-1 | 2025-12-03T18:49:17.565697034Z return self.read(buflen)
object-detection-llm-1 | 2025-12-03T18:49:17.565702334Z ^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565707754Z File "/usr/local/lib/python3.11/ssl.py", line 1168, in read
object-detection-llm-1 | 2025-12-03T18:49:17.565713364Z return self._sslobj.read(len)
object-detection-llm-1 | 2025-12-03T18:49:17.565718775Z ^^^^^^^^^^^^^^^^^^^^^^
object-detection-llm-1 | 2025-12-03T18:49:17.565724145Z File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/base.py", line 204, in handle_a
bort
object-detection-llm-1 | 2025-12-03T18:49:17.565730076Z sys.exit(1)
object-detection-llm-1 | 2025-12-03T18:49:17.565735265Z SystemExit: 1
object-detection-llm-1 | 2025-12-03T18:49:17.565741888Z [2025-12-03 18:49:17 +0000] [8] [INFO] Worker exiting (pid: 8)
orchestrator-1 | 2025-12-03T18:49:17.566293412Z req=da7dfe32-bfb2-4cab-88a8-f70e7fac9e3f Preprocessor "object-detection-llm" responded with stat
us 500
orchestrator-1 | 2025-12-03T18:49:17.566375395Z req=da7dfe32-bfb2-4cab-88a8-f70e7fac9e3f timestamp=2025-12-03T18:49:17.566Z label=Preprocessor "
object-detection-llm" execution_time_ms=75852.51ms cpu_time_ms=119.01ms normalized_cpu_usage_percent=0.01%
object-depth-calculator-1 | 2025-12-03T18:49:17.629973718Z [2025-12-03 18:49:17 +0000] [6] [DEBUG] POST /preprocessor
object-depth-calculator-1 | 2025-12-03T18:49:17.630228135Z DEBUG:root:Received request
object-grouping-1 | 2025-12-03T18:49:17.630400979Z [2025-12-03 18:49:17 +0000] [6] [DEBUG] POST /preprocessor
ocr-clouds-preprocessor-1 | 2025-12-03T18:49:17.630751115Z [2025-12-03 18:49:17 +0000] [7] [DEBUG] POST /preprocessor
object-grouping-1 | 2025-12-03T18:49:17.630774399Z PII:root:Received request
ocr-clouds-preprocessor-1 | 2025-12-03T18:49:17.631024387Z DEBUG:root:Received request
object-sorting-1 | 2025-12-03T18:49:17.631131578Z [2025-12-03 18:49:17 +0000] [8] [DEBUG] POST /preprocessor
object-sorting-1 | 2025-12-03T18:49:17.631507593Z DEBUG:root:Received request
hello-haptics-handler-1 | 2025-12-03T18:49:17.644507278Z Received request
hello-haptics-handler-1 | 2025-12-03T18:49:17.646676464Z No object detection data: can't render!