Fix llm looping on complex graphics #1180
Conversation
jeffbl
left a comment
jeffbl
left a comment
There was a problem hiding this comment.
No red flags. Couple minor nits/questions. Go ahead and merge when ready.
| Args: | ||
| objects (list): List of detected objects with confidence scores | ||
| qwen_output (list): Qwen detection output with bbox_2d and label | ||
| width (int): Image width for normalization |
There was a problem hiding this comment.
In pixels, yes. Thank you!
| json_schema=BBOX_RESPONSE_SCHEMA, | ||
| temperature=0.0, | ||
| parse_json=True | ||
| temperature=0.5, |
There was a problem hiding this comment.
curious why the move back to non-zero temperature?
There was a problem hiding this comment.
It is one of the theories on why this is happening: same problem reported on GitHub.
When the temperatue is default, it happens only sometimes, but when the temperature is set to 0, it happens every time.
I am not 100% sure it's affecting the performance in our specific case (too many variables: model, quantization, engine...), but I'm willing to try since it doesn't seem to affect the accuracy of outputs.
| if object_json is None or len(object_json.get("objects", [])) == 0: | ||
| logging.debug(f"Qwen output received: {qwen_output}") | ||
|
|
||
| if qwen_output is None or len(qwen_output) == 0: |
There was a problem hiding this comment.
No content (204) isn't really an error, if there is legit nothing to extract. Should ideally distinguish between an actual error (something went wrong and nothing to report) vs. everything worked, but there were no objects to extract.
Resolves #1174.
Tried to prevent Qwen from getting stuck while working with complex graphics.
Changes made:
Limited testing on Unicorn. Needs more testing, but the first results are encouraging.
Required Information
Coding/Commit Requirements
New Component Checklist (mandatory for new microservices)
docker-compose.ymlandbuild.yml..github/workflows.README.mdfile that describes what the component does and what it depends on (other microservices, ML models, etc.).OR