Skip to content

Conversation

@jeffbl
Copy link
Member

@jeffbl jeffbl commented Dec 14, 2025

resolves #1184

There were several bugs, such as using "developer" rather than "system" for the role message history, as well as some other tweaks. However, the core issue appears to be that qwen3-vl does not like having a rigid json schema requirement. Main fix appears to be changing to "json_object" and not passing the raw schema. This would likely need to be revisited for other model families.

Tested in override on unicorn using jq script, which returned the "A" responses without fix, and seems to reliably return valid, schema-compliant json with the instructions in the current prompt:

jq '.followup.query = "What are the hairstyles in the photo?"' /home/venicq/follow-up.json | /var/docker/image/bin/sendimagereq /dev/stdin https:/
/unicorn.cim.mcgill.ca/image/render/preprocess/ | jq '.preprocessors."ca.mcgill.a11y.image.preprocessor.text-followup"'

@Samyuktha0104 also tested from monarch using override.

Other tweaks in this PR:

  • re-implemented a prompt override with env variable FOLLOWUP_PROMPT_OVERRIDE, which had been lost in refactoring
  • changed "developer" to "system" for system prompt role
  • fixed some duplicative prompt text
  • tweaks to logging and errors
  • corrected copyright date

Please ensure you've followed the checklist and provide all the required information before requesting a review.
If you do not have everything applicable to your PR, it will not be reviewed!
If you don't know what something is or if it applies to you, ask!

Please note that PRs from external contributors who have not agreed to our Contributor License Agreement will not be considered.
To accept it, include I agree to the [current Contributor License Agreement](/CLA.md) in this pull request.

Don't delete below this line.


Required Information

  • I referenced the issue addressed in this PR.
  • I described the changes made and how these address the issue.
  • I described how I tested these changes.

Coding/Commit Requirements

  • I followed applicable coding standards where appropriate (e.g., PEP8)
  • I have not committed any models or other large files.

New Component Checklist (mandatory for new microservices)

  • I added an entry to docker-compose.yml and build.yml.
  • I created A CI workflow under .github/workflows.
  • I have created a README.md file that describes what the component does and what it depends on (other microservices, ML models, etc.).

OR

  • I have not added a new component in this PR.

@jeffbl jeffbl merged commit be3e3cd into main Dec 14, 2025
2 checks passed
@jeffbl jeffbl deleted the fix-followup-requests branch December 14, 2025 17:31
@jeffbl
Copy link
Member Author

jeffbl commented Dec 15, 2025

@gvzdv ping in case anything I did looks suspicious to you. Note that we're sticking with Qwen/Qwen3-VL-4B-Instruct for now, instead of unsloth, but I don't have firm feelings on this if you think it is a mistake.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

text-followup returns only "A" as responses

2 participants