fix: Fix benchmark results display for paswordless AWS OpenSearch benchmarks #721
+87
−7
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…chmarks Fixes zilliztech#716 Description: When running a benchmark against AWS OpenSearch, username and password are optional and can we left empty so benchmark can use AWS authentiction tokens in environment variables. This works perfectly, and benchamrk runs and does the expected inserts and queries on the AWS OS cluster. But when trying to look at the benchmark results, http://localhost:8502/results displays this error: ``` pydantic.error_wrappers.ValidationError: 2 validation errors for AWSOpenSearchConfig user Empty string! (type=value_error) password Empty string! (type=value_error) File "/Users/jvegas/.cache/uv/archive-v0/DcT73OzatkvN3UXwJBtH_/lib/python3.12/site-packages/vectordb_bench/frontend/pages/results.py", line 62, in <module> main() File "/Users/jvegas/.cache/uv/archive-v0/DcT73OzatkvN3UXwJBtH_/lib/python3.12/site-packages/vectordb_bench/frontend/pages/results.py", line 29, in main allResults = benchmark_runner.get_results() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jvegas/.cache/uv/archive-v0/DcT73OzatkvN3UXwJBtH_/lib/python3.12/site-packages/vectordb_bench/interface.py", line 106, in get_results return ResultCollector.collect(target_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jvegas/.cache/uv/archive-v0/DcT73OzatkvN3UXwJBtH_/lib/python3.12/site-packages/vectordb_bench/backend/result_collector.py", line 18, in collect file_result = TestResult.read_file(json_file, trans_unit=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jvegas/.cache/uv/archive-v0/DcT73OzatkvN3UXwJBtH_/lib/python3.12/site-packages/vectordb_bench/models.py", line 331, in read_file task_config["db_config"] = db.config_cls(**task_config["db_config"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "pydantic/main.py", line 364, in pydantic.main.BaseModel.__init__ ``` Looks like during results loading the system is applying a more strict validation that does not allow empty username or password, which was allowed at test creation. This PR fixes validation to allow for empty username or password in both cases, resulting in viewables results for passwordless AWS OS benchmarks
Collaborator
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: javiervegas, xiaofan-luan The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #716
Description:
When running a benchmark against AWS OpenSearch, username and password are optional and can we left empty so benchmark can use AWS authentiction tokens in environment variables. This works perfectly, and benchamrk runs and does the expected inserts and queries on the AWS OS cluster. But when trying to look at the benchmark results, http://localhost:8502/results displays this error:
Looks like during results loading the system is applying a more strict validation that does not allow empty username or password, which was allowed at test creation. This PR fixes validation to allow for empty username or password in both cases, resulting in viewables results for passwordless AWS OS benchmarks
Before:
After: