Skip to content

Commit 6a24ddd

Browse files
committed
Readme and usage updated
Updated the usage and readme for the project.
1 parent b45e5fb commit 6a24ddd

File tree

3 files changed

+87
-34
lines changed

3 files changed

+87
-34
lines changed

README.md

Lines changed: 79 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,95 @@
11
# TDEI-python-osw-quality-metric
2-
Quality metric calculator for OSW record
2+
Quality metric calculator for OSW record.
3+
4+
# Introduction:
5+
This service calculates the specific quality metric of a given OSW dataset. There are two algorithms that it supports:
6+
- fixed calculation (assumes a random score )
7+
- xn calculation. Calculates the intersection quality metric based on either input sub-polygons or vornoi generated polygons.
8+
9+
## Requirements
10+
python 3.10
11+
12+
## How to run the project with Python3.10
13+
#### Create virtual env
14+
15+
`python3.10 -m venv .venv`
16+
17+
`source .venv/bin/activate`
18+
19+
#### Install requirements
20+
21+
`pip install -r requirements.txt`
22+
23+
#### Set up env file, create a .env file at project root level
24+
25+
```shell
26+
QUALITY_REQ_TOPIC
27+
QUALITY_REQ_SUB
28+
QUALITY_RES_TOPIC
29+
PROVIDER=Azure
30+
QUEUECONNECTION=Endpoint=sb://xxxxxxxxxxxxx
31+
STORAGECONNECTION=DefaultEndpointsProtocol=https;xxxxxxxxxxxxx
32+
MAX_CONCURRENT_MESSAGES=xxx # Optional if not provided defaults to 1
33+
PARTITION_COUNT=xx # Optional number of partitions to use for dask library. defaults to 2
34+
```
35+
Note: Replace the endpoints with the actual endpoints of the environment you want to run the service in
36+
37+
`MAX_CONCURRENT_MESSAGES` is the maximum number of concurrent messages that the service can handle. If not provided, defaults to 1
38+
39+
### Run the Server
40+
41+
`uvicorn src.main:app --reload`
42+
43+
remove `--reload` for non-debug mode
44+
45+
46+
### Run the Server
47+
48+
`uvicorn src.main:app --reload`
49+
50+
### Run Unit tests
51+
52+
#### Run Coverage
53+
`python -m coverage run --source=src -m unittest discover -s tests/unit_tests`
54+
55+
To run a single test use
56+
57+
`python -m unittest tests.unit_tests.service.test_osw_confidence_metric_calculator.TestOSWConfidenceMetric.test_calculate_score`
58+
59+
#### Run Coverage Report
60+
`coverage report`
61+
62+
#### Run Coverage HTML report
63+
`coverage html`
364

465

566
# Incoming message
667

768
```json
869
{
9-
"datasetId":"",
10-
"intersection_file":""
11-
}
70+
"messageType": "mettric-calculation",
71+
"messageId": "message-id-from-msg",
72+
"data": {
73+
"jobId": "0b41ebc5-350c-42d3-90af-3af4ad3628fb",
74+
"data_file": "https://tdeisamplestorage.blob.core.windows.net/osw/test/wenatchee.zip",
75+
"algorithm": "fixed",
76+
"sub_regions_file":""
77+
}
78+
}
1279

1380
```
1481

1582
# Outgoing message
1683
```json
1784
{
18-
"datasetId":"",
19-
"metrics_file":""
85+
"status":"",
86+
"message":"",
87+
"success":true/false,
88+
"dataset_url":"",
89+
"qm_dataset_url":""
2090
}
2191

2292
```
93+
94+
# Run the metrics locally
95+
- Use [test.py](./test.py) to run the metrics on any dataset locally.

src/services/osw_qm_calculator_service.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ def calculate_quality_metric(self, input_file, algorithm_names, output_path, ixn
4545
input_file (str): The path to the input file. (dataset.zip file)
4646
algorithm_names (list): A list of algorithm names to be used for calculating quality metrics.
4747
output_path (str): The path to the output zip file.
48-
48+
ixn_file (str): The path to the sub-regions polygon file.
4949
Returns:
5050
None
5151

test.py

Lines changed: 7 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,13 @@
11
# # Testing document for testing a simple osm input.
22

33
# from src.config import Config
4-
# from src.services.osw_qm_calculator_service import OswQmCalculator
5-
# import os
6-
# # import urllib.parse as urlparse
7-
# from urllib.parse import urlparse
4+
from src.services.osw_qm_calculator_service import OswQmCalculator
85

9-
# config = Config()
10-
11-
# osw_qm_calculator = OswQmCalculator()
12-
13-
# asset_folder = config.get_assets_folder()
14-
# download_folder = config.get_download_folder()
15-
# print(asset_folder)
16-
# print(download_folder)
17-
# test_path = os.path.join(asset_folder, 'osm-input.zip')
18-
# test_op_path = os.path.join(download_folder, 'osm-output.zip')
19-
# # osw_qm_calculator.calculate_quality_metric(test_path, ['fixed'],test_op_path)
20-
21-
# # def get_directory_path(remote_url:str)-> str:
22-
# # # https://tdeisamplestorage.blob.core.windows.net/osw/test_upload/df/fff/500mb_file.zip
23-
# # # should give output test_upload/df/fff/
24-
# # parsed_url = urlparse(remote_url)
25-
# # dirname = os.path.dirname(parsed_url.path)
26-
# # folder_path = dirname.split('/')[2:]
27-
# # folder_path = '/'.join(folder_path)
28-
# # return folder_path
29-
# # print(get_directory_path('https://tdeisamplestorage.blob.core.windows.net/osw/test_upload/df/fff/500mb_file.zip'))
30-
algorithms = "fixed"
6+
osw_qm_calculator = OswQmCalculator(cores_to_use=1) # Configure the number of cores for dask here.
7+
algorithms = "fixed" # Algorithms to use for calculating quality metrics.
318
algorithm_names = algorithms.split(',')
329
for algorithm in algorithm_names:
33-
print(algorithm)
10+
print(algorithm)
11+
12+
result = osw_qm_calculator.calculate_quality_metric('src/assets/osm-input.zip', ['fixed'],'downloads/osm-output.zip') # Calculation code.
13+
print(result)

0 commit comments

Comments
 (0)