Commit 291234c
committed
process updates in batches of 500
Batch processing:
* updates just the declared license in the DB documents using `collection.bulk_write()`
* updates denitions using service API `POST /definitions?force=true`
_NOTE: Updating the DB makes the fix of the declared license immediately available. When the `POST /definitions` request completes, the full DB document will be updated to be in sync with the blob definition._
Additional changes:
* moves global variable definitions based on .env to the initialize() function
* adds DRYRUN flag to check what would run and how many records would be evaluated
* add estimated time to complete
* adds script and function level documentation
* includes timestamps to make it easier to estimate how long it will take to complete a run
* generate filename based on date range and offset to avoid overwriting output files
_NOTE: Azure only supports fetching one blob at a time. Not able to optimize that part of the code. _
_NOTE: Batch size of 500 was selected because that is the max number of coordinates supported in calls to service API `POST /definitions`._1 parent 615c940 commit 291234c
File tree
2 files changed
+169
-84
lines changed- tools/analyze_data_synchronization
2 files changed
+169
-84
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| 13 | + | |
0 commit comments