-
Notifications
You must be signed in to change notification settings - Fork 4
Open
Labels
Description
Description
We've encountered multiple issues with the netcdf-to-ioda and bufr-to-ioda converters producing garbage or corrupted values, most recently within the MetaData group.
These invalid outputs propagate downstream and end up choking the JEDI data assimilation applications.
Rationale
The converters should ensure that only valid, self-consistent data are written to IODA files.
Instead of letting corrupted or uninitialized values pass through (and later break the DA), the converters should detect and safely discard invalid entries during conversion.
Tasks
- Detect uninitialized, NaN, or corrupted values during conversion
- Remove invalid records from all relevant groups (
MetaData,ObsValue,ObsError, etc.) - Add logging and optional warnings summarizing how many records were filtered out (optional)
- Add unit tests to ensure corrupted input data are safely ignored, not propagated (optional)
Expected Outcome
- Converters output only valid, consistent IODA files
- JEDI applications no longer crash due to corrupted or invalid data
- Invalid observations are automatically filtered out at conversion time, without requiring additional QC steps downstream