Skip to content

Harden the ioda converters (bufr and netcdf) #149

@guillaumevernieres

Description

@guillaumevernieres

Description

We've encountered multiple issues with the netcdf-to-ioda and bufr-to-ioda converters producing garbage or corrupted values, most recently within the MetaData group.
These invalid outputs propagate downstream and end up choking the JEDI data assimilation applications.

Rationale

The converters should ensure that only valid, self-consistent data are written to IODA files.
Instead of letting corrupted or uninitialized values pass through (and later break the DA), the converters should detect and safely discard invalid entries during conversion.

Tasks

  • Detect uninitialized, NaN, or corrupted values during conversion
  • Remove invalid records from all relevant groups (MetaData, ObsValue, ObsError, etc.)
  • Add logging and optional warnings summarizing how many records were filtered out (optional)
  • Add unit tests to ensure corrupted input data are safely ignored, not propagated (optional)

Expected Outcome

  • Converters output only valid, consistent IODA files
  • JEDI applications no longer crash due to corrupted or invalid data
  • Invalid observations are automatically filtered out at conversion time, without requiring additional QC steps downstream

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinggfsv17

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions