tasmax/tasmin ERA5 CMORization fails

[malininae]

I was running ERA5 tasmax and tasmin CMORization using the daily cmorization recipe with the recently downloaded grib data and it's been failing for both variables. I tried it with both 2.12 and 2.13 core and it fails at the same stage, yet I successfully CMOrized sfcWind with 2.12 core the other day, so it seems to be variable dependent.

2025-11-18 22:35:57,634 UTC [276027] ERROR   Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/_main.py", line 635, in run
    fire.Fire(ESMValTool())
    ~~~~~~~~~^^^^^^^^^^^^^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ~~~~~~~~~~~~~~~~~~~^
        component,
        ^^^^^^^^^^
    ...<2 lines>...
        treatment='class' if is_class else 'routine',
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        target=component.__name__)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/_main.py", line 463, in run
    self._run(recipe, session, cli_config_dir)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/_main.py", line 519, in _run
    process_recipe(recipe_file=recipe, session=session)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/_main.py", line 145, in process_recipe
    recipe.run()
    ~~~~~~~~~~^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/_recipe/recipe.py", line 1348, in run
    self.tasks.run(max_parallel_tasks=self.session["max_parallel_tasks"])
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/_task.py", line 849, in run
    self._run_sequential()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/_task.py", line 860, in _run_sequential
    task.run()
    ~~~~~~~~^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/_task.py", line 292, in run
    input_files.extend(task.run())
                       ~~~~~~~~^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/_task.py", line 299, in run
    self.output_files = self._run(input_files)
                        ~~~~~~~~~^^^^^^^^^^^^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 783, in _run
    product.apply(step, self.debug)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 569, in apply
    self.cubes,
    ^^^^^^^^^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 581, in cubes
    self._cubes = [ds.load() for ds in self.datasets]  # type: ignore
                   ~~~~~~~^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/dataset.py", line 822, in load
    cube = self._load()
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/dataset.py", line 907, in _load
    result = preprocess(
        result,
    ...<4 lines>...
        **kwargs,
    )
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 457, in preprocess
    _run_preproc_function(
    ~~~~~~~~~~~~~~~~~~~~~^
        function,
        ^^^^^^^^^
    ...<2 lines>...
        input_files=input_files,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    ),
    ^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 397, in _run_preproc_function
    return function(items, **kwargs)
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/cmor/fix.py", line 187, in fix_metadata
    cube_list = fix.fix_metadata(cube_list)
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/cmor/_fixes/native6/era5.py", line 612, in fix_metadata
    cube = safe_convert_units(cube, self.vardef.units)
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/esmvalcore/iris_helpers.py", line 514, in safe_convert_units
    cube.convert_units(units)
    ~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/fs/site5/eccc/crd/ccrn/users/emr001/miniconda3/envs/etool_213_not_source/lib/python3.13/site-packages/iris/cube.py", line 1479, in convert_units
    raise iris.exceptions.UnitConversionError(
    ...<2 lines>...
    )
iris.exceptions.UnitConversionError: Cannot convert from unknown units. The "cube.units" attribute may be set directly.

Here's the recipe I used:

# ESMValTool
# recipe_daily_era5.yml
---
documentation:
  description: Create daily ERA5 data

  title: Daily ERA5 data CMORization and creation

  authors:
    - andela_bouwe
    - kalverla_peter
    - alidoost_sarah

  maintainer:
    - unmaintained

  references:
    - acknow_project
    - hersbach20rmets

  projects:
    - ewatercycle

datasets:
  # For the daily diagnostic, always add the next year, otherwise the last day is not cmor compatible
  - {dataset: ERA5, project: native6, type: reanaly, version: v1, tier: 3, start_year: 2023, end_year: 2024}

preprocessors:
  add_one_day: &add_one_day
    extract_time:
      start_year: 2023
      start_month: 1
      start_day: 1
      end_year: 2024
      end_month: 1
      end_day: 1

  daily_max:
    <<: *add_one_day
    daily_statistics:
      operator: max

diagnostics:
  daily:
    description: Create daily ERA5 data
    scripts:
      rename:
        script: cmorizers/era5.py
    variables:
      tasmax:
        mip: E1hr
        era5_name: maximum_2m_temperature_since_previous_post_processing
        era5_freq: hourly
        preprocessor: daily_max

main_log_debug_era_tasmax_cmor.txt

I'll try to investigate it soon, but if someone has a chance to look at it in the next days, I'll really appreciate it!

[bouweandela]

It looks like translation from grib to cf conventions doesn't happen as expected by our code. I downloaded some data and opened it:

In [1]: import iris

In [2]: cubes = iris.load("maximum_2m_temperature_since_previous_post_processing_2000-2001.grib")

In [3]: cubes
Out[3]: [<iris 'Cube' of UNKNOWN LOCAL PARAM 201.128 / (unknown) (time: 17544; latitude: 721; longitude: 1440)>]

In [4]: cubes[0].units
Out[4]: Unit('unknown')

I haven't worked with GRIB data before, so I can't say whether this has changed or has never worked. Maybe @schlunma can comment when is back from leave?

[malininae]

It looks like translation from grib to cf conventions doesn't happen as expected by our code. I downloaded some data and opened it:

In [1]: import iris

In [2]: cubes = iris.load("maximum_2m_temperature_since_previous_post_processing_2000-2001.grib")

In [3]: cubes
Out[3]: [<iris 'Cube' of UNKNOWN LOCAL PARAM 201.128 / (unknown) (time: 17544; latitude: 721; longitude: 1440)>]

In [4]: cubes[0].units
Out[4]: Unit('unknown')
I haven't worked with GRIB data before, so I can't say whether this has changed or has never worked. Maybe @schlunma can comment when is back from leave?

Thanks so much @bouweandela, it was my assumption too that the units aren't properly caught. I'm just not sure whether it's a min/max temperature issue or more widespread. I'll try testing it before New Year's but not guaranteed. I almost want to say that it is new, because I have a vague recollection of successfully CMORizing some variable from grib (most likely tasmax) at some point. But yes, @schlunma would know better. I think @axel-lauer mentioned that the format changed.

[axel-lauer]

The change in format I noticed was for netCDF files downloaded via era5cli from the Copernicus Climate Data Store. I am not sure about the grib files (haven't tried this in a while).

[malininae]

Hi @bouweandela, @schlunma and @axel-lauer , since I needed this behavior fixed, I went down a rabbit hole.

What I figured out, is, indeed, the units "capturing" is indeed variable specific. From what I checked, the variables that work are 2m_temperature, 10m_u_component_of_wind, 10m_v_component_of_wind, 2m_dewpoint_temperature, so essentially the standard simple variables. The ones that don't work based on my check are maximum_2m_temperature_since_previous_post_processing, minimum_2m_temperature_since_previous_post_processing, surface_pressure, total_precipitation, while not 100% standard, they are important. As an example, if I do iris.load_cube() on the 2m_temperature I get:

<iris 'Cube' of air_temperature / (K) (time: 8760; latitude: 721; longitude: 1440)>

while for the maximum_2m_temperature_since_previous_post_processing I get:

<iris 'Cube' of UNKNOWN LOCAL PARAM 201.128 / (unknown) (time: 8760; latitude: 721; longitude: 1440)>

According to my good friend ChatGPT:

In GRIB, units are not always encoded explicitly in a way Iris can reliably map to CF units. Iris relies on:

* the GRIB parameter ID (table + discipline + number)

* and its internal GRIB → CF mapping

In your case:

cb.attributes['GRIB_PARAM']
# <GRIBCode1 {'table_version': 128, 'centre_number': 98, 'number': 201}>


This tells us:

* GRIB edition: 1

* Local table: 128

* Centre: 98 (ECMWF)

* Parameter number: 201

This is a local ECMWF parameter, not a WMO-standard one. For local GRIB parameters, Iris cannot know the units unless you tell it.
That’s why:

cb.units == Unit('unknown')

Now, the fix for this seems to be not 100% straightforward. I even don't mind implementing it quickly, I just need guidance on what way to take. As far as I understand, there are two ways:

1. On load of grib (???), we add a check that in case the units are unknown, we load the file with eccodes which can actually help us fish out the short_name, long_name and units. As far as I understand we would need to do it here.
2. We add either a dictionary somewhere here with the dictionary of variables or we add a keyword in the recipe along era_name.

Since at CCCma we are interested in fixing it quickly, I'm tagging @k-a-webb, @Row-Bean and @ECCC-dev-Noah.