C96_gcafs_cycled gcdas_aeroanlgenb failure following PR #1991

[RussTreadon-NOAA]

The following GDASApp ctests fail when run from develop at 358a2d5

        306 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201200 (Failed)
        310 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201800 (Failed)
        311 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlinit_202112201800 (Failed)
        312 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlvar_202112201800 (Failed)
        313 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlfinal_202112201800 (Failed)

Examination of C96_gcafs_cycled/logs/2021122012/gcdas_aeroanlgenb.log finds

 0: FATAL from PE     0: NetCDF: Variable not found: get_variable_num_dimension: file:./clm_stddev/stddev.fv_tracer.res.tile1.nc variable: seas5
 0:
 0:
 0: FATAL from PE     0: NetCDF: Variable not found: get_variable_num_dimension: file:./clm_stddev/stddev.fv_tracer.res.tile1.nc variable: seas5
 0:
 0: Image              PC                Routine            Line        Source
 0: libfv3jedilm.so    00001457BB99F307  mpp_mod_mp_mpp_er          80  mpp_util_mpi.inc
 0: libfv3jedilm.so    00001457BB853EA1  fms_io_utils_mod_         190  fms_io_utils.F90
 0: libfv3jedilm.so    00001457BB89C925  netcdf_io_mod_mp_         386  netcdf_io.F90
 0: libfv3jedilm.so    00001457BB8A3365  netcdf_io_mod_mp_         470  netcdf_io.F90
 0: libfv3jedilm.so    00001457BB89CAA4  netcdf_io_mod_mp_        1598  netcdf_io.F90

Variable seas5 is not found in the input file.

GDASApp PR #1991 replaced 'seas1', 'seas2', 'seas3', 'seas4', with 'seas2', 'seas3', 'seas4', 'seas5'. File clm_stddev/stddev.fv_tracer.res.tile1.nc is copied from /scratch3/NCEPDEV/global/role.glopara/fix/gdas/aero/20240806/clim_b

-rw-r--r-- 1 role.glopara global 131170313 Dec 11  2024 rescale.fv_tracer.res.tile1.nc
-rw-r--r-- 1 role.glopara global 131170313 Dec 11  2024 rescale.fv_tracer.res.tile2.nc
-rw-r--r-- 1 role.glopara global 131170313 Dec 11  2024 rescale.fv_tracer.res.tile3.nc
-rw-r--r-- 1 role.glopara global 131170313 Dec 11  2024 rescale.fv_tracer.res.tile4.nc
-rw-r--r-- 1 role.glopara global 131170313 Dec 11  2024 rescale.fv_tracer.res.tile5.nc
-rw-r--r-- 1 role.glopara global 131170313 Dec 11  2024 rescale.fv_tracer.res.tile6.nc
-rw-r--r-- 1 role.glopara global 131170328 Dec 11  2024 stddev.fv_tracer.res.tile1.nc
-rw-r--r-- 1 role.glopara global 131170328 Dec 11  2024 stddev.fv_tracer.res.tile2.nc
-rw-r--r-- 1 role.glopara global 131170328 Dec 11  2024 stddev.fv_tracer.res.tile3.nc
-rw-r--r-- 1 role.glopara global 131170328 Dec 11  2024 stddev.fv_tracer.res.tile4.nc
-rw-r--r-- 1 role.glopara global 131170328 Dec 11  2024 stddev.fv_tracer.res.tile5.nc
-rw-r--r-- 1 role.glopara global 131170328 Dec 11  2024 stddev.fv_tracer.res.tile6.nc

ncdump -hcs stddev.fv_tracer.res.tile1.nc | grep double returns

        double xaxis_1(xaxis_1) ;
        double yaxis_1(yaxis_1) ;
        double zaxis_1(zaxis_1) ;
        double Time(Time) ;
        double so4(Time, zaxis_1, yaxis_1, xaxis_1) ;
        double bc1(Time, zaxis_1, yaxis_1, xaxis_1) ;
        double bc2(Time, zaxis_1, yaxis_1, xaxis_1) ;
        double oc1(Time, zaxis_1, yaxis_1, xaxis_1) ;
        double oc2(Time, zaxis_1, yaxis_1, xaxis_1) ;
        double dust1(Time, zaxis_1, yaxis_1, xaxis_1) ;
        double dust2(Time, zaxis_1, yaxis_1, xaxis_1) ;
        double dust3(Time, zaxis_1, yaxis_1, xaxis_1) ;
        double dust4(Time, zaxis_1, yaxis_1, xaxis_1) ;
        double dust5(Time, zaxis_1, yaxis_1, xaxis_1) ;
        double seas1(Time, zaxis_1, yaxis_1, xaxis_1) ;
        double seas2(Time, zaxis_1, yaxis_1, xaxis_1) ;
        double seas3(Time, zaxis_1, yaxis_1, xaxis_1) ;
        double seas4(Time, zaxis_1, yaxis_1, xaxis_1) ;

The stddev fix files still retains seas1 seas2 seas3 seas4. These variables need to be updated to seas2 seas3 seas4 seas5.

[RussTreadon-NOAA]

@ypwang19 : How did your tests pass for #1991? Do you have an updated set of stddev files for the global clim_b directory?

[RussTreadon-NOAA]

@ypwang19 : Please create new issue in NOAA-EMC/global-workflow: Static Data Update to update the clim_b files.

[ypwang19]

@RussTreadon-NOAA Thank you Russ for catching this. I am working on fixing it.

[RussTreadon-NOAA]

Thank you @ypwang19 for the quick fix.

test_gdasapp_C96_gcafs_cycled was rerun on Hera using the updated aero/clim_b files. All tests Passed. See g-w issue #4295 for details.

The sorc/gdas.cd hash in g-w develop will need to be updated once the revised aero/clim_b files are installed on WCOSS2 and NOAA RDHPCS. Will you open a g-w issue and PR to update the sorc/gdas.cd hash?

[ypwang19]

Thank you @ypwang19 for the quick fix.

test_gdasapp_C96_gcafs_cycled was rerun on Hera using the updated aero/clim_b files. All tests Passed. See g-w issue #4295 for details.

The sorc/gdas.cd hash in g-w develop will need to be updated once the revised aero/clim_b files are installed on WCOSS2 and NOAA RDHPCS. Will you open a g-w issue and PR to update the sorc/gdas.cd hash?

Yes, I will update the hash once the data are installed.