Hi everyone,
I’m trying to run ICON NWP compiled with GCC, but the simulation stops very early with a confusing error. These are the last two lines of log:
mo_initicon:read_dwdfg: read atm_FG fields from igfff00000000_ic.grb2
/opt/icon/model/source/externals/cdi/src/gribapi_utilities.c:114: Error in function `gribGetLong`: `grib_get_long` returned error code -10 for key "localNumberOfExperiment"
What is weird is the fact that I’m using the same input data and configuration as my colleague who does not have this issue and the only (significant) difference between our setups is the compilator and libraries.
I am using a newer version of eccodes than my colleague (2.47.0 vs 2.36.4), but even if downgrade eccodes the issue still persists. The initial conditions which the model fails to read are simply an output from global ICON from DWD processed with iconremap onto a local domain.
One interesting difference I have noticed between the two compilations is the fact that icon –version does not list eccodes amongst other libraries for my colleague but does list it in my case (see below), but for both ldd shows that ICON binary is linked with eccodes. Could that be somehow a culprit? If so, how can I fix it?
What I find strange however is the fact that eccodes grib_ls (regardless of version) cannot find the localNumberOfExperiment key in any grib from any model (not only ICON) I tested, so I’m not even sure why is ICON trying to access that key.
If anyone have encountered this issue or knows why ICON is trying to read that key in the first place, that would be of a great help.
Output of icon –version and ldd for my compilation (not working):
$ icon --version
ICON MPI interface runtime information:
mo_mpi::start_mpi Used MPI version: 3.1
mo_mpi::start_mpi ICON: Globally run on 1 mpi processes.
Single processor run.
executable: /opt/icon/model/build/bin/icon
date: 20260609
time: 090905
user: xxx
host: xxxx (Linux 5.14.0-503.40.1.el9_5.x86_64 x86_64)
version: 2026.04
revision: icon-2026.04-public-0-g8597da45ef4b86323f3fb844caedc4ae5e1ffc01
repository: https://gitlab.dkrz.de/icon/icon-model.git
local branch: release-2026.04-public
application libraries:
ECRAD: ecrad-safeguard-09666303-13-g19ceb48ac61ea1cab148149b65f7a496cb5d6626
infrastructure and support libraries:
ICONMATH: 1.3.0-0-g622c01ba78e4fbb19fdf5f3c9f18a90b1e551389
FORTRAN-SUPPORT: 2.2.1-0-g34246610b17db29f214fb2f95ca1c9087f09b89c
MTIME: 1.3.0-1-g04f02ccbd765104a1570355c5d3fdfcfdad11c4c
CDI:
version: 2.6.0
revision: cdi-2.6.0-1-g60e0a3b9899c8021437686d720702a978a5bb63d
other libraries:
ECCODES: 2.47.0
NetCDF-C: 4.10.0
MPI: Open MPI v5.0.10, package: Open MPI root@n219 Distribution, ident: 5.0.10, repo rev: v5.0.10, Feb 23, 2026
compilers:
Fortran: GNU 14.3.1 (GCC version 14.3.1 20251022 (Red Hat 14.3.1-4))
C: GNU 14.3.1
$ ldd icon
linux-vdso.so.1 (0x00007ffd5f111000)
libxml2.so.2 => /lib64/libxml2.so.2 (0x000014ad14b52000)
libfyaml.so.0 => /opt/icon/libs/install/libfyaml-0.9.6/lib64/libfyaml.so.0 (0x000014ad14a7a000)
libeccodes_f90.so => /opt/icon/libs/install/eccodes-2.47.0/lib64/libeccodes_f90.so (0x000014ad14a47000)
libeccodes.so => /opt/icon/libs/install/eccodes-2.47.0/lib64/libeccodes.so (0x000014ad145e4000)
libnetcdff.so.7 => /opt/icon/libs/install/netcdf-fortran-4.6.3/lib64/libnetcdff.so.7 (0x000014ad144fd000)
libnetcdf.so.22 => /opt/icon/libs/install/netcdf-c-4.10.0/lib64/libnetcdf.so.22 (0x000014ad14382000)
libhdf5_hl.so.320 => /opt/icon/libs/install/hdf5-2.1.1/lib/libhdf5_hl.so.320 (0x000014ad14359000)
libhdf5.so.320 => /opt/icon/libs/install/hdf5-2.1.1/lib/libhdf5.so.320 (0x000014ad13e09000)
libopenblas.so.0 => /opt/icon/libs/install/openblas-0.3.33/lib/libopenblas.so.0 (0x000014ad12e75000)
libjasper.so.7 => /opt/icon/libs/install/jasper-4.2.9/lib64/libjasper.so.7 (0x000014ad12e09000)
libpng16.so.16 => /lib64/libpng16.so.16 (0x000014ad12dd0000)
libjpeg.so.62 => /lib64/libjpeg.so.62 (0x000014ad12d21000)
libz.so.1 => /opt/icon/libs/install/zlib-1.3.2/lib64/libz.so.1 (0x000014ad12d00000)
libsz.so.2 => /opt/icon/libs/install/libaec-1.1.7/lib64/libsz.so.2 (0x000014ad12cf2000)
libaec.so.0 => /opt/icon/libs/install/libaec-1.1.7/lib64/libaec.so.0 (0x000014ad12ce4000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x000014ad12a6e000)
libgfortran.so.5 => /lib64/libgfortran.so.5 (0x000014ad12744000)
libm.so.6 => /lib64/libm.so.6 (0x000014ad12696000)
libmvec.so.1 => /lib64/libmvec.so.1 (0x000014ad1259a000)
libmpi_usempif08.so.40 => /opt/icon/libs/install/openmpi-5.0.10/lib/libmpi_usempif08.so.40 (0x000014ad12558000)
libmpi_usempi_ignore_tkr.so.40 => /opt/icon/libs/install/openmpi-5.0.10/lib/libmpi_usempi_ignore_tkr.so.40 (0x000014ad12546000)
libmpi_mpifh.so.40 => /opt/icon/libs/install/openmpi-5.0.10/lib/libmpi_mpifh.so.40 (0x000014ad124d9000)
libmpi.so.40 => /opt/icon/libs/install/openmpi-5.0.10/lib/libmpi.so.40 (0x000014ad1210c000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000014ad120dd000)
libc.so.6 => /lib64/libc.so.6 (0x000014ad11f04000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x000014ad11ed0000)
libquadmath.so.0 => /lib64/libquadmath.so.0 (0x000014ad11e86000)
/lib64/ld-linux-x86-64.so.2 (0x000014ad14cb9000)
libopen-pal.so.80 => /opt/icon/libs/install/openmpi-5.0.10/lib/libopen-pal.so.80 (0x000014ad11d09000)
libfabric.so.1 => /lib64/libfabric.so.1 (0x000014ad11b79000)
libucp.so.0 => /lib64/libucp.so.0 (0x000014ad11a9f000)
libucs.so.0 => /lib64/libucs.so.0 (0x000014ad11a38000)
libucm.so.0 => /lib64/libucm.so.0 (0x000014ad11a1d000)
libuct.so.0 => /lib64/libuct.so.0 (0x000014ad119e2000)
libpmix.so.2 => /opt/icon/libs/install/openmpi-5.0.10/lib/libpmix.so.2 (0x000014ad11791000)
libevent_core-2.1.so.7 => /lib64/libevent_core-2.1.so.7 (0x000014ad11759000)
libevent_pthreads-2.1.so.7 => /lib64/libevent_pthreads-2.1.so.7 (0x000014ad11754000)
libhwloc.so.15 => /lib64/libhwloc.so.15 (0x000014ad116ed000)
librdmacm.so.1 => /lib64/librdmacm.so.1 (0x000014ad116d0000)
libefa.so.1 => /lib64/libefa.so.1 (0x000014ad116c0000)
libibverbs.so.1 => /lib64/libibverbs.so.1 (0x000014ad1169e000)
libnl-3.so.200 => /lib64/libnl-3.so.200 (0x000014ad1167a000)
libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x000014ad115e5000)
libudev.so.1 => /lib64/libudev.so.1 (0x000014ad1159c000)
libpciaccess.so.0 => /lib64/libpciaccess.so.0 (0x000014ad1158f000)
libOpenCL.so.1 => /lib64/libOpenCL.so.1 (0x000014ad11562000)
libcap.so.2 => /lib64/libcap.so.2 (0x000014ad11555000)
Output of icon –version and ldd from my colleague (working):
$ icon --version
ICON MPI interface runtime information:
mo_mpi::start_mpi Used MPI version: 3.1
mo_mpi::start_mpi ICON: Globally run on 1 mpi processes.
Single processor run.
executable: ./icon
date: 20260609
time: 090746
user: xxx
host: xxxx (Linux 5.14.0-503.40.1.el9_5.x86_64 x86_64)
version: 2025.10
revision: icon-2026.04-21-gf2a870082ad2d988e8f20194d16ffe926a90890d-dirty
repository: git@xxxx/icon-precip/icon.git
local branch: main
model components:
ICON-Land:
version: 2026.04
revision: icon-land-2026.04-3-gbaa26809aa359d74403d6d0f4d4cb01ba48ef4da
application libraries:
ECRAD: ecrad-safeguard-09666303-13-g19ceb48ac61ea1cab148149b65f7a496cb5d6626
infrastructure and support libraries:
MEMMAN: 1.0.0-0-g90bd21b062b0a7a1c6f3b8675cce2098a1ca0045
ICONMATH: 1.3.0-0-g622c01ba78e4fbb19fdf5f3c9f18a90b1e551389
FORTRAN-SUPPORT: 2.2.1-0-g34246610b17db29f214fb2f95ca1c9087f09b89c
YAC:
version: 3.14.0
revision: v3.14.0_p1-0-ge796555f4319712c5f49713409d4e2a56769be22
MTIME: 1.3.0-1-g04f02ccbd765104a1570355c5d3fdfcfdad11c4c
CDI:
version: 2.6.0
revision: cdi-2.6.0-1-g60e0a3b9899c8021437686d720702a978a5bb63d
other libraries:
NetCDF-C: 4.9.2
MPI: Intel(R) MPI Library 2021.14 for Linux* OS...
compilers:
Fortran: Intel 2025.0.4 (Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2025.0.4 Build 20241205)
C: Intel 2025.0.4 (GNU 4.2.1)
$ ldd icon
linux-vdso.so.1 (0x00007ffdc9fb3000)
libz.so.1 => /opt/local/zlib-1.3.1-icx-2025.0.4/lib/libz.so.1 (0x000014981f0ce000)
libsz.so.2 => /opt/local/libaec-1.1.3-icx-2025.0.4/lib64/libsz.so.2 (0x000014981f0c1000)
libnetcdff.so.7 => /opt/local/netcdf_fortran-4.6.1-icx-2025.0.4/lib/libnetcdff.so.7 (0x000014981ed54000)
libnetcdf.so.19 => /opt/local/netcdf_c-4.9.2-icx-2025.0.4/lib/libnetcdf.so.19 (0x000014981eb03000)
libeccodes.so => /opt/local/eccodes-2.36.4-icx-2025.0.4/lib64/libeccodes.so (0x000014981e74a000)
libeccodes_f90.so => /opt/local/eccodes-2.36.4-icx-2025.0.4/lib64/libeccodes_f90.so (0x000014981e720000)
libfyaml.so.0 => /opt/local/libfyaml-0.9-icx-2025.0.4/lib/libfyaml.so.0 (0x000014981e67d000)
libmkl_intel_lp64.so.2 => /opt/intel/oneapi/mkl/2025.0/lib/libmkl_intel_lp64.so.2 (0x000014981d77d000)
libmkl_intel_thread.so.2 => /opt/intel/oneapi/mkl/2025.0/lib/libmkl_intel_thread.so.2 (0x000014981b467000)
libmkl_core.so.2 => /opt/intel/oneapi/mkl/2025.0/lib/libmkl_core.so.2 (0x00001498174b6000)
libiomp5.so => /opt/intel/oneapi/compiler/2025.0/lib/libiomp5.so (0x00001498170be000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000149816e83000)
libhdf5hl_fortran.so.310 => /opt/local/hdf5-1.14.5-icx-2025.0.4/lib/libhdf5hl_fortran.so.310 (0x0000149816e60000)
libhdf5_hl.so.310 => /opt/local/hdf5-1.14.5-icx-2025.0.4/lib/libhdf5_hl.so.310 (0x0000149816e3b000)
libhdf5_fortran.so.310 => /opt/local/hdf5-1.14.5-icx-2025.0.4/lib/libhdf5_fortran.so.310 (0x0000149816dd3000)
libhdf5.so.310 => /opt/local/hdf5-1.14.5-icx-2025.0.4/lib/libhdf5.so.310 (0x00001498169a4000)
libmpifort.so.12 => /opt/intel/oneapi/mpi/2021.14/lib/libmpifort.so.12 (0x00001498165af000)
libmpi.so.12 => /opt/intel/oneapi/mpi/2021.14/lib/libmpi.so.12 (0x000014980c5e9000)
libimf.so => /opt/intel/oneapi/compiler/2025.0/lib/libimf.so (0x000014980c1dd000)
libm.so.6 => /lib64/libm.so.6 (0x000014980c102000)
libc.so.6 => /lib64/libc.so.6 (0x000014980bef9000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000014980bedf000)
libippdc.so.11 => /opt/intel/oneapi/ipp/2022.0/lib/libippdc.so.11 (0x000014980bcd7000)
libipps.so.11 => /opt/intel/oneapi/ipp/2022.0/lib/libipps.so.11 (0x000014980ba98000)
libippcore.so.11 => /opt/intel/oneapi/ipp/2022.0/lib/libippcore.so.11 (0x000014980b890000)
libsvml.so => /opt/intel/oneapi/compiler/2025.0/lib/libsvml.so (0x000014980a24f000)
libirng.so => /opt/intel/oneapi/compiler/2025.0/lib/libirng.so (0x000014980a154000)
libintlc.so.5 => /opt/intel/oneapi/compiler/2025.0/lib/libintlc.so.5 (0x000014980a0f2000)
libzstd.so.1 => /lib64/libzstd.so.1 (0x000014980a01b000)
libxml2.so.2 => /lib64/libxml2.so.2 (0x0000149809e92000)
libcurl.so.4 => /lib64/libcurl.so.4 (0x0000149809df0000)
libifport.so.5 => /opt/intel/oneapi/compiler/2025.0/lib/libifport.so.5 (0x0000149809dc4000)
libifcoremt.so.5 => /opt/intel/oneapi/compiler/2025.0/lib/libifcoremt.so.5 (0x0000149809c53000)
libbz2.so.1 => /lib64/libbz2.so.1 (0x0000149809c40000)
libjasper.so.4 => /lib64/libjasper.so.4 (0x0000149809bee000)
libjpeg.so.62 => /lib64/libjpeg.so.62 (0x0000149809b6d000)
libopenjp2.so.7 => /lib64/libopenjp2.so.7 (0x0000149809b13000)
libaec.so.0 => /opt/local/libaec-1.1.3-icx-2025.0.4/lib64/libaec.so.0 (0x0000149809b05000)
libpng16.so.16 => /lib64/libpng16.so.16 (0x0000149809ace000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000149809ac9000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000149809ac4000)
librt.so.1 => /lib64/librt.so.1 (0x0000149809abf000)
/lib64/ld-linux-x86-64.so.2 (0x000014981f0e9000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x0000149809a91000)
libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x0000149809a67000)
libidn2.so.0 => /lib64/libidn2.so.0 (0x0000149809a46000)
libssh.so.4 => /lib64/libssh.so.4 (0x00001498099d1000)
libpsl.so.5 => /lib64/libpsl.so.5 (0x00001498099bd000)
libssl.so.3 => /lib64/libssl.so.3 (0x00001498098d5000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00001498093bc000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x0000149809366000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x000014980928c000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x0000149809273000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x000014980926c000)
libldap.so.2 => /lib64/libldap.so.2 (0x0000149809203000)
liblber.so.2 => /lib64/liblber.so.2 (0x00001498091f1000)
libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00001498091e3000)
libunistring.so.2 => /lib64/libunistring.so.2 (0x000014980905e000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x000014980904d000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x0000149809044000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x0000149809030000)
libevent-2.1.so.7 => /lib64/libevent-2.1.so.7 (0x0000149808fd7000)
libsasl2.so.3 => /lib64/libsasl2.so.3 (0x0000149808fb7000)
libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x0000149808f94000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x0000149808f65000)
libcrypt.so.2 => /lib64/libcrypt.so.2 (0x0000149808f2b000)
libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x0000149808e8f000)
You can notice that my colleague is using a custom icon source in this case, but they also have no issue when running the ICON built from the official release.
Please let me know if any more information is needed and thank you for your help