has anybody attempted and succeeded to compile and run ICON with the “new” intel compilers icx/icpx/ifx? At University of Vienna, we have been working with version 2025.3.2 of these and while we could compile after some headaches we then found obscure model crashes.
In any case, I was wondering what the perspective on the intel compiler is. We can compile and run without any problems with the classic intel compilers and release 2025-10, but I am suspecting that these might stop working with future releases at some point.
there has been some work to get the llvm-based Intel compilers working on Levante, see here. The changes within ICON are small, you should be able to rebase them easily.
What kind of model crashes do you encounter? Can you maybe post logs?
Sorry for the delay, I made some changes to the config script based on the levante script provided, and then couldn’t get the model to start at all. With that issue fixed, we still get the same crash as before, salinity going negative in the Kara sea (always the same grid point). Here is what the logs show at the crash:
Time step: 29464, model time: 1990-06-03 11:00:00.000 forecast time 153D 11H00M00S
MAXABS VN, W 0.1700423439E+03 0.2396474970E+00
mo_ocean_atmo_coupling:couple_ocean_toatmo_fluxes: starts…
mo_ocean_atmo_coupling:couple_ocean_toatmo_fluxes: received data umfl : F
mo_ocean_atmo_coupling:couple_ocean_toatmo_fluxes: ends.
mo_hydro_ocean_run:ocean_time_step_zstar: Begin of timestep = 7366 datetime: 1990-06-03T11:00:00.000
ocean_solve(CG): surface height: SUM of ocean_solve iteration = 26, residual = 0.94208195935881194280E-09
Salinity too low: -2.811303976897068E-003
Value -2.811303976897068E-003 found at lon= 76.1220655511285 lat=
74.0008956954837 level= 1
FINISH PE: 465 After advect_diffuse_tracer: tracer below threshold
We have had this same crash across several different configurations using the new intel compilers (mpiifx, mpiicx, mpiicpx) in the hpckit 2025.3.1 container. Those other configurations mostly used different flags and optimization levels, and sometimes resulted in the crash happening at a slightly different time (sometimes roughly a year later, sometime a few days before or after).
Running with the ocean, atmosphere, and coupling timesteps halved (and with the ocean timestep pushed even lower) still produced the same crash at the same time (for a specific configuration).
We tried an intermediate version using intelhpckit’s 2024.0.1 version that still had mpiifort (but used mpiicx and mpiicpx) and still had the same crash.
Interestingly, a native compilation on the same machine produced bit-identical output up to the point of the crash (at least for monthly values).
I can provide the actual config and runscripts if needed.
from the bits in the logs, it looks like the horizontal velocity is very high. as a consequence the salinity gets too low - could be a violation of the CFL.
For debugging I would try to start from a checkpoint shortly before this crash with and intel-classic binary and a oneapi binary. pls have a look at the ocean compile flags when reducing optimization. they are different from the general/atmosphere ones (ICON_OCEAN_FCFLAGS)