ecRad error in nested setup

Open Discussion
1

Hi all,
I am running a coupled ICON-JSBACH simulation but want to start the nesting from a restart file from the coarsest domain. I use the start_time option to initialize the smaller domains one after another.
I am running into a Segmentation fault after the second ecRad-call of DOM02 after the boundary interpolation

mo_nwp_ecrad_interface::nwp_ecrad_radiation: ecrad radiation on full grid
490: [l30036:329344:0:329344] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc0e4f7530).

The latest log-file can be found in the run-directory:
/work/bb1337/icon-mpim_10.2025/run/LOG.exp.atm_xpp_nest_R2B4_R2B7_test.run.25376348.o

I am grateful for each hint… segmentations faults are so difficult to understand..

2

Hi,

as a first step, could you please activate ecrad_check_input=.TRUE. in the Radiation Namelist. This performs several checks, including out of physical bounds for input. Maybe this gives already some hints.

3

Hi,
I did as you recommended: LOG.exp.atm_xpp_nest_R2B4_R2B7_test.run.25421049.o

As far as I get it, the input fields from DOM02 are ok, and it computes LW and SW radiations fluxes. It just crashes afterwards.

4

Thanks for trying this option.

The place where it crashes (srtm_taumol16) suggests that it connected either to gases or somehow to the vertical axis. What are your settings for the different greenhouse gases? Where can I find your namelists?

5

Okay. The namelists are in the experiment folder: /work/bb1337/icon-mpim_10.2025/experiments/atm_xpp_nest_R2B4_R2B7_test

In my runscript: /work/bb1337/icon-mpim_10.2025/run/exp.atm_xpp_nest_R2B4_R2B7_test.run
The following gas namelist parameter are set:

irad_o3 = 5
irad_co2 = 4
irad_ch4 = 4
irad_n2o = 4
irad_cfc11 = 4
irad_cfc12 = 4

Is this what you mean with gases?

I read in historical GHG until 2020 and switch to ssp3.7 for the last two years.
Are those informations helpful?

6

Yes, my suspecion is that one of these options provides erroneous values to ecRad (i.e. that it doesn’t work with nesting). Maybe you could try to set these options one by one to some default we are using for NWP, e.g.:

 irad_o3     = 79
 irad_aero   = 6
 vmr_co2     = 425.e-06   ! values representative for 2023
 vmr_ch4     = 1900.e-09
 vmr_n2o     = 334.0e-09
 vmr_o2      = 0.20946
 vmr_cfc11   = 220.e-12
 vmr_cfc12   = 490.e-12

Make sure to not set irad_co2 etc. explicitely to use the default.

7

Unfortunately this does not change anything. The same error occurs at the same time.

8

Ok, tough one :sweat_smile:

So it looks like it is not connected to gases/aerosols which leaves us with the vertical axis. Could you please try:

lvert_nest           = .True.
num_lev              = 90, 60, 60, 60

and/or:

lvert_nest           = .false.
num_lev              = 90, 90, 90, 90

Have you tried running this configuration with GCC instead of Intel? This often provides more helpful output.

9

I know… :face_with_peeking_eye:
The two changes with the vertical nesting also do not change anything - exactly the same error.
Does maybe the line: srtm_gas_optical_depth.F90:162 say something more specific?

No I have not run the configuration with GCC. I will try it!

10

Not really, that’s a look-up-table overflow which is usually due to unreasonable values in pressure (i.e. vertical coordinate) or in GHG concentrations. My previous suggestions aimed at making sure that these values are reasonable and usually these help :sweat_smile:

11

So I have run it with GCC. Check the new log-File: LOG.exp.atm_xpp_nest_R2B4_R2B7_test.run.25476595.o

The error is much more specific:
mo_nh_nest_utilities:prep_bdy_nudging: 1-way nesting: == Boundary nudging: 2
1: At line 416 of file /work/bb1337/icon-mpim_10.2025/src/atm_phy_schemes/mo_turb_vdiff_diag.f90
1: Fortran runtime error: Index ‘91’ of dimension 2 of array ‘pghf’ above upper bound of 90

12

Looks like a PBL diagnostic doesn’t find an index and returns nlev+1 which then leads to a lookup table overflow. Since intel on levante in the standard configure wrappers is using higher optimization, I would suspect that the original error is the one in vdiff, while the ecrad error is only a symptom of that.

Since vdiff is not a typical nwp configuration, I am not sure if nested setups are supported at all and if so, what needs to be done to set it up properly. Maybe someone else can comment - @rwirth @rschnur can you jump in here?

13

The lookup failure is usually a sign of trouble with vdiff. I wasn’t able to figure out exactly why that happens, though.

I’m sorry but that initialization mode you’re trying to use is completely unsupported. You cannot initialize a nest at runtime because that would involve interpolating a whole bunch of jsbach fields from parent to nest, which are hidden inside jsbach itself. And that is not implemented. I had deprioritized this when I wrote the interface between jsbach/vdiff and the NWP physics. You’re probably the first person to try using this mode, so… congratulations, I guess.

As a workaround I can suggest a few things:

  • Try interpolating the restart file. I’ve never done this but maybe someone from the climate division (tvpham maybe?) can help you with that. Their workflow is based on restarts, so something like that may have come up.

  • Use the NWP initialization machinery (init_mode=7) to start from an output file and interpolate that for the nest domain. You can run the restart for the parent domain for a single time step and output the groups group:mode_dwd_fg_in, group:tracer_fg_in, group:jsb_init_vars, -vn, u, v to get a suitable output file. I’m excluding vn in favor of u, v to make the interpolation easier. This won’t give you bit-identical outputs to the restart, but the number of variables to interpolate is much lower.

  • This is Frankensteinesc, but jsbach initializes itself as usual even when the nest is started later. So you could pass a suitable file to jsb_model_nml:ifs_filename and set init_from_ifs=.TRUE. and start the nest as you do now. jsbach uses the 4 soil layer temperatures and soil-moisture index from an IFS file, which you can build yourself from an ICON output:

        cdo -s -f nc4 merge \
              -intlevel,0.07,0.28,1.00,2.83 -selvar,SMI ${src} \
              -intlevel,0.07,0.28,1.00,2.83 -selvar,T_SO ${src} \
              -setvar,SKT -topvalue -selvar,T_SO ${src} \
              -selvar,W_SNOW ${src} \
              ${srctmp} && \
    cdo -s -f nc4 merge \
              -selvar,SKT,W_SNOW ${srctmp} \
              -setvar,SMIL1 -sellevidx,1 -selvar,SMI ${srctmp} \
              -setvar,SMIL2 -sellevidx,2 -selvar,SMI ${srctmp} \
              -setvar,SMIL3 -sellevidx,3 -selvar,SMI ${srctmp} \
              -setvar,SMIL4 -sellevidx,4 -selvar,SMI ${srctmp} \
              -setvar,STL1 -sellevidx,1 -selvar,T_SO ${srctmp} \
              -setvar,STL2 -sellevidx,2 -selvar,T_SO ${srctmp} \
              -setvar,STL3 -sellevidx,3 -selvar,T_SO ${srctmp} \
              -setvar,STL4 -sellevidx,4 -selvar,T_SO ${srctmp} \
              ${tgt}

    The names used here are for a grib file, so they might need tweaking (lowercase). If your source file is from a run with terra, you can get the SMI directly; just output smi. For a jsbach run, you need hydro_wtr_soil{,_fc,_pwp}_sl_boxand compute smi = (water - pwp) / (fc - pwp). Maybe this creates enough consistency between parent and nest to get you over the initialization shock. If you don’t say anything, the nest jsbach will start from a climatology. That is probably the reason for the crash.

1 Like
14

Thanks a lot for the comprehensive answer! Too bad, that I am the first trying this :smiley:

My current R2B4 run from which I want to start the nesting uses the multi-file restart but likely I can restart and output a single restart file.

Option two also sounds relatively easy to do.
From your expertise, which option would you choose?

15

If you have to run the model anyway, I think the simplest approach is the init_mode=7. I’ve done successful starts from ICON-Dream reanalysis data at 6.5km interpolated to 2km (though that was with the default NWP setup, not with jsbach). I don’t think that jsbach will pose additional problems here.

16

Alright, thanks a lot, I’ll give it a try :slight_smile:

17

@rwirth I am able to output group:mode_dwd_fg_in, group:tracer_fg_in, -vn, u, v
but when I try to output group:jsb_init_vars

I get the error: 0: FINISH PE: 0 mo_name_list_output_init::collect_requested_ipz_levels: Input from output_nml: requested pressureinterpolation without specifying levels!

My outputnml looks like this:
&output_nml
output_start = “${start_date}”
output_end = “${end_date}”
output_interval = ‘${atmTimeStep}’
file_interval = ‘P1D’
filetype = 5
dom = -1
mode = 1
output_filename = “${EXPNAME}_jsb_init”
filename_format = “<output_filename>_DOM_”
remap = 0
ml_varlist = ‘group:jsb_init_vars’

/

Did you manage to do it successfully? Maybe you can check my output_nml?

18

This has to be a different issue. The error comes from a namelist check that directly looks at pl_varlist for all domains. If the first entry is not empty and there are no p_levels specified, it triggers that message. Do you have a different namelist with pressure-level vars? Maybe from the pre-restart namelist (those are stored in the restart file and overwrite the defaults when the run is continued)?Try master_nml:read_restart_namelists = .FALSE.to prevent that.

Belay that. The restore happens for all nmls but the output_nmls.

19

I created a new runscript: /work/bb1337/icon-mpim_10.2025/run/exp.atm_xpp_R2B4_90lev_1step_fg.run just to output the respective file. And there are no other output_nmls

20

Sorry, but I’m unable to reproduce that behavior. I’ve generated the atm_nwp_jsbach_test, deleted all output_nmls and added

&output_nml
 output_filename  = '${EXPNAME}_jsb'
 filename_format  = '${EXPNAME}_jsb_ml_<datetime2>'
 filetype         = ${output_format}
 remap            = 0
 output_grid      = .FALSE.
 output_start     = '${first_output_date}'
 output_end       = '${last__output_date}'
 output_interval  = '${atm_output_interval}'
 file_interval    = '${atm_file_interval}'
 include_last     = .FALSE.
 ml_varlist       = 'group:jsb_init_vars'
/

That works as expected. I didn’t try a restart, though.