coupled models crash when MOM6 is compiled with -openmp #28

nikizadehgfdl · 2014-06-24T22:08:07Z

In branch dev/master/2014.06.23 (testing tag) all coupled test cases crash when I compile with -openmp
Here's the traceback:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
fms_MOM6_SIS_LAD_  000000000185D246  mom_tracer_hor_di         303  MOM_tracer_hor_diff.F90
fms_MOM6_SIS_LAD_  0000000001717C50  mom_mp_step_mom_         1062  MOM.F90
fms_MOM6_SIS_LAD_  00000000016C2F27  ocean_model_mod_m         368  ocean_model_MOM.F90
fms_MOM6_SIS_LAD_  00000000004013C7  MAIN__                    687  coupler_main.F90

-t repro does not have this problem.

To reproduce the issue:

fremake --execute --nolink -x /ncrc/home2/Niki.Zadeh/frerts/testing_20140623/MOM6_CM.xml.20140624170350 -p ncrc2.intel -t repro-openmp MOM6_CM_libs_compile

fremake --execute -x /ncrc/home2/Niki.Zadeh/frerts/testing_20140623/MOM6_CM.xml.20140624170350 -p ncrc2.intel -t repro-openmp MOM6_SIS_LAD_FV_compile

frerun -s -r basic -u --no-transfer -x /ncrc/home2/Niki.Zadeh/frerts/testing_20140623/MOM6_CM.xml.20140624170350 -p ncrc2.intel -t repro-openmp MOM6_AM2_MOM6i_1deg

The text was updated successfully, but these errors were encountered:

Zhi-Liang · 2014-06-24T23:05:53Z

Hi Niki,

I will take a look.

Zhi

On Tue, Jun 24, 2014 at 6:08 PM, Niki Zadeh [email protected]
wrote:

In branch dev/master/2014.06.23 (testing tag) all coupled test cases crash
when I compile with -openmp

Here's the traceback:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source

fms_MOM6_SIS_LAD_ 000000000185D246 mom_tracer_hor_di 303
MOM_tracer_hor_diff.F90
fms_MOM6_SIS_LAD_ 0000000001717C50 mom_mp_step_mom_ 1062 MOM.F90
fms_MOM6_SIS_LAD_ 00000000016C2F27 ocean_model_mod_m 368
ocean_model_MOM.F90
fms_MOM6_SIS_LAD_ 00000000004013C7 MAIN__ 687 coupler_main.F90

-t repro does not have this problem.

To reproduce the issue:

fremake --execute --nolink -x
/ncrc/home2/Niki.Zadeh/frerts/testing_20140623/MOM6_CM.xml.20140624170350
-p ncrc2.intel -t repro-openmp MOM6_CM_libs_compile

fremake --execute -x
/ncrc/home2/Niki.Zadeh/frerts/testing_20140623/MOM6_CM.xml.20140624170350
-p ncrc2.intel -t repro-openmp MOM6_SIS_LAD_FV_compile

frerun -s -r basic -u --no-transfer -x
/ncrc/home2/Niki.Zadeh/frerts/testing_20140623/MOM6_CM.xml.20140624170350
-p ncrc2.intel -t repro-openmp MOM6_AM2_MOM6i_1deg

—
Reply to this email directly or view it on GitHub
#28.

nikizadehgfdl · 2014-06-25T19:49:18Z

These crashes always precede by a warning:

WARNING from PE    30: tracer_hordiff: Tracer T has associated 3-d diffusive flux diagnostics.  These are not valid when DIFFUSE_ML_TO_INTERIOR is defined. Use 2-d tracer diffusion diagnostics instead to get accurate total fluxes.

What does this mean?

Zhi-Liang · 2014-06-25T19:54:11Z

Hi Niki,

The non-openmp run also print out same warning message.

Zhi

On Wed, Jun 25, 2014 at 3:49 PM, Niki Zadeh [email protected]
wrote:

These crashes always precede by a warning:

WARNING from PE 30: tracer_hordiff: Tracer T has associated 3-d diffusive flux diagnostics. These are not valid when DIFFUSE_ML_TO_INTERIOR is defined. Use 2-d tracer diffusion diagnostics instead to get accurate total fluxes.

What does this mean?

—

Reply to this email directly or view it on GitHub
#28 (comment)
.

Hallberg-NOAA · 2014-06-25T19:59:53Z

Hi,
    This warning means that there are some diagnostic fields that
  are requested from the diag_table that simply can not be
  calculated with this particular model configuration. It is a
  warning, and it should not presage a model failure.
    - Bob Hallberg Dr. Robert Hallberg

Oceanographer and Ocean Group Head
NOAA Geophysical Fluid Dynamics Laboratory

NOAA GFDL || Phone: (609) 452-6508
Princeton University Forrestal Campus || Home: (732) 463-1274
201 Forrestal Road || Fax: (609) 987-5063
Princeton, New Jersey 08540-6649 || Email: [email protected]

  On 06/25/2014 03:46 PM, Niki Zadeh wrote:

  These crashes always precede by a warning:
    '''
    WARNING from PE 30: tracer_hordiff: Tracer T has associated 3-d
    diffusive flux diagnostics. These are not valid when
    DIFFUSE_ML_TO_INTERIOR is defined. Use 2-d tracer diffusion
    diagnostics instead to get accurate total fluxes.
    '''
    What does this mean?
  —
    Reply to this email directly or view
      it on GitHub.

StephenGriffies · 2014-06-25T20:47:07Z

"Presage" is a great word. We should all make a point of using it more
often.

On Wed, Jun 25, 2014 at 3:59 PM, Robert Hallberg [email protected]
wrote:

Hi,
This warning means that there are some diagnostic fields that
are requested from the diag_table that simply can not be
calculated with this particular model configuration. It is a
warning, and it should not presage a model failure.

Bob Hallberg Dr. Robert Hallberg
Oceanographer and Ocean Group Head
NOAA Geophysical Fluid Dynamics Laboratory

NOAA GFDL || Phone: (609) 452-6508
Princeton University Forrestal Campus || Home: (732) 463-1274
201 Forrestal Road || Fax: (609) 987-5063
Princeton, New Jersey 08540-6649 || Email: [email protected]

On 06/25/2014 03:46 PM, Niki Zadeh wrote:

These crashes always precede by a warning:
'''
WARNING from PE 30: tracer_hordiff: Tracer T has associated 3-d
diffusive flux diagnostics. These are not valid when
DIFFUSE_ML_TO_INTERIOR is defined. Use 2-d tracer diffusion
diagnostics instead to get accurate total fluxes.
'''
What does this mean?
—
Reply to this email directly or view
it on GitHub.

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

Dr. Stephen M. Griffies
NOAA Geophysical Fluid Dynamics Lab
201 Forrestal Road
Princeton, NJ 08542
USA

Zhi-Liang · 2014-06-26T14:59:15Z

Hi Niki,

I made a fix in MOM_tracer_hor_diff.F90. I change is to change argument
Reg from pointer to intent(inout).
I tested in /MOM6_AM2_MOM6i_1deg and it runs OK with openmp and reproduce
answer from no openmp.

Zhi

On Tue, Jun 24, 2014 at 6:08 PM, Niki Zadeh [email protected]
wrote:

In branch dev/master/2014.06.23 (testing tag) all coupled test cases crash
when I compile with -openmp

Here's the traceback:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source

fms_MOM6_SIS_LAD_ 000000000185D246 mom_tracer_hor_di 303
MOM_tracer_hor_diff.F90
fms_MOM6_SIS_LAD_ 0000000001717C50 mom_mp_step_mom_ 1062 MOM.F90
fms_MOM6_SIS_LAD_ 00000000016C2F27 ocean_model_mod_m 368
ocean_model_MOM.F90
fms_MOM6_SIS_LAD_ 00000000004013C7 MAIN__ 687 coupler_main.F90

-t repro does not have this problem.

To reproduce the issue:

fremake --execute --nolink -x
/ncrc/home2/Niki.Zadeh/frerts/testing_20140623/MOM6_CM.xml.20140624170350
-p ncrc2.intel -t repro-openmp MOM6_CM_libs_compile

fremake --execute -x
/ncrc/home2/Niki.Zadeh/frerts/testing_20140623/MOM6_CM.xml.20140624170350
-p ncrc2.intel -t repro-openmp MOM6_SIS_LAD_FV_compile

frerun -s -r basic -u --no-transfer -x
/ncrc/home2/Niki.Zadeh/frerts/testing_20140623/MOM6_CM.xml.20140624170350
-p ncrc2.intel -t repro-openmp MOM6_AM2_MOM6i_1deg

—
Reply to this email directly or view it on GitHub
#28.

adcroft · 2014-08-26T19:32:01Z

What's the status on this issue?

nikizadehgfdl · 2014-08-26T19:54:52Z

It is fixed. Closing.

Zhi-Liang · 2014-08-26T20:42:07Z

Hi Alistair,

I think this issue is solved.

Zhi

On Tue, Aug 26, 2014 at 3:32 PM, Alistair Adcroft (GFDL) <
[email protected]> wrote:

What's the status on this issue?

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

…flux Pass latent heat flux via ice/ocean boundary type

- NOAA-GFDL/SIS2@5c8f644 Merge pull request mom-ocean#28 from NOAA-GFDL/user/nnz/fix_omp_compile - NOAA-GFDL/SIS2@cb454e6 Make SIS2 compile with -openmp - No answer changes.

Corrects OMP directives to allow SIS2 to compile with -openmp

Fixed an issue that had the reduced_k_range diagnostic output use the wrong number of elements to determine the time average. Also fixed an issue that would have caused an issue with reduced_k_range, time averaged diurnal sampling Fixed mom-ocean#28

nikizadehgfdl added the bug label Jun 24, 2014

nikizadehgfdl assigned Zhi-Liang Jun 24, 2014

nikizadehgfdl closed this as completed Aug 26, 2014

nikizadehgfdl unassigned Zhi-Liang May 27, 2015

gustavo-marques pushed a commit to gustavo-marques/MOM6 that referenced this issue Sep 27, 2017

Merge pull request mom-ocean#28 from gustavo-marques/fix_latent_heat_…

32916a0

…flux Pass latent heat flux via ice/ocean boundary type

nikizadehgfdl pushed a commit to nikizadehgfdl/MOM6 that referenced this issue Oct 9, 2017

Merge pull request mom-ocean#28 from nicjhan/diagnostic-tests

0aa6749

MJHarrison-GFDL pushed a commit to MJHarrison-GFDL/MOM6 that referenced this issue Jun 15, 2021

Merge pull request mom-ocean#28 from NOAA-GFDL/user/nnz/fix_omp_compile

5c8f644

Corrects OMP directives to allow SIS2 to compile with -openmp

MJHarrison-GFDL pushed a commit to MJHarrison-GFDL/MOM6 that referenced this issue Aug 16, 2022

Allow a peset to have size 1. mom-ocean#28

0df61c4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

coupled models crash when MOM6 is compiled with -openmp #28

coupled models crash when MOM6 is compiled with -openmp #28

nikizadehgfdl commented Jun 24, 2014

Zhi-Liang commented Jun 24, 2014

nikizadehgfdl commented Jun 25, 2014

Zhi-Liang commented Jun 25, 2014

Hallberg-NOAA commented Jun 25, 2014

StephenGriffies commented Jun 25, 2014

Zhi-Liang commented Jun 26, 2014

adcroft commented Aug 26, 2014

nikizadehgfdl commented Aug 26, 2014

Zhi-Liang commented Aug 26, 2014

coupled models crash when MOM6 is compiled with -openmp #28

coupled models crash when MOM6 is compiled with -openmp #28

Comments

nikizadehgfdl commented Jun 24, 2014

Zhi-Liang commented Jun 24, 2014

nikizadehgfdl commented Jun 25, 2014

Zhi-Liang commented Jun 25, 2014

Hallberg-NOAA commented Jun 25, 2014

StephenGriffies commented Jun 25, 2014

Zhi-Liang commented Jun 26, 2014

adcroft commented Aug 26, 2014

nikizadehgfdl commented Aug 26, 2014

Zhi-Liang commented Aug 26, 2014