Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

Description:

This PR updates the cpld_bmark_p8 tests to a prototype GEFS test case of fully coupled s2swa+IAU+stochastics, with configuration and warm starts from restarts of EP5r2 ensemble member 1 for 2021-03-25 06Z.

The EP5r2 test case was kindly provided by @bingfu-NOAA via @junwang-noaa with aerosol input data and configurations from @lipan-NOAA. The wave element mask has been modified as in TODO.

A separate INPUTDATA_ROOT_BMIC is no longer needed and is removed.

The test suite regression tests basic reproducibility/quality checks, particularly:

• control reproduces itself
• restart reproduces control
• changing number of tasks reproduces control
• changing number of threads reproduces control
• Intel debug version reproduces itself
• GNU and GNU debug versions run
  • GNU debug on hera fails with likely openmpi error:
    140: The OSC pt2pt component does not support MPI_THREAD_MULTIPLE in this release.
    140: Workarounds are to run on a single node, or to use a system with an RDMA
    140: capable network such as Infiniband.
  • GNU and GNU debug on hercules fail with NetCDF HDF error
    Error in handle_err: get_var3_r4 get_vara_real delp_inc NetCDF: HDF error

with caveats across supported platforms. TODO see pre-test matrix

• All tests pass on supported platforms

  • Hera (intel)
  • Hercules (intel)
  • Gaea (intel)
  • Derecho (intel)
    With increased WAV_tasks, hangs 30 minutes into simulation
  • Larger issue of GNU support

• No major diffs from GEFS workflow configuration
  I have tried to note intentional differences. Please let us know if you see differences.

  Depending on aerosol coupling, GOCART .rc files and ExtData directory structure may be revised for consistency with global-workflow.

  This benchmark configuration and case may be updated, particularly with GEFS reforecast or UFS case study.

Test failures may be better served via library/platform support and work will continue in follow up issues, including:

• [] Larger issue of GNU support
• [] Sensitivity to spack-stack updates
• [] individual test results

TODO: Scripts need finalizing once filepaths are in shared space.
Input data is currently in user space on hera:
/scratch1/NCEPDEV/nems/Nick.Szapiro/tasks/input_data/gefs.v13/RT_GEFS/

Commit Message:

* UFSWM - Add GEFS regression test suite from EP5r2 configuration/case

Priority:

• Normal

Git Tracking

UFSWM:

• Closes Update cpld_bmark_p8 with GEFSv13 EP5 configuration #1987
• Closes Add RT test for gocart_on, gccpp_on, nasa_on #1818

Sub component Pull Requests:

• None

UFSWM Blocking Dependencies:

• Update inline post + Add documentation for HSD cases in tests-dev #2498 #2489
  Need to update UPP hash to include UPP PRs already committed (Add aerosol fields in the gefs UPP control files NOAA-EMC/UPP#1052, Add gefs inline post itag file NOAA-EMC/UPP#1060, and Reduce the printouts for GEFS processing. NOAA-EMC/UPP#1075)

Changes

Regression Test Changes (Please commit test_changes.list):

• PR Adds New Tests/Baselines.

Input data Changes:

• New input data.

Library Changes/Upgrades:

• No Updates

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)