Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re-work testing verification, fail file naming consistency, remove lsf support, simplify RT logs. #2061

Merged
merged 95 commits into from
Feb 21, 2024
Merged

Re-work testing verification, fail file naming consistency, remove lsf support, simplify RT logs. #2061

merged 95 commits into from
Feb 21, 2024

Conversation

BrianCurtis-NOAA
Copy link
Collaborator

@BrianCurtis-NOAA BrianCurtis-NOAA commented Dec 22, 2023 •
edited by zach1221
Loading

Commit Queue Requirements:

  • Fill out all sections of this template.
  • All sub component pull requests have been reviewed by their code managers.
  • Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
  • Commit 'test_changes.list' from previous step

Description

This PR updates the rt.sh script:

  • Makes fail_test and fail_compile files consistent in naming
  • Created a new generate_log() function in rt.sh to allow for better control over the information logged after the full RT suite is finished.
    • Log outputs minimized to information that's important to track, and easier to read for developers.
    • Adds test_changes.list file for repository tracking of what tests are changed due to the changes in the PR.
    • Adds test_changes.list file for easy creation of new baselines specific to the PR using rt.sh -b test_changes.out.
  • Ends support for lsf (queue system from old WCOSS systems)

Commit Message

* UFSWM
  * Re-work test verification in rt.sh for developers to use in PR process. (Closes #2058)
  * fail_<test_number> file name upgraded to use compile/test name instead of number.
  * Regression test logs do not clearly indicate when a test fails to run. (Closes #1821)
  * New test_changes.list file:
    * Verification of what tests change for each PR.
    * Used with './rt.sh -c -b test_changes.list` to help create new baselines for each PR.

Priority

  • Normal

Git Tracking

UFSWM

Sub component Pull Requests

  • None

UFSWM Blocking Dependencies

  • None

Changes

Regression Test Changes (Please commit test_changes.list):

  • None Expected

Input data Changes

  • None.

Library Changes/Upgrades

  • No Updates

Testing Log:

  • RDHPCS
    • Hera
    • Orion
    • Hercules
    • Jet
    • Gaea
    • Derecho
  • WCOSS2
    • Dogwood/Cactus
    • Acorn
  • CI
  • opnReqTest (complete task if unnecessary)

BrianCurtis-NOAA and others added 12 commits December 14, 2023 16:06
@BrianCurtis-NOAA
Remove LSF support
6d06965
@BrianCurtis-NOAA
Cleanup wording for RT failures, and why tests are failling
7645d38
merge conflict with rt_utils fixed
d4df8c7
@BrianCurtis-NOAA
Initial push, adds calculation to compare compiles and tests to make …
e83f033 
…sure all are run/completed
@BrianCurtis-NOAA
Re-work testing verification to match STESTS_FILE
e6a72e0
@BrianCurtis-NOAA
Merge branch 'cleanup_failures' of github.com:BrianCurtis-NOAA/ufs-we…
e781d2b 
…ather-model into rt_pretest
@BrianCurtis-NOAA
Brought in changes to wording for tests, and fixed any testing bugs
2768362
@BrianCurtis-NOAA
Fix error with logic in new verification function, and testing bugfixes
7d7d4ba
@BrianCurtis-NOAA
Fixed issue with log dir test name having extra underscore
fba71de
@BrianCurtis-NOAA
Small adjustment of text
4d1c08c
@BrianCurtis-NOAA
Re-work verification a little to reduce the number of if statements.
d2d27c7
@BrianCurtis-NOAA
Merge branch 'develop' of github.com:ufs-community/ufs-weather-model …
70c02c1 
…into rt_pretest
@BrianCurtis-NOAA BrianCurtis-NOAA changed the title Rt pretest Add pre-test option to rt.sh (-p), re-work testing verification, fail file naming consistency. Dec 22, 2023
@BrianCurtis-NOAA BrianCurtis-NOAA changed the title Add pre-test option to rt.sh (-p), re-work testing verification, fail file naming consistency. Add pre-test option to rt.sh (-p), re-work testing verification, fail file naming consistency, remove lfs support. Dec 22, 2023
@BrianCurtis-NOAA BrianCurtis-NOAA mentioned this pull request Jan 11, 2024
Closed
@BrianCurtis-NOAA
Copy link
Collaborator Author

@zach1221 it looks like the Derecho issue would block any PR at this point, should we proceed with testing? Have the ORT's been successful?

I don't think there's another PR ready at the moment. We could continue testing against this PR today, on the other machines, while Derecho is worked on. I've got ORTs running on Hera now. @jkbk2004 do you want to wait until Derecho is ready?

@zach1221 can you submit jobs in sequential mode even if rocoto/eclfow are not functional ?

This would take an extremely long time.

@zach1221
Copy link
Collaborator

@zach1221 it looks like the Derecho issue would block any PR at this point, should we proceed with testing? Have the ORT's been successful?

I don't think there's another PR ready at the moment. We could continue testing against this PR today, on the other machines, while Derecho is worked on. I've got ORTs running on Hera now. @jkbk2004 do you want to wait until Derecho is ready?

@zach1221 can you submit jobs in sequential mode even if rocoto/eclfow are not functional ?

This would take an extremely long time.

It's still running since last night. Hopefully not too much longer.

@BrianCurtis-NOAA BrianCurtis-NOAA added No Baseline Change No Baseline Change Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. labels Feb 20, 2024
@zach1221
Copy link
Collaborator

Jet is mostly done. Just had one test timeout that I'm rerunning.

@BrianCurtis-NOAA
Copy link
Collaborator Author

What changes were made in Derecho to allow these tests to pass? I didn't see any Derecho changes in that regard.

@zach1221
Copy link
Collaborator

What changes were made in Derecho to allow these tests to pass? I didn't see any Derecho changes in that regard.

I'm about to commit them.

@zach1221 zach1221 mentioned this pull request Feb 21, 2024
Closed
@grantfirl grantfirl mentioned this pull request Feb 21, 2024
Merged
14 tasks
@zach1221
Copy link
Collaborator

Ok we should be ready to proceed with final review here. @jkbk2004 fyi

@BrianCurtis-NOAA
Copy link
Collaborator Author

BrianCurtis-NOAA commented Feb 21, 2024 •
edited by zach1221
Loading

Should be good to go for review/merge.
@jkbk2004 @SadeghTabas-NOAA

@FernandoAndrade-NOAA FernandoAndrade-NOAA merged commit 6988662 into ufs-community:develop Feb 21, 2024
@CSyl CSyl mentioned this pull request Mar 13, 2024
Merged
@grantfirl grantfirl mentioned this pull request Mar 28, 2024
Merged
@DeniseWorthen DeniseWorthen mentioned this pull request Dec 4, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DusanJovic-NOAA DusanJovic-NOAA DusanJovic-NOAA left review comments

@FernandoAndrade-NOAA FernandoAndrade-NOAA FernandoAndrade-NOAA approved these changes

@jkbk2004 jkbk2004 jkbk2004 approved these changes

@SadeghTabas-NOAA SadeghTabas-NOAA Awaiting requested review from SadeghTabas-NOAA

Assignees

@BrianCurtis-NOAA BrianCurtis-NOAA

Labels
No Baseline Change No Baseline Change Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked.
Projects
None yet
Milestone
No milestone
7 participants
@BrianCurtis-NOAA @DeniseWorthen @junwang-noaa @zach1221 @jkbk2004 @DusanJovic-NOAA @FernandoAndrade-NOAA