feature/global_ens_priorities_456 #332
Conversation
Refs: NOAA-EMC#5
Iteration 3: global_ens atmos & headline: stats
NOTE: The WMO stats related scripts were not touched as these will not be run in operations. Describe testing already performed for this Pull Request:
Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
Has the code been checked to ensure that no errors occur during the execution? Yes Do these updates/additions include sufficient testing updates? Yes |
|
@malloryprow and @StevenSimon-NOAA , I have kicked off all of the stats processing. All are underway and some are completed. Please review. |
There are a few errors that we need to navigate through. But let's start with the precip stats drivers first. The ecme precip stats job has a syntax error around line 40000. The cmce precip stats job has a bunch of LNET errors in debugging section, but it is not immediately clear what the source is. The naefs precip stats job has a similar syntax error as ecme prcip stats. The gefs precip stats job has the same unknown LNET error as the cmce precip stats job. I will see if there is anything more to these unknown LNET errors. In the meantime, I will delve into the other stats jobs. |
|
Noted with the ecme and naefs precip. I'll fix that. What are these LNET errors? I searched "LNET" in /lfs/h2/emc/ptmp/shelley.melchior/EVS_out/jevs_global_ens_cmce_atmos_precip_stats.o93093411 and I'm not getting anything back. |
Try a search with just "Error" for both cmce and gefs precip stats. It should appear towards the bottom of both logfiles. |
|
That isn't related to anything EVS or METplus pretty sure that is something with WCOSS2 nodes. |
|
@ShelleyMelchior-NOAA Can you please rerun dev/drivers/scripts/stats/global_ens/jevs_global_ens_cmce_atmos_precip_stats.sh, dev/drivers/scripts/stats/global_ens/jevs_global_ens_ecme_atmos_precip_stats.sh, dev/drivers/scripts/stats/global_ens/jevs_global_ens_gefs_atmos_precip_stats.sh, and dev/drivers/scripts/stats/global_ens/jevs_global_ens_naefs_atmos_precip_stats.sh? |
|
All previous stats processing has completed. I have re-submitted the 4 precip drivers noted. |
|
jevs_global_ens_ecme_atmos_precip_stats.o93101037 completed and this run does not have the previously mentioned error. |
|
All jobs are completed. I ran my script to grep for "bad" words on the files. The only concerning ones I found were for the previously run precip jobs but the new logs don't have those errors. |
|
This is excellent! Does that seem to satisfy iteration 3 testing? |
|
I looked over the large stat files. I think iteration 3 is good. Should I remove iteration 4 from this PR since there is #348? |
|
@malloryprow I would say yes! If this PR gets merged, @MarcelCaron-NOAA can bring your changes into his and be super mindful of any conflicts! Does that sound good, @ShelleyMelchior-NOAA? |
|
I can also confirm that all large global_ens stats files are fully accounted for. I am going over the small stats files right now. |
|
Excellent! Yes, iteration 4 will get covered in Marcel's PR #348. |
|
So here's the deal with the global_ens small stats files. In most cases, the counted quantities are less than the expected quantities for each field by model. But the log files indicate no apparent errors. So my conjecture right now is that these discrepancies are due to the recent WCOSS2 issues causing stops and discontinuities in the prep space. Does that make sense in this situation? |
|
I added file checks so we don't see any errors anymore when there is missing data. I added a check too to make sure all the ensemble members are present. |
In that case, I think we are in good shape with the global_ens small stats files as well. |
|
Before I merge, I just want to comment, WOW! You did a ton of work on this Mallory. I mean, I know you did a lot of work, but seeing it in lines of changed code in the formal code review is quite staggering. Well done! Thank you for addressing these critical EE2 items for the global_ens component. |
I second these kudos. These updates to the global_ens component have already helped me better navigate the global_ens logfiles, especially those pesky "red herring errors" that pop up from time to time. Many thanks Mallory for the time and effort you put into these updates! We are now one step closer (pun intended) to getting this EVS component ready for production. Onto global_ens plots! |
|
Thank you @ShelleyMelchior-NOAA and @StevenSimon-NOAA! |
Since global_ens is such a large component and I am not the developer of this code, we are going to iteratively work through this PR, so the usual PR information will be found in the comments for each iteration.
This PR can be merged once all iterations are completed.
Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
Set-up
Pull Request Checklist
Review the source issue metadata (required labels, projects, and milestone).
Complete the PR description above.
Ensure the PR title matches the feature branch name.
Check the following:
Instructions provided on how to run
Developer's name is replaced by ${user} where necessary throughout the code
Check that the ecf file has all the proper definitions of variables
Check that the jobs file has all the proper settings of COMIN and COMOUT and other input variables
Check to see that the output directory structure is followed
Be sure that you are not using MET utilities outside the METplus wrapper structure
After submitting the PR, select Development issue with the original issue number.
After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
Close the linked issue.