Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: File names are not correct when calculating variability indices for certain realizations. #1105

Open
ShihengDuan opened this issue Jul 17, 2024 · 3 comments

Comments

@ShihengDuan
Copy link

What happened?

I'm using r*i1p1f1 in the parameter file to calculate variability indices for all realizations but with i1p1f1 (only change r numbers). This seems to work but the output file name does not follow the realization name:

metrics_results : /p/user_pub/climate_work/duan5/Noise/DuanPMP-New/CanESM5-psl-historical-CBF
Converting units by  divide 100.0
 -----  CanESM5  ---------------------
model_path: /p/css03/esgf_publish/CMIP6/CMIP/CCCma/CanESM5/historical/r1i1p1f1/Amon/psl/gn/v20190306/psl_Amon_CanESM5_historical_r1i1p1f1_gn_185001-201412.nc
 ---  r*i1p1f1  ---
model_lf_path: None
Warning: Data now ends at 2014 instead of 2015
Converting units by  divide 100.0
INFO::2024-07-17 14:40::pcmdi_metrics:: Results saved to a json file: /p/user_pub/climate_work/duan5/Noise/DuanPMP-New/CanESM5-psl-historical-CBF/var_mode_NAO_EOF1_stat_cmip6_historical_mo_atm_CanESM5_r*i1p1f1_1850-2014.json
2024-07-17 14:40:40,049 [INFO]: base.py(write:251) >> Results saved to a json file: /p/user_pub/climate_work/duan5/Noise/DuanPMP-New/CanESM5-psl-historical-CBF/var_mode_NAO_EOF1_stat_cmip6_historical_mo_atm_CanESM5_r*i1p1f1_1850-2014.json
2024-07-17 14:40:40,049 [INFO]: base.py(write:251) >> Results saved to a json file: /p/user_pub/climate_work/duan5/Noise/DuanPMP-New/CanESM5-psl-historical-CBF/var_mode_NAO_EOF1_stat_cmip6_historical_mo_atm_CanESM5_r*i1p1f1_1850-2014.json
model_path: /p/css03/esgf_publish/CMIP6/CMIP/CCCma/CanESM5/historical/r1i1p1f1/Amon/psl/gn/v20190429/psl_Amon_CanESM5_historical_r1i1p1f1_gn_185001-201412.nc
 ---  r*i1p1f1  ---
model_lf_path: None

The model path is correct with r1i1p1f1 but not the file names.

What did you expect to happen? Are there are possible answers you came across?

The file name should follow the model path name.

Minimal Complete Verifiable Example (MVCE)

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

pcmdi-metrics 3.3.4 pypi_0 pypi

@lee1043
Copy link
Contributor

lee1043 commented Jul 19, 2024

@ShihengDuan Thank you for reporting this. I am currently on vacation but should be able to work on this in a few weeks after I return. Do you have a workaround during the mean time?

@ShihengDuan
Copy link
Author

I'm using * for now and it should be fine for most models. But for CanESM5, there are lots of ensembles with varying physics schemes (which we don't need). So it would be better if we can only get the ensembles with different initial conditions.

@ShihengDuan
Copy link
Author

I tried to use xsearch and only select the ensemble members I want to keep. Here is a workaround that is slow but seems to work.

import sys
sys.path.insert(0, '/p/user_pub/xclim/persist/software/xsearch/')
import xsearch as xs
import argparse
import subprocess
import os

def get_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('--model', type=str, default='CanESM5')
    parser.add_argument('--scenario', type=str, default='historical')
    parser.add_argument('--variability', type=str, choices=['PNA', 'NAM', 'SAM', 'NAO'])
    args = vars(parser.parse_args())
    return args

def modify_template_file(template_path, new_file_path, new_lines):
    # Read the content of the template file
    with open(template_path, 'r') as file:
        content = file.readlines()

    # Add new lines to the beginning
    new_content = new_lines + content

    # Save the modified content to a new file
    if os.path.exists(new_file_path):
        os.remove(new_file_path)
    with open(new_file_path, 'w') as file:
        file.writelines(new_content)


args = get_args()
model = args['model']
scenario = args['scenario']
variability = args['variability']
var = 'psl'
print('model: ', model, ' \nscenario: ', scenario)
files = list(xs.findPaths(scenario, var, 'mon', model = model).keys())
print(sorted(files))
print(len(files))
template = 'parameter_template_psl.py'
for file in files:
    member = file.split('/')[-6]
    # print(member)
    if 'i1p1f' in member:
        print(member)
        new_lines = [
            "scenario = "+"'"+scenario+"'\n",
            "gcm_name = "+"'"+model+"'\n",
            "realization = "+"'"+member+"'\n",
            "modpath = "+"'"+file+"'\n",
            "variability_mode = "+"'"+variability+"'\n"
        ]
        modify_template_file(template, variability+'-'+model+'-'+member+'.py', new_lines)

        command = f"variability_modes_driver.py -p "+variability+"-"+model+"-"+member+".py > "+variability+"-"+model+"-"+member+".out"
        subprocess.run(command, shell=True)
       

parameter_template_psl.py only contains parameters that are not relevant to the model and realizations (e.g., CBF, debug options, msyear)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants