There are nearly perfect matches between GSI/Hofx and UFO/HofX values when hydrometeor control variables include liquid and ice clouds. For example, these two figures shows comparisons of ATMS-N20 HofX values without bias correction (BC) and with BC. These are made by using IODA files transformed from the default standalone GSI outputs with Cori's "anavinfo" file /work2/noaa/da/cmartin/UFO_eval/geovals/GSI/fix/global_anavinfo.l127.txt
Same comparisons are made with Andrew's testing files for UFO evaluations in /work2/noaa/da/acollard/UFO_eval/data/gsi_geovals_l127/

However, there are inconsistencies between GSI and UFO with my IODA testing files. Liquid and ice clouds are used as state variables in both GSI and UFO. However, there are slight difference in the GSI "anavinfo" file as shown above.

@ADCollard @CoryMartin-NOAA It turns out that all liquid and ice cloud content are zero in current IODA geoval files for AMSUA and ATMS. Standalone test produce zero-ed cloud with Cory's "anavinfo" files. Here is a files of "mass_content_of_cloud_liquid_water_in_atmosphere_layer" dumped from from "atms_n20_geoval_2024021900.nc4": /work2/noaa/da/jianjun/UFO_eval/data/gsi_geovals_l127/nofgat_feb2024/20240815/qlqi.Andrew/atms_n20.ql.content.txt

I am not exactly sure how the clouds are zero-ed. However, it is likely because "cw" is presented in the "anavinfo". It should be omitted in all-sky configurations.

@ADCollard It makes UFO HOfX close to GSI HofX to read "cloud fraction" from geoval. Of course, this "cloud fraction" has to be saved from GFS background. Here is a comparison of ATMS_N20 channel 1 hofx differences (UFO hofX - GSI hofX) versus Obs-hofX in GSI without bias correction or quality control. Only liquid and ice clouds are used in this comparison.

Here is another comparison of ATMS_N20 channel 1 hofx differences (UFO hofX - GSI hofX) versus Obs-hofX in GSI without bias correction or quality control. All five hydrometeors are used in this comparison.

Now, there is an issue in bias correction. UFO HofX values don't match GSI HofX after bias correction even though those values match before bias correction. Here is a comparison between ATMS-N20 Channel 1 UFO Hofx and GSI hofX after bias correction with cloud area fraction as an input from geoval and use only liquid and ice clouds in both GSI and UFO. It is a bit puzzling since the same bias correction coefficient files are use in both GSI and UFO.

Here is another comparison of ATMS-N20 channel 15 HofX. Note, there is no bias correction for this channel in GSI or UFO.

It looks like there is an inconsistency in surface type. Here are differences between JEDI and GSI. HofX for ATMS-N20 without bias correction. Note, only liquid and ice clouds are used as cloud inputs.

It is seems that bias correction works now using cloud-fraction from geoval. However, UFO hofx, before or after bias correction, just don't match GSI hofx over non-water surface. Differences are relatively large at channels 16-22. The issue seems to be related with some difference in surface characters instead of cloud cover since hofx values match over water.

JEDI hofx - GSI hofx, without bias correction, channel 3:

JEDI hofx - GSI hofx, after bias correction, channel 3:

JEDI hofx - GSI hofx, without bias correction, channel 18:

JEDI hofx - GSI hofx, after bias correction, channel 18:

It turns out that a configuration "Cloud_Seeding: true" in the UFO yaml sets cloud threshold values for CRTM inputs not only over ocean but also over other types of surfaces. However, all cloud values are set to be zero for CRTM inputs in GSI over non-water surfaces. These discrepancies cause differences between GSI and UFO Hofx over land and ice areas. Here are two figures showing nearly identical ATMS-N20 HofX at channels 3 and 18 after bias correction when Cloud_Seeding is set as "false". A new configuration will be added to calculate HofX without clouds over non-water surfaces.

It turns out that there is no error inflation due to scattering effects for ATMS. Here is the line where it is turned off:
https://github.com/NOAA-EMC/GSI/blob/develop/src/gsi/qcmod.f90#L3673
I tried to update the UFO code, but later decide to assign "-999" for ATMS scattering indices after they are calculated in the Yaml.

  # Calculate scattering index from observation
  - filter: Variable Assignment
    assignments:
    - name: SIRetFromObs@DerivedMetaData
      type: float
      function:
        name: SCATRetMW@ObsFunction
        options:
          scatret_ch238: 1
          scatret_ch314: 2
          scatret_ch890: 16
          scatret_types: [ObsValue]
  # scattering index is actually not used in GSI for ATMS.
  - filter: Variable Assignment
    assignments:
    - name: SIRetFromObs@DerivedMetaData
      type: float
      value: -999.0
      

Here are figures to show differences between those final observational errors for channel 3 and all channels before and after this inflation is turned off.

Some updates in quality control and gross error check are added in an UFO PR: https://github.com/JCSDA-internal/ufo/pull/3547
However, there is still a puzzle why numbers of used observations don't match between JEDI and GSI:

@ADCollard @emilyhcliu Now JEDI/UFO can fully reproduce ATMS-N20 and ATMS_NPP results made by GSI after the scattering error inflation and cloud fraction threshold in GSI are made to be consistent with or same as in UFO. Here are differences in Obs errors, Hofx, and number of data passing quality control for ATMS_N20. Note: some Hofx values are missing in GSI. That causes large differences between Hofx. However, it is not a concern since these data are tossed out in GSI and JEDI.

Here are differences in Obs errors, Hofx, and number of data passing quality control for ATMS_NPP:

There is one ATMS-N21 observation that doesn't pass QC in GSI but passes in JEDI.

Good work, @jianjunj !

Corrected the post about ATMS-N21: There is one observation that doesn't pass QC in GSI but passes in JEDI.

Corrected the post about ATMS-N21: There is one observation that doesn't pass QC in GSI but passes in JEDI.

@jianjunj That's OK. It is great that AMTS in JEDI can reproduce GSI for the phase-1 tests.

@ADCollard @emilyhcliu Here are results for AMSUA-N19. JEDI results are nearly identical to GSI values in terms of observational errors. There is one more observation passing QC in JEDI than in GSI.

The bias correction is turned off in above results by adding the channel number in the Yaml

  obs bias:
    variables without bc: [brightnessTemperature]
    channels: 14

However, I think we want to get this number from the observation chronicle table:
https://github.com/NOAA-EMC/jcb-gdas/blob/c9f2d79833de83f342055140cac679e6e7ec9ff7/observation_chronicle/atmosphere/amsua_n19.yaml#L32

channel_values:
#       sim act bcd  err   errcld  ermax    x0     x1
......
  13: [  1,  1,  1,  0.80,  0.80,  4.50,  0.000,  0.000 ]
  14: [  1,  1,  0,  4.00,  4.00,  4.50,  0.000,  0.000 ]   # No bias correction of Ch.14
  15: [  1,  1,  1,  3.50, 18.00,  4.50,  0.030,  0.200 ]

Not sure whether we are able to do that at the moment.

@ADCollard @emilyhcliu It turned out I no longer have the old statistics plots in which bias correction was conducted for channel 14. Please see the updated post.

@CoryMartin-NOAA Is there a way to parm the channel number for which bias correction is turned off from the observation_chronicle table into the Yaml file for satellite data?

Here is the table
https://github.com/NOAA-EMC/jcb-gdas/blob/c9f2d79833de83f342055140cac679e6e7ec9ff7/observation_chronicle/atmosphere/amsua_n19.yaml#L32

channel_values:
#       sim act bcd  err   errcld  ermax    x0     x1
......
  13: [  1,  1,  1,  0.80,  0.80,  4.50,  0.000,  0.000 ]
  14: [  1,  1,  0,  4.00,  4.00,  4.50,  0.000,  0.000 ]   # No bias correction of Ch.14
  15: [  1,  1,  1,  3.50, 18.00,  4.50,  0.030,  0.200 ]

I'd like to add number 14 in
https://github.com/NOAA-EMC/jcb-gdas/blob/c9f2d79833de83f342055140cac679e6e7ec9ff7/observations/atmosphere/amsua_n19.yaml.j2#L34:
like:

  obs bias:
    variables without bc: [brightnessTemperature]
    channels: 14

@jianjunj I'm not 100% sure what you're asking, but if you're saying you'd prefer the exclusion of bias correction for this channel to be written in the template YAML and not the chronicle, I think we want these sort of choices made in the chronicle. @danholdaway any comments?

@CoryMartin-NOAA @danholdaway We can parm the column for "bcd" in the chronicle [1,1,....,1,0,1]. What I want is the index number of 14 where 'bcd' is 0.
-- It is the channel number that is wanted.

@CoryMartin-NOAA I'd prefer bias correction is determined by the chronicle table. However, bias correction was actually conducted without the specific configuration in the observation Yaml file. Here are figures made before and after the new configuration is added:

@jianjunj I think we need to add a function to JCB to handle this case. It will transcribe 1,0,1 into channels: 14. There's a similar case to this, which is the simulated channel setting in the YAML, which transcribes e.g. 1,1,1 into the actual channel numbers, e.g. 13,14,15. Take a look here: https://github.com/NOAA-EMC/jcb/blob/336d96c980892146de9fa76e2dfe55ce41038b2f/src/jcb/observation_chronicle/observation_chronicle.py#L155

If you can add another conditional that is along the lines of "if variable_name == 'not_bias_corrected':" that transcribes 0,1,0 into 14. Then in the AMSUA-N19 YAML you can have:

obs bias:
  variables without bc: [brightnessTemperature]
  channels: {{observation_from_jcb}}_not_bias_corrected

Does that YAML work if all channels are bias corrected? I.e. if the resulting YAML is:

obs bias:
  variables without bc: [brightnessTemperature]
  channels: 

Can you check that? If so then we can just go ahead and add that section to basically every YAML (that has an associated chronicle). If it doesn't work then can you update UFO so it would work like that? Otherwise we'd have instead something (slightly undesirable) like:

{% if {{observation_from_jcb}}_not_bias_corrected is defined %}
obs bias:
  variables without bc: [brightnessTemperature]
  channels: {{observation_from_jcb}}_not_bias_corrected
{% endif %}

Thanks @danholdaway . I think it works for observations when all channels are bias corrected, though I only tested ATMS and AMSUA data. Bias correction is off for ATMS channel 15. However, all input bias correction predictors are zero for ATMS channel 15. As a result, this issue didn't pop up.

@danholdaway The configuration with empty channels does not work. JCB assigns "null" to the empty channels and then JEDI complains about "nil" channels.

obs bias:
  variables without bc: [brightnessTemperature]
  channels: 

Can you have JCB return an empty string when all channels are bias corrected? It might be quite messy to have the if statements, with the section of obs bias coming and going depending on the time.

@danholdaway I had a look at JCB, but not really sure how change that output of the not-defined value to an empty string in it. In addition, it turns out I am actually using @CoryMartin-NOAA 's build of JCB:
https://github.com/NOAA-EMC/GDASApp/blob/develop/ush/ufoeval/run_ufo_hofx_test.sh#L80

@danholdaway Just had a test. UFO does not like an empty string. It reads an empty input in as "nil":
variables without bc => (brightnessTemperature) , channels => (nil) ,

Then it has an error:
"ufo/ObsFilters/testFilters" failed with unhandled eckit::Exception: Bad Conversion: Cannot convert (nil) (Nil) to std::string

Thanks for checking that out.

@danholdaway I think it is the "eckit" that complains the empty input in YAML. Here is the error message:

Running case 0: ufo/ObsFilters/testFilters ...
Test "ufo/ObsFilters/testFilters" failed with unhandled eckit::Exception: Bad Conversion: Cannot convert (nil) (Nil) to std::string @ (/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/cache/build_stage/spack-stage-eckit-1.24.5-lxshla5iuewtipytiq6zn7z5git23m4g/spack-src/src/eckit/value/Content.cc +75 badConversion)^[[0m
Stack trace: backtrace [1] stack has 28 addresses

I just had a test with "channels: 0" as an input for observations in which all channels are bias-corrected. It works. We can list an empty string there. Then change "null" to "0" after JCB changes changes the empty to "null". However, @emilyhcliu and I discussed about it and think it is a bit confusing and your suggestion is actually a better choice.

{% if {{observation_from_jcb}}_not_bias_corrected is defined %}
obs bias:
 variables without bc: [brightnessTemperature]
 channels: {{observation_from_jcb}}_not_bias_corrected
{% endif %}       

Ok sounds good. I was hoping to avoid jinja2 logic in the YAML files that live in jcb-gdas but this is will have to be the exception for now, until we think of a different way.

@danholdaway I am thinking about adding the channels to be bias-corrected instead of the list not to be bias-corrected in the UFO yaml. It would be straight forward with my draft PR in the JCB if we can do that.

I had a look at the UFO and found that it didn't work to simply add the list of channels for which we want bias-corrected. It seems that it requires effort to make that work. Therefore, I decided to let JCB return "-999" for the channels without bias correction when all channels are bias corrected. That works in UFO. I don't think the number "-999" is as confusing as "0".

Here are my changes in JCB NOAA-EMC/jcb#15 and JCB-GDAS https://github.com/NOAA-EMC/jcb-gdas/pull/57/files.