JPEG-XL #80
JPEG-XL #80
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Codecov Report
@@ Coverage Diff @@
## main #80 +/- ##
=====================================
Coverage 0.00% 0.00%
=====================================
Files 8 9 +1
Lines 679 710 +31
=====================================
- Misses 679 710 +31
Continue to review full report at Codecov.
|
| dataarray = dataarray.round().clip(min=0, max=3).astype(np.int8) | ||
| dataarray.attrs = serialize_attrs(dataarray.attrs) | ||
| # Convert 3's to NaNs as they should be No Data/Space | ||
| dataarray = dataarray.where(dataarray["variable"] != 3) |
There was a problem hiding this comment.
I think this line implicitly converted the cloud masks back to float64 (because int8 can't represent NaNs)?
There was a problem hiding this comment.
Ah, possibly, the cloud masks, when I open them up, are int8, so possibly something else is happening to the NaNs
| @@ -0,0 +1,182 @@ | |||
| """Thin wrapper around imagecodecs.JpegXl. | |||
There was a problem hiding this comment.
We could consider moving this file into a separate repository? Maybe it's fine in satip for now?
There was a problem hiding this comment.
I think its fine here, if this is the only place we are using JPEGXL for now
|
Sorry, I should've said: I haven't tried running But |
No worries! If the generate test plot runs, then it does work end to end, the test plots uses all the same steps as the other scripts, so it should be fine! |
|
Might merge this now, just to make it easier working on the conversion script |
Pull Request
Description
Use JPEG-XL to compress satellite Zarr images.
Fixes #66
Fixes #67
int8, with NaNs represented as-1(I think the previous code saved as float32?)generate_test_plots.pystacked_eumetsat_datatodata(Issue Renamestacked_eumetsat_datatodata; and renamevariabletochannels? #66).gitignoreI've changed the channels_chunk_size from 11 to 1 because the imagecodecs interface to JPEG-XL doesn't yet know how to decompress multiple images per JPEG-XL file (so it has to store each channel and each timestep as a separate file).
@jacobbieker This might also mean that we might want to consider using large image sizes (in pixels). If the compressed images are too small (much less than 1 MByte on average) then we might want to consider using larger image sizes, I'd guess.
How Has This Been Tested?
Checklist: