Add collective metadata functions to the low level API #2224
Conversation
The HDF5 documentations says H5P[get/set]_all_coll_metadata_ops can take file, group, dataset, datatype, link, or attribute access property list identifiers. Only file, link and dataset access property lists appear to be exposed by h5py.
Codecov ReportBase: 90.01% // Head: 89.30% // Decreases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## master #2224 +/- ##
==========================================
- Coverage 90.01% 89.30% -0.72%
==========================================
Files 17 17
Lines 2394 2394
==========================================
- Hits 2155 2138 -17
- Misses 239 256 +17
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
|
Just driving by to show my support. These optimizations are critical for any non-trivial level of scaling and I hope these can be added to h5py soon |
|
Hi, thanks @jchelly for implementing this! May I ask what the status of this PR? I was looking for the exact low-level API functions in h5py and found this PR! I am more than happy to contribute to the code for any remaining work so this will be available in h5py in the future. |
I'd like to add the following functions to the low level API:
For context: I'm using h5py on a HPC cluster to process simulation outputs stored as HDF5. The code is distributed over multiple compute nodes which have 128 cores each. In order to make use of all of the cpu cores I run python using mpi4py with one process per core. I'm using collective I/O to read the input simulation data and write out the results.
This puts quite a load on the Lustre parallel file system, and I think it's probably because every process accesses the files independently for metadata operations. I'm hoping that can be alleviated by having HDF5 do all file access in collective mode so that only a few processes per node need to access the file system.
For my use case I just need to put the whole file in collective metadata mode. To do that I've added get/set_all_coll_metadata_ops() and get/set_coll_metadata_write() methods to h5p.PropFAID. The HDF5 documentation says that H5Pset_all_coll_metadata_ops() can also be called on group, dataset, datatype, link, or attribute access property lists. Of those I think h5py only exposes link and dataset access property lists so I also added get/set_all_coll_metadata_ops() to h5p.PropLAID and h5p.PropDAID.