Skip to content

Uninitialized variable length sequences are returned as scalars instead of empty arrays #321

Open
@mattjala

Description

When reading from elements in a dataset of variable-length type, uninitialized elements are returned as scalars. With the library API, uninitialized vlen types are considered to be length-zero arrays.

Test program in C to generate an erroneous response from HSDS using the REST VOL:

int main() {
  #define COUNT 10
  hid_t file_id = H5I_INVALID_HID;
  hid_t dset_id = H5I_INVALID_HID;
  hvl_t rbuf[COUNT];
  const hsize_t dims[] = {COUNT};

  hid_t vlen_id = H5Tvlen_create(H5T_NATIVE_INT);

  file_id = H5Fcreate("/home/test_user1/tfile.c", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
  hid_t space_id = H5Screate_simple(1, dims, NULL);
  
  dset_id = H5Dcreate2(file_id, "dset_vlen", vlen_id, space_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
  H5Dread(dset_id, vlen_id, space_id, H5S_ALL, H5P_DEFAULT, (void*) rbuf);
  
  H5Treclaim(vlen_id, space_id, H5P_DEFAULT, (void*) rbuf);
}

Server response:

{..."value": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], ...}

Expected response:

{..."value": [[], [], [], [], [], [], [], [], [], []], ...}

Activity

jreadey

jreadey commented on Mar 4, 2024

@jreadey
Member

I think this is working as designed from the HSDS point of view..
Take a look at the h5pyd test here: https://github.com/HDFGroup/h5pyd/blob/master/test/hl/test_vlentype.py#L151.
The bytes returned from HSDS in this case (dset is {[], []}) are:

b'\x00\x00\x00\x00\x00\x00\x00\x00'

i.e. 2 elements returned. The 1st element is a zero-length array, the 2nd element also.
By contrast, if the vlen dataset had: {[0,]j, [0,]} the bytes returned would be:

b'\x02\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00'

mattjala

mattjala commented on Mar 4, 2024

@mattjala
ContributorAuthor

Looks like it's correct when HSDS returns the variable lengths types in binary instead of JSON. Changing the VOL to use binary instead of JSON should be faster for this as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    Uninitialized variable length sequences are returned as scalars instead of empty arrays · Issue #321 · HDFGroup/hsds