Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Optional In-Built ThreadLocal Arrays for Metadata #59

Merged
merged 4 commits into from
Jan 2, 2020

Conversation

jamesmcclain
Copy link
Member

@jamesmcclain jamesmcclain commented Jul 24, 2019

Adds

  • get_metadata_domain_list(long token, int dataset, int attempts, int band_number, String[][] domain_list)
  • int get_metadata(long token, int dataset, int attempts, int band_number, String domain, String[][] list)
  • int get_metadata_item(long token, int dataset, int attempts, int band_number, String key, String domain, String[] value)

A string array of the form String[1][0] should be passed as the last argument to the first one. When the call returns, the array will be of the form String[1][n] with the n domain list items stored within. Similarly for the second one. The third one should be passed a String array of the form String[1], upon return it will contain the metadata item.

Depends on #58
Closes #54

@jamesmcclain jamesmcclain changed the title Add Metadata Tight-Loop Experiment Add Optional In-Built ThreadLocal Arrays for Metadata Jul 27, 2019
@jamesmcclain
Copy link
Member Author

jamesmcclain commented Jul 27, 2019

  noop get_metadata_domain_list get_domain_list reallocate 16x1024 get_domain_list reallocate 1024x1024 get_domain_list preallocate GDALGetMetadataDomainList
Native -O0 -ggdb3 1261.098 1796.401       242.306
Native -O4 512.643 797.272       246.73
Java -O0 -ggdb3 2287 3761 5653 77936 3466  
Java -O4 779 1299 3195 75866 1719  

The table contains timings in milliseconds for doing 2**28 get_metadata_domain_list calls. The first column contains timings for performing noop operations to establish a baseline. The second column contains timings for doing the get_metadata_domain_list call (in the native context) and using the Java get_metadata_domain_list with built-in thread local storage (the latter mechanism is introduced in this pull request). The next three columns show results from using the Java get_metadata_domain_list method without built-in thread local storage (the previously-existing mechanism) in various ways. The last column gives timings obtained from using GDALGetMetadataDomainList directly. The bindings themselves were compiled at -O0 and also at -O4 for these tests.

These numbers do not really show it, but I would still suspect that large-enough, preallocated, thread-local byte arrays would be the best for lowest latency.

@metasim
@pomadchin

@metasim
Copy link

metasim commented Jul 29, 2019

@jamesmcclain My 2¢
If I understand what's happening here, this approach concerns me as it uses effectively uses a string array as a mutable call-by-reference variable in a way that's not typically seen in JVM code. It makes the end user think about memory management through the use of arrays in a way that feels unconventional.

What I'd prefer to see is these methods just return the data that was requested, without any pre-allocated array being passed in... IOW:

String[] get_metadata_domain_list(long token, int dataset, int attempts, int band_number)
// Not quire abou this one... really want a Map<String, String>...
String[][] get_metadata(long token, int dataset, int attempts, int band_number, String domain)
String get_metadata_item(long token, int dataset, int attempts, int band_number, String key, String domain)

@jamesmcclain
Copy link
Member Author

I would like to go ahead a merge this since it does address the metadata performance concerns. We can revisit the issue if the API is not what is wanted (please keep in mind that this library is intended to be a down-and-dirty interface, not an API per se [GDALRasterSource is the API]).

@jamesmcclain jamesmcclain merged commit 1d7ddea into geotrellis:master Jan 2, 2020
@jamesmcclain jamesmcclain deleted the metadata branch January 2, 2020 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Q: How to determine buffer sizes to use for metadata query?
2 participants