Unified data access interface #352
Conversation
sc1f
commented
Dec 26, 2018
sc1f
commented
- Move data ingestion and metadata inference into C++
- Create a unified API for accessing data regardless of underlying format (columnar vs. row-based)
- Ported inference functions return native C++ types
| new_pdata.push(Object.assign({}, pdata, chunk)); | ||
| } | ||
| pdata = new_pdata; | ||
| } else { |
There was a problem hiding this comment.
This is a bit tricky - unsure if we will need this again in the future. It was oroginalyl added to prevent vecFromJSArray() calls in C++ from blowing up the wasm heap, but data accessor no longer calls this per column.
There was a problem hiding this comment.
We do call vecFromJSArray() for each load, update, and delete operation instead of per-column, so there is still the danger of blowing up the WASM heap even with this change.
I was working on a version of the accessor that did not rely on the JS side passing in column names and data types, as we already have that information stored on the C++ side (thus avoiding the vecFromJSArray() casting), but I ran into some non-deterministic memory access out of bounds issues.
Theoretically it's possible to access the already-created gnode in make_table, grab the schema from there, and then grab the names and types. Actually doing it, however, creates a bunch of errors that I haven't been able to nail down yet.
|
Thanks! This is a great addition, very nice work! |
