Implement to_arrow in C++ for JS/Python
#850
Conversation
texodus
force-pushed
the
to-arrow
branch
2 times, most recently
from
36aff39 to
1125ade
Compare
…lta return arrow by default add dictionary array generation, tests for start/end col/row
|
|
There was a problem hiding this comment.
Thanks for the PR! This is a really cool feature, great performance improvement, better portability, smaller asset size! I've done a review of this offline and added 2 commits:
- Remove
@arrow/es5-esmnow that this is entirely implemented in C++, saving ~200k in addition to the runtime performance improvements. - Adds
PSP_CHECK_ARROW_STATUSto wrapStatus::invalid()in aPSP_COMPLAIN_AND_ABORTto squash the build warnings and provide better error messages.
|
Also fixed longstanding test flap, which was due to incorrect Javascript |
|
This looks great! Can't wait to try it in JS. Are all the arrow date types properly handled in this update? |
|
Yes - date32 and date64 are handled as date types, while timestamp is handled as a datetime. |
|
sweet! will wait for the next rc to bundle it in my data ext. that uses this lib. would be a nice way to close this year & thank you guys for this awesome data analytics lib & contribs! |
This PR uses the C++ Apache Arrow library to serialize Perspective views into the Arrow format. It includes:
0.15.1, which requires a rebuild of Docker images.update()) using C++ to_arrow, and implemented in Python.WebsocketClosedErrorbreaking in streaming example - clients closing websocket should no longer break lingering callbacks.PerspectiveManagerto use_PerspectiveCallbackCacheinstead of a dictionary.remote.pyexample in Python, which usesto_arrowandrow_deltato stream changed rows to clients every millisecond.arrow.cppintoarrow_loaderandarrow_writerfor cleanliness.scalar_vec_to_valandscalar_vec_to_stringmethods in C++ - usescalar_to_valinstead.