Full support for creating and writing to data chunks #85
Description
The current (high-level) API provides very limited (barely usable) support for creating data chunks from scratc, and no support for writing data values to the chunk. This support is highly desirable, because it would allow efficiently loading arbitrary data using the appender (using DuckDBAppender.appendDataChunk
).
My current plan is to extend the current DuckDBVector
interfaces to allow writes, and ensure these writes make their way to the underlying vectors in DuckDB. There are some interesting challenges and tradeoffs in making these writes efficient; ideally flushing to underlying vectors could be batched, but this might mean the JS data and the DuckDB data could be out of sync for some time, which could lead to surprising behavior.
I also plan to make it easier to create data chunks from scratch, by allowing their types to be specified using DuckDBType
helpers instead of low-level duckdb.LogicalType
handles.