Skip to content

Full support for creating and writing to data chunks #85

Closed
@jraymakers

Description

The current (high-level) API provides very limited (barely usable) support for creating data chunks from scratc, and no support for writing data values to the chunk. This support is highly desirable, because it would allow efficiently loading arbitrary data using the appender (using DuckDBAppender.appendDataChunk).

My current plan is to extend the current DuckDBVector interfaces to allow writes, and ensure these writes make their way to the underlying vectors in DuckDB. There are some interesting challenges and tradeoffs in making these writes efficient; ideally flushing to underlying vectors could be batched, but this might mean the JS data and the DuckDB data could be out of sync for some time, which could lead to surprising behavior.

I also plan to make it easier to create data chunks from scratch, by allowing their types to be specified using DuckDBType helpers instead of low-level duckdb.LogicalType handles.

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions