Arrow String View Type #10481
Merged
Arrow String View Type #10481
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@bkietz you might be interested to take a look |
|
@pdet apache/arrow#39852 just merged. With a development build of PyArrow, now you can create Arrow objects using the StringView type like this: import pyarrow as pa
pa.array(['foo', 'bar'], type=pa.string_view()) |
bkietz
reviewed
Feb 7, 2024
ianmcook
reviewed
Feb 8, 2024
bkietz
suggested changes
Feb 13, 2024
Tishj
reviewed
Feb 14, 2024
Tishj
reviewed
Feb 14, 2024
Tishj
reviewed
Feb 14, 2024
Tishj
reviewed
Feb 14, 2024
Tishj
reviewed
Feb 14, 2024
Tishj
reviewed
Feb 14, 2024
| D_ASSERT(!IsInline()); | ||
| return ref.offset; | ||
| } | ||
| static constexpr uint8_t max_inlined_bytes = 12; |
Tishj
suggested changes
Feb 14, 2024
|
One curiosity is that it seems that the last version of Polars can read string views produced by DuckDB (I'll add a proper test for that tomorrow), but I'm not yet sure what is happening internally in Polars when calling I've seen a blog post that they changed their internal representation to string views, but I'm not sure what operations they do yet, maybe would be interesting to do a quick benchmark on that. :-) |
jorisvandenbossche
added a commit
to apache/arrow
that referenced
this pull request
Feb 27, 2024
…iffer in the C Data Interface (#40156) ### Rationale for this change Attempt to draw more attention to the fact that the buffer listing / number of buffers differ between the main Format spec and the C Data Interface, for the Binary View layout. Triggered by feedback from implementing this in duckdb at duckdb/duckdb#10481 (comment) Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
zanmato1984
pushed a commit
to zanmato1984/arrow
that referenced
this pull request
Feb 28, 2024
…iffer in the C Data Interface (apache#40156) ### Rationale for this change Attempt to draw more attention to the fact that the buffer listing / number of buffers differ between the main Format spec and the C Data Interface, for the Binary View layout. Triggered by feedback from implementing this in duckdb at duckdb/duckdb#10481 (comment) Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
thisisnic
pushed a commit
to thisisnic/arrow
that referenced
this pull request
Mar 8, 2024
…iffer in the C Data Interface (apache#40156) ### Rationale for this change Attempt to draw more attention to the fact that the buffer listing / number of buffers differ between the main Format spec and the C Data Interface, for the Binary View layout. Triggered by feedback from implementing this in duckdb at duckdb/duckdb#10481 (comment) Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
2 tasks
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements the production and consumption of Arrow String Views.
By default, we will still produce regular strings unless
produce_arrow_string_viewis set.In that case, all strings produced will be string views.
If necessary, we could offer users the option to specify which columns they would prefer to be produced in one way or another. However, it currently seems to me that if a system implements operations on string views, it will most likely always prefer that.
Another point to consider is that the production of string views, especially for strings that are not inlined, can be performed over multiple data buffers. Perhaps, at some point, we might also want to allow users to specify this if it could be advantageous for them. Currently, we use only one data buffer for all strings.
Because PyArrow is still in the process of implementing support for creating String Views in their API (see apache/arrow#39633), we only test over string views that we produce ourselves. Since it is much more relevant to verify our compliance with the Arrow Spec using their own strings, maybe we should wait to consider merging this PR until we can properly test our interoperability with PyArrow String Views.
cc @ianmcook