46 Pull requests merged by 29 people

• Minor: Document output schema of LogicalPlan::Aggregate and LogicalPl…

  #14047 merged Jan 9, 2025

• Unparsing optimized (> 2 inputs) unions

  #14031 merged Jan 9, 2025

• feat(optimizer): Enable filter pushdown on window functions

  #14026 merged Jan 8, 2025

• Fix: ensure that compression type is also taken into consideration during ListingTableConfig infer_options

  #14021 merged Jan 8, 2025

• Encapsulate fields of OrderingEquivalenceClass (make field non pub)

  #14037 merged Jan 8, 2025

• Update petgraph requirement from 0.6.2 to 0.7.1

  #14045 merged Jan 8, 2025

• Fix error on array_distinct when input is empty #13810

  #14034 merged Jan 8, 2025

• Encapsulate fields of EquivalenceGroup

  #14039 merged Jan 8, 2025

• Encapsulate fields of EquivalenceProperties

  #14040 merged Jan 8, 2025

• Use partial aggregation schema for spilling to avoid column mismatch in GroupedHashAggregateStream

  #13995 merged Jan 8, 2025

• Added references to IDE documentation for dev containers

  #14014 merged Jan 7, 2025

• Minor: Remove redundant implementation of StringArrayType

  #14023 merged Jan 7, 2025

• Minor: make nested functions public and implement Default trait

  #14030 merged Jan 6, 2025

• Minor: Improve zero partition check when inserting into MemTable

  #14024 merged Jan 6, 2025

• url dependancy update

  #14019 merged Jan 6, 2025

• Update sqllogictest requirement from =0.24.0 to =0.26.0

  #14017 merged Jan 6, 2025

• [Minor] refactor: make ArraySort public for broader access

  #14006 merged Jan 6, 2025

• Use workspace rust-version for all workspace crates

  #14009 merged Jan 6, 2025

• Chore: update wasm-supported crates, add tests

  #14005 merged Jan 6, 2025

• Minor: Rewrite LogicalPlan::max_rows for Join and Union, made it easier to understand

  #14012 merged Jan 6, 2025

• FIX: Out of bounds error when inserting into MemTable with zero partitions

  #14011 merged Jan 6, 2025

• fix: Preserve session id when using ctx.enable_url_table()

  #14004 merged Jan 6, 2025

• feat: support RightAnti for SortMergeJoin

  #13680 merged Jan 5, 2025

• fix: Avoid re-wrapping planning errors Err(DataFusionError::Plan) for use in plan_datafusion_err

  #14000 merged Jan 5, 2025

• feat(substrait): modular substrait producer

  #13931 merged Jan 5, 2025

• FIX : Incorrect NULL handling in BETWEEN expression

  #14007 merged Jan 5, 2025

• Consolidate csv_opener.rs and json_opener.rs into a single example (#…

  #13981 merged Jan 5, 2025

• feat(substrait): introduce consume_rel and consume_expression

  #13963 merged Jan 4, 2025

• Optimize CASE expression for "expr or expr" usage.

  #13953 merged Jan 4, 2025

• Update doc example to remove deprecated DELIMITER option for external tables

  #14002 merged Jan 4, 2025

• Improve deserialize_to_struct example

  #13958 merged Jan 4, 2025

• Minor: sort requirement check for Last function's merge_batch

  #13980 merged Jan 3, 2025

• Update release README for datafusion-cli publishing

  #13982 merged Jan 3, 2025

• Update substrait requirement from 0.50 to 0.51

  #13978 merged Jan 3, 2025

• doc-gen: migrate scalar functions (string) documentation 2/4

  #13925 merged Jan 3, 2025

• Change trigger, rename hash_collision.yml to extended.yml and add comments

  #13988 merged Jan 3, 2025

• Update itertools requirement from 0.13 to 0.14

  #13965 merged Jan 3, 2025

• Move hash collision test to run only when merging to main

  #13973 merged Jan 2, 2025

• Update rstest requirement from 0.23.0 to 0.24.0

  #13977 merged Jan 2, 2025

• Extract postgres container from sqllogictest, update datafusion-testing pin

  #13971 merged Jan 2, 2025

• Include license and notice files in more crates

  #13985 merged Jan 2, 2025

• docs: Add datafusion python 43.1.0 blog post to events page

  #13974 merged Jan 2, 2025

• Update release instructions for 44.0.0

  #13959 merged Jan 2, 2025

• fix(datafusion-functions-nested): arrow-distinct now work with null rows

  #13966 merged Jan 2, 2025

• Add swap_inputs to SMJ

  #13984 merged Jan 2, 2025

• Parallelize pruning utf8 fuzz test

  #13947 merged Jan 2, 2025

15 Pull requests opened by 13 people

• WIP: Proposed interface for physical plan invariant checking.

  #13986 opened Jan 2, 2025

• feat(datafusion-functions-aggregate): add support for lists and other nested types in `min` and `max`

  #13991 opened Jan 2, 2025

• Add H2O.ai Database-like Ops benchmark to dfbench (groupby support)

  #13996 opened Jan 3, 2025

• test: Add plan execution during tests for bounded source

  #14013 opened Jan 5, 2025

• Improve performance of `find_in_set` function

  #14020 opened Jan 6, 2025

• Improve perfomance of `reverse` function

  #14025 opened Jan 6, 2025

• fix: yield when the next file is ready to open to prevent CPU starvation

  #14028 opened Jan 6, 2025

• chore: deprecate `ValuesExec` in favour of `MemoryExec`

  #14032 opened Jan 7, 2025

• Simplify error handling in case.rs (#13990)

  #14033 opened Jan 7, 2025

• Refactor into `LexOrdering::collapse`, `LexRequirement::collapse` avoid clone

  #14038 opened Jan 7, 2025

• Fix bug in `nth_value` when `ignoreNulls` is true and no nulls in values

  #14042 opened Jan 8, 2025

• 'array_repeat' if the repeat count value is 0, return NULL instead of empty array

  #14046 opened Jan 8, 2025

• docs(ci): use up-to-date protoc with docs.rs

  #14048 opened Jan 8, 2025

• WIP: Reduce time spent normalizing

  #14049 opened Jan 8, 2025

• minor: re-export TypeSignatureClass from the datafusion-expr package

  #14051 opened Jan 9, 2025

18 Issues closed by 6 people

• Unparse `UNION` plan with multiple inputs to SQL text

  #13621 closed Jan 9, 2025

• Inference of ListingTableConfig does not work (anymore) for compressed json file

  #14016 closed Jan 8, 2025

• Schema error when spilling with multiple aggregations

  #13949 closed Jan 8, 2025

• Document how to use `.devcontainer`

  #13969 closed Jan 7, 2025

• Vulnerable version of `url` dependancy needs update

  #14018 closed Jan 6, 2025

• Assert for invariants in tests and debug builds

  #594 closed Jan 6, 2025

• Fix rust-version key in workspace Cargo.toml to inherit from workspace

  #9214 closed Jan 6, 2025

• Out of bounds error when inserting into MemTable with zero partitions

  #14010 closed Jan 6, 2025

• Preserve session context id when `ctx.enable_url_table()`

  #14003 closed Jan 6, 2025

• SortMergeJoin: Add RightAnti join support

  #13472 closed Jan 5, 2025

• datafusion-cli displays error prefix twice

  #13979 closed Jan 5, 2025

• [substrait] customizable producer

  #13901 closed Jan 5, 2025

• Incorrect `NULL` handling in `BETWEEN` expression

  #13976 closed Jan 5, 2025

• Consolidate csv_opener.rs and json_opener.rs into a single example

  #13955 closed Jan 5, 2025

• Optimize CASE expression for "expr or expr" usage

  #11638 closed Jan 4, 2025

• Refactor sqllogictest to extract postgres functionality into a separate file

  #13948 closed Jan 2, 2025

• `array_distinct` fails when getting null as list

  #13961 closed Jan 2, 2025

• Improve speed of `datafusion::fuzz fuzz_cases::pruning::test_fuzz_utf8` test

  #13946 closed Jan 2, 2025

17 Issues opened by 13 people

• Optimize filtered SortMergeJoin to avoid producing small/empty batches

  #14050 opened Jan 8, 2025

• Implement xxhash algorithms as part of the expression API

  #14044 opened Jan 8, 2025

• Optimize `date_part` Minute by avoiding unnecessary computation

  #14043 opened Jan 8, 2025

• `ignoreNulls` in `nth_value` doesn't work when no nulls are in input

  #14041 opened Jan 8, 2025

• Un-cancellable Query when hitting many large files.

  #14036 opened Jan 7, 2025

• Define extension API for user-defined invariants.

  #14029 opened Jan 6, 2025

• Support pruning on `starts_with`

  #14027 opened Jan 6, 2025

• Enhance msrv check to check all crates

  #14022 opened Jan 6, 2025

• Panic in a query with NATURAL JOIN (SQLancer)

  #14015 opened Jan 6, 2025

• Release DataFusion `45.0.0`

  #14008 opened Jan 4, 2025

• Doc attribution: make `user_doc` to work with predefined consts.

  #14001 opened Jan 3, 2025

• Support fast group accumulator for `first` and `last`

  #13998 opened Jan 3, 2025

• supports_filters_pushdown is invoked more than once on a single Custom Data Source

  #13994 opened Jan 3, 2025

• TPCDS 49, 70, 72, 86 Failed

  #13993 opened Jan 3, 2025

• Simplify error handling in case.rs

  #13990 opened Jan 2, 2025

• min and max should support lists as well

  #13987 opened Jan 2, 2025

• Update ClickBench benchmarks with DataFusion `44.0.0`

  #13983 opened Jan 2, 2025

37 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• doc-gen: migrate scalar functions (encoding & regex) documentation

  #13919 commented on Jan 8, 2025 • 6 new comments

• feat: add `AsyncCatalogProvider` helpers for asynchronous catalogs

  #13800 commented on Jan 7, 2025 • 4 new comments

• Feat/ffi enter tokio runtime

  #13937 commented on Jan 7, 2025 • 1 new comment

• verify TPC-DS results

  #13791 commented on Jan 3, 2025 • 1 new comment

• [substrait] Add support for ExtensionTable

  #13772 commented on Jan 9, 2025 • 1 new comment

• Deprecate ScalarUDFImpl::return_type

  #13717 commented on Jan 3, 2025 • 1 new comment

• [EPIC] Run full sqllogic / sqlite test suite against DataFusion

  #13811 commented on Jan 2, 2025 • 0 new comments

• Functionality of `array_repeat` udf

  #13872 commented on Jan 9, 2025 • 0 new comments

• [EPIC] A collection of items to improve developer / CI speed

  #13813 commented on Jan 9, 2025 • 0 new comments

• [DISCUSSION] Make it easier and faster to query remote files (S3, iceberg, etc)

  #13456 commented on Jan 9, 2025 • 0 new comments

• External sorting not working for (maybe only for string columns??)

  #12136 commented on Jan 9, 2025 • 0 new comments

• Move CPU Bound Tasks off Tokio Threadpool

  #13692 commented on Jan 9, 2025 • 0 new comments

• Fail on optimization cycles

  #11288 commented on Jan 5, 2025 • 0 new comments

• Do not push down filter through distinct on

  #12943 commented on Jan 3, 2025 • 0 new comments

• Add snapshot testing to CLI & set up AWS mock

  #13672 commented on Jan 9, 2025 • 0 new comments

• Support specific `GroupsAccumulator` for `median`

  #13681 commented on Jan 3, 2025 • 0 new comments

• chore: Migration Guide

  #13849 commented on Jan 6, 2025 • 0 new comments

• Custom scalar to sql overrides support for DuckDB Unparser dialect

  #13915 commented on Jan 9, 2025 • 0 new comments

• Find a way to communicate the ordering of a file back with the existi…

  #13933 commented on Jan 2, 2025 • 0 new comments

• sql result discrepency with sqlite, postgres and duckdb bug #2

  #13782 commented on Jan 4, 2025 • 0 new comments

• Add H2O.ai Database-like Ops benchmark to `dfbench`

  #7209 commented on Jan 5, 2025 • 0 new comments

• [EPIC] Improve examples to make them easier to navigate

  #11172 commented on Jan 5, 2025 • 0 new comments

• [Epic] Remove Sort Merge Join Experimental status

  #9846 commented on Jan 5, 2025 • 0 new comments

• metadata column support

  #13975 commented on Jan 6, 2025 • 0 new comments

• Implement SQLancer (a end-to-end SQL fuzz testing library)

  #11030 commented on Jan 6, 2025 • 0 new comments

• sql odd case of rounding compared to duckdb and postgresql

  #13781 commented on Jan 6, 2025 • 0 new comments

• Memory account not adding up in SortExec

  #10073 commented on Jan 7, 2025 • 0 new comments

• Automatically check "invariants"

  #13652 commented on Jan 7, 2025 • 0 new comments

• Introduce ProjectionMask To Allow Nested Projection Pushdown

  #2581 commented on Jan 7, 2025 • 0 new comments

• sql result discrepency with sqlite, postgres and duckdb

  #13780 commented on Jan 7, 2025 • 0 new comments

• multiply overflow in stats.rs

  #13775 commented on Jan 7, 2025 • 0 new comments

• FFI Execution Plans that spawn threads panic

  #13851 commented on Jan 7, 2025 • 0 new comments

• parquet RowGroup pruning for `Dictionary(Decimal)` type incorrect

  #13821 commented on Jan 8, 2025 • 0 new comments

• Add `union_extract` function

  #11081 commented on Jan 8, 2025 • 0 new comments

• datafusion-substrait API docs on docs.rs are broken

  #13853 commented on Jan 8, 2025 • 0 new comments

• Exponential planning time (100s of seconds) with `UNION` and `ORDER BY` queries

  #13748 commented on Jan 8, 2025 • 0 new comments

• Jan 1, 2025: This week(s) in DataFusion

  #13970 commented on Jan 8, 2025 • 0 new comments