34 Pull requests opened by 23 people

• [SPARK-50005][SQL] Enhance method verifyNotReadPath to identify subqueries hidden in the filter conditions.

  #48640 opened Oct 24, 2024

• [DRAFT] New collation precedence

  #48641 opened Oct 24, 2024

• [SPARK-50101][SQL] Fix collated behavior of StringToMap expression

  #48642 opened Oct 24, 2024

• [DOCS] Add note on compatibility with Confluent Schema Registry in Avro Data Source Guide

  #48645 opened Oct 24, 2024

• [SPARK-49563][SQL] Add SQL pipe syntax for the WINDOW operator

  #48649 opened Oct 25, 2024

• [SPARK-50112] Allowing the TransformWithState operator to use Avro encoding

  #48650 opened Oct 25, 2024

• [SPARK-50113][CONNECT][PYTHON][TESTS] Compatibility check should respect `ONLY_SUPPORTED_WITH_SPARK_CONNECT`

  #48651 opened Oct 25, 2024

• [SPARK-50110][SQL] Fix `from_csv`: parse fails when data contains spaces before and after

  #48653 opened Oct 25, 2024

• [SPARK-50016][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_2067`: `UNSUPPORTED_PARTITION_TRANSFORM`

  #48655 opened Oct 25, 2024

• [TYPING] Add type overloads for inplace dataframe operations

  #48662 opened Oct 25, 2024

• [SPARK-50175][SQL] Change collation precedence calculation

  #48663 opened Oct 25, 2024

• [SPARK-50130][SQL][PYTHON] Add DataFrame APIs for scalar and exists subqueries

  #48664 opened Oct 25, 2024

• [SPARK-50118][CONNET] Reset isolated state cache when tasks are running

  #48665 opened Oct 26, 2024

• [SPARK-50137][HIVE] Avoid fallback to Hive-incompatible ways when table creation fails by thrift exception

  #48668 opened Oct 26, 2024

• [SPARK-50140][BUILD] Upgrade `zstd-jni` to 1.5.6-7

  #48671 opened Oct 28, 2024

• [SPARK-50166][SQL][BUILD] Fix shade and relocation rule of `sql/core` module

  #48675 opened Oct 28, 2024

• [SPARK-50146][PYTHON][CONNECT] Configurable schema validation when creating DataFrames from Arrow tables

  #48677 opened Oct 28, 2024

• [WIP] Added Logistic Matrix Factorization(LMF) and Item2Vec models

  #48681 opened Oct 28, 2024

• [SPARK-50151][SS][RocksDB Hardening] - Fix ineffective file reuse bug in the new file management change

  #48685 opened Oct 28, 2024

• [SPARK-50152][SS] Support handleInitialState with state data source reader

  #48686 opened Oct 28, 2024

• [SPARK-50102][SQL][CONNECT] Add shims need for missing public SQL methods.

  #48687 opened Oct 29, 2024

• [SPARK-50153][SQL] Add `name` to `RuleExecutor` to make printing `QueryExecutionMetrics`'s logs clearer

  #48688 opened Oct 29, 2024

• [SPARK-50154][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_0043`: `INVALID_RESET_COMMAND_FORMAT`

  #48689 opened Oct 29, 2024

• [SPARK-50011][INFRA] Add a separate docker file for doc build

  #48690 opened Oct 29, 2024

• [SPARK-50156][SQL] Integrate `_LEGACY_ERROR_TEMP_2113` into `UNRECOGNIZED_STATISTIC`

  #48692 opened Oct 29, 2024

• [SPARK-50157][SQL] Using SQLConf provided by SparkSession first.

  #48693 opened Oct 29, 2024

• [SPARK-50160][SQL][KAFKA] KafkaWriteTask: allow customizing record timestamp

  #48695 opened Oct 29, 2024

• [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener

  #48697 opened Oct 29, 2024

• [SPARK-50162][SS] Add tests for loading snapshot with given version for transformWithState operator state and state data source reader

  #48710 opened Oct 30, 2024

• changing files

  #48716 opened Oct 31, 2024

• [SPARK-50184][SS][TESTS]Make test case 'SPARK-48997: maintenance threads with exceptions unload only themselves' use a self-cleaning `rootLocation`

  #48717 opened Oct 31, 2024

• [SPARK-50188][CONNECT] When the connect client starts, print the server's webUrl

  #48720 opened Oct 31, 2024

• [SPARK-50189][SQL] Upgrade ICU4J to `76.1`

  #48721 opened Oct 31, 2024

• [SPARK-50190][PYTHON] Remove direct dependency of Numpy from Histogram

  #48722 opened Oct 31, 2024

29 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-50017] Support Avro encoding for TransformWithState operator - ValueState, ListState

  #48401 commented on Oct 31, 2024 • 45 new comments

• [SPARK-49249][SPARK-49122] Artifact isolation in Spark Classic

  #48120 commented on Oct 31, 2024 • 21 new comments

• [SPARK-50092][SQL] Fix PostgreSQL connector behaviour for multidimensional arrays

  #48625 commented on Oct 31, 2024 • 19 new comments

• [SPARK-50087] Robust handling of boolean expressions in CASE WHEN for MsSqlServer and future connectors

  #48621 commented on Oct 29, 2024 • 15 new comments

• [SPARK-50032][SQL] Allow use of fully qualified collation name

  #48546 commented on Oct 30, 2024 • 15 new comments

• [SPARK-49899][PYTHON][SS] Support deleteIfExists for TransformWithStateInPandas

  #48373 commented on Oct 31, 2024 • 8 new comments

• [SPARK-50096][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_2150`: `TUPLE_SIZE_EXCEEDS_LIMIT`

  #48631 commented on Oct 30, 2024 • 5 new comments

• [SPARK-49676][SS][PYTHON] Add Support for Chaining of Operators in transformWithStateInPandas API

  #48124 commented on Oct 30, 2024 • 5 new comments

• [SPARK-37178][ML] Add Target Encoding to ml.feature

  #48347 commented on Oct 28, 2024 • 4 new comments

• [SPARK-49490][SQL] Add benchmarks for initCap

  #48501 commented on Oct 29, 2024 • 4 new comments

• [SPARK-50055][SQL] Add TryMakeInterval alternative

  #48580 commented on Oct 30, 2024 • 4 new comments

• [SPARK-49883][SS] State Store Checkpoint Structure V2 Integration with RocksDB and RocksDBFileManager

  #48355 commented on Oct 24, 2024 • 2 new comments

• [SPARK-49884][SS] State Store Checkpoint Structure V2 Backward compatibility Tests

  #48356 commented on Oct 24, 2024 • 2 new comments

• [SPARK-50083][SQL] Integrate `_LEGACY_ERROR_TEMP_1231` into `PARTITIONS_NOT_FOUND`

  #48614 commented on Oct 30, 2024 • 2 new comments

• [SPARK-46679][SQL] Fix for SparkUnsupportedOperationException Not found an encoder of the type T, when using Parameterized class

  #48304 commented on Oct 29, 2024 • 1 new comment

• [SPARK-50028][CONNECT] Replace global locks in Spark Connect server listener with fine-grained locks

  #48544 commented on Oct 29, 2024 • 1 new comment

• [SPARK-49601][SS][PYTHON] Support Initial State Handling for TransformWithStateInPandas

  #48005 commented on Oct 25, 2024 • 1 new comment

• [SPARK-48284][SQL] Fix UTF8String indexOf behaviour for empty string search

  #46581 commented on Oct 30, 2024 • 0 new comments

• [WIP] [SPARK-50127] Implement Avro encoding for MapState and PrefixKeyScanStateEncoder

  #48629 commented on Oct 25, 2024 • 0 new comments

• [SPARK-50091][SQL] Handle case of aggregates in left-hand operand of IN-subquery

  #48627 commented on Oct 29, 2024 • 0 new comments

• [SPARK-44884][Core] Create _SUCCESS marker file for spark write with …

  #47439 commented on Oct 31, 2024 • 0 new comments

• [SPARK-49482][SQL] Refactor V2 parquet datasource

  #47947 commented on Oct 30, 2024 • 0 new comments

• [SPARK-50033][SQL] Add a hint to logical.Aggregate() node

  #48523 commented on Oct 31, 2024 • 0 new comments

• [SPARK-37687][K8S] Removes the compile time dependency on the OkHttp http client

  #48446 commented on Oct 27, 2024 • 0 new comments

• [SPARK-49992][SQL] Session level collation should not impact DDL queries

  #48436 commented on Oct 29, 2024 • 0 new comments

• [SPARK-49789][SQL] Handling of generic parameter with bounds while creating encoders

  #48252 commented on Oct 30, 2024 • 0 new comments

• [SPARK-49730][SQL] classify syntax errors for pgsql, mysql, sqlserver and h2

  #48368 commented on Oct 31, 2024 • 0 new comments

• [SPARK-47193][SQL] Ensure SQL conf is propagated to executors when actions are called on RDD returned by `Dataset#rdd`

  #48325 commented on Oct 29, 2024 • 0 new comments

• [TEST-ONLY] Directly invoke functions in JVM side

  #48273 commented on Oct 28, 2024 • 0 new comments