Skip to content

spark/v3.2 code does not build against Scala 2.13 #3372

Closed
@wypoon

Description

@wypoon

It has been proposed to build Spark 3 support -- at least for Spark 3.2 -- against both Scala 2.12 and 2.13.
The code in spark/v3.2 currently does not compile against Scala 2.13.
Some of the sources of source incompatibility are:

  • scala.Predef.conforms (deprecated since 2.11) has been removed in 2.13.
  • scala.Seq[+A] in 2.13 is an alias for scala.collection.immutable.Seq[A] instead of scala.collection.Seq[A]. Some Spark APIs that Iceberg call take scala.Seq parameters, instead of parameters explicitly declared as scala.collection.Seq. (The problem is that the APIs are called from Java code, not Scala code. Java lacks the type alias, so the Java code currently use scala.collection.Seq, but that cannot be used in 2.13 where a scala.Seq is required.) Another manifestation of this issue is scala.collection.mutable.Buffer cannot be used where a scala.Seq is required.
  • scala.collection.JavaConversions has been removed in 2.13 (it is used in some tests).

Incidentally, scala.collection.JavaConverters is still in 2.13 but is deprecated and use of scala.jdk.CollectionConverters is recommended instead (but that is only in 2.13, not in 2.12).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      spark/v3.2 code does not build against Scala 2.13 · Issue #3372 · apache/iceberg