Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

1 test error after migrating to Spark 4.0.0-preview2 #40

Open
tribbloid opened this issue Dec 19, 2024 · 4 comments
Open

1 test error after migrating to Spark 4.0.0-preview2 #40

tribbloid opened this issue Dec 19, 2024 · 4 comments

Comments

@tribbloid
Copy link

on test "join-right-on":

org.scalatest.exceptions.TestFailedException: List((None, None, 3, "c"), (Some(2), Some(20), 2, "b")) did not equal List((Some(2), Some(20), 2, "b"), (None, None, 3, "c"))
	at app//org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:397)
	at app//org.scalatest.matchers.should.Matchers.shouldEqual(Matchers.scala:6893)
	at app//org.scalatest.matchers.should.Matchers.shouldEqual$(Matchers.scala:1808)
	at app//org.scalatest.matchers.should.Matchers$.shouldEqual(Matchers.scala:7725)
	at app//org.virtuslab.iskra.test.SparkUnitTest.shouldEqual(SparkUnitTest.scala:20)
	at app//org.virtuslab.iskra.test.JoinTest.testFun$proxy3$1(JoinTest.scala:87)
	at app//org.virtuslab.iskra.test.JoinTest.$init$$$anonfun$3(JoinTest.scala:67)
	at app//org.scalatest.Transformer.apply$$anonfun$1(Transformer.scala:22)
	at app//org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at app//org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:31)
	at app//org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at app//org.scalatest.Transformer.apply(Transformer.scala:22)
	at app//org.scalatest.Transformer.apply(Transformer.scala:21)
	at app//org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:206)
	at app//org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
	at app//org.scalatest.TestSuite.withFixture$(TestSuite.scala:138)
	at app//org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1563)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:212)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.runTest$$anonfun$1(AnyFunSuiteLike.scala:216)
	at app//org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:216)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:47)
	at app//org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1563)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.runTests$$anonfun$1(AnyFunSuiteLike.scala:249)
	at app//org.scalatest.SuperEngine.traverseSubNodes$1$$anonfun$1(Engine.scala:413)
	at app//scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at app//scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at app//scala.collection.immutable.List.foreach(List.scala:334)
	at app//org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:429)
	at app//org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at app//org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:249)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:47)
	at app//org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1563)
	at app//org.scalatest.Suite.run(Suite.scala:1114)
	at app//org.scalatest.Suite.run$(Suite.scala:564)
	at app//org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1563)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.run$$anonfun$1(AnyFunSuiteLike.scala:253)
	at app//org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:253)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:47)
	at app//org.virtuslab.iskra.test.SparkUnitTest.org$scalatest$BeforeAndAfterAll$$super$run(SparkUnitTest.scala:7)
	at app//org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at app//org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:217)
	at app//org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:135)
	at app//org.virtuslab.iskra.test.SparkUnitTest.run(SparkUnitTest.scala:7)
	at app//co.helmethair.scalatest.runtime.Executor.runScalatests(Executor.java:130)
	at app//co.helmethair.scalatest.runtime.Executor.executeSuite(Executor.java:86)
	at app//co.helmethair.scalatest.runtime.Executor.executeTest(Executor.java:53)
	at app//co.helmethair.scalatest.runtime.Executor.lambda$executeSuite$2(Executor.java:82)
	at java.base@17.0.13/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base@17.0.13/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)
	at java.base@17.0.13/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510)
	at java.base@17.0.13/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base@17.0.13/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base@17.0.13/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base@17.0.13/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base@17.0.13/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at app//co.helmethair.scalatest.runtime.Executor.executeSuite(Executor.java:82)
	at app//co.helmethair.scalatest.runtime.Executor.executeTest(Executor.java:60)
	at app//co.helmethair.scalatest.ScalatestEngine.execute(ScalatestEngine.java:55)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at app//org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at app//org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at app//org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:124)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:99)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:94)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:63)
	at java.base@17.0.13/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base@17.0.13/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base@17.0.13/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base@17.0.13/java.lang.reflect.Method.invoke(Method.java:569)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:92)
	at jdk.proxy1/jdk.proxy1.$Proxy4.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:200)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:132)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:103)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:63)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:121)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

haven't figure out the cause yet
@prolativ
Copy link
Collaborator

Thanks for reporting.
Actually an attempt to bump spark to 4.0.0-preview2 caused a few more issues for me:

  • I had to add //> using jvm 17 to project.scala because that's the minimal java version required by spark 4 (it was 11 by default on my machine)
  • join-right-on failed because the rows of the data frame were reordered; that's quite surprising but I guess spark doesn't give any guarantees on ordering unless one explicitly uses .orderBy on a data frame (which is not yet supported in iskra, waiting for a major refactor that I'm slowly working on right now). The fix might be to change tests to compare sets of values rather than lists
  • The test in Workers.scala failed to because of the same reason, combined with some differences in how data frames are printed. I would need to rethink how self-standing examples could be reused as end-to-end tests (that might require parsing the console output or changing the approach to such tests entirely)

@tribbloid
Copy link
Author

tribbloid commented Dec 20, 2024 •
edited
Loading

I see, I used a copy-pasted gradle build (unfortunately as a submodule), should I publish it as an independent fork?

BTW, the following test JVM argument will be required for Spark 4 itself and all dependents:

"-XX:+IgnoreUnrecognizedVMOptions",
        "--add-opens=java.base/java.lang=ALL-UNNAMED",
        "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED",
        "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED",
        "--add-opens=java.base/java.io=ALL-UNNAMED",
        "--add-opens=java.base/java.net=ALL-UNNAMED",
        "--add-opens=java.base/java.nio=ALL-UNNAMED",
        "--add-opens=java.base/java.util=ALL-UNNAMED",
        "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED",
        "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED",
        "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED",
        "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
        "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED",
        "--add-opens=java.base/sun.security.action=ALL-UNNAMED",
        "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED",
        "-Djdk.reflect.useDirectMethodHandle=false",
        "-Dio.netty.tryReflectionSetAccessible=true"

@tribbloid
Copy link
Author

sorry @prolativ do you encounter some difficulty in reproducing this testing environment in scala-cli? would you like me to publish this project under sbt or gradle?

@prolativ
Copy link
Collaborator

I did reproduce the issue directly in this repo, which is built using scala-cli.
I guess you meant spark 4, not scala 4 😆
Currently I'm struggling to find time to implement the redesign of iskra'a API that's been boiling in my head for some time already, but that's not that easy 😅 So making it run on spark 4, which doesn't have a stable release yet, is in no measure my focus at the moment, even though this doesn't seem to require that much work. The build too used for publishing shouldn't matter at all from the library's users

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tribbloid @prolativ