Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

spark3 | scala 2.12 | jackson ScalaObjectMapper | NoSuchMethodError #283

Closed
kennydataml opened this issue Apr 28, 2020 · 13 comments
Closed

spark3 | scala 2.12 | jackson ScalaObjectMapper | NoSuchMethodError #283

kennydataml opened this issue Apr 28, 2020 · 13 comments
Assignees
@tr11
Labels
bug Something isn't working

Comments

@kennydataml
Copy link

Describe the bug

Loading dat file produces a NoSuchMethodError

To Reproduce

sc.addFile(copybook)
sc.addFile(dat)
df = spark.read.option.format("cobol")\
  .option("copybook", SparkFiles.get(copybook))\
  .load(SparkFiles.get(dat))

Expected behaviour

Should load successfully

Screenshots

20/04/28 19:09:53 INFO DefaultSource: Cobrix 'spark-cobol' build 2.0.7 (2020-04-14T12:00:03)
Traceback (most recent call last):
  File "/tmp/spark-0a0a39a6-a906-4b6c-af92-2ece5fcb2776/k8s_spark_cobol_adls.py", line 77, in <module>
    .load(SparkFiles.get(datafile1))
  File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 160, in load
  File "/opt/spark/python/lib/py4j-0.10.8.1-src.zip/py4j/java_gateway.py", line 1286, in __call__
  File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 98, in deco
  File "/opt/spark/python/lib/py4j-0.10.8.1-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o58.load.
: java.lang.NoSuchMethodError: com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.$init$(Lcom/fasterxml/jackson/module/scala/experimental/ScalaObjectMapper;)V
        at za.co.absa.cobrix.spark.cobol.source.parameters.CobolParametersParser$$anon$1.<init>(CobolParametersParser.scala:499)
        at za.co.absa.cobrix.spark.cobol.source.parameters.CobolParametersParser$.getOccursMappings(CobolParametersParser.scala:499)
        at za.co.absa.cobrix.spark.cobol.source.parameters.CobolParametersParser$.parse(CobolParametersParser.scala:215)
        at za.co.absa.cobrix.spark.cobol.source.DefaultSource.createRelation(DefaultSource.scala:56)
        at za.co.absa.cobrix.spark.cobol.source.DefaultSource.createRelation(DefaultSource.scala:48)
        at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:339)
        at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:240)
        at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:229)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:229)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:190)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)

20/04/28 19:09:53 INFO SparkContext: Invoking stop() from shutdown hook

Additional context

Using spark3 pyspark on Kubernetes. Require to use sc.addFile and SparkFiles.get due to limitations of kubernetes with ADLS2
@kennydataml kennydataml added the bug Something isn't working label Apr 28, 2020
@kennydataml
Copy link
Author

error is due to conflicting jackson versions with Spark3. Had to pull this Git repo and build the jar with sbt clean package. changed the dependencies.scala to match with spark 3 before building.

@yruslan
Copy link
Collaborator

yruslan commented Apr 30, 2020

Hi,

Thanks for the bug report!

I'm going to reopen it since spark-cobol should support Spark 3 out of the box.

@yruslan yruslan reopened this Apr 30, 2020
@tr11
Copy link
Collaborator

tr11 commented May 6, 2020

@kenny-bui-slalom, which version did you use to make it work?

@kennydataml
Copy link
Author

spark 3.0.0-preview2 currently uses jackson version 2.10.0

@tr11
Copy link
Collaborator

tr11 commented May 7, 2020

Yeah, tried with 3.0.0-preview2 and had similar issues.
Just changing the scala version to 2.12 fixed it for me even with the jackson version 2.10.3.

I'll add a PR to specifically list the dependency as an override.

@yruslan
Copy link
Collaborator

yruslan commented May 11, 2020

@tr11 , Thanks for the fix!

@yruslan
Copy link
Collaborator

yruslan commented May 11, 2020

@tr11 , after the explicit jackson dependency is added, spark-cobol is failing to build.

  java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.JsonMappingException.<init>(Ljava/io/Closeable;Ljava/lang/String;)V
  at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:61)
  at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
  at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:718)
  at za.co.absa.cobrix.spark.cobol.parameters.CobolParametersParser$.getOccursMappings(CobolParametersParser.scala:498)
  at za.co.absa.cobrix.spark.cobol.parameters.CobolParametersParser$.parse(CobolParametersParser.scala:213)

@yruslan
Copy link
Collaborator

yruslan commented May 12, 2020

I've played with shading our own version of Jackson in cobol-parser. It works well, but cobol-parser becomes a 3M artifact (from original 700K). And shading disrupts Maven's dependency management somewhat. So I think I'll just going to write a tiny JSON parser for the OCCURS parameters. It will be tied to the specific structure of the parameters JSON.

@tr11
Copy link
Collaborator

tr11 commented May 12, 2020

Makes sense. Since we already import ANTLR, we can use the grammar at https://github.com/antlr/grammars-v4/blob/master/json/JSON.g4. I can very easily build the visitor for that.

@yruslan
Copy link
Collaborator

yruslan commented May 12, 2020

Cool, thanks for the link!

@tr11 tr11 mentioned this issue May 13, 2020
Merged
@yruslan yruslan closed this as completed May 14, 2020
@prakki79
Copy link

I am getting this same error in Spark-3.0.0 with spark-cobol_2.12-2.0.7

java.lang.NoSuchMethodError: com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.$init$(Lcom/fasterxml/jackson/module/scala/experimental/ScalaObjectMapper;)V

I see this bug is fixed, is that I am missing anything?

@yruslan
Copy link
Collaborator

yruslan commented Aug 11, 2020

spark-cobol 2.0.7 does not have this fix. Try spark-cobol_2.12-2.1.0.

@prakki79
Copy link

@yruslan Thank you, it worked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tr11 tr11

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tr11 @yruslan @prakki79 @kennydataml