Problem with the submitting Spark job of Morpheus Examples jar #932
Description
I have the following on my cluster of machines:
OS: Centos 7
Spark: version 2.4.3
Scala: Version 2.12.8
I have run the (./gradlew allJar -x test) in the root directory of the Morpheus project with the following error:
Task 'allJar' not found in root project 'okapi'. Some candidates are: 'docJar', 'jar', 'jmhJar'.
So I had to run it with 'jar' option and got jar file for all modules inside the /build/libs directory.
I want to submit spark job of one of the examples inside the generated 'morpheus-examples/build/libs/morpheus-examples-0.4.3-SNAPSHOT.jar'
I have put the required jars in the Spark class path (spark/jars) directory, such as:
- morpheus-spark-cypher-0.4.3-SNAPSHOT.jar
- okapi-api-0.4.3-SNAPSHOT.jar
- okapi-relational-0.4.3-SNAPSHOT.jar
but when I submitted the job like this
spark/bin/spark-submit --class 'org.opencypher.morpheus.examples.CaseClassExample' --master local morpheus-examples/build/libs/morpheus-examples-0.4.3-SNAPSHOT.jar
I always face this error:
Exception in thread "main" java.lang.NoSuchMethodError: scala.App.$init$(Lscala/App;)V
at org.opencypher.morpheus.util.App.(App.scala:34)
......
This problem I think because the Scala Version and Spark Version comparability but I installed every thing like the versions inside the gradle parameters as stated in the beginning.
thanks in advance for your help !