Skip to content

Problem with the submitting Spark job of Morpheus Examples jar #932

Open
@MohamedRagabAnas

Description

I have the following on my cluster of machines:
OS: Centos 7
Spark: version 2.4.3
Scala: Version 2.12.8

I have run the (./gradlew allJar -x test) in the root directory of the Morpheus project with the following error:
Task 'allJar' not found in root project 'okapi'. Some candidates are: 'docJar', 'jar', 'jmhJar'.

So I had to run it with 'jar' option and got jar file for all modules inside the /build/libs directory.

I want to submit spark job of one of the examples inside the generated 'morpheus-examples/build/libs/morpheus-examples-0.4.3-SNAPSHOT.jar'

I have put the required jars in the Spark class path (spark/jars) directory, such as:

  • morpheus-spark-cypher-0.4.3-SNAPSHOT.jar
  • okapi-api-0.4.3-SNAPSHOT.jar
  • okapi-relational-0.4.3-SNAPSHOT.jar

but when I submitted the job like this

spark/bin/spark-submit --class 'org.opencypher.morpheus.examples.CaseClassExample' --master local morpheus-examples/build/libs/morpheus-examples-0.4.3-SNAPSHOT.jar

I always face this error:

Exception in thread "main" java.lang.NoSuchMethodError: scala.App.$init$(Lscala/App;)V
at org.opencypher.morpheus.util.App.(App.scala:34)
......

This problem I think because the Scala Version and Spark Version comparability but I installed every thing like the versions inside the gradle parameters as stated in the beginning.

thanks in advance for your help !

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions