Skip to content

Commit

Permalink
[SPARK-13529][BUILD] Move network/* modules into common/network-*
Browse files Browse the repository at this point in the history
## What changes were proposed in this pull request?
As the title says, this moves the three modules currently in network/ into common/network-*. This removes one top level, non-user-facing folder.

## How was this patch tested?
Compilation and existing tests. We should run both SBT and Maven.

Author: Reynold Xin <rxin@databricks.com>

Closes apache#11409 from rxin/SPARK-13529.
  • Loading branch information
rxin committed Feb 29, 2016
1 parent cca79fa commit 9e01dcc
Show file tree
Hide file tree
Showing 115 changed files with 9 additions and 8 deletions.
File renamed without changes.
File renamed without changes.
File renamed without changes.
4 changes: 2 additions & 2 deletions dev/sparktestsupport/modules.py
Original file line number Diff line number Diff line change
Expand Up @@ -473,11 +473,11 @@ def __hash__(self):
dependencies=[],
source_file_regexes=[
"yarn/",
"network/yarn/",
"common/network-yarn/",
],
sbt_test_goals=[
"yarn/test",
"network-yarn/test",
"common/network-yarn/test",
],
test_tags=[
"org.apache.spark.tags.ExtendedYarnTest"
Expand Down
2 changes: 1 addition & 1 deletion docs/job-scheduling.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ In YARN mode, start the shuffle service on each `NodeManager` as follows:
1. Build Spark with the [YARN profile](building-spark.html). Skip this step if you are using a
pre-packaged distribution.
2. Locate the `spark-<version>-yarn-shuffle.jar`. This should be under
`$SPARK_HOME/network/yarn/target/scala-<version>` if you are building Spark yourself, and under
`$SPARK_HOME/common/network-yarn/target/scala-<version>` if you are building Spark yourself, and under
`lib` if you are using a distribution.
2. Add this jar to the classpath of all `NodeManager`s in your cluster.
3. In the `yarn-site.xml` on each node, add `spark_shuffle` to `yarn.nodemanager.aux-services`,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,8 @@ List<String> buildClassPath(String appClassPath) throws IOException {
String scala = getScalaVersion();
List<String> projects = Arrays.asList("core", "repl", "mllib", "graphx",
"streaming", "tools", "sql/catalyst", "sql/core", "sql/hive", "sql/hive-thriftserver",
"yarn", "launcher", "network/common", "network/shuffle", "network/yarn");
"yarn", "launcher",
"common/network-common", "common/network-shuffle", "common/network-yarn");
if (prependClasses) {
if (!isTesting) {
System.err.println(
Expand Down
2 changes: 1 addition & 1 deletion make-distribution.sh
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ cp "$SPARK_HOME"/assembly/target/scala*/*assembly*hadoop*.jar "$DISTDIR/lib/"
cp "$SPARK_HOME"/examples/target/scala*/spark-examples*.jar "$DISTDIR/lib/"
# This will fail if the -Pyarn profile is not provided
# In this case, silence the error and ignore the return code of this command
cp "$SPARK_HOME"/network/yarn/target/scala*/spark-*-yarn-shuffle.jar "$DISTDIR/lib/" &> /dev/null || :
cp "$SPARK_HOME"/common/network-yarn/target/scala*/spark-*-yarn-shuffle.jar "$DISTDIR/lib/" &> /dev/null || :

# Copy example sources (needed for python and SQL)
mkdir -p "$DISTDIR/examples/src/main"
Expand Down
6 changes: 3 additions & 3 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -87,13 +87,13 @@

<modules>
<module>common/sketch</module>
<module>common/network-common</module>
<module>common/network-shuffle</module>
<module>tags</module>
<module>core</module>
<module>graphx</module>
<module>mllib</module>
<module>tools</module>
<module>network/common</module>
<module>network/shuffle</module>
<module>streaming</module>
<module>sql/catalyst</module>
<module>sql/core</module>
Expand Down Expand Up @@ -2442,7 +2442,7 @@
<id>yarn</id>
<modules>
<module>yarn</module>
<module>network/yarn</module>
<module>common/network-yarn</module>
</modules>
</profile>

Expand Down

0 comments on commit 9e01dcc

Please sign in to comment.