Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import implementation from OSMesa project #60

Merged
merged 41 commits into from
Mar 22, 2019
Merged

Import implementation from OSMesa project #60

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
b927f93
Clean out project; Remove obsolete code and related documentation
jpolchlo Mar 1, 2019
72470ae
Update README
jpolchlo Mar 1, 2019
f4d6075
Freshen SBT project configs
jpolchlo Mar 1, 2019
a0ccffc
Remove chatty scalac warnings
jpolchlo Mar 1, 2019
acac58c
Import code from external project
jpolchlo Mar 1, 2019
fa2d035
Include SPI registry
jpolchlo Mar 1, 2019
98ce096
Remove extraneous version identifier
jpolchlo Mar 1, 2019
b7092c1
Remove outdated console setup
jpolchlo Mar 1, 2019
82ee939
Import test suite
jpolchlo Mar 1, 2019
f12c970
Bump version number
jpolchlo Mar 4, 2019
115ccb8
Bring in missing SAX parser from OSMesa (oops)
jpolchlo Mar 5, 2019
4f17b6b
Small config fixes
jpolchlo Mar 5, 2019
d47c826
Move files to right place in tree
jpolchlo Mar 5, 2019
da8643c
Make constructGeometries work for inputs from Change streams as well …
jpolchlo Mar 5, 2019
9b78222
Update README
jpolchlo Mar 5, 2019
b01b60f
Remove raster package
jpolchlo Mar 6, 2019
378b5be
Move Geocode out of ProcessOSM
jpolchlo Mar 13, 2019
c4f14f4
Address PR comments (remove caching facilities, improve conversion fu…
jpolchlo Mar 14, 2019
4bcf487
Fix test
jpolchlo Mar 14, 2019
b1569e8
Make tests pass
jpolchlo Mar 14, 2019
ca6a156
Adjust version number [skipci]
jpolchlo Mar 14, 2019
cfd541a
Unused
mojodna Mar 14, 2019
9558453
Ignore benchmark artifacts
mojodna Mar 14, 2019
0ad9c7a
Remove benchmarks referencing unused code
mojodna Mar 14, 2019
ec0e127
Remove unused imports
mojodna Mar 14, 2019
57ce394
geotrellis.spark.io.hadoop._ _is_ required
mojodna Mar 14, 2019
d161297
Upgrade dependencies
mojodna Mar 14, 2019
a51bdae
Style tweaks
mojodna Mar 14, 2019
b118b33
Additional docs for ProcessOSM entrypoints
mojodna Mar 14, 2019
1260e13
Make sure we are using compressed internal representations for member…
jpolchlo Mar 15, 2019
c2e1571
Reorganize library components; simplify and rename main user-facing i…
jpolchlo Mar 18, 2019
b2a0bbd
Update README to use new struture
jpolchlo Mar 18, 2019
23c6b83
Improve description of `toGeometry`'s output [skip ci]
jpolchlo Mar 18, 2019
e68d7e8
Fix slight README issue
jpolchlo Mar 18, 2019
1b7c43f
Fix tests for new structure
jpolchlo Mar 18, 2019
78820aa
Update docs to include section on internal package and compressed mem…
jpolchlo Mar 19, 2019
51f0827
Make tests work
jpolchlo Mar 19, 2019
bf2b5fc
Clean up
jpolchlo Mar 19, 2019
bb64732
Upgrade to `org.locationtech` organization for JTS (bumps GT and Geom…
jpolchlo Mar 19, 2019
230627d
Adjust copy method definitions for CoordinateSequence subclasses
jpolchlo Mar 22, 2019
12e2da2
Fix tests
jpolchlo Mar 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Freshen SBT project configs
  • Loading branch information
@jpolchlo
jpolchlo committed Mar 22, 2019
commit f4d6075b4b508a35e56494c77ff7ced4239176f8
254 changes: 151 additions & 103 deletions build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,53 @@
name := "vectorpipe"

description := "Convert Vector data to VectorTiles with GeoTrellis."
description := "Import OSM data and output to VectorTiles with GeoTrellis."

organization := "com.azavea"
import Dependencies._

organizationName := "Azavea"
lazy val commonSettings = Seq(
organization := "com.azavea",

scalaVersion in ThisBuild := "2.11.12"
organizationName := "Azavea",

val common = Seq(
version := Version.vectorpipe,

cancelable in Global := true,

scalaVersion in ThisBuild := Version.scala,

scalacOptions := Seq(
"-deprecation",
"-unchecked",
"-feature",
"-language:implicitConversions",
"-language:reflectiveCalls",
"-language:higherKinds",
"-language:postfixOps",
"-language:existentials",
"-language:experimental.macros",
"-feature",
"-Ywarn-value-discard",
"-Ywarn-dead-code",
"-Ywarn-numeric-widen",
"-Ypartial-unification",
"-Ypatmat-exhaust-depth", "100"
),

scalacOptions in (Compile, doc) += "-groups",

/* For Monocle's Lens auto-generation */
addCompilerPlugin("org.scalamacros" %% "paradise" % "2.1.0" cross CrossVersion.full),

resolvers ++= Seq(
"locationtech-releases" at "https://repo.locationtech.org/content/groups/releases",
Resolver.bintrayRepo("azavea", "maven")
Resolver.bintrayRepo("lonelyplanet", "maven"),
Resolver.bintrayRepo("kwark", "maven"), // Required for Slick 3.1.1.2, see https://github.com/azavea/raster-foundry/pull/1576
Resolver.bintrayRepo("bkirwi", "maven"), // Required for `decline` dependency
"locationtech-releases" at "https://repo.locationtech.org/content/repositories/releases/",
"locationtech-snapshots" at "https://repo.locationtech.org/content/repositories/snapshots/",
"geosolutions" at "http://maven.geo-solutions.it/",
"boundless" at "https://repo.boundlessgeo.com/main/",
"osgeo" at "http://download.osgeo.org/webdav/geotools/",
"apache.commons.io" at "https://mvnrepository.com/artifact/commons-io/commons-io"
),

initialCommands in console :=
Expand All @@ -37,53 +72,32 @@ val common = Seq(
.withJTS
""".stripMargin,

scalacOptions := Seq(
"-deprecation",
"-Ypartial-unification",
"-Ywarn-value-discard",
"-Ywarn-dead-code",
"-Ywarn-numeric-widen",
"-language:implicitConversions",
"-language:reflectiveCalls"
),

resolvers ++= Seq(
Resolver.bintrayRepo("lonelyplanet", "maven"),
Resolver.bintrayRepo("bkirwi", "maven"), // Required for `decline` dependency
"locationtech-releases" at "https://repo.locationtech.org/content/repositories/releases/",
"locationtech-snapshots" at "https://repo.locationtech.org/content/repositories/snapshots/",
"geosolutions" at "http://maven.geo-solutions.it/",
"boundless" at "https://repo.boundlessgeo.com/main/",
"osgeo" at "http://download.osgeo.org/webdav/geotools/",
"apache.commons.io" at "https://mvnrepository.com/artifact/commons-io/commons-io"
),

scalacOptions in (Compile, doc) += "-groups",

/* For Monocle's Lens auto-generation */
addCompilerPlugin("org.scalamacros" %% "paradise" % "2.1.0" cross CrossVersion.full),

libraryDependencies ++= Seq(
"org.locationtech.geotrellis" %% "geotrellis-vectortile" % Version.geotrellis exclude("com.google.protobuf", "protobuf-java"),
"org.locationtech.geotrellis" %% "geotrellis-s3" % Version.geotrellis exclude("com.google.protobuf", "protobuf-java"),
"org.locationtech.geomesa" %% "geomesa-spark-jts" % Version.geomesa,
"org.apache.spark" %% "spark-hive" % Version.spark % "provided",
"org.spire-math" %% "spire" % Version.spire,
"org.typelevel" %% "cats-core" % Version.cats,
"com.monovore" %% "decline" % Version.decline,
"org.scalatest" %% "scalatest" % Version.scalaTest % "test",
"com.github.julien-truffaut" %% "monocle-core" % Version.monocle,
"com.github.julien-truffaut" %% "monocle-macro" % Version.monocle,
"com.github.seratch" %% "awscala" % "0.6.1",
"org.apache.commons" % "commons-compress" % "1.16.1",
"com.google.protobuf" % "protobuf-java" % "2.5.0",
"io.dylemma" %% "xml-spac" % "0.3",
"javax.media" % "jai_core" % "1.1.3" % "test" from "http://download.osgeo.org/webdav/geotools/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar",
),

parallelExecution in Test := false,
fork in Test := false,
testOptions in Test += Tests.Argument("-oDF")
updateOptions := updateOptions.value.withGigahorse(false),

shellPrompt := { s => Project.extract(s).currentProject.id + " > " },

assemblyMergeStrategy in assembly := {
case "reference.conf" | "application.conf" => MergeStrategy.concat
case PathList("META-INF", xs@_*) =>
xs match {
case ("MANIFEST.MF" :: Nil) => MergeStrategy.discard
// Concatenate everything in the services directory to keep GeoTools happy.
case ("services" :: _ :: Nil) =>
MergeStrategy.concat
// Concatenate these to keep JAI happy.
case ("javax.media.jai.registryFile.jai" :: Nil) | ("registryFile.jai" :: Nil) | ("registryFile.jaiext" :: Nil) =>
MergeStrategy.concat
case (name :: Nil) => {
// Must exclude META-INF/*.([RD]SA|SF) to avoid "Invalid signature file digest for Manifest main attributes" exception.
if (name.endsWith(".RSA") || name.endsWith(".DSA") || name.endsWith(".SF"))
MergeStrategy.discard
else
MergeStrategy.first
}
case _ => MergeStrategy.first
}
case _ => MergeStrategy.first
}
)

val release = Seq(
Expand All @@ -97,57 +111,91 @@ val release = Seq(
homepage := Some(url("https://geotrellis.github.io/vectorpipe/"))
)

lazy val lib = project.in(file(".")).settings(common, release)

assemblyShadeRules in assembly := {
val shadePackage = "com.azavea.shaded.demo"
Seq(
ShadeRule.rename("com.google.common.**" -> s"$shadePackage.google.common.@1")
.inLibrary("com.azavea.geotrellis" %% "geotrellis-cassandra" % Version.geotrellis).inAll,
ShadeRule.rename("io.netty.**" -> s"$shadePackage.io.netty.@1")
.inLibrary("com.azavea.geotrellis" %% "geotrellis-hbase" % Version.geotrellis).inAll,
ShadeRule.rename("com.fasterxml.jackson.**" -> s"$shadePackage.com.fasterxml.jackson.@1")
.inLibrary("com.networknt" % "json-schema-validator" % "0.1.7").inAll,
ShadeRule.rename("org.apache.avro.**" -> s"$shadePackage.org.apache.avro.@1")
.inLibrary("com.azavea.geotrellis" %% "geotrellis-spark" % Version.geotrellis).inAll
)
}

assemblyMergeStrategy in assembly := {
case s if s.startsWith("META-INF/services") => MergeStrategy.concat
case "reference.conf" | "application.conf" => MergeStrategy.concat
case "META-INF/MANIFEST.MF" | "META-INF\\MANIFEST.MF" => MergeStrategy.discard
case "META-INF/ECLIPSEF.RSA" | "META-INF/ECLIPSEF.SF" => MergeStrategy.discard
case "META-INF/ECLIPSE_.RSA" | "META-INF/ECLIPSE_.SF" => MergeStrategy.discard
case s if s.startsWith("META-INF/") && s.endsWith("SF") => MergeStrategy.discard
case s if s.startsWith("META-INF/") && s.endsWith("RSA") => MergeStrategy.discard
case s if s.startsWith("META-INF/") && s.endsWith("DSA") => MergeStrategy.discard
case _ => MergeStrategy.first
}
val vpExtraSettings = Seq(
libraryDependencies ++= Seq(
// gtGeomesa exclude("com.google.protobuf", "protobuf-java") exclude("org.locationtech.geomesa",
// "geomesa-accumulo-datastore"),
gtGeotools exclude ("com.google.protobuf", "protobuf-java"),
"com.github.seratch" %% "awscala" % "0.6.1",
"org.scalaj" %% "scalaj-http" % "2.3.0",
sparkHive % Provided,
sparkJts,
gtS3 exclude ("com.google.protobuf", "protobuf-java") exclude ("com.amazonaws", "aws-java-sdk-s3"),
gtSpark exclude ("com.google.protobuf", "protobuf-java"),
gtVectorTile exclude ("com.google.protobuf", "protobuf-java"),
decline,
jaiCore,
gtVector,
cats,
scalactic,
scalatest,
circeCore,
circeGeneric,
circeExtras,
circeParser,
circeOptics,
circeJava8,
circeYaml,
"com.softwaremill.macmemo" %% "macros" % "0.4",
"com.amazonaws" % "aws-java-sdk-s3" % "1.11.340" % Provided
),

Test / fork := true,
Test / baseDirectory := (baseDirectory.value).getParentFile,
Test / parallelExecution := false,
Test / testOptions += Tests.Argument("-oDF")

)

// /* Microsite Settings
// *
// * To generate the microsite locally, use `sbt makeMicrosite`.
// * To publish the site to Github, use `sbt publishMicrosite`.
// *
// * Spark deps must not be marked `provided` while doing these, or you will get errors.
// */

// enablePlugins(MicrositesPlugin)
// enablePlugins(SiteScaladocPlugin)

// micrositeName := "VectorPipe"
// micrositeDescription := "Convert Vector data into VectorTiles"
// micrositeAuthor := "GeoTrellis Team at Azavea"
// micrositeGitterChannel := false
// micrositeOrganizationHomepage := "https://www.azavea.com/"
// micrositeGithubOwner := "geotrellis"
// micrositeGithubRepo := "vectorpipe"
// micrositeBaseUrl := "/vectorpipe"
// micrositeDocumentationUrl := "/vectorpipe/latest/api/#vectorpipe.package" /* Location of Scaladocs */

/* Main project */
lazy val vectorpipe = project
.in(file("."))
.settings(commonSettings, vpExtraSettings, release)

/* Benchmarking suite.
* Benchmarks can be executed by first switching to the `bench` project and then by running:
jmh:run -t 1 -f 1 -wi 5 -i 5 .*Bench.*
*/
lazy val bench = project.in(file("bench")).settings(common).dependsOn(lib).enablePlugins(JmhPlugin)

/* Microsite Settings
*
* To generate the microsite locally, use `sbt makeMicrosite`.
* To publish the site to Github, use `sbt publishMicrosite`.
*
* Spark deps must not be marked `provided` while doing these, or you will get errors.
*/

enablePlugins(MicrositesPlugin)
enablePlugins(SiteScaladocPlugin)

micrositeName := "VectorPipe"
micrositeDescription := "Convert Vector data into VectorTiles"
micrositeAuthor := "GeoTrellis Team at Azavea"
micrositeGitterChannel := false
micrositeOrganizationHomepage := "https://www.azavea.com/"
micrositeGithubOwner := "geotrellis"
micrositeGithubRepo := "vectorpipe"
micrositeBaseUrl := "/vectorpipe"
micrositeDocumentationUrl := "/vectorpipe/latest/api/#vectorpipe.package" /* Location of Scaladocs */
lazy val bench = project
.in(file("bench"))
.settings(commonSettings)
.dependsOn(vectorpipe)
.enablePlugins(JmhPlugin)




// assemblyShadeRules in assembly := {
// val shadePackage = "com.azavea.shaded.demo"
// Seq(
// ShadeRule.rename("com.google.common.**" -> s"$shadePackage.google.common.@1")
// .inLibrary("com.azavea.geotrellis" %% "geotrellis-cassandra" % Version.geotrellis).inAll,
// ShadeRule.rename("io.netty.**" -> s"$shadePackage.io.netty.@1")
// .inLibrary("com.azavea.geotrellis" %% "geotrellis-hbase" % Version.geotrellis).inAll,
// ShadeRule.rename("com.fasterxml.jackson.**" -> s"$shadePackage.com.fasterxml.jackson.@1")
// .inLibrary("com.networknt" % "json-schema-validator" % "0.1.7").inAll,
// ShadeRule.rename("org.apache.avro.**" -> s"$shadePackage.org.apache.avro.@1")
// .inLibrary("com.azavea.geotrellis" %% "geotrellis-spark" % Version.geotrellis).inAll
// )
// }
36 changes: 36 additions & 0 deletions project/Dependencies.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
import sbt._

object Dependencies {
val decline = "com.monovore" %% "decline" % Version.decline
val sparkHive = "org.apache.spark" %% "spark-hive" % Version.spark
val sparkStreaming = "org.apache.spark" %% "spark-streaming" % Version.spark
val sparkJts = "org.locationtech.geomesa" %% "geomesa-spark-jts" % Version.geomesa
val gtGeomesa = "org.locationtech.geotrellis" %% "geotrellis-geomesa" % Version.geotrellis
val gtGeotools = "org.locationtech.geotrellis" %% "geotrellis-geotools" % Version.geotrellis
val gtS3 = "org.locationtech.geotrellis" %% "geotrellis-s3" % Version.geotrellis
val gtSpark = "org.locationtech.geotrellis" %% "geotrellis-spark" % Version.geotrellis
val gtSparkTestKit = "org.locationtech.geotrellis" %% "geotrellis-spark-testkit" % Version.geotrellis % "test"
val gtVector = "org.locationtech.geotrellis" %% "geotrellis-vector" % Version.geotrellis
val gtShapefile = "org.locationtech.geotrellis" %% "geotrellis-shapefile" % Version.geotrellis
val gtVectorTile = "org.locationtech.geotrellis" %% "geotrellis-vectortile" % Version.geotrellis
val vectorpipe = "com.azavea" %% "vectorpipe" % Version.vectorpipe
val cats = "org.typelevel" %% "cats-core" % Version.cats
val scalactic = "org.scalactic" %% "scalactic" % Version.scalactic
val scalatest = "org.scalatest" %% "scalatest" % Version.scalatest % "test"
val jaiCore = "javax.media" % "jai_core" % "1.1.3" % "test" from "http://download.osgeo.org/webdav/geotools/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar"
val hbaseCommon = "org.apache.hbase" % "hbase-common" % "1.3.1"
val hbaseClient = "org.apache.hbase" % "hbase-client" % "1.3.1"
val hbaseServer = "org.apache.hbase" % "hbase-server" % "1.3.1"
val geomesaHbaseDatastore = "org.locationtech.geomesa" % "geomesa-hbase-datastore_2.11" % Version.geomesa
val kryo = "com.esotericsoftware" % "kryo-shaded" % Version.kryo
val snakeyaml = "org.yaml" % "snakeyaml" % Version.snakeyaml
val circeCore = "io.circe" %% "circe-core" % Version.circe
val circeGeneric = "io.circe" %% "circe-generic" % Version.circe
val circeExtras = "io.circe" %% "circe-generic-extras" % Version.circe
val circeParser = "io.circe" %% "circe-parser" % Version.circe
val circeOptics = "io.circe" %% "circe-optics" % Version.circe
val circeJava8 = "io.circe" %% "circe-java8" % Version.circe
val circeYaml = "io.circe" %% "circe-yaml" % "0.8.0"
val logging = "com.typesafe.scala-logging" %% "scala-logging" % Version.scalaLogging
val commonsIO = "commons-io" % "commons-io" % Version.commonsIO
}