Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more benchmarks and fix old measurements #59

Merged
merged 1 commit into from
Mar 23, 2020
Merged

Add more benchmarks and fix old measurements #59

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 56 additions & 14 deletions benchmark/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,31 +17,73 @@ $ sbt

| | Dataset | Resolution | delaunay + rasterization, ms/op |
|-----------------------------|----------|------------|---------------------------------|
|**PDAL (native rasterizer)** |red-rocks | 6.9375 | **878.513 ± 102.391** |
|PDAL (JVM rasterizer) |red-rocks | 6.9375 | 1877.393 ± 1127.682 |
|GT |red-rocks | 6.9375 | 3301.495 ± 846.053 |
|**PDAL (native rasterizer)** |red-rocks | 3.4687 | **6223.862 ± 610.427** |
|PDAL (JVM rasterizer) |red-rocks | 3.4687 | 13023.915 ± 1695.589 |
|GT |red-rocks | 3.4687 | 95570.479 ± 11310.479 |
|**PDAL (native rasterizer)** |red-rocks | 1.7344 | **11113.148 ± 5216.283** |
|PDAL (JVM rasterizer) |red-rocks | 1.7344 | 15685.002 ± 1306.871 |
|GT |red-rocks | 1.7344 | 179422.928 ± 22267.424 |
|**PDAL (native rasterizer)** |red-rocks | 0.8672 | **10907.534 ± 2789.634** |
|PDAL (JVM rasterizer) |red-rocks | 0.8672 | 20032.444 ± 9338.665 |
|GT |red-rocks | 0.8672 | 204982.543 ± 86162.431 |
|**PDAL (native rasterizer)** |red-rocks | 6.9375 | **382.476 ± 44.347** |
|PDAL (JVM rasterizer) |red-rocks | 6.9375 | 735.026 ± 10.594 |
|GT |red-rocks | 6.9375 | 2319.073 ± 171.407 |
|**PDAL (native rasterizer)** |red-rocks | 3.4687 | **5573.016 ± 493.018** |
|PDAL (JVM rasterizer) |red-rocks | 3.4687 | 12579.721 ± 2345.998 |
|GT |red-rocks | 3.4687 | 88464.543 ± 10692.569 |
|**PDAL (native rasterizer)** |red-rocks | 1.7344 | **8083.490 ± 1786.544** |
|PDAL (JVM rasterizer) |red-rocks | 1.7344 | 16152.638 ± 4003.070 |
|GT |red-rocks | 1.7344 | 158994.121 ± 23280.155 |
|**PDAL (native rasterizer)** |red-rocks | 0.8672 | **7697.157 ± 396.745** |
|PDAL (JVM rasterizer) |red-rocks | 0.8672 | 16710.832 ± 2661.915 |
|GT |red-rocks | 0.8672 | 153764.608 ± 12090.790 |

#### Legend:
* PDAL (native rasterizer) - benchmarks a DEMRasterSource that uses PDAL Delaunay and uses native (C++) Rasterizer.
* PDAL (JVM rasterizer) - benchmarks a DEMRasterSource that uses PDAL Delaunay and uses JVM Rasterizer.
* GT - benchmarks a DEMRasterSource that uses GeoTrellis Delaunay and uses JVM Rasterizer.

#### Conclusion

PDAL Delaunay + Native rasterization makes a lot of sense and it is at it behaves at
least two times faster comparing to other methods. It makes no sense to use GeoTrellis Delaunay Triangulation at this point.

### Reproject 3/15/2020

| | Dataset | Resolution | delaunay + rasterization, ms/op |
|--------------------|----------|------------|---------------------------------|
|PDAL reproject |red-rocks | 6.9375 | 1154.920 ± 334.336 |
|GT region reproject |red-rocks | 6.9375 | **976.935 ± 43.521** |
|PDAL reproject |red-rocks | 6.9375 | 531.069 ± 47.114 |
|GT region reproject |red-rocks | 6.9375 | **527.233 ± 15.586** |

#### Legend:
* PDAL reproject - benchmarks reprojection done on the PDAL side via [filters.reprojection](https://pdal.io/stages/filters.reprojection.html).
* GT region reproject - benchmarks reprojection done using GeoTrellis capabilities.

#### Conclusion

GeoTrellis reprojection tends to be a slightly faster.
The performance is pretty much the same with taking into account the standard deviation.
We decided to rely on the GeoTrellis reprojection capabilities at this point.

# PDAL JNI Bindings benchmarks

This benchmark is done against [PDAL Java](https://github.com/PDAL/java) bindings to get the sense of what is slow in the DEMRasterSource.
It does not use the `RasterSource` abstraction and uses only [PDAL Java](https://github.com/PDAL/java) API.

To reproduce the table below you can use the following command:

```bash
$ sbt
> project benchmark
> jmh:run -i 10 -wi 5 -f1 -t1 .*ReadEPTBench.*
```

### 3/23/2020

```
# VM options: -Xmx4G
# Cores: 4
```

| | Dataset | Resolution | delaunay + rasterization, ms/op |
|----------------------------|----------|------------|---------------------------------|
|Pipeline execute |red-rocks | 6.9375 | 207.858 ± 18.166 |
|Pipeline Delaunay execute |red-rocks | 6.9375 | 388.973 ± 76.595 |
|Pipeline Delaunay rasterize |red-rocks | 6.9375 | 419.780 ± 63.084 |

#### Legend:
* Pipeline execute - benchmarks raw Pipeline execution without generating a triangulation mesh.
* Pipeline Delaunay execute - benchmarks a Pipeline execution with generating a triangulation mesh.
* Pipeline Delaunay rasterize - benchmarks Pipeline execution with generating a triangulation mesh with the mesh rasterization.
31 changes: 21 additions & 10 deletions benchmark/src/main/scala/geotrellis/pointcloud/raster/ept/DEMRasterSourceBench.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,10 @@ import java.util.concurrent.TimeUnit
class DEMRasterSourceBench {
val catalogPath = "../pointcloud/src/test/resources/red-rocks"

var rs: DEMRasterSource = _
var gtrs: GeoTrellisDEMRasterSource = _
var jrs: JavaDEMRasterSource = _
/** This bench benchmarks the actual PDAL reads, not the RasterSource initialization time. */
val rs = DEMRasterSource(catalogPath)
val gtrs = GeoTrellisDEMRasterSource(catalogPath)
val jrs = JavaDEMRasterSource(catalogPath)

/**
* jmh:run -i 3 -wi 1 -f1 -t1 .*DEMRasterSourceBench.*
Expand Down Expand Up @@ -75,15 +76,25 @@ class DEMRasterSourceBench {
* [info] Benchmark Mode Cnt Score Error Units
* [info] DEMRasterSourceBench.DEMReprojectRasterSourceReadAll avgt 10 976.935 ± 43.521 ms/op
* [info] DEMRasterSourceBench.JavaDEMReprojectRasterSourceReadAll avgt 10 1154.920 ± 334.336 ms/op
*
* 03/23/2020
* [info] Benchmark Mode Cnt Score Error Units
* [info] DEMRasterSourceBench.DEMRasterSourceReadAll avgt 10 382.476 ± 44.347 ms/op
* [info] DEMRasterSourceBench.DEMRasterSourceReadAll2 avgt 10 5573.016 ± 493.018 ms/op
* [info] DEMRasterSourceBench.DEMRasterSourceReadAll4 avgt 10 8083.490 ± 1786.544 ms/op
* [info] DEMRasterSourceBench.DEMRasterSourceReadAll8 avgt 10 7697.157 ± 396.745 ms/op
* [info] DEMRasterSourceBench.DEMReprojectRasterSourceReadAll avgt 10 527.233 ± 15.586 ms/op
* [info] DEMRasterSourceBench.GeoTrellisDEMRasterSourceReadAll avgt 10 2319.073 ± 171.407 ms/op
* [info] DEMRasterSourceBench.GeoTrellisDEMRasterSourceReadAll2 avgt 10 88464.543 ± 10692.569 ms/op
* [info] DEMRasterSourceBench.GeoTrellisDEMRasterSourceReadAll4 avgt 10 158994.121 ± 23280.155 ms/op
* [info] DEMRasterSourceBench.GeoTrellisDEMRasterSourceReadAll8 avgt 10 153764.608 ± 12090.790 ms/op
* [info] DEMRasterSourceBench.JavaDEMRasterSourceReadAll avgt 10 735.026 ± 10.594 ms/op
* [info] DEMRasterSourceBench.JavaDEMRasterSourceReadAll2 avgt 10 12579.721 ± 2345.998 ms/op
* [info] DEMRasterSourceBench.JavaDEMRasterSourceReadAll4 avgt 10 16152.638 ± 4003.070 ms/op
* [info] DEMRasterSourceBench.JavaDEMRasterSourceReadAll8 avgt 10 16710.832 ± 2661.915 ms/op
* [info] DEMRasterSourceBench.JavaDEMReprojectRasterSourceReadAll avgt 10 531.069 ± 47.114 ms/op
*/

@Setup(Level.Invocation)
def setupData(): Unit = {
rs = DEMRasterSource(catalogPath)
gtrs = GeoTrellisDEMRasterSource(catalogPath)
jrs = JavaDEMRasterSource(catalogPath)
}

@Benchmark
def GeoTrellisDEMRasterSourceReadAll(): Option[Raster[MultibandTile]] = gtrs.read()

Expand Down
110 changes: 110 additions & 0 deletions benchmark/src/main/scala/geotrellis/pointcloud/raster/ept/ReadEPTBench.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
/*
* Copyright 2020 Azavea
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package geotrellis.pointcloud.raster.ept

import geotrellis.pointcloud.raster.rasterize.triangles.PDALTrianglesRasterizer

import geotrellis.raster.{MultibandTile, Raster, RasterExtent}
import geotrellis.vector.Extent
import io.pdal.{PointView, TriangularMesh}
import io.pdal.pipeline.{ENil, FilterDelaunay, ReadEpt}
import cats.syntax.option._
import org.openjdk.jmh.annotations._

import scala.collection.JavaConverters._
import java.util.concurrent.TimeUnit

@BenchmarkMode(Array(Mode.AverageTime))
@State(Scope.Benchmark)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
class ReadEPTBench {
val catalogPath = "../pointcloud/src/test/resources/red-rocks"

/** Params are used to match [[DEMRasterSourceBench]] behavior. **/
val extent @ Extent(exmin, eymin, exmax, eymax) = Extent(481968.0, 4390186.0, 482856.0, 4391074.0)
val readEpt: ReadEpt = ReadEpt(
filename = catalogPath,
bounds = s"([$exmin, $eymin], [$exmax, $eymax])".some,
resolution = 6.9375.some
)

/**
* jmh:run -i 10 -wi 5 -f1 -t1 .*ReadEPTBench.*
*
* 03/23/2020 #1
* [info] Benchmark Mode Cnt Score Error Units
* [info] ReadEPTBench.EPTRasterize avgt 10 419.780 ± 63.084 ms/op
* [info] ReadEPTBench.EPTRead avgt 10 207.858 ± 18.166 ms/op
* [info] ReadEPTBench.EPTReadTriangulation avgt 10 388.973 ± 76.595 ms/op
*/
@Benchmark
def EPTReadTriangulation(): Option[TriangularMesh] = {
val pipeline = readEpt ~ FilterDelaunay() toPipeline

try {
if (pipeline.validate()) {
pipeline.execute

pipeline
.getPointViews()
.asScala
.toList
.headOption
.map { _.getTriangularMesh() }
} else None
} finally pipeline.close()
}

@Benchmark
def EPTRead(): Option[PointView] = {
val pipeline = readEpt ~ ENil toPipeline

try {
if (pipeline.validate()) {
pipeline.execute

pipeline
.getPointViews()
.asScala
.toList
.headOption
} else None
} finally pipeline.close()
}

@Benchmark
def EPTRasterize(): Option[Raster[MultibandTile]] = {
val pipeline = readEpt ~ FilterDelaunay() toPipeline

try {
if (pipeline.validate()) {
pipeline.execute

pipeline
.getPointViews()
.asScala
.toList
.headOption
.map {
PDALTrianglesRasterizer
.native(_, RasterExtent(extent, 128, 128))
.mapTile(MultibandTile(_))
}
} else None
} finally pipeline.close()
}
}