Skip to content
This repository has been archived by the owner on Dec 20, 2018. It is now read-only.

TODO: Merge raster to existing layer #3

Open
vsimko opened this issue Nov 5, 2016 · 5 comments
Open

TODO: Merge raster to existing layer #3

vsimko opened this issue Nov 5, 2016 · 5 comments
Assignees

Comments

@vsimko
Copy link
Member

vsimko commented Nov 5, 2016

No description provided.

@aklink
Copy link
Member

aklink commented Jul 6, 2017

ManyLayersToMultibandLayer should solve this for both Tile and MultibandTile layers.

@vsimko
Copy link
Member Author

vsimko commented Aug 30, 2017

e.g. merging single-band raster to existing single-band layer (NA values ignored, existing values overwritten)

@aklink
Copy link
Member

aklink commented Nov 28, 2017

OK. We have used the term "Merge" for two different things in the past.
ManyLayersToMultibandLayer is LayerStacking (e.g. 3 bands + 3 bands = 6 bands).
What we need here is an overlapping Mosaicing (e.g. 3 bands + 3bands := 3bands ).

@aklink
Copy link
Member

aklink commented Feb 21, 2018

WIP: merge (mosaicing / update)

Some ToDo's:

  • check zoom levels (and/or projection), probably enable ZoomResample (Pyramiding)
  • reduce updating/merge to changed area only (currently merge applies to whole layer) - maybe this is no issue, but maybe runtime concerns
  • error handling in case merge/update fails (assert same zoom level etc.)

@aklink aklink self-assigned this Feb 21, 2018
@aklink
Copy link
Member

aklink commented Mar 8, 2018

An other approach from Gitter (probably solving "unnecessary untouched tiles updates"), not tested yet:

  // initial steps skipped...
    val (zoom, reprojected): (Int, RDD[(SpatialKey, Tile)] with Metadata[TileLayerMetadata[SpatialKey]]) =
      TileLayerRDD(tiled, rasterMetaData)
        .reproject(WebMercator, layoutScheme, Bilinear)
    val attributeStore = HadoopAttributeStore(hdfsPath)

    val updater = HadoopLayerUpdater(hdfsPath)
    val writer = HadoopLayerWriter(hdfsPath)

    val keyIndex: KeyIndexMethod[SpatialKey] = ZCurveKeyIndexMethod
    val layerId = LayerId("hrdem", zoom)

    if(attributeStore.layerExists(layerId)) {
      // update existing layer using a custom merge function 
      updater.update(layerId, reprojected, (v1:Tile, v2:Tile) => v1.combineDouble(v2) { (z1, z2) => {
        if (isData(z1) && isData(z2)) (z1 + z2) / 2
        else if (isData(z1)) z1
        else if (isData(z2)) z2
        else {
          Double.NaN // TODO: investigate why returning NODATA is not producing the expected output 
        }
      }
      })
    } else {
      // get computed keybounds
      val initial = reprojected.metadata.bounds.get
      // arbitrary enlarge keybounds to ensure updates would be possible anywhere
      val enlarged = KeyBounds(
        initial.minKey.copy(
          col = 0,
          row = 0
        ),
        initial.maxKey.copy(
          col = Int.MaxValue,
          row = Int.MaxValue
        ))
      writer.write(layerId, reprojected, keyIndex.createIndex(enlarged))
    }

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants