This is a WIP PR to drive comments and discussion about the proposed merge of geomesa_pyspark module of GeoMesa into GeoPySpark and the subsequent promotion of GeoPySpark from just GeoTrellis/VectorPipe Python bindings to a project that brings general Geo* support to Python/PySpark.

Major thing of note and discussion points.

• YARN distribution of resources
  • Part of the functionality that has been ported over from geomesa_pyspark is the ability to package up runtime resources and ship them out when the spark context is started as a YARN job. This allows for jobs to be run on a cluster without requiring GeoPySpark to be installed on the worker nodes.
  • More testing needs to be done around this to ensure it works for GeoTrellis and VectorPipe.
• GeoMesa bindings
  • Verified able to query out of GeoMesa on HBase.
  • The geomesa-*-spark-runtime jar is required for this to work. Integrating this into the install-jar workflow might help end user experience.
• Discuss aligning VectorPipe PB / Shapely Geometries / SimpleFeatures
  • Allow for workflows such as pulling Polygons from GeoMesa to use in masking operation of GeoTrellis Tiles or creating VectorTiles from vectors pulled from GeoMesa. This is something that would probably need to be done on the Spark level and wired though Python.
• Kryo Serialization Change
  • Changed to using GeoMesa's Kryo registration for simple features.
  • Need to verify this still works with VectorPipe.
  • Does this break backwards compatibility? Does that matter?

A beta release of this is available here. The GeoMesa pre-release that includes the complementary code is here with my working branch here. The changes in GeoMesa are to bind the pyUDT function in AbstractGeometryUDT to geopyspark.GeometryUDT. This fixes the Non issue we've been seeing.

I've included this in the README but for demonstration and reference purposes I'll include here a code sample of how to pull features from GeoMesa. This uses the YARN packaging mentioned above.

import geopyspark as gps
from pyspark import SparkContext
from pyspark.sql import SQLContext

conf = gps.geopyspark_conf(appName="test", master="yarn")
sc = SparkContext(conf=conf)
spark = SQLContext(sc)

params = { "hbase.catalog": "catalog" }
feature = "gdelt"
df = spark.read\
    .format("geomesa")\
    .options(**params)\
    .option("geomesa.feature", feature)\
    .load()

df.createOrReplaceTempView("gdelt")

spark.sql("select * from gdelt where st_contains(st_makeBBOX(0.0, 0.0, 90.0, 90.0), geom) limit 10").show()

+---------+--------------------+-------------------+--------------------+
|eventCode|          actor1Name|                dtg|                geom|
+---------+--------------------+-------------------+--------------------+
|      042|              TAIWAN|2017-01-01 00:00:00|POINT (6.73333 0....|
|      043|             VATICAN|2017-01-01 00:00:00|POINT (6.73333 0....|
|      161|SAO TOME AND PRIN...|2017-01-01 00:00:00|POINT (6.73333 0....|
|      042|              TAIWAN|2017-01-01 00:00:00|POINT (6.73333 0....|
|      043|              POLICE|2017-01-01 00:00:00|POINT (5.4851 5.4...|
|      160|CONSTITUTIONAL COURT|2017-01-01 00:00:00|POINT (8.78333 3.75)|
|      173|              PRISON|2017-01-01 00:00:00|POINT (8.78333 3.75)|
|      173|   EQUATORIAL GUINEA|2017-01-01 00:00:00|POINT (8.78333 3.75)|
|      042|            CAMEROON|2017-01-01 00:00:00|POINT (9.241 4.1527)|
|      051|             NIGERIA|2017-01-01 00:00:00|POINT (6.08333 4.75)|
+---------+--------------------+-------------------+--------------------+