Skip to content
This repository has been archived by the owner on Mar 6, 2019. It is now read-only.

EMR zeppelin %sql error with TIleUDT #50

Open
vpipkt opened this issue Feb 9, 2018 · 3 comments
Open

EMR zeppelin %sql error with TIleUDT #50

vpipkt opened this issue Feb 9, 2018 · 3 comments
Labels

Comments

@vpipkt
Copy link
Contributor

vpipkt commented Feb 9, 2018

Do this on EMR zep

val layer[astraea.spark.rasterframes.datasource.geotrellis.Layer] = ???
val rf = spark.read.geotrellis
    .loadRF(layer)
rf.createOrReplaceTempView("someTableFoo")

Then in next paragraph:

%sql
SELECT * FROM someTableFoo

You get:

org.apache.spark.SparkException: Can not load in UserDefinedType org.apache.spark.sql.gt.types.TileUDT for user class geotrellis.raster.Tile.
	at org.apache.spark.sql.types.UDTRegistration$$anonfun$getUDTFor$1.apply(UDTRegistration.scala:84)
	at org.apache.spark.sql.types.UDTRegistration$$anonfun$getUDTFor$1.apply(UDTRegistration.scala:73)
	at scala.Option.map(Option.scala:146)
	at org.apache.spark.sql.types.UDTRegistration$.getUDTFor(UDTRegistration.scala:73)
	at org.apache.spark.sql.catalyst.encoders.RowEncoder$.org$apache$spark$sql$catalyst$encoders$RowEncoder$$serializerFor(RowEncoder.scala:83)
	at org.apache.spark.sql.catalyst.encoders.RowEncoder$$anonfun$2.apply(RowEncoder.scala:160)
	at org.apache.spark.sql.catalyst.encoders.RowEncoder$$anonfun$2.apply(RowEncoder.scala:159)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
	at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:186)
	at org.apache.spark.sql.catalyst.encoders.RowEncoder$.org$apache$spark$sql$catalyst$encoders$RowEncoder$$serializerFor(RowEncoder.scala:159)
	at org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(RowEncoder.scala:61)
	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:68)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:632)
	at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:691)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.zeppelin.spark.SparkSqlInterpreter.interpret(SparkSqlInterpreter.java:116)
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:97)
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:498)
	at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
	at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
@vpipkt
Copy link
Contributor Author

vpipkt commented Feb 9, 2018

this is with raster-frames on commit bb060fbb56ef6686b7a793114be7a025311790e3

@metasim metasim added the bug label Feb 12, 2018
@metasim
Copy link
Contributor

metasim commented Mar 16, 2018

@vpipkt Any clue if this is still an issue? Not quite sure how this use case is any different from other standard use cases.

@vpipkt
Copy link
Contributor Author

vpipkt commented Mar 16, 2018

Will check this today with 0.6.0-RC7

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants