Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

folium with pyspark-folium type is not serializable #878

Closed
HADDADENICAR opened this issue Jun 8, 2018 · 1 comment
Closed

folium with pyspark-folium type is not serializable #878

HADDADENICAR opened this issue Jun 8, 2018 · 1 comment
Labels
duplicate The issue is a duplicate of a previous issue wontfix This problem or request is outside the scope of folium

Comments

@HADDADENICAR
Copy link

HADDADENICAR commented Jun 8, 2018 •
edited
Loading

I m trying to use folium with pyspark , when i run the foreach method of Spark dataframe Rdd i got this error :

PicklingError: Could not serialize object: TypeError: can't pickle weakref objects

and this is my code :

 import findspark
 findspark.init()
 from pyspark import SparkContext, SparkConf, SparkSession

  #readmultiple csv with pyspark
  spark2 = SparkSession \
  .builder \
  .appName("Python Spark SQL basic example") \
  .getOrCreate()

  df = spark2.read.csv("Desktop/train  /train.csv",header=True).select("pickup_datetime","pickup_longitude","pickup_latitude","dropoff_datetime","dropoff_longitude","dropoff_latitude")

  df.rdd.foreach(lambda row : folium.CircleMarker(location=(float(row.pickup_latitude),
                              float(row.pickup_longitude)),
                    radius=20,
                    color="#0A8A9F",fill=True).add_to(folium_map_pickups))
@Conengmo
Copy link
Member

Conengmo commented Jun 8, 2018

Similar problem as #796. Indeed you cannot pickle folium objects, and we're not going to fix this in folium. Unless someone has an idea how to reasonably do this.

I advice you to collect your data from pyspark and create a folium map with markers outside of pyspark. You cannot create a map with millions of markers anyway, so you won't need parallel processing to add markers.

@Conengmo Conengmo closed this as completed Jun 8, 2018
@Conengmo Conengmo added duplicate The issue is a duplicate of a previous issue wontfix This problem or request is outside the scope of folium labels Jun 8, 2018
This was referenced Jun 8, 2018
Closed
Open
@bwest2397 bwest2397 mentioned this issue Oct 20, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
duplicate The issue is a duplicate of a previous issue wontfix This problem or request is outside the scope of folium
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Conengmo @HADDADENICAR