Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support running Pandas UDFs on GPUs in Python processes. #640

Merged
merged 72 commits into from
Sep 11, 2020
Merged

Support running Pandas UDFs on GPUs in Python processes. #640

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
4063857
Support Pandas UDF on GPU
firestarman Jul 21, 2020
94f3b22
Fix an error when running rapids.worker.
firestarman Jul 21, 2020
96e35aa
Pack python files
firestarman Jul 22, 2020
dccf977
Add API to init GPU context in python process
firestarman Jul 27, 2020
ef36d5e
Support limiting the number of python workers
firestarman Jul 31, 2020
93966b4
Support memory limitaion for Python processes
firestarman Aug 4, 2020
3b2e527
Imporve the memory computation for Python workers
firestarman Aug 5, 2020
1e4895a
Support setting max size of RMM pool
firestarman Aug 6, 2020
c877506
Support more types of Pandas UDF
firestarman Aug 11, 2020
d0d15c2
Use maxsize for max pool size when not specified.
firestarman Aug 18, 2020
61b3589
Support two more types of Pandas UDF
firestarman Aug 27, 2020
65e497d
Add tests for udfs and basic support for accelerated arrow exchange with
revans2 Aug 12, 2020
ec2cec6
Support Pandas UDF on GPU
firestarman Jul 21, 2020
d18ff4e
Fix an error when running rapids.worker.
firestarman Jul 21, 2020
616d1da
Pack python files
firestarman Jul 22, 2020
c5557ca
Add API to init GPU context in python process
firestarman Jul 27, 2020
2fcc2aa
Support limiting the number of python workers
firestarman Jul 31, 2020
31a9fb3
Support memory limitaion for Python processes
firestarman Aug 4, 2020
f71f4de
Imporve the memory computation for Python workers
firestarman Aug 5, 2020
d4791af
Support setting max size of RMM pool
firestarman Aug 6, 2020
4b88939
Support more types of Pandas UDF
firestarman Aug 11, 2020
d5d156e
Use maxsize for max pool size when not specified.
firestarman Aug 18, 2020
37dc856
Support two more types of Pandas UDF
firestarman Aug 27, 2020
29e39ea
Use the columnar version rule for Scalar Pandas UDF
firestarman Sep 1, 2020
1033f23
Updates the RapidsMeta of plans for Pandas UDF
firestarman Sep 2, 2020
5e28772
Remove the unnecessary env variable
firestarman Sep 2, 2020
3f94ac8
Correct some doc styles to pass mvn verification
firestarman Sep 3, 2020
6f24ca2
add udf test
shotai Sep 3, 2020
5ea3ab6
Merge branch 'pandas-udf' of https://github.com/firestarman/spark-rap…
shotai Sep 3, 2020
69d5b54
Support process pool for Python workers
firestarman Sep 3, 2020
9b25c2e
merge udftest
shotai Sep 3, 2020
c8ab681
add cudf test
shotai Sep 3, 2020
b81fe94
Add a config to disable/enable Pandas UDF on GPU.
firestarman Sep 4, 2020
908bb93
add more test case with cudf
shotai Sep 4, 2020
963c821
refactor udf test
shotai Sep 4, 2020
868ca3a
Python: Not init GPU if no cuda device specified
firestarman Sep 4, 2020
92c10a6
resolve conflict
shotai Sep 4, 2020
e05e4b6
resolve conflict
shotai Sep 4, 2020
69b1ec0
Update the config doc
firestarman Sep 7, 2020
1f56cd9
skip udf in premerge
shotai Sep 7, 2020
e400f5a
add pyarrow in docker
shotai Sep 7, 2020
6435eaa
disable udf test in premerge
shotai Sep 7, 2020
c33b41c
Merge pull request #4 from firestarman/pandas-test-mg
firestarman Sep 8, 2020
53959db
Merge branch 'branch-0.2' into pandas-udf-col
firestarman Sep 8, 2020
732505b
Move gpu init to `try...catch`
firestarman Sep 8, 2020
5a074ac
Remove numpy, it will include in pandas installation. Update readme.
shotai Sep 8, 2020
b80a201
update doc with pandas udf support
shotai Sep 8, 2020
7514ac1
update integration dockerfile
shotai Sep 8, 2020
db06504
Merge branch 'pandas-udf' of https://github.com/firestarman/spark-rap…
shotai Sep 8, 2020
7817e5e
Update getting-started-on-prem.md
shotai Sep 8, 2020
0f72025
Update getting-started-on-prem.md
shotai Sep 8, 2020
35a3008
Update getting-started-on-prem.md
shotai Sep 8, 2020
ec7848e
Update getting-started-on-prem.md
shotai Sep 8, 2020
be65f58
Add warning log when python worker reuse enabled
firestarman Sep 8, 2020
1757afa
Replace GpuSemaphore with PythonWorkerSemaphore
firestarman Sep 8, 2020
537e594
Remove the warning log for python worker reuse enabled
firestarman Sep 9, 2020
60cf951
remove udf marker, add comment, update jenkins script for udf_cudf test
shotai Sep 9, 2020
6c9e86b
update doc in pandas udf section
shotai Sep 9, 2020
58fab52
update dockerfile for integration test
shotai Sep 9, 2020
283b6a2
Merge branch 'pandas-udf' of https://github.com/firestarman/spark-rap…
shotai Sep 9, 2020
eee4d05
Update the name of conf for python gpu enabled.
firestarman Sep 10, 2020
8924ad8
add marker for cudf udf test
shotai Sep 10, 2020
7eba830
update comment in test start script
shotai Sep 10, 2020
f838ae0
remove old config
shotai Sep 10, 2020
803fcf4
Not init gpu memory when python on gpu is disabled
firestarman Sep 10, 2020
984082b
remove old config
shotai Sep 10, 2020
127ab08
Merge branch 'pandas-udf' of https://github.com/firestarman/spark-rap…
shotai Sep 10, 2020
6156298
import cudf lib normally
shotai Sep 10, 2020
beabf8b
update import cudf
shotai Sep 10, 2020
47ffc98
Check python module conf only when python gpu enabeld
firestarman Sep 10, 2020
b1c9be5
update dynamic config for udf enable
shotai Sep 10, 2020
9860ee6
Merge branch 'pandas-udf' of https://github.com/firestarman/spark-rap…
shotai Sep 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
resolve conflict
  • Loading branch information
@shotai
shotai committed Sep 4, 2020
commit e05e4b67aa095bf227f5939e2118217bde9d1fd4
4 changes: 0 additions & 4 deletions sql-plugin/src/main/scala/com/nvidia/spark/rapids/python/PythonWorkerSemaphore.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
/*
<<<<<<< HEAD
* Copyright (c) 2019-2020, NVIDIA CORPORATION.
=======
* Copyright (c) 2020, NVIDIA CORPORATION.
>>>>>>> 3f94ac8b608e311c181892fc72756d894627037f
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
4 changes: 0 additions & 4 deletions ...rc/main/scala/org/apache/spark/sql/rapids/execution/python/GpuAggregateInPandasExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
/*
<<<<<<< HEAD
* Copyright (c) 2019-2020, NVIDIA CORPORATION.
=======
* Copyright (c) 2020, NVIDIA CORPORATION.
>>>>>>> 3f94ac8b608e311c181892fc72756d894627037f
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
10 changes: 0 additions & 10 deletions .../src/main/scala/org/apache/spark/sql/rapids/execution/python/GpuArrowEvalPythonExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,4 @@
/*
<<<<<<< HEAD
* Copyright (c) 2019-2020, NVIDIA CORPORATION.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
=======
* Copyright (c) 2020, NVIDIA CORPORATION.
*
* Licensed to the Apache Software Foundation (ASF) under one or more
Expand All @@ -18,7 +9,6 @@
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
>>>>>>> 3f94ac8b608e311c181892fc72756d894627037f
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
Expand Down
4 changes: 0 additions & 4 deletions ...n/scala/org/apache/spark/sql/rapids/execution/python/GpuFlatMapCoGroupsInPandasExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
/*
<<<<<<< HEAD
* Copyright (c) 2019-2020, NVIDIA CORPORATION.
=======
* Copyright (c) 2020, NVIDIA CORPORATION.
>>>>>>> 3f94ac8b608e311c181892fc72756d894627037f
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
4 changes: 0 additions & 4 deletions ...ain/scala/org/apache/spark/sql/rapids/execution/python/GpuFlatMapGroupsInPandasExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
/*
<<<<<<< HEAD
* Copyright (c) 2019-2020, NVIDIA CORPORATION.
=======
* Copyright (c) 2020, NVIDIA CORPORATION.
>>>>>>> 3f94ac8b608e311c181892fc72756d894627037f
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
4 changes: 0 additions & 4 deletions ...ugin/src/main/scala/org/apache/spark/sql/rapids/execution/python/GpuMapInPandasExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
/*
<<<<<<< HEAD
* Copyright (c) 2019-2020, NVIDIA CORPORATION.
=======
* Copyright (c) 2020, NVIDIA CORPORATION.
>>>>>>> 3f94ac8b608e311c181892fc72756d894627037f
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
4 changes: 0 additions & 4 deletions sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/python/GpuPythonHelper.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
/*
<<<<<<< HEAD
* Copyright (c) 2019-2020, NVIDIA CORPORATION.
=======
* Copyright (c) 2020, NVIDIA CORPORATION.
>>>>>>> 3f94ac8b608e311c181892fc72756d894627037f
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
4 changes: 0 additions & 4 deletions ...n/src/main/scala/org/apache/spark/sql/rapids/execution/python/GpuWindowInPandasExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
/*
<<<<<<< HEAD
* Copyright (c) 2019-2020, NVIDIA CORPORATION.
=======
* Copyright (c) 2020, NVIDIA CORPORATION.
>>>>>>> 3f94ac8b608e311c181892fc72756d894627037f
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
4 changes: 0 additions & 4 deletions sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/python/RowUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
/*
<<<<<<< HEAD
* Copyright (c) 2019-2020, NVIDIA CORPORATION.
=======
* Copyright (c) 2020, NVIDIA CORPORATION.
>>>>>>> 3f94ac8b608e311c181892fc72756d894627037f
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down