Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FIX-#4479: Prevent users from using a local filepath when performing a distributed write #4484

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

FIX-#4479: Prevent users from using a local filepath when performing a distributed write #4484

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
aca8db5
FIX-#4479: Prevent users from using a local filepath when performing …
RehanSD May 23, 2022
e622fcf
lint
RehanSD May 23, 2022
be74c8e
Fix docstring
RehanSD May 23, 2022
7b1dff6
Update modin/core/execution/ray/implementations/utils.py
RehanSD May 26, 2022
a313a79
Update modin/core/execution/ray/implementations/utils.py
RehanSD May 26, 2022
1dd8636
Update modin/core/execution/ray/implementations/pandas_on_ray/io/io.py
RehanSD May 26, 2022
4191088
Update modin/core/execution/ray/implementations/pandas_on_ray/io/io.py
RehanSD May 26, 2022
8c7120b
Address review comments
RehanSD May 26, 2022
dcadcf4
Address review comments and fix util for non-local paths and non-exis…
RehanSD May 26, 2022
b6552ba
Update modin/core/io/utils.py
RehanSD May 27, 2022
0608f9e
Update modin/core/io/utils.py
RehanSD May 27, 2022
cc80818
Add testing
RehanSD May 27, 2022
8081d9c
Use fsspec and default to pandas
RehanSD Jun 2, 2022
9c108af
Use fsspec more and default to pandas
RehanSD Jun 3, 2022
f1bc60a
Add support for windows
RehanSD Jun 3, 2022
a7acd81
Fix windows support
RehanSD Jun 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add support for windows 
Signed-off-by: Rehan Durrani <rehan@ponder.io>
  • Loading branch information
@RehanSD
RehanSD committed Jun 3, 2022
commit f1bc60aa1e0fea64b4fa4929f224e1fcc17b0d29
8 changes: 6 additions & 2 deletions modin/core/io/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
import re
import fsspec

IS_FILE_ONLY_REGEX = re.compile("[^\/]*\.\w+") # noqa: W605
IS_FILE_ONLY_REGEX = re.compile(f"[^\\{os.sep}]*\.\w+") # noqa: W605


def is_local_path(path) -> bool:
Expand Down Expand Up @@ -45,13 +45,17 @@ def is_local_path(path) -> bool:
parent_dir = os.getcwd()
else:
# If we are passed a full path, we want to remove the filename from it.
parent_dir = "/".join(path.split("/")[:-1])
parent_dir = os.sep.join(path.split(os.sep)[:-1])
fs = fsspec.core.url_to_fs(parent_dir)[0] # Grab just the FileSystem object
if hasattr(
fs, "local_file"
): # If the FS does not have the `local_file` attr, it is not local.
# We still need to check that it is not a mounted file - as fsspec treats mounted
# files the same as local ones, but we want to distinguish between local and mounted.
if os.name == "nt" and parent_dir[:3] == "D:\\":
# In Windows, os.path.abspath(os.sep) will give us the C Drive, but we want the
# D drive to also be marked as local.
return True
local_device_id = os.stat(os.path.abspath(os.sep)).st_dev
path_device_id = os.stat(parent_dir).st_dev
return path_device_id == local_device_id
Expand Down
6 changes: 5 additions & 1 deletion modin/pandas/test/test_io.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2393,7 +2393,11 @@ def test_is_local_path():
assert is_local_path(
os.getcwd()
), "Current Working Directory incorrectly flagged as not local!"
new_file = os.getcwd() + "/modin-example-file"
new_file = os.getcwd() + "/modin-example-file.extension"
assert is_local_path(
new_file
), "Non-existent file under current working directory incorrectly flagged as not local!"
new_file_in_curr_dir = "modin-example-file.extension"
assert is_local_path(
new_file_in_curr_dir,
), "Non-existent file without absolute path incorrectly flagged as not local!"