BUG: jupyter loading of serialized modin objects from ray tasks converts to pandas objects #7408
Description
Modin version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest released version of Modin.
-
I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow this guide.)
Reproducible Example
import ray
ray.init()
@ray.remote
def test():
import modin.pandas as pd
df = pd.DataFrame({
'A': [1, 2, 3],
'B': [4, 5, 6]
})
print(type(df)) # <class 'modin.pandas.dataframe.DataFrame'>
return df
df = ray.get(test.remote())
print(type(df)) # <class 'pandas.core.frame.DataFrame'>Issue Description
Hi! i am trying to integrate modin with jupyter environments and noticed unexpectedly that when dataframes are passed to/from ray tasks, they serialize to pandas objects. While this is fine for toy examples, I am concerned about unexpected OOMs popping up during interactive work with large datasets.
Expected Behavior
My expectation is the objects would deserialize to the same objects with whatever cluster refs to continue manipulating the underlying data.
Error Logs
No errors. Just inline type changing.
Installed Versions
INSTALLED VERSIONS
commit : 1c4d173
python : 3.11.9
python-bits : 64
OS : Darwin
OS-release : 23.6.0
Version : Darwin Kernel Version 23.6.0: Wed Jul 31 20:49:39 PDT 2024; root:xnu-10063.141.1.700.5~1/RELEASE_ARM64_T6000
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8
Modin dependencies
modin : 0.32.0+6.g1c4d173d
ray : 2.35.0
dask : 2024.10.0
distributed : 2024.10.0
pandas dependencies
pandas : 2.2.3
numpy : 2.1.3
pytz : 2024.2
dateutil : 2.9.0.post0
pip : 24.2
Cython : None
sphinx : None
IPython : 8.29.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.10.0
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.4
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 18.0.0
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.14.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2024.2
qtpy : None
pyqt5 : None