Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dask indexing problem with cupy #11266

Open
miguelcarcamov opened this issue Jul 31, 2024 · 5 comments
Open

Dask indexing problem with cupy #11266

miguelcarcamov opened this issue Jul 31, 2024 · 5 comments
Labels
array bug Something is broken gpu

Comments

@miguelcarcamov
Copy link

miguelcarcamov commented Jul 31, 2024 •
edited
Loading

Hi, I want to index an array and then set those values to something.

So, this is the example that DOES NOT work.

Minimal Complete Verifiable Example:

import dask.array as da
import dask

dask.config.set({"array.backend": "cupy"})

data = da.random.random((20000,3))
condition = data[:,0] > 0.5
idx = da.flatnonzero(condition).compute()
data[idx] *= -1

Error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/miguel/.conda/envs/pyralysis-env/lib/python3.12/site-packages/dask/array/core.py", line 1926, in __setitem__
    dsk = setitem_array(out, self, key, value)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miguel/.conda/envs/pyralysis-env/lib/python3.12/site-packages/dask/array/slicing.py", line 1843, in setitem_array
    raise ValueError(
ValueError: could not broadcast input array from shape(9961, 3) into shape (3,)

Anything else we need to know?:
The example above works with numpy as backend, works with only numpy and cupy.

Environment:

  • Dask version: dask==2024.6.0
  • Python version: Python 3.12.4
  • Operating System: Manjaro Linux x86_64
  • Install method (conda, pip, source): conda environment, package installed via pip
@github-actions github-actions bot added the needs triage Needs a response from a contributor label Jul 31, 2024
@quasiben
Copy link
Member

quasiben commented Aug 5, 2024

Hmm, in the example above, vis is not defined. If I replace vis with data I do get the error you posted above. I'm not sure what's going on yet. When @rjzamora returns from PTO maybe he has an idea. I'll spend some time investigating and see I can figure out what's going on

@miguelcarcamov
Copy link
Author

Hmm, in the example above, vis is not defined. If I replace vis with data I do get the error you posted above. I'm not sure what's going on yet. When @rjzamora returns from PTO maybe he has an idea. I'll spend some time investigating and see I can figure out what's going on

Yep, sorry I have fixed the example above so users can reproduce the error. Thank you for investigating the issue :)

@phofl phofl added gpu and removed needs triage Needs a response from a contributor labels Aug 7, 2024
@rjzamora rjzamora added array bug Something is broken labels Aug 14, 2024
@rjzamora
Copy link
Member

As far as I can tell, setitem_array (used to build the graph for the data[idx] *= -1 assignment) uses some code that isn't compatible with cupy. For example, there is a np.ndarray instance check instead of a is_arraylike.

There also seems to be very recent code that uses numpy explicitly in a way that will not work when the underlying data is cupy (for example #11267 and #11291 will further break your example). cc @phofl (just for awareness - not because you have any responsibility to fix anything)

I'll try to find a way to address the functionality described in this issue, but it does look like there are a large number of other legitimate dask-array failures in gpuCI at the moment as well...

@miguelcarcamov
Copy link
Author

@rjzamora thank you for the answer, now the bug is more clear to me, let me know once there is some fixes on this please :)

@rjzamora rjzamora mentioned this issue Oct 9, 2024
Merged
3 tasks
@rjzamora
Copy link
Member

rjzamora commented Oct 9, 2024

let me know once there is some fixes on this please

Thanks for the bump @miguelcarcamov - I'm hopeful that #11421 will resolve this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
array bug Something is broken gpu
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@quasiben @miguelcarcamov @rjzamora @phofl