You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Basically, the embedding of keys into nested data structures creates a large overhead of task objects. For the workload I investigated, this appears to have contributed up to 50% of all tasks. (Take that number with a grain of salt.)
We should avoid using these nested data structures for keys entirely. In array-code, I've identified the usage of concrete (the example above) as a culprit that can be trivially removed. However, concatenate3 and related functions are other culprits that require a bit more rewriting.
The text was updated successfully, but these errors were encountered:
While investigating dask/distributed#8958, I noticed this:
Basically, the embedding of keys into nested data structures creates a large overhead of task objects. For the workload I investigated, this appears to have contributed up to 50% of all tasks. (Take that number with a grain of salt.)
We should avoid using these nested data structures for keys entirely. In array-code, I've identified the usage of
concrete
(the example above) as a culprit that can be trivially removed. However,concatenate3
and related functions are other culprits that require a bit more rewriting.The text was updated successfully, but these errors were encountered: