Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use functools.cache more #11554

Open
dcherian opened this issue Nov 21, 2024 · 4 comments
Open

Use functools.cache more #11554

dcherian opened this issue Nov 21, 2024 · 4 comments
Labels
needs triage Needs a response from a contributor

Comments

@dcherian
Copy link
Contributor

It'd be great to stick a cache on dask.array.svg.svg and normalize_chunks.
These seem like highly cacheable functions, and would help with pydata/xarray#8902. The svg would help with the ds._repr_html_() of that dataset.
@github-actions github-actions bot added the needs triage Needs a response from a contributor label Nov 21, 2024
@phofl phofl mentioned this issue Nov 22, 2024
Closed
3 tasks
@fjetter
Copy link
Member

fjetter commented Nov 22, 2024

normalize_chunks is a bit weird since it accepts all kinds of input, some of them are mutable and therefore not hashable

@fjetter
Copy link
Member

fjetter commented Nov 22, 2024

If it's always a tuple in xarray we can likely offer a "cachable" version (or we can cache on a best effort basis)

@dcherian
Copy link
Contributor Author

We are indeed sending dask tuples always!

https://github.com/pydata/xarray/blob/1bb867d573390509dbc0379f0fd318a6985dab45/xarray/core/dataset.py#L246-L258

preferred_chunks comes from the storage library is always a tuple of ints.

chunk_shape can be either a tuple of ints or with some "auto" sprinkled in : ('auto', 721, 1440)

@Illviljan
Copy link
Contributor

#10648 moves stuff around to encourage using the normalized shape.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
needs triage Needs a response from a contributor
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dcherian @fjetter @Illviljan