Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rerun a subset of keys #800

Open
pepeiborra opened this issue Mar 14, 2021 · 3 comments
Open

Rerun a subset of keys #800

pepeiborra opened this issue Mar 14, 2021 · 3 comments

Comments

@pepeiborra
Copy link
Contributor

pepeiborra commented Mar 14, 2021 •
edited
Loading

I want a version of shakeRunDatabase that takes in a set of rule keys to use as a dirtiness seed. Shake should then rerun only these rules and anything that depends on them, and assume that all other rule keys haven't changed.

The expected Shake overhead should be proportional to the number of rules dirtied and not to the number of rule dependencies.

It is ok if a Shake option is needed to enable this functionality and results in a higher space/time overhead in other operations
@pepeiborra
Copy link
Contributor Author

I assume that this will require extending Shake to track reverse dependencies. @hsenag given your dag experience, do you know what are the pitfalls here?

@pepeiborra
Copy link
Contributor Author

I have a rough, but working implementation of this. PR soon!

@pepeiborra pepeiborra mentioned this issue Mar 16, 2021
Closed
@hsenag
Copy link

hsenag commented Mar 21, 2021

I assume that this will require extending Shake to track reverse dependencies. @hsenag given your dag experience, do you know what are the pitfalls here?

I guess the main thing is that it adds more overhead. Firstly the obvious point about space/time to store and update the reverse dependencies. But also having a bidirectionally connected graph significantly increases the chances of memory leaks and means you need more complicated locking and update protocols to avoid races. I don't know much about the Shake graph and haven't looked at the detail of your PR, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Collect reverse dependencies pepeiborra/shake
2 participants
@pepeiborra @hsenag