Implement a sqlite-based store for the cache #6023
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File system operations on OS X are pretty slow, and untarring a large archive of mypy cache information can get pretty slow. Work around this by using a sqlite database to store the entire cache in one file. To do this we introduce a generic interface for storing metadata, called `MetadataStore`. It presents an essentially key/value interface. We provide two implementations, one using the existing file system backing and one using sqlite. It is enabled with the option `--sqlite-cache`, but is not the default yet. I'm not sure what the right thing to do about testing is. I've tested it with the default changed, and everything passes.
ilevkivskyi
approved these changes
Dec 7, 2018
misc/convert-cache.py
Outdated
|
|
||
| for s in input.list_all(): | ||
| # print("Copying", s) | ||
| assert output.write(s, input.read(s), input.getmtime(s)) |
There was a problem hiding this comment.
I would give a reasonable error message if this fails. Also, do we need to perform a cleanup for partially written cache to disk in this case?
There was a problem hiding this comment.
It's just a debugging script, so I dunno if it matters?
There was a problem hiding this comment.
Up to you, but if we expect that someone except us will use it (I assume), then think it is better to do it.
| structure of files. | ||
| * A hokey sqlite backed implementation, which basically simulates | ||
| the file system in an effort to work around poor file system performance | ||
| on OS X. |
There was a problem hiding this comment.
What about Windows? Is it also fast or also slow?
There was a problem hiding this comment.
I have no idea at all
mypy/metastore.py
Outdated
|
|
||
| class SqliteMetadataStore(MetadataStore): | ||
| def __init__(self, cache_dir_prefix: str) -> None: | ||
| if cache_dir_prefix.startswith('/dev/null'): |
There was a problem hiding this comment.
Why startswith and not ==? I would also documment this behavior in a docstring.
gvanrossum
reviewed
Dec 7, 2018
emmatyping
reviewed
Dec 7, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
File system operations on OS X are pretty slow, and untarring a large
archive of mypy cache information can get pretty slow. Work around
this by using a sqlite database to store the entire cache in one file.
To do this we introduce a generic interface for storing metadata,
called
MetadataStore. It presents an essentially key/valueinterface. We provide two implementations, one using the existing file
system backing and one using sqlite.
It is enabled with the option
--sqlite-cache, but is not the defaultyet.
I'm not sure what the right thing to do about testing is. I've tested
it with the default changed, and everything passes.