Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pickling support to ScmProjectTree #3189

Closed
stuhood opened this issue Apr 14, 2016 · 2 comments
Closed

Add pickling support to ScmProjectTree #3189

stuhood opened this issue Apr 14, 2016 · 2 comments
Assignees

Comments

@stuhood
Copy link
Member

stuhood commented Apr 14, 2016

ScmProjectTree is very not-pickleable, primarily because it eagerly constructs a GitReader object.

Lazily constructing a GitReader and/or excluding the reader from pickling would go a long way.

@stuhood stuhood added the engine label Apr 14, 2016
stuhood added a commit that referenced this issue Apr 19, 2016
In order to properly invalidate for watchman events, we need to record that we walked symlinks in the product graph. This review adds symlink handling to `engine.exp.fs`, and improves the type safety of filesystem operations by requiring that a PathGlobs object matches exactly one of `Files`, `Dirs` or `Links`.

- Replace `fs.Path` with `fs.Stat` subclasses: `fs.{File,Dir,Link}`
- Add explicit support to PathGlobs for matching directories. A PathGlobs object matches either Files or Dirs, but not both.
- Resolve symlinks by recursively requesting `Files`/`Dirs` for a `Path` projected from a `ReadLink`
- Replace `RecursiveSubDirectories` with use of a `PathGlobs` object recursively matching `Dirs`.
- Add ProjectTree.{lstat, readlink, listdir}
- Split `test_fs.py` from `test_path_globs.py`, and prepare to test ScmProjectTree (see [#3189](#3189))
- Include tests in `test_fs.py` to validate the actual filesystem events that occurred.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/124108442

Bugs closed: 3121, 3190

Reviewed at https://rbcommons.com/s/twitter/r/3691/
@stuhood
Copy link
Member Author

stuhood commented Apr 26, 2016

This codepath will be heavily exercised by https://github.com/pantsbuild/pants/blob/master/tests/python/pants_test/engine/exp/test_fs.py#L164 once it is pickleable.

@peiyuwang
Copy link
Contributor

Submitted as 265d140

Reviewed at https://rbcommons.com/s/twitter/r/3770/
Will follow up the remaining test failures in #3281

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants