Make running individual Python tests less painful #6791

soltanmm · 2016-06-03T22:19:40Z

Invoke ./tools/run_tests/build_python.sh. Invoke ./py27/bin/python -m $TEST_MODULE_NAME. Do a not-sucked-into-/dev/null-dance.

soltanmm · 2016-06-03T22:34:14Z

n.b. most of the additional lines are top-level directory duplications to work around the lack of namespace package support in auditwheel (and, in a strange twist of fate, this hack works with wheels, but not with installing from source, so build_python.sh needs to do a double-install-dance, but users won't). Those particular additions are not necessary for this PR per se, but I found it easier to fix them in this lump of work rather than after the fact.

The only tests I'm unsure about succeeding are the performance smoke tests. Following that the DNM will be removed and reviewers assigned.

soltanmm · 2016-06-04T01:40:26Z

I... dislike... Mac OS X and it's almost-being-Linux-but-everyone-treating-it-slightly-differently-cascading-into-my-face. It is... unpleasant.

soltanmm · 2016-06-04T02:22:47Z

@nathanielmanistaatgoogle This was basically what I rebased your commits on top of for debugging poirposes 🐬.

nathanielmanistaatgoogle · 2016-06-04T21:07:40Z

tools/distrib/python/grpcio_tools/grpc/tools/protoc.py

+  """Run the protocol buffer compiler with the given command-line arguments.
+
+  Args:
+    command_arguments: a list (yes, it *must* be a list) of strings representing


No need for parenthetical, although I appreciate the sentiment.

nathanielmanistaatgoogle · 2016-06-04T21:23:22Z

"Make running individual Python tests less painful" is too brief a commit log message - may I trouble you for "before, the difficulty was ..." and "with this change, the developer experience is ..." embellishments?

jtattermusch · 2016-06-04T22:47:35Z

How exactly is running individual python tests painful? Are you aware you can do run_tests.py -l python -r REGEX where regex is matched against the name of the python test?

jtattermusch · 2016-06-04T22:49:25Z

Btw, ./tools/run_tests/build_python.sh is not something you should invoke yourself (at least that's the idea for other languages and I don't think python should be different). If you want to just build, you can run run_tests.py --build_only, which is going to work well for other languages as well.

soltanmm · 2016-06-04T22:49:29Z

@jtattermusch The output gets eaten on interrupt, the PID of the process has to be guessed at so GDB attachment takes some guesswork on each run, and working around any of those requires an encyclopedic knowledge of our environment variable names. These details will be added in the updated commit message.

soltanmm · 2016-06-04T22:55:54Z

@jtattermusch Build-only might be available as an option on run_tests, yes, but the point is that whatever it generates at the current time results in a mishmash of difficulties for getting something as simple as process attachment via gdb (or cygdb, which is also indirectly addressed here; run_tests doesn't provide debug options to generate the virtualenvs while build_python.sh is low-enough-level that it can by invoking against the right python build [usually python-dbg]). Even if I wanted to modify run_tests to work with --build_only for debugging purposes, build_python.sh would still need the additional capabilities to handle that (and that's what's shown here).

soltanmm · 2016-06-04T23:39:52Z

Rebased. Also removed tox as it's now unused.

soltanmm · 2016-06-05T01:04:50Z

Also the changes up through ecc3cfa will play a major role in getting the Python-on-Windows tests to work on Jenkins^{Because indirect-things-cascading-into-everything-else} (coming soon to a PR near you).

kpayson64 · 2016-06-05T20:17:29Z

I'm not fully convinced that putting unit tests in a separate module is best way to go. In most (all) python modules I've encountered, unit tests get packaged with the module they test. I can't find much discussion of the pros/cons to this approach online, so to be solidly convinced this is the way we should be doing things, I would need more information around what specifically doesn't work, and why a separate package is the best (or only) way to solve that problem.

The two problems that I understand are addressed here are Windows testing and gdb support. I don't have much context for the Windows issues, and it seems like the build_python.sh changes combined with gdb --args python <unit-test> is sufficient to use gdb on a unit test.

Also, I think we should hold off on merging this until the namespace issues that cause us to duplicate large chunks of code into grpcio_tools are resolved.

kpayson64 · 2016-07-01T15:44:26Z

setup.py

@@ -119,9 +116,13 @@
 # For Python3.4, this is OSX 10.6, but we need Thread Local Support (__thread)
 if 'darwin' in sys.platform and PY3:
  mac_target = sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET')
+  # The working computer's mac os x version defines what frameworks are
+  # available to build against.
+  mac_version = platform.mac_ver()[0]


Can a mac not build against older frameworks? I'm concerned about us compiling binaries against a high MACOSX_DEPLOYMENT_TARGET. I'm pretty sure our macos jenkins workers are post 10.7, but they are able to build against that target.

I got errors building locally against 10.7 (missing standard library header files). If that sounds familiar to you and you have an environment fix s.t. the proper configuration is the builder's responsibility, I'd be more than happy to go back to 10.7.

Arguably, it should be able to (heck, it builds against 10.6 sans __thread), but for whatever reason I kept getting tripped up by missing headers.

kpayson64 · 2016-07-01T15:57:10Z

Just the one comment, then LGTM.

kpayson64 · 2016-07-01T17:01:02Z

LGTM

Moves all tests into a separate package. This does not change existing supported means of running tests (e.g. through run_tests.py).

Before this change, running Python tests individually required building a tox environment via the run_tests script and then specifying long environment variables to filter out just the test we wanted to run (and then we wouldn't be able to get the output on interrupt, nor would we have an easy way of determining the PID of the process for debugger attachment). Now invoking the build_python.sh script creates a workable python virtual environment that includes all necessary libraries and tests (s.t. running a single test is now possible by just knowing the module name). This does not change existing supported means of running tests (e.g. through run_tests.py). An additional way of running individual tests has been introduced. Following invocation of `./tools/run_tests/build_python.sh` (or run_tests.py), one may invoke ./$VENV/bin/python -m $TEST_MODULE_NAME and acquire a single running process that *is* the test process (rather than a parent of the process). $VENV is the virtual environment name specified to `build_python.sh` (defaults to `py27`) and $TEST_MODULE_NAME is what it says on the tin.

We'll need to fix coverage testing in the future anyway (see grpc#6894).

soltanmm · 2016-07-02T01:28:17Z

All possibly relevant test failures are flakes of the Jenkins infrastructure. Marking ready to merge.

nathanielmanistaatgoogle · 2016-07-02T13:26:57Z

Luhguhtum.

Woohoo!

soltanmm added lang/Python disposition/DO NOT MERGE labels Jun 3, 2016

googlebot added the cla: yes label Jun 3, 2016

soltanmm force-pushed the pain branch 3 times, most recently from 25aeac4 to cc10012 Compare June 4, 2016 01:37

soltanmm force-pushed the pain branch from cc10012 to 4a90ecb Compare June 4, 2016 01:47

soltanmm removed the disposition/DO NOT MERGE label Jun 4, 2016

soltanmm assigned nathanielmanistaatgoogle and kpayson64 Jun 4, 2016

nathanielmanistaatgoogle reviewed Jun 4, 2016
View reviewed changes

soltanmm force-pushed the pain branch from 4a90ecb to cb2b312 Compare June 4, 2016 22:51

soltanmm force-pushed the pain branch 2 times, most recently from c7a2629 to 01d37bb Compare June 4, 2016 23:33

soltanmm mentioned this pull request Jul 1, 2016

Fix Python interop unit-tests on Windows #7195

Merged

kpayson64 reviewed Jul 1, 2016
View reviewed changes

soltanmm force-pushed the pain branch from 8823695 to 9083c63 Compare July 1, 2016 16:42

soltanmm force-pushed the pain branch from 9083c63 to 16180d5 Compare July 1, 2016 17:15

soltanmm-google added 11 commits July 1, 2016 10:18

Build Python3 grpcio-tools on OS X

d5fee35

Add programmatic access to protoc in grpcio-tools

ebf81e7

Ignore cython debug information

b926ef2

Organize Python tests to use grpcio-tools directly

1ff429d

Moves all tests into a separate package. This does not change existing supported means of running tests (e.g. through run_tests.py).

Remove tox

ac586ba

Sanitize grpcio-tools command arguments

6db60b9

Fall back to default python for test virtualenvs

0bd13ed

Remove gcov special-casing for Python tests

1c062bd

We'll need to fix coverage testing in the future anyway (see grpc#6894).

Fix job_spec invocation for Python run_tests

e6a23e2

Regenerate project files

4763678

soltanmm force-pushed the pain branch from 16180d5 to 4763678 Compare July 1, 2016 17:18

soltanmm mentioned this pull request Jul 1, 2016

Make grpcio-tools compile targeting Mac OS X 10.6 #7208

Closed

soltanmm added the disposition/ready to merge label Jul 2, 2016

nathanielmanistaatgoogle mentioned this pull request Jul 2, 2016

Make handlers optional at server construction #7160

Merged

soltanmm mentioned this pull request Jul 6, 2016

Added str->byte conversion in grpcio_tools #7237

Closed

soltanmm assigned nicolasnoble Jul 6, 2016

kpayson64 merged commit dcca468 into grpc:master Jul 6, 2016

soltanmm deleted the pain branch July 6, 2016 18:10

matt-kwong mentioned this pull request May 2, 2018

Running tests with Python3.5 and 3.6 actually runs with 2.7 #15253

Closed

lock bot locked as resolved and limited conversation to collaborators Jan 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make running individual Python tests less painful #6791

Make running individual Python tests less painful #6791

soltanmm commented Jun 3, 2016

soltanmm commented Jun 3, 2016

soltanmm commented Jun 4, 2016

soltanmm commented Jun 4, 2016

nathanielmanistaatgoogle Jun 4, 2016

nathanielmanistaatgoogle commented Jun 4, 2016

jtattermusch commented Jun 4, 2016

jtattermusch commented Jun 4, 2016

soltanmm commented Jun 4, 2016

soltanmm commented Jun 4, 2016

soltanmm commented Jun 4, 2016

soltanmm commented Jun 5, 2016

kpayson64 commented Jun 5, 2016

kpayson64 Jul 1, 2016

soltanmm Jul 1, 2016

soltanmm Jul 1, 2016

kpayson64 commented Jul 1, 2016

kpayson64 commented Jul 1, 2016

soltanmm commented Jul 2, 2016

nathanielmanistaatgoogle commented Jul 2, 2016

Make running individual Python tests less painful #6791

Make running individual Python tests less painful #6791

Conversation

soltanmm commented Jun 3, 2016

soltanmm commented Jun 3, 2016

soltanmm commented Jun 4, 2016

soltanmm commented Jun 4, 2016

nathanielmanistaatgoogle Jun 4, 2016

Choose a reason for hiding this comment

nathanielmanistaatgoogle commented Jun 4, 2016

jtattermusch commented Jun 4, 2016

jtattermusch commented Jun 4, 2016

soltanmm commented Jun 4, 2016

soltanmm commented Jun 4, 2016

soltanmm commented Jun 4, 2016

soltanmm commented Jun 5, 2016

kpayson64 commented Jun 5, 2016

kpayson64 Jul 1, 2016

Choose a reason for hiding this comment

soltanmm Jul 1, 2016

Choose a reason for hiding this comment

soltanmm Jul 1, 2016

Choose a reason for hiding this comment

kpayson64 commented Jul 1, 2016

kpayson64 commented Jul 1, 2016

soltanmm commented Jul 2, 2016

nathanielmanistaatgoogle commented Jul 2, 2016