Initial adoption of Ginkgo in Kubernetes e2e tests #3855
Conversation
|
@filbranden this looks to be a reasonable proposal. Is there is time frame for the complete conversion? We don't want to add to the conversion work by continuing either bash scripts or test conversions to go w/o ginkgo/gomega if that is the intended end goal, so I'd suggest adding to the proposal that all future e2e work be in ginkgo/gomega while porting of old tests is done to ginkgo/gomega. |
|
Can you add a parameter for the report file rather than just hardcoding it as I haven't really reviewed yet, either, just giving this a once over for what I'd want out of it. |
|
@zmerlynn Ok I'll include a --reports_dir or similar flag and only enable the JUnit custom reporter if that path is set. I'm leaning towards doing a flag with a directory name instead of a file name since when we run parallel tests one XML is generated per worker: http://onsi.github.io/ginkgo/#generating-junit-xml-output I think specifying a directory should be enough control for us, I don't see a reason why we would need the files to be called something other than |
|
@satnam6502 can you please take a look at the code in particular |
|
Generating a directory is fine, Jenkins can slurp entire wildcards. |
|
The godeps stuff is voodoo to me. The rest LGTM, modulo the --reports_dir. You still owe me an |
|
I've looked at driver.go and as far as I can tell (which might not be very far) I think it is fine. |
|
One thing I can't tell from this is whether we continue to create a fresh client for each test. Is that still the case? |
|
@satnam6502 Yes still the case, each test gets See this one here for example: |
|
That's fine. I was wrong about the rate limit being applied to each client connection. I think the rate limit applies to each instance of the apiserver. So having separate clients is not buying us anything here. So it is fine I think to switch back to using a shared client for all the tests -- if that makes things easier or run faster. |
This commit uses `godep` to vendor these libraries into our project. They are not yet in use but we plan to use them in `test/e2e` in a follow up commit.
In order to adopt ginkgo incrementally, let's start by replacing test/e2e/driver.go with a call to ginkgo runner and convert each of the other tests to a small Decscribe() snippet that simply calls the legacy TestXYZ function. From this basis we can take further incremental steps by converting individual tests to native ginkgo format, using Fail() for all failure cases, using By() for logs, enabling JUnit reports, etc. Tested: - cmd/e2e builds and `make check` passes. - Running _output/bin/.../e2e on an alive cluster works. - Running the full hack/e2e-test.sh works as expected.
Use ginkgo's native support for JUnit in order to generate the XML file. This is a first step in better integration of our e2e tests with Jenkins. In order to improve the logged information, we will probably need to have more native ginkgo tests but this step allows us to see what Jenkins can already do with this information and what we need to tweak to improve it. Tested by running the full e2e tests and inspecting the contents of junit.xml on the top of the tree. Textual output is still generated on the console to keep the current goe2e.sh logs available until the full conversion of our Jenkins instance to use the JUnit XML is completed.
|
OK I added --report_dir and modified the Jenkins setup to save the junit.xml file on top of the Jenkins tree. PTAL, I'd like to get this one merged today so that we can start making progress on converting the Go tests to native Ginkgo. Tested by running the full |
|
LGTM except for minor nit on last commit. |
Use the E2E_REPORT_DIR global environment variable to define the
location where the JUnit XML reports should be saved.
Modify the Jenkins e2e.sh script to export that variable pointing to the
top of the Jenkins build tree.
Tested by running `E2E_REPORT_DIR=${PWD}/.. hack/e2e-test.sh` and
confirmed ../junit.xml was generated and looked good.
|
LGTM. Travis isn't going to say anything about hack/jenkins, so I'm merging this so we can see if we broke anything quickly. |
Initial adoption of Ginkgo in Kubernetes e2e tests
|
Awesome thanks! @rrati I think this means you can send us a PR to bring e.g. https://github.com/rrati/kubernetes/blob/ginkgo/test/ginkgo/pods_test.go into our tree. (I wouldn't do network_test.go just now because that test has been flaky for a while... I think the others are better candidates.) |
|
I believe I've also modified the Jenkins config to slurp junit*.xml (which On Wed Jan 28 2015 at 4:35:07 PM Filipe Brandenburger <
|
|
@zmerlynn I'm really looking forward to seeing this! Will be very helpful and at least solve one of our current P0's which is the visibility of what inside goe2e.sh went bonkers when it fails... :-) |
These commits are a first step in migrating e2e tests to Ginkgo and implementing #3837.
The last commit also starts generating JUnit XML output for the tests which should allow for cleaner better integration of the e2e tests with Jenkins.
My proposal is:
hack/e2e-suite/and ofhack/e2e.go, having a single standalonecmd/e2ethat can be used to run all or a subset of the end-to-end tests on an existing test cluster.@satnam6502 @rrati @timothysc @roberthbailey @zmerlynn