Use python to define the E2E test workflow for kfctl. #4148

jlewi · 2019-09-18T22:53:45Z

We need to write an E2E test for kfctl upgrade (E2E test for kfctl upgrade kfctl#35).

Before we do that we want to remove ksonnet from our existing E2E test.

The E2E test for upgrades will be very similar to the kfctl E2E test
so it makes sense to convert that test to python before writing
the upgrade test.
Related to Migrate E2E tests off ksonnet #3035 migrate e2e tests off ksonnet

This change is

jlewi · 2019-09-23T15:14:18Z

So close.
Workflow ran and most steps completed successfully
http://testing-argo.kubeflow.org/workflows/kubeflow-test-infra/kubeflow-presubmit-kfctl-go-iap-istio-4148-96825a4-7760-946c?tab=workflow

It looks like delete test-dir failed.

No longs show up if we filter by pod label (probably because of the known stackdriver issue).
But if we get logs for the pod id it works.

The error is

rm: cannot remove '/mnt/test-data-volume/kubeflow-presubmit-kfctl-go-iap-istio-4148-96825a4-7760-946c/src/kubeflow/kubeflow/output/artifacts/logs/.nfs000000000070af1c0000003b': Device or resource busy

jlewi · 2019-09-23T15:21:29Z

Basic auth
http://testing-argo.kubeflow.org/workflows/kubeflow-test-infra/kubeflow-presubmit-kfctl-go-basic-auth-4148-96825a4-7760-9d9a?tab=workflow

Failed on build-deploy the error is

INFO Error: couldn't generate KfApp: (kubeflow.error): Code 500 with message: coordinator Generate failed for gcp: Could not configure basic auth; environment variable KUBEFLOW_USERNAME not set

jlewi · 2019-09-25T18:42:57Z

Looks like both basic-auth and iap workflows ran and only the step test_dir delete failed.

jlewi · 2019-09-25T20:55:25Z

Tests passed
https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubeflow_kubeflow/4148/kubeflow-presubmit/1176949535604215811/

* We need to write an E2E test for kfctl upgrade (kubeflow/kfctl#35). Before we do that we want to remove ksonnet from our existing E2E test. The E2E test for upgrades will be very similar to the kfctl E2E test so it makes sense to convert that test to python before writing the upgrade test. * Related to kubeflow#3035 migrate e2e tests off ksonnet * Code to setup default profile should not return an error if the secret already exists. This prevents us from calling apply multiple times. * Related to kubeflow/issues/3810; kfctl apply fails if we rerun it * Fixing working directory of juptyer test. * Update prow_config.yaml to use the new py_funcs for the kfctl_e2e tests * Update kfctl_go_test.py to determine whether we are using basic_auth from the KFDef spec rather than the command line arguments.

jlewi · 2019-09-25T21:11:26Z

/assign @gabrielwen
/assign @yanniszark

@yanniszark @gabrielwen This is ready for review. The tests passed before I rebased. The current run was triggered by the rebase but I expect them to pass as well.

gabrielwen

/lgtm

gabrielwen · 2019-09-26T17:35:15Z

py/kubeflow/kubeflow/ci/kfctl_e2e_workflow.py

+               "--project=kubeflow-ci",
+               # TODO(jlewi): Do we need a GITHUB_TOKEN? I'm guessing that
+               # was for ksonnet.
+               # "--github_token=$(GITHUB_TOKEN)",


seems like we can remove this?

gabrielwen · 2019-09-26T17:35:36Z

py/kubeflow/kubeflow/ci/kfctl_e2e_workflow.py

+                "testing.test_deploy",
+                "--project=kubeflow-ci",
+                # TODO(jlewi): Do we still need a GITHUB_TOKEN?
+                # "--github_token=$(GITHUB_TOKEN)",


* Once kfctl upgrade is ready we will have to update this test to actually run. * kubeflow#4148 needs to be submitted first. * We will also need to update the test after kubeflow#4187 is merged. That splits kfctl_go_test.py into separate build and deploy steps.

jlewi · 2019-09-26T21:15:38Z

@yanniszark Did you want to take a look at this?

gabrielwen · 2019-09-27T06:12:02Z

/lgtm

jlewi · 2019-09-27T09:35:31Z

/approve

k8s-ci-robot · 2019-09-27T09:37:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jlewi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jlewi · 2019-09-27T11:42:33Z

Looks like the workflow
http://testing-argo.kubeflow.org/workflows/kubeflow-test-infra/kubeflow-presubmit-jupyterui-release-4148-1909805-8608-4f44?tab=workflow

Failed. I'm not sure why we are even running that workflow got triggered. I suspect an issue with computing the modified files because of the clone depth.

jlewi · 2019-09-27T11:42:39Z

/test all

jlewi · 2019-09-27T11:46:36Z

INFO|2019-09-27T09:38:16|/src/kubeflow/testing/py/kubeflow/testing/util.py|45| Running: git merge-base HEAD remotes/origin/master 
cwd=/src/kubeflow/kubeflow
INFO|2019-09-27T09:38:16|/src/kubeflow/testing/py/kubeflow/testing/util.py|60| Subprocess output:

WARNING|2019-09-27T09:38:17|/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py|164| git merge-base failed; see https://github.com/kubeflow/kubeflow/issues/3523. Diff will be computed against the current master and therefore files not changed in the PR might be considered when determining which tests to trigger
...
INFO|2019-09-27T09:39:30|/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py|239| Triggering workflow jupyterui-release because components/jupyter-web-app/backend/kubeflow_jupyter/common/api.py in dir components/jupyter-web-app/* is modified.

* Use python to define the E2E test workflow for kfctl. * We need to write an E2E test for kfctl upgrade (kubeflow/kfctl#35). Before we do that we want to remove ksonnet from our existing E2E test. The E2E test for upgrades will be very similar to the kfctl E2E test so it makes sense to convert that test to python before writing the upgrade test. * Related to kubeflow#3035 migrate e2e tests off ksonnet * Code to setup default profile should not return an error if the secret already exists. This prevents us from calling apply multiple times. * Related to kubeflow/issues/3810; kfctl apply fails if we rerun it * Fixing working directory of juptyer test. * Update prow_config.yaml to use the new py_funcs for the kfctl_e2e tests * Update kfctl_go_test.py to determine whether we are using basic_auth from the KFDef spec rather than the command line arguments. * Address comments.

* We need to write an E2E test for kfctl upgrade (kubeflow/kfctl#35). * Once kfctl upgrade is ready we will have to update this test to actually run. * kubeflow#4148 needs to be submitted first. * We will also need to update the test after kubeflow#4187 is merged. That splits kfctl_go_test.py into separate build and deploy steps.

* Use python to define the E2E test workflow for kfctl. * We need to write an E2E test for kfctl upgrade (kubeflow/kfctl#35). Before we do that we want to remove ksonnet from our existing E2E test. The E2E test for upgrades will be very similar to the kfctl E2E test so it makes sense to convert that test to python before writing the upgrade test. * Related to kubeflow#3035 migrate e2e tests off ksonnet * Code to setup default profile should not return an error if the secret already exists. This prevents us from calling apply multiple times. * Related to kubeflow/issues/3810; kfctl apply fails if we rerun it * Fixing working directory of juptyer test. * Update prow_config.yaml to use the new py_funcs for the kfctl_e2e tests * Update kfctl_go_test.py to determine whether we are using basic_auth from the KFDef spec rather than the command line arguments. * Address comments.

k8s-ci-robot added the do-not-merge/work-in-progress label Sep 18, 2019

googlebot added the cla: yes label Sep 18, 2019

k8s-ci-robot added the size/L label Sep 18, 2019

k8s-ci-robot requested review from lluunn and pdmack September 18, 2019 22:54

jlewi force-pushed the kfctl_py_test branch from 1faeee6 to bed4311 Compare September 19, 2019 18:25

k8s-ci-robot added size/XL and removed size/L labels Sep 19, 2019

This was referenced Sep 19, 2019

Use a py_func and not ksonnet to define the unittest E2E workflow; update kf_unittests to create a valid workflow kubeflow/testing#463

Merged

Split kfctl_go_test.py into separate python functions for building and deploying kubeflow kubeflow/kfctl#36

Open

jlewi force-pushed the kfctl_py_test branch from b1dcdb4 to 96825a4 Compare September 22, 2019 14:37

jlewi force-pushed the kfctl_py_test branch from 96825a4 to 2ed8bd0 Compare September 24, 2019 00:33

jlewi force-pushed the kfctl_py_test branch from b7c3aac to 18aa073 Compare September 25, 2019 21:09

jlewi marked this pull request as ready for review September 25, 2019 21:10

k8s-ci-robot removed the do-not-merge/work-in-progress label Sep 25, 2019

k8s-ci-robot assigned gabrielwen and yanniszark Sep 25, 2019

This was referenced Sep 26, 2019

kfctl#36 Split build and deploy to separate tests #4187

Merged

e2e: existing_arrikto: initial e2e test #4154

Merged

gabrielwen reviewed Sep 26, 2019

View reviewed changes

k8s-ci-robot added the lgtm label Sep 26, 2019

jlewi mentioned this pull request Sep 26, 2019

Kfctl upgrade py test #4191

Closed

Address comments.

1909805

k8s-ci-robot removed the lgtm label Sep 26, 2019

k8s-ci-robot added the lgtm label Sep 27, 2019

k8s-ci-robot added the approved label Sep 27, 2019

jlewi mentioned this pull request Sep 27, 2019

Files modified in PR not computed correctly because clone depth = 1 kubeflow/testing#473

Closed

k8s-ci-robot merged commit 4a3766b into kubeflow:master Sep 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use python to define the E2E test workflow for kfctl. #4148

Use python to define the E2E test workflow for kfctl. #4148

jlewi commented Sep 18, 2019 •

edited

Loading

jlewi commented Sep 23, 2019

jlewi commented Sep 23, 2019

jlewi commented Sep 25, 2019

jlewi commented Sep 25, 2019

jlewi commented Sep 25, 2019

gabrielwen left a comment

gabrielwen Sep 26, 2019

gabrielwen Sep 26, 2019

jlewi commented Sep 26, 2019

gabrielwen commented Sep 27, 2019

jlewi commented Sep 27, 2019

k8s-ci-robot commented Sep 27, 2019

jlewi commented Sep 27, 2019

jlewi commented Sep 27, 2019

jlewi commented Sep 27, 2019

Use python to define the E2E test workflow for kfctl. #4148

Use python to define the E2E test workflow for kfctl. #4148

Conversation

jlewi commented Sep 18, 2019 • edited Loading

jlewi commented Sep 23, 2019

jlewi commented Sep 23, 2019

jlewi commented Sep 25, 2019

jlewi commented Sep 25, 2019

jlewi commented Sep 25, 2019

gabrielwen left a comment

Choose a reason for hiding this comment

gabrielwen Sep 26, 2019

Choose a reason for hiding this comment

gabrielwen Sep 26, 2019

Choose a reason for hiding this comment

jlewi commented Sep 26, 2019

gabrielwen commented Sep 27, 2019

jlewi commented Sep 27, 2019

k8s-ci-robot commented Sep 27, 2019

jlewi commented Sep 27, 2019

jlewi commented Sep 27, 2019

jlewi commented Sep 27, 2019

jlewi commented Sep 18, 2019 •

edited

Loading