Use Argo rather than Airflow to run our E2E tests #358

jlewi · 2018-01-31T13:22:27Z

An Argo workflow to replace our Airflow workflow for running E2E tests
Create a ksonnet application to manage our E2E workflows.
Argo should be much better to manage and use for our E2E tests
Related to Use Argo workflow engine for CI/CD or releases #205
Need to modify some of the steps in the workflow to activate the service account.
Here's a passing run
kubeflow-test-infra/training-presubmit-1-5-01133157
Update the Docker image used by our prow jobs to trigger the workflow rather than use Airflow.
Delete all the files related to E2E pipelines using Airflow.

This change is

* This will facilitate replacing Airflow with Argo because it will allow us to not rely on xcom to pass information between steps. * Instead we will just pick a label based on the prow environment variables.

coveralls · 2018-01-31T13:34:05Z

Coverage remained the same at 31.746% when pulling b6c5efc on jlewi:argo_e2e into 330eb92 on tensorflow:master.

coveralls · 2018-01-31T13:34:05Z

Coverage remained the same at 31.746% when pulling 8227da5 on jlewi:argo_e2e into 330eb92 on tensorflow:master.

jimexist · 2018-01-31T13:56:34Z

py/release.py

+    n = datetime.datetime.now()
+    version_tag = n.strftime("v%Y%m%d") + "-" + commit
+  logging.info("Using version tag: %s", version_tag)
+  image = image_base + ":" + version_tag


are we using python 3.6 yet?

No; although I suspect updating our test infra to be 3.6 compatible shouldn't be too bad.

scripts defined within it.

Switch to HEAD on kubeflow_testing.

yupbank · 2018-02-02T22:18:21Z

Review status: 0 of 19 files reviewed at latest revision, 4 unresolved discussions, some commit checks failed.

.gitmodules, line 3 at r3 (raw file):

[submodule "kubeflow_testing"]
	path = kubeflow_testing
	url = https://github.com/jlewi/testing.git

do we want to leave a personal repo here?

py/test_runner.py, line 137 at r3 (raw file):

  logging.getLogger().setLevel(logging.INFO) # pylint: disable=too-many-locals

  if os.getenv("GOOGLE_APPLICATION_CREDENTIALS"):

Can we make this a ensure logged in with google function, since it's being used multi places

py/test_util.py, line 162 at r3 (raw file):

  else:
    dir_name = os.path.dirname(output_path)
    if not os.path.exists(dir_name):

this can simply use from distutils.dir_util import mkpath; mkpath(dir_name)

Comments from Reviewable

jlewi · 2018-02-02T22:43:04Z

Review status: 0 of 17 files reviewed at latest revision, 4 unresolved discussions.

py/test_runner.py, line 137 at r3 (raw file):

Previously, yupbank (Peng Yu) wrote…

Can we make this a ensure logged in with google function, since it's being used multi places

How does that work?

py/test_util.py, line 162 at r3 (raw file):

from distutils.dir_util import mkpath
I think I prefer not to rely on distutils just to make a directory.

.gitmodules, line 3 at r3 (raw file):

Previously, yupbank (Peng Yu) wrote…

do we want to leave a personal repo here?

Good point. I think we should actually get rid of using submodules.

Comments from Reviewable

yupbank · 2018-02-02T23:25:21Z

Review status: 0 of 17 files reviewed at latest revision, 3 unresolved discussions.

py/test_runner.py, line 137 at r3 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

How does that work?

def activate_service_account_if_credential_set():
    if not os.getenv("GOOGLE_APPLICATION_CREDENTIALS"):
        logging.info("GOOGLE_APPLICATION_CREDENTIALS is set; configuring gcloud "
            "to use service account.")
    util.run(["gcloud", "auth", "activate-service-account", "--key-file=" + os.getenv("GOOGLE_APPLICATION_CREDENTIALS")])

    and reuse this function  https://github.com/tensorflow/k8s/pull/358/files#diff-aff0df9f619a7c6e4d212b4d51634268R171

https://github.com/tensorflow/k8s/pull/358/files#diff-775fd702f5201a69d32ea89b5fa390f7R708
https://github.com/tensorflow/k8s/pull/358/files#diff-a21c288bd7c0a85dba041751952f4e29R137

Comments from Reviewable

yupbank · 2018-02-02T23:31:09Z

Review status: 0 of 17 files reviewed at latest revision, 3 unresolved discussions.

py/test_util.py, line 162 at r3 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

from distutils.dir_util import mkpath
I think I prefer not to rely on distutils just to make a directory.

In [3]: from distutils.dir_util import mkpath

In [4]: print mkpath.__doc__
Create a directory and any missing ancestor directories.

    If the directory already exists (or if 'name' is the empty string, which
    means the current directory, which of course exists), then do nothing.
    Raise DistutilsFileError if unable to create some directory along the way
    (eg. some sub-path exists, but is a file rather than a directory).
    If 'verbose' is true, print a one-line summary of each mkdir to stdout.
    Return the list of directories actually created.

this function is doing exactly what is being done here https://github.com/jlewi/k8s/blob/a6f02438aeda5af2b8f3db72bc95244363f890d2/py/test_util.py#L162-L173
and distutils is a standard lib of python

Comments from Reviewable

jose5918 · 2018-02-03T00:12:03Z

test/workflows/environments/kubeflow/spec.json

@@ -0,0 +1,4 @@
+{
+  "server": "https://35.196.185.88",


Where is this value being used?

This is a ksonnet application that allows you to run the workflows. This is an environment that points at the cluster we use for E2E testing.

jlewi · 2018-02-03T01:42:52Z

/test all

jlewi · 2018-02-03T01:49:58Z

/test all

jlewi · 2018-02-03T01:52:55Z

/test all

gaocegege

I found a huge file in the PR, which adds 56122 lines: test/workflows/environments/kubeflow/.metadata/swagger.json. And it contains the API docs for Kubernetes, do we need to add it to the repo?

jlewi · 2018-02-03T01:59:03Z

/test all

jlewi · 2018-02-03T02:03:34Z

/test all

jlewi · 2018-02-03T02:05:46Z

/test all

jlewi · 2018-02-03T02:07:55Z

@gaocegege Yes. That's part of the ksonnet APP and the point of ksonnet apps is to check it into source control so that your configs are versioned.

jlewi · 2018-02-03T02:20:37Z

/test all

jlewi · 2018-02-03T17:43:55Z

/test all

jlewi · 2018-02-03T18:21:05Z

/test all

jlewi · 2018-02-03T18:30:57Z

/test all

jlewi · 2018-02-03T18:34:56Z

/test all

jlewi · 2018-02-03T18:39:07Z

/test all

jlewi · 2018-02-04T01:28:40Z

/test all

jlewi · 2018-02-04T18:12:27Z

/test all

jlewi added 4 commits January 30, 2018 16:43

Add an option to release.py to specify the tag for the image to use.

50a42fd

* This will facilitate replacing Airflow with Argo because it will allow us to not rely on xcom to pass information between steps. * Instead we will just pick a label based on the prow environment variables.

Start writing an Argo workflow to replace the Airflow workflow.

3921dc2

Start working on the parameters.

82fb9a0

Add code to create symbolic link.

b6c5efc

jimexist reviewed Jan 31, 2018

View reviewed changes

jlewi added 3 commits January 31, 2018 06:25

Pull in kubeflow_testing as a submodule so that we can use the python

d986bf9

scripts defined within it.

latest.

ec01981

Fix test util.

f43273b

bhack mentioned this pull request Jan 31, 2018

Generate Workflow OpenAPI models and spec argoproj/argo-workflows#707

Closed

jlewi added 9 commits January 31, 2018 20:47

Use service account.

6ba2f44

run gcloud auth login

47129b3

Fix the path.

3907f32

Fix import.

8895927

Exclude submodule.

d75180d

Fix namespace.

94e8ee5

Merge remote-tracking branch 'upstream/master' into argo_e2e

1568b01

Add credentials.

cb6d0c4

Fix the teardown step.

6db9fbe

jlewi changed the title ~~[Wip] Argo workflow for running E2E tests~~ Argo workflow for running E2E tests Feb 1, 2018

jlewi requested a review from gaocegege February 1, 2018 22:05

Now that the python scripts are checked in

863e75b

Switch to HEAD on kubeflow_testing.

Remove kubeflow_testing as a submodule.

a6f0243

Location of kubeflow/testing is changed.

22cd048

jose5918 reviewed Feb 3, 2018

View reviewed changes

Use the service account function.

425594c

gaocegege reviewed Feb 3, 2018

View reviewed changes

jlewi added 9 commits February 3, 2018 10:51

Fix the workflow.

9c100f9

Fix it.

58a1e44

Fix paths.

2e096cc

Fix path.

5d87458

Set workingDir.

91a8c8e

Fix path.

754b50a

Fix the test.

8851265

Fix lint.

52182fe

Fix lint.

8227da5

jlewi added 2 commits February 4, 2018 10:33

Autoformat jsonnet files.

a21a009

Fix the README.

6f85585

jlewi merged commit 55058f2 into kubeflow:master Feb 4, 2018

jlewi mentioned this pull request Feb 4, 2018

Use Argo workflow engine for CI/CD or releases #205

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Argo rather than Airflow to run our E2E tests #358

Use Argo rather than Airflow to run our E2E tests #358

jlewi commented Jan 31, 2018 •

edited

Loading

coveralls commented Jan 31, 2018

coveralls commented Jan 31, 2018 •

edited

Loading

jimexist Jan 31, 2018

jlewi Feb 1, 2018

yupbank commented Feb 2, 2018

jlewi commented Feb 2, 2018

yupbank commented Feb 2, 2018

yupbank commented Feb 2, 2018

jose5918 Feb 3, 2018

jlewi Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

gaocegege left a comment

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 4, 2018

jlewi commented Feb 4, 2018

Use Argo rather than Airflow to run our E2E tests #358

Use Argo rather than Airflow to run our E2E tests #358

Conversation

jlewi commented Jan 31, 2018 • edited Loading

coveralls commented Jan 31, 2018

coveralls commented Jan 31, 2018 • edited Loading

jimexist Jan 31, 2018

Choose a reason for hiding this comment

jlewi Feb 1, 2018

Choose a reason for hiding this comment

yupbank commented Feb 2, 2018

jlewi commented Feb 2, 2018

yupbank commented Feb 2, 2018

yupbank commented Feb 2, 2018

jose5918 Feb 3, 2018

Choose a reason for hiding this comment

jlewi Feb 3, 2018

Choose a reason for hiding this comment

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

gaocegege left a comment

Choose a reason for hiding this comment

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 3, 2018

jlewi commented Feb 4, 2018

jlewi commented Feb 4, 2018

jlewi commented Jan 31, 2018 •

edited

Loading

coveralls commented Jan 31, 2018 •

edited

Loading