Hi @Nick-Harvey. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

@jlewi just in case you didn't see the update in #282 .

"I've attempted a few squashs and rebases at this point, and I've just ended up with a tangled mess of a repo 😄. I'm sure I could figure it out and get it all sorted given some time, but I'm a bit swamped so, I took the easy way out 🥂. Here's a fresh PR with the same content: #522"

/ok-to-test

/approve
/lgtm

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: texasmichelle

The full list of commands accepted by this bot can be found here.

The pull request process is described here

/test all

/retest

@cwbeitel @gaocegege any chance I can get your help finding out why the test seems to be failing. The logs in argo look blank to me http://testing-argo.kubeflow.org/workflows/kubeflow-test-infra/kubeflow-examples-presubmit-gis-522-9aecf80-1405-9055?tab=workflow&nodeId=kubeflow-examples-presubmit-gis-522-9aecf80-1405-9055-1251014382&sidePanel=logs%3Akubeflow-examples-presubmit-gis-522-9aecf80-1405-9055-1251014382%3Amain

There are instructions here for fetching logs when they don't show up in Argo. This happens because Argo depends on the pod being around and this will be garbage collected.

In this case we can fetch the logs from stackdriver. Here's the query I used

resource.type="k8s_container"
resource.labels.cluster_name="kubeflow-testing"
resource.labels.pod_name="kubeflow-examples-presubmit-gis-522-9aecf80-1405-9055-1251014382"
resource.labels.container_name="main"

The error is

Traceback (most recent call last): File "tfjob_test.py", line 31, in <module> from py import tf_job_client #pylint: disable=no-name-in-module ImportError: No module named py

This is due to the fact that the kubeflow/tf-operator was refactored and the import changed and the test is pulling in master at HEAD.

I'll file a separate issue to get this fixed.

Thanks @jlewi & @zabbasi for this fix! @Nick-Harvey if you rebase, the tests should pass

/lgtm