Skip to content

Commit

Permalink
postsubmit should push image to gcr.io/kubeflow-images-public (#831)
Browse files Browse the repository at this point in the history 
* postsubmit should push image to gcr.io/kubeflow-images-public

* We want to automatically publish public versions of our images on
  postsubmit.

* This is needed to make managing releases tractable.

Related to:

* Fix prow_config.yaml

* * The overriding of parameters wasn't being passed correctly to the individual
  template steps; as a result I think we working
  gcr.io/kubeflow-ci/test-worker:latest and not the pinned versions in
  prow_config.yaml

* We fixed that because we want to override the container registry the
  image is pushed to based on params. This had the effect of using
  the pinned test-worker image in prow_config.yaml. I believe this image
  is out of date (missing some changes to the py client) so we need to
  upgrade that image.
  • Loading branch information
@jlewi @k8s-ci-robot
jlewi authored and k8s-ci-robot committed Sep 22, 2018
1 parent c45db42 commit d994e24
Show file tree
Hide file tree
Showing 3 changed files with 35 additions and 20 deletions.
16 changes: 14 additions & 2 deletions prow_config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,22 @@ workflows:
name: v1
params:
tfJobVersion: v1alpha1
testWorkerImage: gcr.io/kubeflow-ci/test-worker:v20180615-e6b2059-e3b0c4
testWorkerImage: gcr.io/kubeflow-ci/test-worker:v20180822-da7cefc-e3b0c4
- app_dir: kubeflow/tf-operator/test/workflows
component: workflows
name: v2
job_types:
- presubmit
params:
registry: "gcr.io/kubeflow-ci"
tfJobVersion: v1alpha2
testWorkerImage: gcr.io/kubeflow-ci/test-worker:v20180615-e6b2059-e3b0c4
testWorkerImage: gcr.io/kubeflow-ci/test-worker:v20180822-da7cefc-e3b0c4
- app_dir: kubeflow/tf-operator/test/workflows
component: workflows
name: v2
job_types:
- postsubmit
params:
registry: "gcr.io/kubeflow-public-images"
tfJobVersion: v1alpha2
testWorkerImage: gcr.io/kubeflow-ci/test-worker:v20180822-da7cefc-e3b0c4
36 changes: 18 additions & 18 deletions test/workflows/components/workflows.libsonnet
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
},

// overrides is a dictionary of parameters to provide in addition to defaults.
parts(namespace, name, overrides={}):: {
parts(namespace, name, overrides):: {
// Workflow to run the e2e test.
e2e(prow_env, bucket):
local params = $.defaultParams + overrides;
Expand Down Expand Up @@ -338,7 +338,7 @@
],
},
}, // checkout
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("build", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("build", [
"python",
"-m",
"py.release",
Expand All @@ -348,21 +348,21 @@
"--project=" + project,
"--version_tag=" + versionTag,
]), // build
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("py-test", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("py-test", [
"python",
"-m",
"kubeflow.testing.test_py_checks",
"--artifacts_dir=" + artifactsDir,
"--src_dir=" + srcDir,
]), // py test
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("py-lint", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("py-lint", [
"python",
"-m",
"kubeflow.testing.test_py_lint",
"--artifacts_dir=" + artifactsDir,
"--src_dir=" + srcDir,
]), // py lint
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("setup-cluster", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("setup-cluster", [
"python",
"-m",
"py.deploy",
Expand All @@ -374,7 +374,7 @@
"--accelerator=nvidia-tesla-k80=1",
"--junit_path=" + artifactsDir + "/junit_setupcluster.xml",
]), // setup cluster
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("setup-kubeflow", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("setup-kubeflow", [
"python",
"-m",
"py.deploy",
Expand All @@ -388,7 +388,7 @@
"--tf_job_version=" + params.tfJobVersion,
"--junit_path=" + artifactsDir + "/junit_setupkubeflow.xml",
]), // setup cluster
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("run-chief", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("run-chief", [
"python",
"-m",
"py.test_runner",
Expand All @@ -406,7 +406,7 @@
"--params=name=master-is-chief,namespace=default,image=" + testServerImage,
"--junit_path=" + artifactsDir + "/junit_chief.xml",
]), // run worker0
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("run-worker0", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("run-worker0", [
"python",
"-m",
"py.test_runner",
Expand All @@ -424,7 +424,7 @@
"--params=name=worker0-is-chief,namespace=default,image=" + testServerImage,
"--junit_path=" + artifactsDir + "/junit_worker0.xml",
]), // run worker0
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("run-tests", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("run-tests", [
"python",
"-m",
"py.test_runner",
Expand All @@ -441,7 +441,7 @@
"--tfjob_version=" + params.tfJobVersion,
"--junit_path=" + artifactsDir + "/junit_e2e.xml",
]), // run tests
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("run-gpu-tests", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("run-gpu-tests", [
"python",
"-m",
"py.test_runner",
Expand All @@ -458,7 +458,7 @@
"--tfjob_version=" + params.tfJobVersion,
"--junit_path=" + artifactsDir + "/junit_gpu-tests.xml",
]), // run gpu_tests
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("run-clean-pod-all", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("run-clean-pod-all", [
"python",
"-m",
"py.test_runner",
Expand All @@ -473,7 +473,7 @@
"--verify_clean_pod_policy=All",
"--junit_path=" + artifactsDir + "/junit_clean-pod-all-tests.xml",
]), // run clean_pod_all
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("run-clean-pod-running", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("run-clean-pod-running", [
"python",
"-m",
"py.test_runner",
Expand All @@ -488,7 +488,7 @@
"--verify_clean_pod_policy=Running",
"--junit_path=" + artifactsDir + "/junit_clean-pod-running-tests.xml",
]), // run clean_pod_running
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("run-clean-pod-none", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("run-clean-pod-none", [
"python",
"-m",
"py.test_runner",
Expand All @@ -503,7 +503,7 @@
"--verify_clean_pod_policy=None",
"--junit_path=" + artifactsDir + "/junit_clean-pod-none-tests.xml",
]), // run clean_pod_none
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("estimator-runconfig", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("estimator-runconfig", [
"python",
"-m",
"py.test_runner",
Expand All @@ -518,7 +518,7 @@
"--verify_runconfig",
"--junit_path=" + artifactsDir + "/junit_estimator-runconfig-tests.xml",
]), // run estimator_runconfig
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("invalid-tfjob", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("invalid-tfjob", [
"python",
"-m",
"py.test_invalid_job",
Expand All @@ -527,15 +527,15 @@
"--params=name=invalid-tfjob,namespace=default",
"--artifacts_dir=" + artifactsDir,
]), // invalid-tfjob
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("create-pr-symlink", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("create-pr-symlink", [
"python",
"-m",
"kubeflow.testing.prow_artifacts",
"--artifacts_dir=" + outputDir,
"create_pr_symlink",
"--bucket=" + bucket,
]), // create-pr-symlink
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("teardown-cluster", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("teardown-cluster", [
"python",
"-m",
"py.deploy",
Expand All @@ -545,7 +545,7 @@
"--project=" + project,
"--junit_path=" + artifactsDir + "/junit_teardown.xml",
]), // setup cluster
$.parts(namespace, name).e2e(prow_env, bucket).buildTemplate("copy-artifacts", [
$.parts(namespace, name, overrides).e2e(prow_env, bucket).buildTemplate("copy-artifacts", [
"python",
"-m",
"kubeflow.testing.prow_artifacts",
Expand Down
3 changes: 3 additions & 0 deletions test/workflows/environments/releasing/globals.libsonnet
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
registry: "gcr.io/kubeflow-images-public",
}

0 comments on commit d994e24

Please sign in to comment.