Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Cadence] Use of helm install --wait causes the jobs to not be created #1176

Open
drewhemm opened this issue Oct 21, 2020 · 2 comments
Open

[Cadence] Use of helm install --wait causes the jobs to not be created #1176

drewhemm opened this issue Oct 21, 2020 · 2 comments

Comments

@drewhemm
Copy link

Describe the bug
When installing this chart with the --wait flag, because of the post-install hook on the resources in cadence/templates/server-job.yaml, they do not get created until Helm considers the app installed, but because the pods will not come online until the jobs have run, there is a circular dependency.

Steps to reproduce the issue:
Attempt to install the chart using helm install --wait or helm upgrade --install --wait

Expected behavior
The chart should install correctly, waiting for all resources to be created
@drewhemm
Copy link
Author

Here is some output from running helm with --debug that shows the circular dependency very clearly:

client.go:108: [debug] creating 190 resource(s)
wait.go:53: [debug] beginning wait for 190 resources with timeout of 30m0s
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready

Helm is going to wait for these pods to be ready, and they never will be because the jobs have not run. Here is how the debug output looks when the --wait flag is absent:

client.go:108: [debug] creating 197 resource(s)
client.go:108: [debug] creating 1 resource(s)
client.go:463: [debug] Watching for changes to Job test-cadence-schema-setup with timeout of 10m0s
client.go:491: [debug] Add/Modify event for test-cadence-schema-setup: ADDED
client.go:530: [debug] test-cadence-schema-setup: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:491: [debug] Add/Modify event for test-cadence-schema-setup: MODIFIED
client.go:530: [debug] test-cadence-schema-setup: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:491: [debug] Add/Modify event for test-cadence-schema-setup: MODIFIED
client.go:108: [debug] creating 1 resource(s)
client.go:463: [debug] Watching for changes to Job test-cadence-schema-update with timeout of 10m0s
client.go:491: [debug] Add/Modify event for test-cadence-schema-update: ADDED
client.go:530: [debug] test-cadence-schema-update: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:491: [debug] Add/Modify event for test-cadence-schema-update: MODIFIED
client.go:530: [debug] test-cadence-schema-update: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:491: [debug] Add/Modify event for test-cadence-schema-update: MODIFIED
client.go:254: [debug] Starting delete for "test-cadence-schema-setup" Job
client.go:254: [debug] Starting delete for "test-cadence-schema-update" Job

@sagikazarmark
Copy link
Member

Thanks @drewhemm !

We'll look into it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sagikazarmark @drewhemm