-
Notifications
You must be signed in to change notification settings - Fork 40k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How about supports ETL job scheduler? #39411
Comments
There was an idea for that, but was rejected eventually from core. You'll have to use some 3-rd party solution to provide such functionality or write your own controller on top of TPR. You can sync with @sdminonne, he's interested in this as well
Already available.
Yup, #30243 is the way to solve this.
Not sure what you mean with this, so I'm leaving this open until that gets cleared out. |
@Yancey1989 if you have this kind of job (please confirm) and you want to state the dependency in your manifest files you may need a custom controller with built on top of Third Party Resources (TPR). My company has some interested in this kind of tool and we're trying to build one even if very simple. Don't hesitate to contact me (on slack) if you need more info. |
@soltysh @sdminonne , thanks for your reply.
This mean, we assume each job is a vertex, each dependency relation is an edge. The graph consisting of these vertices and edges can not have a ring, otherwise these jobs will fall into deadlock. @sdminonne The repo is very helpful for us, I'll have a thorough understanding of this, and maybe contact you on slack, thank you! :) |
Since workflow is not part of k8s and other problems have already appropriate issues, I'm closing this one. |
We have many ETL jobs and preparing transfer these jobs running on k8s. Here is my demands and ideas:
RestartPolicy=Never
will start a new pod, Maximum number of failures or failure backoff policy for Jobs #30243 also discusses this.kubectl create -f xxx.yaml
, at the same time,apiserver
will check whether a loop will be formed.Anyone has some great idea for this? I also glad to submit a PR for the feature.
The text was updated successfully, but these errors were encountered: