Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How about supports ETL job scheduler? #39411

Closed
Yancey1989 opened this issue Jan 4, 2017 · 4 comments
Closed

How about supports ETL job scheduler? #39411

Yancey1989 opened this issue Jan 4, 2017 · 4 comments
Assignees
Labels
area/batch kind/support Categorizes issue or PR as a support question. sig/apps Categorizes an issue or PR as relevant to SIG Apps.

Comments

@Yancey1989
Copy link
Contributor

Yancey1989 commented Jan 4, 2017

We have many ETL jobs and preparing transfer these jobs running on k8s. Here is my demands and ideas:

  1. Jobs on k8s should have dependency relation.
  2. Support CRON syntax, CronJob can do this.
    • A job will run if and only the state of it's dependency job is Complete.
    • A job can be configurated without CRON if it has an dependency job.
  3. Support configuration a maximum number of failures backoff policy.
  4. We can add the ETL jobs with kubectl create -f xxx.yaml, at the same time, apiserver will check whether a loop will be formed.

Anyone has some great idea for this? I also glad to submit a PR for the feature.

@soltysh soltysh self-assigned this Jan 4, 2017
@soltysh soltysh added area/batch sig/apps Categorizes an issue or PR as relevant to SIG Apps. labels Jan 4, 2017
@soltysh
Copy link
Contributor

soltysh commented Jan 4, 2017

  1. Jobs on k8s should have dependency relation.
    A job will run if and only the state of it's dependency job is Complete.
    A job can be configurated without CRON if it has an dependency job.

There was an idea for that, but was rejected eventually from core. You'll have to use some 3-rd party solution to provide such functionality or write your own controller on top of TPR. You can sync with @sdminonne, he's interested in this as well

Support CRON syntax, CronJob can do this.

Already available.

Support configuration a maximum number of failures backoff policy.

For now, RestartPolicy=Never will start a new pod, #30243 also discusses this.

Yup, #30243 is the way to solve this.

We can add the ETL jobs with kubectl create -f xxx.yaml, at the same time, apiserver will check whether a loop will be formed.

Not sure what you mean with this, so I'm leaving this open until that gets cleared out.

@soltysh soltysh added the kind/support Categorizes issue or PR as a support question. label Jan 4, 2017
@sdminonne
Copy link
Contributor

@Yancey1989 if you have this kind of job (please confirm) and you want to state the dependency in your manifest files you may need a custom controller with built on top of Third Party Resources (TPR). My company has some interested in this kind of tool and we're trying to build one even if very simple. Don't hesitate to contact me (on slack) if you need more info.

@Yancey1989
Copy link
Contributor Author

Yancey1989 commented Jan 5, 2017

@soltysh @sdminonne , thanks for your reply.

We can add the ETL jobs with kubectl create -f xxx.yaml, at the same time, apiserver will check whether a loop will be formed.

This mean, we assume each job is a vertex, each dependency relation is an edge. The graph consisting of these vertices and edges can not have a ring, otherwise these jobs will fall into deadlock.

@sdminonne The repo is very helpful for us, I'll have a thorough understanding of this, and maybe contact you on slack, thank you! :)

@soltysh
Copy link
Contributor

soltysh commented Jan 5, 2017

Since workflow is not part of k8s and other problems have already appropriate issues, I'm closing this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/batch kind/support Categorizes issue or PR as a support question. sig/apps Categorizes an issue or PR as relevant to SIG Apps.
Projects
None yet
Development

No branches or pull requests

3 participants