Can you contrast what you are proposing to PetSets (which are generally intended to solve this)?

Also, when suggesting things like this, we prefer to have a discussion in the GitHub Kubernetes repo in an issue, create a proposal, get consensus, and only then open a feature in the feature repo.

Also, I think the next sig-apps call would be a great place to discuss this issue - @michelleN if necessary can we make room on the agenda?

@smarterclayton you just cannot deploy whole openstack as PetSet. Also as I said " then can be extended to support post deployment actions."

PetSets are super cool, but sometimes you will have dependency between k8s job and daemonset.

Can we create PoC to show our idea? Also I would like to add that this stuff would be more similar to controllers than PetSets itself.

@smarterclayton I was instructed by @idvoretskyi (sig-openstack) to do so.

@pigmej Could this be implemented as a third party controller? If not why?

This sounds a lot like #1704

@philips Could be, I didn't put any technical details by intention though for now. We could also reuse some parts of "new" workflow (#25067). But still we would like to have it "in" kubernetes and not to have yet another separate framework that uses kubernetes to do stuff.

But yeah #1704 sounds kinda similar too, though AppController is more application/workload orineted, while #1704 is more workflow oriented.

Having dependencies in annotations would be very intuitive for user, also it would be super easy to modify and adjust it.

Yes, it does sound like #1704.

Please also see the decision for Workflow:

#24781 (comment)
#25067

This will need to be implemented outside the core.

@bgrant0607 I'm aware of workflow decision that's why I put 'new' workflow ;-) Thanks for links though.

And yes the idea was to implement it outside core. But there may be some need to modify core (not major though).

@pigmej - Do you have an example of these annotations and some write up on what would happen when the AppController is enabled? (Take an existing kubernetes non-trivial example, add annotations and describe how the annotations affect the example)

We're doing PoC of our idea, you can expect updates this week. We will provide further details soon.

@dims, sorry for long time without reply, but we're doing PoC for that.

@bgrant0607, @smarterclayton, @michelleN (sorry, I am unsure which of you I should address about it) we came up with a PoC: https://github.com/Mirantis/k8s-AppController and we would like to make a small demo of it on @kubernetes/sig-apps meeting. Can we schedule this demo for tomorrow meeting?

@nebril You are confirmed for the meeting tomorrow. :) looking forward to this PoC.

Can you please open a PR with a proposal for the set of changes you'd include? That'll help us understand whether we recommend you build this as an extension or a third party resource

Also, anything you can do to build this as an external component is great, since that helps the rest of the ecosystem get better.

@smarterclayton it's built as external component, it follows design of other controllers, we're reusing 3rd party resoruces as much as possible. Currently you spawn it as a POD, but in future we want to spawn it as a ReplicaSet, and use similar logic as other controllers to give HA.

Currently we did not require changing k8s core at all. We plan to add some improvements to k8s core in general that will improve UX of stuff that we use in AppController (3rd party resources, cli integration etc). We may spot some bugs in not that frequently used areas and that for sure brings huge value to any k8s deployment.

We have also this kubernetes/enhancements#42 opened.

The act of creating a feature is usually "we think this should be part of
Kube" - that means "there needs to be a design proposal approved and
reviewed". For instance, an extension component that just works well with
Kube doesn't need a features PR - you can just go do it. If it's something
that is "part of kube" generally that's where things start a) needing a
feature and b) need a proposal. If this would ever be something part of
"official" kube (and I use that word loosely), it's best that you submit a
proposal that discusses the design goals, how it fits with other parts of
the system, etc. Helm is a great example - it's mostly external to Kube
(layered on top) but it's not actually considered part of kube, and so it's
certainly possible that Kube would add features that overlap with Helm.

On Wed, Aug 17, 2016 at 2:44 PM, Jędrzej Nowak notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton it's built as
external component, it follows design of other controllers, we're reusing
3rd party resoruces as much as possible. Currently you spawn it as a POD,
but in future we want to spawn it as a ReplicaSet, and use similar logic as
other controllers to give HA.

Currently we did not require changing k8s core at all. We plan to add
some improvements to k8s core in general that will improve UX of stuff that
we use in AppController (3rd party resources, cli integration etc). We may
spot some bugs in not that frequently used areas and that for sure brings
huge value to any k8s deployment.

We have also this kubernetes/enhancements#42
kubernetes/enhancements#42 opened.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#29453 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p4c8YpnqV0ZwXZUpgeKDnRLS1L_sks5qg1acgaJpZM4JSzv3
.

We're kind of vague in the features repo about this, which maybe we need to
correct, but the majority of things in features require changes to Kube.
If you don't intend to change kube at all, you wouldn't need a proposal,
but we should clarify that.

On Wed, Aug 17, 2016 at 6:43 PM, Clayton Coleman ccoleman@redhat.com
wrote:

The act of creating a feature is usually "we think this should be part of
Kube" - that means "there needs to be a design proposal approved and
reviewed". For instance, an extension component that just works well with
Kube doesn't need a features PR - you can just go do it. If it's something
that is "part of kube" generally that's where things start a) needing a
feature and b) need a proposal. If this would ever be something part of
"official" kube (and I use that word loosely), it's best that you submit a
proposal that discusses the design goals, how it fits with other parts of
the system, etc. Helm is a great example - it's mostly external to Kube
(layered on top) but it's not actually considered part of kube, and so it's
certainly possible that Kube would add features that overlap with Helm.

On Wed, Aug 17, 2016 at 2:44 PM, Jędrzej Nowak notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton it's built as
external component, it follows design of other controllers, we're reusing
3rd party resoruces as much as possible. Currently you spawn it as a POD,
but in future we want to spawn it as a ReplicaSet, and use similar logic as
other controllers to give HA.

Currently we did not require changing k8s core at all. We plan to add
some improvements to k8s core in general that will improve UX of stuff that
we use in AppController (3rd party resources, cli integration etc). We may
spot some bugs in not that frequently used areas and that for sure brings
huge value to any k8s deployment.

We have also this kubernetes/enhancements#42
kubernetes/enhancements#42 opened.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#29453 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p4c8YpnqV0ZwXZUpgeKDnRLS1L_sks5qg1acgaJpZM4JSzv3
.

I'm interested in something like this as well, ran into kubernetes/helm along the way. @pigmej, have you looked at kubernetes/helm? I'm curious how your use case doesn't work with it. I'm currently evaluating it against my own, but it seems like the best place to start.

@pigmej - There is lots of discussion happening in the #sig-apps channel in the kubernetes slack about app controller. We'd like to get an update and have more discussion around this at the next sig-apps meeting. Please ping me when you have a moment (slack: @mnoorali or michelle@deis.com).

@michelleN Ok, please add to agenda, we will join slack too (though we're mostly Europe based)

I'm also interested in this/similar functionality. Just wanted to ask why the dependencies in the POC (https://github.com/Mirantis/k8s-AppController) are expressed as separate object vs creating an object that has all resources declared in it? Like AWS CloudFormation or Google Cloud Deployment Manager?
I'm interested in building an extensible PaaS-like system on top of Kube using Third Party Resources and driving their creation using a declarative description of a graph of dependencies of resources. So this AppController looks like something I'm looking for but I was thinking of a more general solution that can work with unknown resources (any TPRs) based on their metadata (annotations).
I apologise if this is a wrong place to ask this question.

@ash2k the intent of creating dependencies as separate objects was to allow for elasticity within a cluster - you could deploy one application with one graph and then reuse already deployed services to deploy second application in your cluster.

What do you mean by "working with unknown resources"? We plan to support all or most of k8s objects in AppController, what else would you like to orchestrate?

@ash2k also besides stuff that @nebril mentioned. Separated dependencies allow you to modify them independently. Which means you can change graph (extend) without touching resources itself. Also you can put deps and resource in one file (as a list).

@nebril, @pigmej I guess it depends on the use case. In my case it will not be common to share resources between graphs. Also, it may be possible to have an "external resource" resource type to include pre-existing resources into a graph.

What do you mean by "working with unknown resources"? We plan to support all or most of k8s objects in AppController, what else would you like to orchestrate?

I mean if I create a new resource type using Third Party Resource then I want to be able to orchestrate it (e.g. create instances of it) with the system that is being discussed here. i.e. this system should be able to orchestrate resources that are not defined yet (it may impose certain restrictions on what it can orchestrate - e.g. it may require certain metadata to exist on resources and/or their definitions to specify/configure desired behaviour).

Also, it may be possible to have an "external resource" resource type to include pre-existing resources into a graph.

This is supported ootb in AppController. You just define dependency with it.

Orchestration of 3rd party resources is not possible now, because all orchestrated types has to have defined checks in Go code. But we may change that in near future (that's why we want to validate current approach and take required steps to make it better).

I am going to echo what @smarterclayton said. I think this functionality can live nicely outside of kubernetes as an external controller. Since the POC proves that case can we close this issue out @pigmej?

@philips kubernetes/enhancements#42 we're targeting 1.5 core with AC. We initially wanted to do it as external component, but being part of core would allow us to do more stuff, to be integrated more tightly with K8S.

Plan for now is:

• validate approach (to spot weak points etc)
• make proper design/proposal aiming core
• refactor/code again some parts.

What do you think about that @philips ? We will also have kinda Q&A meeting on sig-apps next wednesday (cc @kubernetes/sig-apps )

@pigmej I really don't want to add more stuff to Core. We added scheduled jobs in v1.4, which I really think doesn't require core either. What would you need out of third party resources to not add this to core?

I agree with @philips and @smarterclayton. We need to hold the line on keeping core, well, core. This should apply to all new features no matter where they come from. Adding new functionality to core should be very carefully considered and should take multiple release cycles. It should be easier, faster and safer to implement new features outside of core.

If there are things that you need in core to make that plan work then those are the things that we should expedite.

@pigmej

but being part of core would allow us to do more stuff,

Can you provide more details? What are the "more stuff"? I am working on an application controller outside Kubernetes too. So I want to understand it better. Thanks in advance.

@xiang90 one example issue I can think of is lack of describe in kubectl for ThirdPartyResources, which causes the user experience to fall (he needs to extract the data about these definitions from json output). AppController could really benefit from having Dependency and Definition as part of the core API.

Other limitation imposed by usage of ThirdPartyResources is that they cannot be namespaced - we have workarounds around that, but it would be nice to not hack namespaces with labels.

There are more things, but we need to work on our AppController proposal to properly define our arguments for changes in core.

@pigmej @nebril do we need this issue anymore?

@dims: Those labels are not set on the issue: area/area/platform/openstack.

@dims: Those labels are not set on the issue: area/platform/openstack.

Closing as per discussion in #25067 (comment)