Skip to content

Running leader election on Karpenter running outside of cluster #4216

Closed
@garvinp-stripe

Description

Description

What problem are you trying to solve?
I am currently running Karpenter "outside" of the cluster, specifically Karpenter itself isn't able to discover API server via K8s Services however we have other method of service discovery to find the API server. We provide Karpenter with a kubeconfig and uses the KUBECONFIG env var. When attempting to run Karpenter with this configuration we run into

 failed to setup manager: unable to find leader election namespace: not running in-cluster, please specify LeaderElectionNamespace

it seems like the controller runtime that implements leader election allows us to run outside but we need to provide the leaderelectionnamespace
https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/leaderelection/leader_election.go#L116

I don't see any configuration that allows us to specify this at this time.

How important is this feature to you?
This is pretty important since we won't be able to run Karpenter in leader elected mode.

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Activity

abeer-stripe

abeer-stripe commented on Jul 27, 2023

@abeer-stripe

I was able to figure out the solution for us here, and it seems it would work in the general case as well. The solution is just passing LeaderElectionNamespace as system.Namespace() here:

https://github.com/aws/karpenter-core/blob/d34a21d5648d0abd91c58fd60b5a5344b7308498/pkg/operator/operator.go#L110-L112

system.Namespace() is already used here in the same file - https://github.com/aws/karpenter-core/blob/d34a21d5648d0abd91c58fd60b5a5344b7308498/pkg/operator/operator.go#L96C71-L96C90

What this does is it avoid this part of the code in controller-runtime when running leader-election.
https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/leaderelection/leader_election.go#L74

I can raise a PR if this seems agreeable.

ellistarn

ellistarn commented on Jul 27, 2023

@ellistarn
Contributor

SGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      Running leader election on Karpenter running outside of cluster · Issue #4216 · aws/karpenter-provider-aws