Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create clusters concurrently while setting up federation e2e #26655

Closed
madhusudancs opened this issue Jun 1, 2016 · 7 comments
Closed

Create clusters concurrently while setting up federation e2e #26655

madhusudancs opened this issue Jun 1, 2016 · 7 comments
Assignees
@madhusudancs
Labels
priority/backlog Higher priority than priority/awaiting-more-evidence.
Milestone
v1.4

Comments

@madhusudancs
Copy link
Contributor

The time to bring up a federation (of clusters) to run e2e test is very long right now. Much of that work can be done concurrently. For example GCE nodes can be created concurrently, etc. At the minimum we should run the iterations in this loop concurrently.

@colhom I am not able to assign issues to you. So I am assigning it to myself now and CC'ing you. But please assume that I am assigning this to you :) Also, please feel free to re-assign.

cc @quinton-hoole @nikhiljindal @kubernetes/sig-cluster-federation
@madhusudancs madhusudancs added this to the next-candidate milestone Jun 1, 2016
@madhusudancs madhusudancs self-assigned this Jun 1, 2016
@ghost ghost added the team/control-plane label Jun 1, 2016
@ghost
Copy link

ghost commented Jun 1, 2016

Sounds like a good idea. How long are you seeing? If it's less than 30 minutes, lets do this optimization later. For now, lets focus on getting the e2e tests in and passing :-)

@madhusudancs
Copy link
Contributor Author

madhusudancs commented Jun 1, 2016 •
edited
Loading

It is right at about 30 minutes, give or take a few minutes depending on various factors not in our direct control. Also, I am blowing through my ISP's monthly data usage budget. For example. we stage the server tars for each cluster right now. Each of them is about 300MB. Plus 100MB for federation-apiserver image (it is not clear if this is being uploaded each time). But roughly, every time I bring up the federation, I spend about a gigabyte. It is only 1st of the month and I am already feeling bad :)

@colhom
Copy link

colhom commented Jun 1, 2016

How do we want to mux the log output from the concurrent kube-up invocations in a maximally jenkins-friendly manner?

  • Line by line with line prefixes?
  • N-line batch groups with header label sections?
  • Write each to a file, print out the file path and cat each file out at the end?

@ghost
Copy link

ghost commented Jun 1, 2016

Fair enough @madhusudancs , that sounds worth fixing (but only after the tests are running and passing :-) )

@madhusudancs
Copy link
Contributor Author

madhusudancs commented Jun 2, 2016 •
edited
Loading

@colhom I have been thinking about that, but I don't have a good answer. I did not think about your option 2, i.e.

  • N-line batch groups with header label sections?

I think doing just 3

  • Write each to a file, print out the file path and cat each file out at the end?

Will lead to a bad experience. For example, people won't be able to look at the output in the in Jenkins console output until the end.

I think line prefix is the quickest solution and which is what we should do in the short term.

In the medium term, we should split up the steps to bring up a cluster and run the individual steps one after the other, but parallelize the individual steps themselves. For example, we can create the MIGs that create nodes for each cluster in parallel. But we can wait for that before moving on to the next step of creating network rules, etc.

Also, it is worth discussing this with @mikedanese and see if the cluster deployment simplification work that he is doing can help here.

@quinton-hoole Ack.

@ghost ghost modified the milestones: v1.4, next-candidate Jul 25, 2016
@ghost
Copy link

ghost commented Jul 25, 2016

I gather that this is done now.

@madhusudancs
Copy link
Contributor Author

Work is all done, the PR is under review. I will close it once the PR is merged.

@madhusudancs madhusudancs mentioned this issue Jul 26, 2016
Merged
@ghost ghost added the priority/backlog Higher priority than priority/awaiting-more-evidence. label Aug 12, 2016
k8s-github-robot pushed a commit that referenced this issue Aug 13, 2016
Merge pull request #29515 from madhusudancs/fed-makefile
2b34988 
Automatic merge from submit-queue

A build and deploy script to manage cluster federation lifecycle and a Makefile wrapper to drive that process.

Also includes a sample config file to describe clusters.

The build script implements the following things:
1. Generates the required configs.
2. Builds the hyperkube binary and the corresponding docker image.
3. Pushes the image to a specified repository.
4. Pulls the federation installer docker images.
5. Builds the Kubernetes clusters described the config.json file.
6. Pushes the federation components to one of the Kubernetes clusters
   built in the previous step.
7. Also turns down the federation components and the Kubernetes
   clusters.

**NOTE**: Installer images are right now being pulled from my public repository of docker images. I am working on pushing them to our release repository.

```release-note
Cluster Federation components can now be built and deployed using the make command. Please see federation/README.md for details.
```

cc @kubernetes/sig-cluster-federation @colhom 

Fixes: Issue #26655
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@madhusudancs madhusudancs

Labels
priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Milestone
v1.4
Development

No branches or pull requests
3 participants
@madhusudancs @colhom @goltermann