(CLI) Conventions

Because in general tools are independent and have their own ways of being configured or run

Perhaps it doesn't have to be that way. I'd like to suggest the following;

• Python
• Click library
• Yaml

It looks like we'll have several projects contributing stuff and a lot of people will want to expose running a test in a nice way with a cli and maybe even as a subcommand of their project. I know I want that.

Aforementioned Click lets you compose these things pretty easily. For example, on @jeremyeder's side of things they have this runner that I could simply import as a subcommand if it were to adopt the convention.

Exposing it in such a way is more user friendly than having users pull this repo and individually read and understand each subdirectory.

Furthermore I think each cli should be able to parse a yaml config file (just like kubectl) that defines a test. See this webframework comparison benchmark for inspiration.

In Summary

I should be able to pull this repo, cd into the subdirectory of interest, and have a cli.py wrapper I can immediately try out either against example yaml definitions or with my own flags.

I should also easily be able to import into my own project whichever subdirectories of this repo I find relevant.

Thoughts?

Meeting occurred, notes were taken:
https://docs.google.com/document/d/1x1r65mXbX_RgLA0HTlsb8QRQqW0xB5XSM1vrMb2naQo/edit

@Zilman - there will be very different things here, e.g. running kubemark will be drastically different than running cluster-loader or dedicated component benchmarks and not all of them will have it's own CLI. Kubemark uses kubectl, dedicated benchmarks and e2e tests will probably use framework from the main repo, etc. OTOH cluster loader probably should have a CLI, but I don't think it will be shared with anything else, so we'll go with whatever is easiest.

For the existing e2e tests(density, load) i.e. density test: should.allow.starting.100.pods.per.node it would be nice to switch them to: should.allow.starting.10.pods.per.core and during the test calculate how many pods should be running on the single node. At this moment biggest density test is 100 pods per node so in case of running it on bare metal(depending on hardware configuration) it could give only about 2 pods per core.

@sjug is going to discuss the first part of this at the @kubernetes/sig-testing tomorrow, Tue Oct 4. And then in further depth at the @kubernetes/sig-scalability meeting on Thursday Oct 6, which we've agreed will own the "perf-tests" repo.

@pskrzyns - I don't think 10pods/core are already supported by Kubelet (@yujuhong), plus this issue is about moving tests to separate repo, not updating them:)

@pskrzyns - I don't think 10pods/core are already supported by Kubelet (@yujuhong), plus this issue is about moving tests to separate repo, not updating them:)

Agreed. For the pods per core discussion, please move it to the original issue: kubernetes/kubernetes#25762

Issues go stale after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale