Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parrallelize e2e test suite #3335

Open
AustinAbro321 opened this issue Dec 16, 2024 · 2 comments
Open

Parrallelize e2e test suite #3335

AustinAbro321 opened this issue Dec 16, 2024 · 2 comments

Comments

@AustinAbro321
Copy link
Contributor

AustinAbro321 commented Dec 16, 2024

Describe what should be investigated or refactored

The e2e test suite is broken into two main segments: tests that require a cluster and tests that don't. Tests with file names starting from 0-19 don't require a cluster while tests with filenames prefixed by a number 20+ do require a cluster. Order matters for some e2e tests that require a cluster, mainly for test 20, which runs zarf init. Tests that are not run in parallel, always run in order. The lower the number on the prefix of the filename the earlier the test runs.

Some tests that don't requiring a cluster already run in parallel. The only side effect from these tests are files being created. This will effect some tests, for example, there are two tests that use the component-choice package. Since we test that this package creates files, and we clean up the files after the tests run, these tests can flake if run in parallel. This situations should be easy to resolve by using packages specific to each test rather than general examples. Parallelizing these tests may cause storage limitation flakes on the runners as well since we'd be creating / pulling several packages.

Running tests that require a cluster in parallel will provide additional challenges. Each test will have to run on it's own cluster. This would take additional setup, compute resources, and storage resources. Another option is to run multiple tests on the cluster at the same time. However, care would have to be taken to ensure that each package is different and deploys to a different namespace. Needing to know which namespaces to avoid will likely become arduous as additional tests are added. We should look into how Kubernetes structures their end to end tests and take inspiration from that.

Once we've parallelized the tests we can remove the numbers prefixed by them, as order will no longer be required. We would also have to update the testing guide

@soltysh
Copy link
Contributor

soltysh commented Dec 17, 2024

Order matters for some e2e tests that require a cluster, mainly for test 20, which runs zarf init

Can we make that test not depend on order? Being run in serial is fine, but each tests should be a complete unit. Even if that requires running zarf init more than once, wdyt?

@AustinAbro321
Copy link
Contributor Author

The amount of time zarf init takes makes it tough to run for each test without parallelizing. zarf init takes about a minute and a half so running it for each test would increase the total e2e test time from about 20 minutes to 50 minutes. This wouldn't include the time to remove Zarf from the cluster / setup a new cluster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants