You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe what should be investigated or refactored
The e2e test suite is broken into two main segments: tests that require a cluster and tests that don't. Tests with file names starting from 0-19 don't require a cluster while tests with filenames prefixed by a number 20+ do require a cluster. Order matters for some e2e tests that require a cluster, mainly for test 20, which runs zarf init. Tests that are not run in parallel, always run in order. The lower the number on the prefix of the filename the earlier the test runs.
Some tests that don't requiring a cluster already run in parallel. The only side effect from these tests are files being created. This will effect some tests, for example, there are two tests that use the component-choice package. Since we test that this package creates files, and we clean up the files after the tests run, these tests can flake if run in parallel. This situations should be easy to resolve by using packages specific to each test rather than general examples. Parallelizing these tests may cause storage limitation flakes on the runners as well since we'd be creating / pulling several packages.
Running tests that require a cluster in parallel will provide additional challenges. Each test will have to run on it's own cluster. This would take additional setup, compute resources, and storage resources. Another option is to run multiple tests on the cluster at the same time. However, care would have to be taken to ensure that each package is different and deploys to a different namespace. Needing to know which namespaces to avoid will likely become arduous as additional tests are added. We should look into how Kubernetes structures their end to end tests and take inspiration from that.
Once we've parallelized the tests we can remove the numbers prefixed by them, as order will no longer be required. We would also have to update the testing guide
The text was updated successfully, but these errors were encountered:
Order matters for some e2e tests that require a cluster, mainly for test 20, which runs zarf init
Can we make that test not depend on order? Being run in serial is fine, but each tests should be a complete unit. Even if that requires running zarf init more than once, wdyt?
The amount of time zarf init takes makes it tough to run for each test without parallelizing. zarf init takes about a minute and a half so running it for each test would increase the total e2e test time from about 20 minutes to 50 minutes. This wouldn't include the time to remove Zarf from the cluster / setup a new cluster.
Describe what should be investigated or refactored
The e2e test suite is broken into two main segments: tests that require a cluster and tests that don't. Tests with file names starting from 0-19 don't require a cluster while tests with filenames prefixed by a number 20+ do require a cluster. Order matters for some e2e tests that require a cluster, mainly for test 20, which runs
zarf init
. Tests that are not run in parallel, always run in order. The lower the number on the prefix of the filename the earlier the test runs.Some tests that don't requiring a cluster already run in parallel. The only side effect from these tests are files being created. This will effect some tests, for example, there are two tests that use the component-choice package. Since we test that this package creates files, and we clean up the files after the tests run, these tests can flake if run in parallel. This situations should be easy to resolve by using packages specific to each test rather than general examples. Parallelizing these tests may cause storage limitation flakes on the runners as well since we'd be creating / pulling several packages.
Running tests that require a cluster in parallel will provide additional challenges. Each test will have to run on it's own cluster. This would take additional setup, compute resources, and storage resources. Another option is to run multiple tests on the cluster at the same time. However, care would have to be taken to ensure that each package is different and deploys to a different namespace. Needing to know which namespaces to avoid will likely become arduous as additional tests are added. We should look into how Kubernetes structures their end to end tests and take inspiration from that.
Once we've parallelized the tests we can remove the numbers prefixed by them, as order will no longer be required. We would also have to update the testing guide
The text was updated successfully, but these errors were encountered: