ci: cache test images for `integration`, `VM` and `module` tests #7599

DmitriyLewen · 2024-09-26T07:15:51Z

Description

We got a lot of 429 errors for the tests.
To avoid loading the test images on every run, we need to cache them.

Test runs:

save images in cache - https://github.com/DmitriyLewen/trivy/actions/runs/11050734568
take cache for tests - https://github.com/DmitriyLewen/trivy/actions/runs/11050825154
saved cache:

Note

This PR doesn't fix all 429 errors.

We can still get error when requesting tags for trivy-test-images and trivy-test-vm-images (to make sure that fs contains all required image):

trivy/magefiles/fixture.go

Lines 23 to 32 in 8128ecc

    
           tags, err := crane.ListTags(testImages) 
        
           if err != nil { 
        
           	return err 
        
           } 
        
           for _, tag := range tags { 
        
           	fileName := tag + ".tar.gz" 
        
           	filePath := filepath.Join(dir, fileName) 
        
           	if exists(filePath) { 
        
           		continue 
        
           	}

TestFanal_Library_DockerLessMode uses remote images:

trivy/pkg/fanal/test/integration/library_test.go

Lines 156 to 159 in ab0fd0d

    
           _, _ = cli.ImageRemove(ctx, tt.remoteImageName, dimage.RemoveOptions{ 
        
           	Force:         true, 
        
           	PruneChildren: true, 
        
           })

Related issues

Close ci: cache trivy-test-images #7598

Checklist

I've read the guidelines for contributing to this repository.
I've followed the conventions in the PR title.
I've added tests that prove my fix is effective or that my feature works.
I've updated the documentation with the relevant information (if needed).
I've added usage information (if the PR introduces new options)
I've included a "before" and "after" example to the description (if the PR is a user interface change).

DmitriyLewen · 2024-09-26T08:37:38Z

.github/workflows/test.yaml

+        id: restore-test-images
+        with:
+          path: integration/testdata/fixtures/images
+          key: cache-test-images-2024-09-10 # trivy-test-images last update date


@knqyf263 I added trivy-test-images (trivy-test-vm-images) last update date to add the ability to control and influence the cache.
Tell me if you have other ideas.

I want to automatically invalidate the cache when a new image is added to trivy-test-images rather than modifying the key manually. How about something like this?

- name: Generate image list digest id: image-digest run: | IMAGE_LIST=$(skopeo list-tags docker://ghcr.io/aquasecurity/trivy-test-images) # skopeo is pre-installed DIGEST=$(echo "$IMAGE_LIST" | sha256sum | cut -d' ' -f1) echo "digest=$DIGEST" >> $GITHUB_OUTPUT - name: Restore test images from cache uses: actions/cache@v4 id: restore-test-images with: path: integration/testdata/fixtures/images key: cache-test-images-${{ steps.image-digest.outputs.digest }} restore-keys: cacche-test-images- - name: Download missing test images if: steps.restore-test-images.outputs.cache-hit != 'true' run: mage test:fixtureContainerImages

It doesn't work if the existing images are updated, though.

Also, we don't need a step to save cache because it automatically creates a new cache.

On a cache miss, the action automatically creates a new cache if the job completes successfully.

https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#matching-a-cache-key

It doesn't work if the existing images are updated, though.

Ideally, we should calculate a cache key from each image manifest, but a list of images should be enough at the moment. We can think about it later.

REPO="ghcr.io/aquasecurity/trivy-test-images" TAGS=$(crane ls $REPO) COMBINED_DIGEST="" for TAG in $TAGS; do DIGEST=$(crane manifest $REPO:$TAG | sha256sum | cut -d' ' -f1) COMBINED_DIGEST="${COMBINED_DIGEST}${DIGEST}" done FINAL_DIGEST=$(echo "$COMBINED_DIGEST" | sha256sum | cut -d' ' -f1) echo "digest=$FINAL_DIGEST" >> $GITHUB_OUTPUT

Ideally, we should calculate a cache key from each image manifest, but a list of images should be enough at the moment. We can think about it later.

we should definitely do this in the future, because when updating the test image - the digest will remain the same

we should definitely do this in the future, because when updating the test image - the digest will remain the same

Exactly. However, since we pin the digest, it is less likely to happen unless we intentionally change the digest.
https://github.com/aquasecurity/trivy-test-images/blob/bcbd1ca7b1a9d001b7f21f41310b8a17bc355dd2/copy-images.sh#L16

@knqyf263 Used your logic and removed the save step c015645

.github/workflows/test.yaml

knqyf263 · 2024-09-26T09:36:20Z

.github/workflows/test.yaml

+        id: restore-test-images
+        with:
+          path: integration/testdata/fixtures/images
+          key: cache-test-images-2024-09-10 # trivy-test-images last update date


I want to automatically invalidate the cache when a new image is added to trivy-test-images rather than modifying the key manually. How about something like this?

- name: Generate image list digest id: image-digest run: | IMAGE_LIST=$(skopeo list-tags docker://ghcr.io/aquasecurity/trivy-test-images) # skopeo is pre-installed DIGEST=$(echo "$IMAGE_LIST" | sha256sum | cut -d' ' -f1) echo "digest=$DIGEST" >> $GITHUB_OUTPUT - name: Restore test images from cache uses: actions/cache@v4 id: restore-test-images with: path: integration/testdata/fixtures/images key: cache-test-images-${{ steps.image-digest.outputs.digest }} restore-keys: cacche-test-images- - name: Download missing test images if: steps.restore-test-images.outputs.cache-hit != 'true' run: mage test:fixtureContainerImages

It doesn't work if the existing images are updated, though.

knqyf263 · 2024-09-26T09:36:34Z

.github/workflows/test.yaml

+        id: restore-test-images
+        with:
+          path: integration/testdata/fixtures/images
+          key: cache-test-images-2024-09-10 # trivy-test-images last update date


Also, we don't need a step to save cache because it automatically creates a new cache.

On a cache miss, the action automatically creates a new cache if the job completes successfully.

https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#matching-a-cache-key

knqyf263 · 2024-09-26T09:51:08Z

In short, the problem with the current logic is that the test fails whenever a test image is added. Then, we need to update the cache key manually. I would like to avoid this somehow by leveraging a cache key and restore keys.
https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#cache-hits-and-misses

knqyf263 · 2024-09-26T11:10:08Z

.github/workflows/test.yaml

+      - name: Download test images
+        if: steps.restore-test-images.outputs.cache-hit != 'true'
+        run: mage test:fixtureContainerImages
+


We don't need this step anymore. It is transparently called.

trivy/magefiles/magefile.go

Line 277 in bbc8e1d

mg.Deps(t.FixtureContainerImages)

And it's efficient as it skips images if it exists locally.

trivy/magefiles/fixture.go

Lines 30 to 32 in bbc8e1d

if exists(filePath) {

continue

}

Now, we call this check twice, which is just time-consuming (and API-consuming).

Right, since the cache will still be saved at the end of the job!

Removed download steps in 0609e5d

knqyf263 · 2024-09-26T11:13:20Z

I saw an error.

Post job cleanup.
/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/trivy/trivy --files-from manifest.txt --use-compress-program zstdmt
Failed to save: Unable to reserve cache with key cache-test-images-e57[2](https://github.com/aquasecurity/trivy/actions/runs/11050771503/job/30699240630?pr=7599#step:14:2)ef093aa79559e3e3fee0916b10fa8804dd0a8836d11eaea83e22c4f8047a, another job may be creating this cache. More details: Cache already exists. Scope: refs/pull/7599/merge, Key: cache-test-images-e572ef09[3](https://github.com/aquasecurity/trivy/actions/runs/11050771503/job/30699240630?pr=7599#step:14:3)aa79559e3e3fee0916b10fa8804dd0a8836d11eaea83e22c4f8047a, Version: f079f76b9e5c88a6d1d7a2a554815bd7b73babbae6f49b403add394d403a41c4

https://github.com/aquasecurity/trivy/actions/runs/11050771503/job/30699240630?pr=7599

But it looks like the module integration test was kicked at the same time and already created the cache with the same key. I don't think it's a problem. Just sharing information.
https://github.com/aquasecurity/trivy/actions/runs/11050771503/job/30699240939?pr=7599

knqyf263 · 2024-09-26T11:27:13Z

I confirmed the cache is restored correctly and images are not pulled.

Run actions/cache@v4
Received 0 of 1848896025 (0.0%), 0.0 MBs/sec
Received 75497472 of 1848896025 (4.1%), 36.0 MBs/sec
Received 171966464 of 1848896025 (9.3%), 54.6 MBs/sec
Received 281018368 of 1848896025 (15.2%), 67.0 MBs/sec
Received 385875968 of 1848896025 (20.9%), 73.6 MBs/sec
Received 482344960 of 1848896025 (26.1%), 76.6 MBs/sec
Received 583008256 of 1848896025 (31.5%), 79.4 MBs/sec
Received 683671552 of 1848896025 (37.0%), 81.5 MBs/sec
Received 784334848 of 1848896025 (42.4%), 83.1 MBs/sec
Received 884998144 of 1848896025 (47.9%), 84.4 MBs/sec
Received 985661440 of 1848896025 (53.3%), 85.4 MBs/sec
Received 1090519040 of 1848896025 (59.0%), 86.6 MBs/sec
Received 1178599424 of 1848896025 (63.7%), 86.4 MBs/sec
Received [12](https://github.com/aquasecurity/trivy/actions/runs/11051066331/job/30700170044?pr=7599#step:6:13)79262720 of 1848896025 (69.2%), 87.1 MBs/sec
Received [13](https://github.com/aquasecurity/trivy/actions/runs/11051066331/job/30700170044?pr=7599#step:6:14)75731712 of 1848896025 (74.4%), 87.4 MBs/sec
Received [14](https://github.com/aquasecurity/trivy/actions/runs/11051066331/job/30700170044?pr=7599#step:6:15)55423488 of 1848896025 (78.7%), 86.7 MBs/sec
Received [15](https://github.com/aquasecurity/trivy/actions/runs/11051066331/job/30700170044?pr=7599#step:6:16)51892480 of 1848896025 (83.9%), 87.0 MBs/sec
Received 1652555776 of 1848896025 (89.4%), 87.5 MBs/sec
Received 1753219072 of 1848896025 (94.8%), 88.0 MBs/sec
Received 1844701721 of 1848896025 (99.8%), 87.9 MBs/sec
Cache Size: ~1763 MB (1848896025 B)
/usr/bin/tar -xf /home/runner/work/_temp/46c2dffa-9fd7-4890-9ec7-13d232adc733/cache.tzst -P -C /home/runner/work/trivy/trivy --use-compress-program unzstd
Received 1848896025 of 1848896025 (100.0%), 83.9 MBs/sec
Cache restored successfully
Cache restored from key: cache-test-images-e572ef093aa79559e3e3fee09[16](https://github.com/aquasecurity/trivy/actions/runs/11051066331/job/30700170044?pr=7599#step:6:17)b10fa8804dd0a8836d11eaea83e22c4f8047a

knqyf263 · 2024-09-26T11:28:16Z

.github/workflows/test.yaml

+
+      - name: Restore test images from cache
+        uses: actions/cache@v4
+        id: restore-test-images


nit: We no longer need id.

We want to merge this PR soon. We can delete it later.

knqyf263 · 2024-09-26T11:35:56Z

Why is cache missing...?

Cache not found for input keys: cache-test-images-e572ef093aa79559e3e3fee0916b10fa8804dd0a8836d11eaea83e22c4f8047a, cache-test-images-

knqyf263 · 2024-09-26T11:37:21Z

By any chance, does the merge queue not share the cache with the normal workflow?

knqyf263 · 2024-09-26T11:40:41Z

I guess this is relevant. The merge queue creates a dedicated branch, which cannot access the cache in the cache-test-images branch.
https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache

DmitriyLewen · 2024-09-26T11:46:30Z

Workflow runs can restore caches created in either the current branch or the default branch (usually main)

UUIC Once we cache images in main branch, new branches created by the merge queue will also be able to access that cache (from the main branch), right?

knqyf263 · 2024-09-26T11:49:56Z

new branches created by the merge queue will also be able to access that cache (from the main branch), right?

Yes. We need to successfully pull images in the main branch only once .

* feat(vm): Support direct filesystem (aquasecurity#7058) Signed-off-by: yusuke.koyoshi <yusuke.koyoshi@bizreach.co.jp> * feat(cli)!: delete deprecated SBOM flags (aquasecurity#7266) Signed-off-by: knqyf263 <knqyf263@gmail.com> * feat(vm): support the Ext2/Ext3 filesystems (aquasecurity#6983) * fix(plugin): do not call GitHub content API for releases and tags (aquasecurity#7274) Signed-off-by: knqyf263 <knqyf263@gmail.com> * fix(java): Return error when trying to find a remote pom to avoid segfault (aquasecurity#7275) Co-authored-by: DmitriyLewen <dmitriy.lewen@smartforce.io> * fix(flag): incorrect behavior for deprected flag `--clear-cache` (aquasecurity#7281) * refactor(misconf): remove file filtering from parsers (aquasecurity#7289) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(vuln): Add `--detection-priority` flag for accuracy tuning (aquasecurity#7288) Signed-off-by: knqyf263 <knqyf263@gmail.com> * docs: add auto-generated config (aquasecurity#7261) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * fix(terraform): add aws_region name to presets (aquasecurity#7184) * perf(misconf): do not convert contents of a YAML file to string (aquasecurity#7292) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * refactor(misconf): remove unused universal scanner (aquasecurity#7293) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * perf(misconf): use json.Valid to check validity of JSON (aquasecurity#7308) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): load only submodule if it is specified in source (aquasecurity#7112) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): support for policy and bucket grants (aquasecurity#7284) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): do not set default value for default_cache_behavior (aquasecurity#7234) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): iterator argument support for dynamic blocks (aquasecurity#7236) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Co-authored-by: simar7 <1254783+simar7@users.noreply.github.com> * chore(deps): bump the common group across 1 directory with 7 updates (aquasecurity#7305) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * docs: update client/server docs for misconf and license scanning (aquasecurity#7277) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * docs: update links to packaging.python.org (aquasecurity#7318) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * perf(misconf): optimize work with context (aquasecurity#6968) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * refactor: replace ftypes.Gradle with packageurl.TypeGradle (aquasecurity#7323) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * docs: update air-gapped docs (aquasecurity#7160) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * docs(misconf): Update callsites to use correct naming (aquasecurity#7335) * chore(deps): bump the common group with 9 updates (aquasecurity#7333) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(misconf): change default TLS values for the Azure storage account (aquasecurity#7345) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * refactor(misconf): highlight only affected rows (aquasecurity#7310) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): wrap Azure PortRange in iac types (aquasecurity#7357) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): scanning support for YAML and JSON (aquasecurity#7311) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): variable support for Terraform Plan (aquasecurity#7228) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix: safely check if the directory exists (aquasecurity#7353) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * chore(deps): bump the aws group across 1 directory with 7 updates (aquasecurity#7358) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(server): add internal `--path-prefix` flag for client/server mode (aquasecurity#7321) Signed-off-by: knqyf263 <knqyf263@gmail.com> * chore(deps): bump trivy-checks (aquasecurity#7350) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * refactor(misconf): use slog (aquasecurity#7295) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): ignore duplicate checks (aquasecurity#7317) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): init frameworks before updating them (aquasecurity#7376) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): support deprecating for Go checks (aquasecurity#7377) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(python): use minimum version for pip packages (aquasecurity#7348) * docs: add pkg flags to config file page (aquasecurity#7370) * feat(misconf): Add support for using spec from on-disk bundle (aquasecurity#7179) * fix(report): escape `Message` field in `asff.tpl` template (aquasecurity#7401) * fix(misconf): use module to log when metadata retrieval fails (aquasecurity#7405) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): support for ignore by nested attributes (aquasecurity#7205) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): do not filter Terraform plan JSON by name (aquasecurity#7406) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): port and protocol support for EC2 networks (aquasecurity#7146) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * chore: fix allow rule of ignoring test files to make it case insensitive (aquasecurity#7415) * fix(secret): use only line with secret for long secret lines (aquasecurity#7412) * chore: update CODEOWNERS (aquasecurity#7398) Signed-off-by: knqyf263 <knqyf263@gmail.com> * feat(server): Make Trivy Server Multiplexer Exported (aquasecurity#7389) * feat(report): export modified findings in JSON (aquasecurity#7383) Signed-off-by: knqyf263 <knqyf263@gmail.com> * fix(sbom): use `NOASSERTION` for licenses fields in SPDX formats (aquasecurity#7403) * fix(misconf): do not register Rego libs in checks registry (aquasecurity#7420) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * chore(deps): Bump trivy-checks (aquasecurity#7417) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Co-authored-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): do not recreate filesystem map (aquasecurity#7416) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(secret): use `.eyJ` keyword for JWT secret (aquasecurity#7410) * fix(misconf): fix infer type for null value (aquasecurity#7424) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(aws): handle ECR repositories in different regions (aquasecurity#6217) Signed-off-by: Kevin Conner <kev.conner@getupcloud.com> * fix: logger initialization before flags parsing (aquasecurity#7372) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * fix(nodejs): check all `importers` to detect dev deps from pnpm-lock.yaml file (aquasecurity#7387) * test: add integration plugin tests (aquasecurity#7299) * feat(sbom): set User-Agent header on requests to Rekor (aquasecurity#7396) Signed-off-by: Bob Callaway <bcallaway@google.com> * fix(helm): explicitly define `kind` and `apiVersion` of `volumeClaimTemplate` element (aquasecurity#7362) * chore(deps): Bump trivy-checks and pin OPA (aquasecurity#7427) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Co-authored-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(java): add `test` scope support for `pom.xml` files (aquasecurity#7414) * fix(license): add license handling to JUnit template (aquasecurity#7409) * feat(go): use `toolchain` as `stdlib` version for `go.mod` files (aquasecurity#7163) * release: v0.55.0 [main] (aquasecurity#7271) * fix(license): stop spliting a long license text (aquasecurity#7336) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * refactor(java): add error/statusCode for logs when we can't get pom.xml/maven-metadata.xml from remote repo (aquasecurity#7451) * chore(helm): bump up Trivy Helm chart (aquasecurity#7441) * chore(deps): bump the common group across 1 directory with 19 updates (aquasecurity#7436) Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * chore(deps): bump the aws group with 6 updates (aquasecurity#7468) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(oracle): Update EOL date for Oracle 7 (aquasecurity#7480) * fix(report): change a receiver of MarshalJSON (aquasecurity#7483) Signed-off-by: knqyf263 <knqyf263@gmail.com> * fix(report): fix error with unmarshal of `ExperimentalModifiedFindings` (aquasecurity#7463) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * docs(oci): Add a note About the expected Media Type for the Trivy-DB OCI Artifact (aquasecurity#7449) * feat(license): improve license normalization (aquasecurity#7131) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: DmitriyLewen <dmitriy.lewen@smartforce.io> Co-authored-by: knqyf263 <knqyf263@gmail.com> * docs(db): add a manifest example (aquasecurity#7485) Signed-off-by: knqyf263 <knqyf263@gmail.com> * revert(java): stop supporting of `test` scope for `pom.xml` files (aquasecurity#7488) * docs: refine go docs (aquasecurity#7442) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * chore(vex): suppress openssl vulnerabilities (aquasecurity#7500) Signed-off-by: knqyf263 <knqyf263@gmail.com> * chore(deps): bump alpine from 3.20.0 to 3.20.3 (aquasecurity#7508) * chore(vex): add `CVE-2024-34155`, `CVE-2024-34156` and `CVE-2024-34158` in `trivy.openvex.json` (aquasecurity#7510) * fix(java): use `dependencyManagement` from root/child pom's for dependencies from parents (aquasecurity#7497) * refactor: split `.egg` and `packaging` analyzers (aquasecurity#7514) * feat(misconf): Register checks only when needed (aquasecurity#7435) * fix(misconf): Fix logging typo (aquasecurity#7473) * chore(deps): bump go-ebs-file (aquasecurity#7513) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(sbom): parse type `framework` as `library` when unmarshalling `CycloneDX` files (aquasecurity#7527) * refactor(misconf): pass options to Rego scanner as is (aquasecurity#7529) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(sbom): export bom-ref when converting a package to a component (aquasecurity#7340) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: amf <amf@macbook.local> Co-authored-by: knqyf263 <knqyf263@gmail.com> * perf(misconf): use port ranges instead of enumeration (aquasecurity#7549) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): Fixed scope for China Cloud (aquasecurity#7560) * docs(misconf): Add more info on how to use arbitrary JSON/YAML scan feat (aquasecurity#7458) * chore(deps): remove broken replaces for opa and discovery (aquasecurity#7600) * ci: cache test images for `integration`, `VM` and `module` tests (aquasecurity#7599) * ci: add `workflow_dispatch` trigger for test workflow. (aquasecurity#7606) * chore(deps): bump the common group across 1 directory with 20 updates (aquasecurity#7604) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * fix(db): check `DownloadedAt` for `trivy-java-db` (aquasecurity#7592) * fix: allow access to '..' in mapfs (aquasecurity#7575) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * test: use a local registry for remote scanning (aquasecurity#7607) Signed-off-by: knqyf263 <knqyf263@gmail.com> * fix(misconf): escape all special sequences (aquasecurity#7558) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): add ability to disable checks by ID (aquasecurity#7536) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Co-authored-by: Simar <simar@linux.com> * feat(suse): added SUSE Linux Enterprise Micro support (aquasecurity#7294) Signed-off-by: Marcus Meissner <meissner@suse.de> Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * fix(misconf): disable DS016 check for image history analyzer (aquasecurity#7540) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * ci: split `save` and `restore` cache actions (aquasecurity#7614) * refactor: fix auth error handling (aquasecurity#7615) Signed-off-by: knqyf263 <knqyf263@gmail.com> * feat(secret): enhance secret scanning for python binary files (aquasecurity#7223) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * feat(java): add empty versions if `pom.xml` dependency versions can't be detected (aquasecurity#7520) Co-authored-by: Teppei Fukuda <knqyf263@gmail.com> * test: use loaded image names (aquasecurity#7617) Signed-off-by: knqyf263 <knqyf263@gmail.com> * ci: don't use cache for `setup-go` (aquasecurity#7622) * feat: support multiple DB repositories for vulnerability and Java DB (aquasecurity#7605) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): Support `--skip-*` for all included modules (aquasecurity#7579) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Co-authored-by: nikpivkin <nikita.pivkin@smartforce.io> * chore: add prefixes to log messages (aquasecurity#7625) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: simar7 <1254783+simar7@users.noreply.github.com> * fix(misconf): Disable deprecated checks by default (aquasecurity#7632) * chore(deps): Bump trivy-checks to v1.1.0 (aquasecurity#7631) * fix(secret): change grafana token regex to find them without unquoted (aquasecurity#7627) * feat: support RPM archives (aquasecurity#7628) Signed-off-by: knqyf263 <knqyf263@gmail.com> * fix(misconf): not to warn about missing selectors of libraries (aquasecurity#7638) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * release: v0.56.0 [main] (aquasecurity#7447) * fix(db): fix javadb downloading error handling [backport: release/v0.56] (aquasecurity#7646) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Co-authored-by: Nikita Pivkin <nikita.pivkin@smartforce.io> * release: v0.56.1 [release/v0.56] (aquasecurity#7648) * fix(sbom): add options for DBs in private registries [backport: release/v0.56] (aquasecurity#7691) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: Teppei Fukuda <knqyf263@gmail.com> * fix(redhat): include arch in PURL qualifiers [backport: release/v0.56] (aquasecurity#7702) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: Teppei Fukuda <knqyf263@gmail.com> * release: v0.56.2 [release/v0.56] (aquasecurity#7694) * Make liveness probe configurable (#3) --------- Signed-off-by: yusuke.koyoshi <yusuke.koyoshi@bizreach.co.jp> Signed-off-by: knqyf263 <knqyf263@gmail.com> Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Kevin Conner <kev.conner@getupcloud.com> Signed-off-by: Bob Callaway <bcallaway@google.com> Signed-off-by: Marcus Meissner <meissner@suse.de> Co-authored-by: yusuke-koyoshi <92022336+yusuke-koyoshi@users.noreply.github.com> Co-authored-by: Teppei Fukuda <knqyf263@gmail.com> Co-authored-by: Aruneko <yuki.fujita@bizreach.co.jp> Co-authored-by: Colm O hEigeartaigh <coheigea@users.noreply.github.com> Co-authored-by: DmitriyLewen <dmitriy.lewen@smartforce.io> Co-authored-by: afdesk <work@afdesk.com> Co-authored-by: Nikita Pivkin <nikita.pivkin@smartforce.io> Co-authored-by: Alberto Donato <albertodonato@users.noreply.github.com> Co-authored-by: simar7 <1254783+simar7@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Itay Shakury <itay@itaysk.com> Co-authored-by: DmitriyLewen <91113035+DmitriyLewen@users.noreply.github.com> Co-authored-by: aasish-r <aasishrampalli1997@gmail.com> Co-authored-by: Ori <59772293+orizerah@users.noreply.github.com> Co-authored-by: Kevin Conner <kev.conner@gmail.com> Co-authored-by: Bob Callaway <bobcallaway@users.noreply.github.com> Co-authored-by: vhash <29121316+LucasVanHaaren@users.noreply.github.com> Co-authored-by: psibre <psibre@users.noreply.github.com> Co-authored-by: Aqua Security automated builds <54269356+aqua-bot@users.noreply.github.com> Co-authored-by: s-reddy1498 <41355782+s-reddy1498@users.noreply.github.com> Co-authored-by: Squiddim <82903357+Squiddim@users.noreply.github.com> Co-authored-by: Pierre Baumard <pierre.baumard@cnav.fr> Co-authored-by: Lior Kaplan <lior@kaplanopensource.co.il> Co-authored-by: amf <amf@macbook.local> Co-authored-by: bloomadcariad <adam.bloom@cariad.us> Co-authored-by: Sylvain Baubeau <lebauce@gmail.com> Co-authored-by: Simar <simar@linux.com> Co-authored-by: Marcus Meissner <meissner@suse.de> Co-authored-by: Samuel Gaist <samuel.gaist@idiap.ch>

…asecurity#7599)

ci(test): cache test images for integration tests

c75cead

DmitriyLewen self-assigned this Sep 26, 2024

DmitriyLewen added 3 commits September 26, 2024 13:23

ci(test): use trivy-test-images last update date for cache key

49461b0

ci(test): cache test VM images

208edcd

ci(test): cache test images for module tests

620ca42

DmitriyLewen changed the title ~~ci(test): cache test images for tests~~ ci: cache test images for integration, VM and module tests Sep 26, 2024

DmitriyLewen marked this pull request as ready for review September 26, 2024 08:33

DmitriyLewen requested a review from knqyf263 as a code owner September 26, 2024 08:33

DmitriyLewen commented Sep 26, 2024

View reviewed changes

knqyf263 reviewed Sep 26, 2024

View reviewed changes

refactor: use digest of image list for cache key

c015645

knqyf263 reviewed Sep 26, 2024

View reviewed changes

refactor: remove download steps

0609e5d

knqyf263 approved these changes Sep 26, 2024

View reviewed changes

knqyf263 enabled auto-merge September 26, 2024 11:28

knqyf263 added this pull request to the merge queue Sep 26, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 26, 2024

knqyf263 added this pull request to the merge queue Sep 26, 2024

Merged via the queue into aquasecurity:main with commit fea7250 Sep 26, 2024
12 checks passed

DmitriyLewen deleted the ci/cache-test-images branch September 26, 2024 12:02

fhielpos pushed a commit to giantswarm/trivy-upstream that referenced this pull request Dec 20, 2024

ci: cache test images for integration, VM and module tests (aqu…

1cbb451

…asecurity#7599)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: cache test images for `integration`, `VM` and `module` tests #7599

ci: cache test images for `integration`, `VM` and `module` tests #7599

DmitriyLewen commented Sep 26, 2024 •

edited

Loading

DmitriyLewen Sep 26, 2024

knqyf263 Sep 26, 2024 •

edited

Loading

knqyf263 Sep 26, 2024

knqyf263 Sep 26, 2024

DmitriyLewen Sep 26, 2024

knqyf263 Sep 26, 2024

DmitriyLewen Sep 26, 2024 •

edited

Loading

knqyf263 Sep 26, 2024 •

edited

Loading

knqyf263 Sep 26, 2024

knqyf263 commented Sep 26, 2024

knqyf263 Sep 26, 2024

DmitriyLewen Sep 26, 2024 •

edited

Loading

knqyf263 commented Sep 26, 2024 •

edited

Loading

knqyf263 commented Sep 26, 2024

knqyf263 Sep 26, 2024

knqyf263 Sep 26, 2024

knqyf263 commented Sep 26, 2024

knqyf263 commented Sep 26, 2024

knqyf263 commented Sep 26, 2024

DmitriyLewen commented Sep 26, 2024

knqyf263 commented Sep 26, 2024

	tags, err := crane.ListTags(testImages)
	if err != nil {
	return err
	}
	for _, tag := range tags {
	fileName := tag + ".tar.gz"
	filePath := filepath.Join(dir, fileName)
	if exists(filePath) {
	continue
	}

	_, _ = cli.ImageRemove(ctx, tt.remoteImageName, dimage.RemoveOptions{
	Force: true,
	PruneChildren: true,
	})

ci: cache test images for integration, VM and module tests #7599

ci: cache test images for integration, VM and module tests #7599

Conversation

DmitriyLewen commented Sep 26, 2024 • edited Loading

Description

Note

Related issues

Checklist

Choose a reason for hiding this comment

knqyf263 Sep 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DmitriyLewen Sep 26, 2024 • edited Loading

Choose a reason for hiding this comment

knqyf263 Sep 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knqyf263 commented Sep 26, 2024

Choose a reason for hiding this comment

DmitriyLewen Sep 26, 2024 • edited Loading

Choose a reason for hiding this comment

knqyf263 commented Sep 26, 2024 • edited Loading

knqyf263 commented Sep 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knqyf263 commented Sep 26, 2024

knqyf263 commented Sep 26, 2024

knqyf263 commented Sep 26, 2024

DmitriyLewen commented Sep 26, 2024

knqyf263 commented Sep 26, 2024

ci: cache test images for `integration`, `VM` and `module` tests #7599

ci: cache test images for `integration`, `VM` and `module` tests #7599

DmitriyLewen commented Sep 26, 2024 •

edited

Loading

knqyf263 Sep 26, 2024 •

edited

Loading

DmitriyLewen Sep 26, 2024 •

edited

Loading

knqyf263 Sep 26, 2024 •

edited

Loading

DmitriyLewen Sep 26, 2024 •

edited

Loading

knqyf263 commented Sep 26, 2024 •

edited

Loading