diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
index ee0f3701c114..5bc20d744c5e 100644
--- a/.github/CONTRIBUTING.md
+++ b/.github/CONTRIBUTING.md
@@ -59,8 +59,6 @@ The following command installs `buildkitd` and `buildctl` to `/usr/local/bin`:
make && sudo make install
```
-You can also use `make binaries-all` to prepare `buildkitd.containerd_only` and `buildkitd.oci_only`.
-
To build containerized `moby/buildkit:local` and `moby/buildkit:local-rootless` images:
```bash
make images
@@ -152,7 +150,7 @@ otherwise cleanup our project.
### Debian GNU/Linux 10
Add `kernel.unprivileged_userns_clone=1` to `/etc/sysctl.conf` (or `/etc/sysctl.d`) and run `sudo sysctl -p`.
@@ -16,8 +40,8 @@ This step is not needed for RHEL/CentOS 8 and later.
### Fedora, before kernel 5.13
You may have to disable SELinux, or run BuildKit with `--oci-worker-snapshotter=fuse-overlayfs`.
-### Container-Optimized OS from Google
-:warning: Currently unsupported. See [#879](https://github.com/moby/buildkit/issues/879).
+
+
## Known limitations
* Using the `overlayfs` snapshotter requires kernel >= 5.11 or Ubuntu kernel.
@@ -77,6 +101,9 @@ $ rootlesskit buildkitd --oci-worker-snapshotter=native
### Error related to `newuidmap` or `/etc/subuid`
See https://rootlesscontaine.rs/getting-started/common/subuid/
+### Error `Options:[rbind ro]}]: operation not permitted`
+Make sure to mount an `emptyDir` volume on `/home/user/.local/share/buildkit` .
+
## Containerized deployment
### Kubernetes
diff --git a/docs/solver.md b/docs/solver.md
deleted file mode 100644
index 45b81c5cb078..000000000000
--- a/docs/solver.md
+++ /dev/null
@@ -1,161 +0,0 @@
-## Buildkit solver design
-
-The solver is a component in BuildKit responsible for parsing the build definition and scheduling the operations to the workers for execution.
-
-Solver package is heavily optimized for deduplication of work, concurrent requests, remote and local caching and different per-vertex caching modes. It also allows operations and frontends to call back to itself with new definition that they have generated.
-
-The implementation of the solver is quite complicated, mostly because it is supposed to be performant with snapshot-based storage layer and distribution model using layer tarballs. It is expected that calculating the content based checksum of snapshots between every operation or after every command execution is too slow for common use cases and needs to be postponed to when it is likely to have a meaningful impact. Ideally, the user shouldn't realize that these optimizations are taking place and just get intuitive caching. It is also hoped that if some implementations can provide better cache capabilities, the solver would take advantage of that without requiring significant modification.
-
-In addition to avoiding content checksum scanning the implementation is also designed to make decisions with minimum available data. For example, for remote caching sources to be effective the solver will not require the cache to be loaded or exists for all the vertexes in the graph but will only load it for the final node that is determined to match cache. As another example, if one of the inputs (for example image) can produce a definition based cache match for a vertex, and another (for example local source files) can only produce a content-based(slower) cache match, the solver is designed to detect it and skip content-based check for the first input(that would cause a pull to happen).
-
-### Build definition
-
-The solver takes in a build definition in the form of a content addressable operation definition that forms a graph.
-
-A vertex in this graph is defined by these properties:
-
-```go
-type Vertex interface {
- Digest() digest.Digest
- Options() VertexOptions
- Sys() interface{}
- Inputs() []Edge
- Name() string
-}
-
-type Edge struct {
- Index Index
- Vertex Vertex
-}
-
-type Index int
-```
-
-Every vertex has a content-addressable digest that represents a checksum of the definition graph up to that vertex including all of its inputs. If two vertexes have the same checksum, they are considered identical when they are executing concurrently. That means that if two other vertexes request a vertex with the same digest as an input, they will wait for the same operation to finish.
-
-The vertex digest can only be used for comparison while the solver is running and not between different invocations. For example, if parallel builds require using `docker.io/library/alpine:latest` image as one of the operations, it is pulled only once. But if a build using `docker.io/library/alpine:latest` was built earlier, the checksum based on that name can't be used for finding if the vertex was already built because the image might have changed in the registry and "latest" tag might be pointing to another image.
-
-`Sys()` method returns an object that is used to resolve the executor for the operation. This is how a definition can pass logic to the worker that will execute the task associated with the vertex, without the solver needing to know anything about the implementation. When the solver needs to execute a vertex, it will send this object to a worker, so the worker needs to be configured to understand the object returned by `Sys()`. The solver itself doesn't care how the operations are implemented and therefore doesn't define a type for this value. In LLB solver this value would be with type `llb.Op`.
-
-`Inputs()` returns an array of other vertexes the current vertex depends on. A vertex may have zero inputs. After an operation has executed, it returns an array of return references. If another operation wants to depend on any of these references they would define an input with that vertex and an index of the reference from the return array(starting from zero). Inputs need to be contained in the `Digest()` of the vertex - two vertexes with different inputs should never have the same digest.
-
-Options contain extra information that can be associated with the vertex but what doesn't change the definition(or equality check) of it. Normally this is either a hint to the solver, for example, to ignore cache when executing. It can also be used for associating messages with the vertex that can be helpful for tracing purposes.
-
-
-### Operation interface
-
-Operation interface is how the solver can evaluate the properties of the actual vertex operation. These methods run on the worker, and their implementation is determined by the value of `vertex.Sys()`. The solver is configured with a "resolve" function that can convert a `vertex.Sys()` into an `Op`.
-
-```go
-// Op is an implementation for running a vertex
-type Op interface {
- // CacheMap returns structure describing how the operation is cached.
- // Currently only roots are allowed to return multiple cache maps per op.
- CacheMap(context.Context, int) (*CacheMap, bool, error)
- // Exec runs an operation given results from previous operations.
- // Note that this is not the process execution but can have any definition.
- Exec(ctx context.Context, inputs []Result) (outputs []Result, err error)
-}
-
-type CacheMap struct {
- // Digest is a base digest for operation that needs to be combined with
- // inputs cache or selectors for dependencies.
- Digest digest.Digest
- Deps []struct {
- // Optional digest that is merged with the cache key of the input
- Selector digest.Digest
- // Optional function that returns a digest for the input based on its
- // return value
- ComputeDigestFunc ResultBasedCacheFunc
- }
-}
-
-type ResultBasedCacheFunc func(context.Context, Result) (digest.Digest, error)
-
-
-// Result is an abstract return value for a solve
-type Result interface {
- ID() string
- Release(context.Context) error
- Sys() interface{}
-}
-```
-
-There are two functions that every operation defines. One describes how to calculate a cache key for a vertex and another how to execute it.
-
-`CacheMap` is a description for calculating the cache key. It contains a digest that is combined with the cache keys of the inputs to determine the stable checksum that can be used to cache the operation result. For the vertexes that don't have inputs(roots), it is important that this digest is a stable secure checksum. For example, in LLB this digest is a manifest digest for container images or a commit SHA for git sources.
-
-`CacheMap` may also define optional selectors or content-based cache functions for its inputs. A selector is combined with the input cache key and useful for describing when different parts of an input are being used, and inputs cache key needs to be customized. Content-based cache function allows computing a new cache key for an input after it has completed. In LLB this is used for calculating cache key based on the checksum of file contents of the input snapshots.
-
-`Exec` executes the operation defined by a vertex by passing in the results of the inputs.
-
-
-### Shared graph
-
-After new build request is sent to the solver, it first loads all the vertexes to the shared graph structure. For status tracking, a job instance needs to be created, and vertexes are loaded through jobs. A job ID is assigned to every vertex. If vertex with the same digest has already been loaded to the shared graph, a new job ID is appended to the existing record. When the job finishes, it removes all of its references from the loaded vertex. The resources are released if no more references remain.
-
-Loading a vertex also creates a progress writer associated with it and sets up the cache sources associated with the specific vertex.
-
-After vertexes have been loaded to the job, it is safe to request a result from an edge pointing to a previously loaded vertex. To do this `build(ctx, Edge) (CachedResult, error)` method is called on the static scheduler instance associated with the solver.
-
-### Scheduler
-
-The scheduler is a component responsible for invoking the individual operations needed to find the result for the graph. While the build definition is defined with vertexes, the scheduler is solving edges. In the case of LLB solver, a result of a solved edge is associated with a snapshot. Usually, to solve an edge, the input edges need to be solved first and this can be done concurrently, but there are many exceptions like edge may be cached but its input might be not, or solving one input might cause a cache hit while solving others would just be wasteful. Scheduler tries do handle all these cases.
-
-The scheduler is implemented as a single threaded non-blocking event loop. The single threaded constraint is for simplicity and might be removed in the future - currently, it is not known if this would have any performance impact. All the events in the scheduler have one fixed sender and receiver. The interface for interacting with the scheduler is to create a "pipe" between a sender and a receiver. One or both sides of the pipe may be an edge instance of the graph. If a pipe is added it to the scheduler and an edge receives an event from the pipe, the scheduler will "unpark" that edge so it can process all the events it had received.
-
-The unpark handler for an edge needs to be non-blocking and execute quickly. The edge will process the data from the incoming events and update its internal state. When calling unpark, the scheduler has already separated out the sender and receiver sides of the pipes that in the code are referred as incoming and outgoing requests. The incoming requests are usually requests to retrieve a result or a cache key from an edge. If it appears that an edge doesn't have enough internal state to satisfy the requests, it can make new pipes and register them with the scheduler. These new pipes are generally of two types: ones asking for some async function to be completed and others that request an input edge to reach a specific state first.
-
-To avoid bugs and deadlocks in this logic, the unpark method needs to follow the following rules. If unpark has finished without completing all incoming requests it needs to create outgoing requests. Similarly, if an incoming request remains pending, at least one outgoing request needs to exist as well. Failing to comply with this rule will cause the scheduler to panic as a precaution to avoid leaks and hiding errors.
-
-### Edge state
-
-During unpark, edge state is incremented until it can fulfill the incoming requests.
-
-An edge can be in the following states: initial, cache-fast, cache-slow, completed. Completed edge contains a reference to the final result, in-progress edge may have zero or more cache keys.
-
-The initial state is the starting state for any edge. If a state has reached a cache-fast state, it means that all the definition based cache key lookups have been performed. Cache-slow means that content-based cache lookup has been performed as well. If possible, the scheduler will avoid looking up the slow keys of inputs if they are unnecessary for solving current edge.
-
-The unpark method is split into four phases. The first phase processes all incoming events (responses from outgoing requests or new incoming requests) that caused the unpark to be called. These contain responses from async functions like calls to get the cachemap, execution result or content-based checksum for an input, or responses from input edges when their state or number of cache keys has changed. All the results are stored in edge's internal state. For the new cache keys, a query is performed to determine if any of them can create potential matches to the current edge.
-
-After that, if any of the updates caused changes to edge's properties, a new state is calculated for the current vertex. In this step, all potential cache keys from inputs can cause new cache keys for the edge to be created and the status of an edge might be updated.
-
-Third, the edge will go over all of its incoming requests, to determine if the current internal state is sufficient for satisfying them all. There are a couple of possibilities how this check may end up. If all requests can be completed and there are no outgoing requests the requests finish and unpark method returns. If there are outgoing requests but the edge has reached the completed state or all incoming requests have been canceled, the outgoing requests are canceled. This is an async operation as well and will cause unpark to be called again after completion. If this condition didn't apply but requests could be completed and there are outgoing requests, then the incoming request is answered but not completed. The receiver can then decide to cancel this request if needed. If no new data has appeared to answer the incoming requests, the desired state for an edge is determined for an edge from the incoming requests, and we continue to the next step.
-
-The fourth step sets up outgoing requests based on the desired state determined in the third step. If the current state requires calling any async functions to move forward then it is done here. We will also loop through all the inputs to determine if it is important to raise their desired state. Depending on what inputs can produce content based cache keys and what inputs have already returned possible cache matches, the desired state for inputs may be raised at different times.
-
-When an edge needs to resolve an operation to call the async `CacheMap` and `Exec` methods, it does so by calling back to the shared graph. This makes sure that two different edges pointing to the same vertex do not execute twice. The result values for the operation that is shared by the edges is also cached until the vertex is cleaned up. Progress reporting is also handled and forwarded to the job through this shared vertex instance.
-
-Edge state is cleaned up when a final job that loaded the vertexes that they are connected to is discarded.
-
-
-### Cache providers
-
-Cache providers determine if there is a result that matches the cache keys generated during the build that could be reused instead of fully reevaluating the vertex and its inputs. There can be multiple cache providers, and specific providers can be defined per vertex using the vertex options.
-
-There are multiple backend implementations for cache providers, in-memory one used in unit tests, the default local one using bbolt and one based on cache manifests in a remote registry.
-
-Simplified cache provider has following methods:
-
-```go
-Query(...) ([]*CacheKey, error)
-Records(ck *CacheKey) ([]*CacheRecord, error)
-Load(ctx context.Context, rec *CacheRecord) (Result, error)
-Save(key *CacheKey, s Result) (*ExportableCacheKey, error)
-```
-
-Query method is used to determine if there exist a possible cache link between the input and a vertex. It takes parameters provided by `op.CacheMap` and cache keys returned by the calling the same method on its inputs.
-
-If a cache key has been found, the matching records can be asked for them. A cache key can have zero or more records. Having a record means that a cached result can be loaded for a specific vertex. The solver supports partial cache chains, meaning that not all inputs need to have a cache record to match cache for a vertex.
-
-Load method is used to load a specific record into a result reference. This value is the same type as the one returned by the `op.Exec` method.
-
-Save allows adding more records to the cache.
-
-### Merging edges
-
-One final piece of solver logic allows merging two edges into one when they have both returned the same cache key. In practice, this appears for example when a build uses image references `alpine:latest` and `alpine@sha256:abcabc` in its definition and they actually point to the same image. Another case where this appears is when same source files from different sources are being used as part of the build.
-
-After scheduler has called `unpark()` on an edge it checks it the method added any new cache keys to its state. If it did it will check its internal index if another active edge already exists with the same cache key. If it does it performs some basic validation, for example checking that the new edge has not explicitly asked cache to be ignored, and if it passes, merges the states of two edges.
-
-In the result of the merge, the edge that was checked is deleted, its ongoing requests are canceled and the incoming ones are added to the original edge.
\ No newline at end of file
diff --git a/examples/README.md b/examples/README.md
index e43ccda44654..0e0c13c17410 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -34,7 +34,7 @@ Different versions of the example scripts show different ways of describing the
- `./buildkit1` - cloning git repositories has been separated for extra concurrency.
- `./buildkit2` - uses git sources directly instead of running `git clone`, allowing better performance and much safer caching.
- `./buildkit3` - allows using local source files for separate components eg. `./buildkit3 --runc=local | buildctl build --local runc-src=some/local/path`
-- `./buildkit4` - uses MergeOp to optimize copy chains for better caching behavior (see `docs/merge+diff.md` for more details)
+- `./buildkit4` - uses MergeOp to optimize copy chains for better caching behavior (see `docs/dev/merge-diff.md` for more details)
- `./dockerfile2llb` - can be used to convert a Dockerfile to LLB for debugging purposes
- `./nested-llb` - shows how to use nested invocation to generate LLB
- `./gobuild` - shows how to use nested invocation to generate LLB for Go package internal dependencies
diff --git a/examples/build-using-dockerfile/README.md b/examples/build-using-dockerfile/README.md
index b0087ad9c903..1ec17bdf198b 100644
--- a/examples/build-using-dockerfile/README.md
+++ b/examples/build-using-dockerfile/README.md
@@ -1,6 +1,5 @@
# `build-using-dockerfile` example
-:information_source: [BuildKit has been integrated to `docker build` since Docker 18.06.](https://docs.docker.com/develop/develop-images/build_enhancements/)
The `build-using-dockerfile` CLI is just provided as an example for writing a BuildKit client application.
For people familiar with `docker build` command, `build-using-dockerfile` is provided as an example for building Dockerfiles with BuildKit using a syntax similar to `docker build`.
diff --git a/examples/buildctl-daemonless/buildctl-daemonless.sh b/examples/buildctl-daemonless/buildctl-daemonless.sh
index 15885dadb783..ab181d16c7e2 100755
--- a/examples/buildctl-daemonless/buildctl-daemonless.sh
+++ b/examples/buildctl-daemonless/buildctl-daemonless.sh
@@ -19,7 +19,7 @@ set -eu
# * addr
# * log
tmp=$(mktemp -d /tmp/buildctl-daemonless.XXXXXX)
-trap "kill \$(cat $tmp/pid); wait \$(cat $tmp/pid) || true; rm -rf $tmp" EXIT
+trap "kill \$(cat $tmp/pid) || true; wait \$(cat $tmp/pid) || true; rm -rf $tmp" EXIT
startBuildkitd() {
addr=
diff --git a/examples/buildkit0/buildkit.go b/examples/buildkit0/buildkit.go
index 78f988d8584d..24810e730ff5 100644
--- a/examples/buildkit0/buildkit.go
+++ b/examples/buildkit0/buildkit.go
@@ -33,7 +33,7 @@ func main() {
}
func goBuildBase() llb.State {
- goAlpine := llb.Image("docker.io/library/golang:1.17-alpine")
+ goAlpine := llb.Image("docker.io/library/golang:1.19-alpine")
return goAlpine.
AddEnv("PATH", "/usr/local/go/bin:"+system.DefaultPathEnvUnix).
AddEnv("GOPATH", "/go").
diff --git a/examples/buildkit1/buildkit.go b/examples/buildkit1/buildkit.go
index 98793d0ab6e2..9f8201b2bddf 100644
--- a/examples/buildkit1/buildkit.go
+++ b/examples/buildkit1/buildkit.go
@@ -33,7 +33,7 @@ func main() {
}
func goBuildBase() llb.State {
- goAlpine := llb.Image("docker.io/library/golang:1.17-alpine")
+ goAlpine := llb.Image("docker.io/library/golang:1.19-alpine")
return goAlpine.
AddEnv("PATH", "/usr/local/go/bin:"+system.DefaultPathEnvUnix).
AddEnv("GOPATH", "/go").
diff --git a/examples/buildkit2/buildkit.go b/examples/buildkit2/buildkit.go
index 7a88562cf72e..5ae6b201678b 100644
--- a/examples/buildkit2/buildkit.go
+++ b/examples/buildkit2/buildkit.go
@@ -33,7 +33,7 @@ func main() {
}
func goBuildBase() llb.State {
- goAlpine := llb.Image("docker.io/library/golang:1.17-alpine")
+ goAlpine := llb.Image("docker.io/library/golang:1.19-alpine")
return goAlpine.
AddEnv("PATH", "/usr/local/go/bin:"+system.DefaultPathEnvUnix).
AddEnv("GOPATH", "/go").
diff --git a/examples/buildkit3/buildkit.go b/examples/buildkit3/buildkit.go
index fa985f18eb84..58de1ebe0666 100644
--- a/examples/buildkit3/buildkit.go
+++ b/examples/buildkit3/buildkit.go
@@ -34,7 +34,7 @@ func main() {
}
func goBuildBase() llb.State {
- goAlpine := llb.Image("docker.io/library/golang:1.17-alpine")
+ goAlpine := llb.Image("docker.io/library/golang:1.19-alpine")
return goAlpine.
AddEnv("PATH", "/usr/local/go/bin:"+system.DefaultPathEnvUnix).
AddEnv("GOPATH", "/go").
diff --git a/examples/buildkit4/buildkit.go b/examples/buildkit4/buildkit.go
index c20f13ea0c05..0bae9e7899e8 100644
--- a/examples/buildkit4/buildkit.go
+++ b/examples/buildkit4/buildkit.go
@@ -37,7 +37,7 @@ func main() {
}
func goBuildBase() llb.State {
- goAlpine := llb.Image("docker.io/library/golang:1.17-alpine")
+ goAlpine := llb.Image("docker.io/library/golang:1.19-alpine")
return goAlpine.
AddEnv("PATH", "/usr/local/go/bin:"+system.DefaultPathEnvUnix).
AddEnv("GOPATH", "/go").
diff --git a/examples/dockerfile2llb/main.go b/examples/dockerfile2llb/main.go
index 94455565b1ec..2fd693a4ae80 100644
--- a/examples/dockerfile2llb/main.go
+++ b/examples/dockerfile2llb/main.go
@@ -2,9 +2,9 @@ package main
import (
"context"
+ "encoding/json"
"flag"
- "io/ioutil"
- "log"
+ "io"
"os"
"github.com/moby/buildkit/client/llb"
@@ -12,40 +12,66 @@ import (
"github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb"
"github.com/moby/buildkit/solver/pb"
"github.com/moby/buildkit/util/appcontext"
+ "github.com/sirupsen/logrus"
)
type buildOpt struct {
- target string
+ target string
+ partialImageConfigFile string
+ partialMetadataFile string
}
func main() {
+ if err := xmain(); err != nil {
+ logrus.Fatal(err)
+ }
+}
+
+func xmain() error {
var opt buildOpt
flag.StringVar(&opt.target, "target", "", "target stage")
+ flag.StringVar(&opt.partialImageConfigFile, "partial-image-config-file", "", "Output partial image config as a JSON file")
+ flag.StringVar(&opt.partialMetadataFile, "partial-metadata-file", "", "Output partial metadata sa a JSON file")
flag.Parse()
- df, err := ioutil.ReadAll(os.Stdin)
+ df, err := io.ReadAll(os.Stdin)
if err != nil {
- panic(err)
+ return err
}
caps := pb.Caps.CapSet(pb.Caps.All())
- state, img, bi, err := dockerfile2llb.Dockerfile2LLB(appcontext.Context(), df, dockerfile2llb.ConvertOpt{
+ state, img, _, err := dockerfile2llb.Dockerfile2LLB(appcontext.Context(), df, dockerfile2llb.ConvertOpt{
MetaResolver: imagemetaresolver.Default(),
Target: opt.target,
LLBCaps: &caps,
})
if err != nil {
- log.Printf("err: %+v", err)
- panic(err)
+ return err
}
- _ = img
- _ = bi
-
dt, err := state.Marshal(context.TODO())
if err != nil {
- panic(err)
+ return err
+ }
+ if err := llb.WriteTo(dt, os.Stdout); err != nil {
+ return err
+ }
+ if opt.partialImageConfigFile != "" {
+ if err := writeJSON(opt.partialImageConfigFile, img); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func writeJSON(f string, x interface{}) error {
+ b, err := json.Marshal(x)
+ if err != nil {
+ return err
+ }
+ if err := os.RemoveAll(f); err != nil {
+ return err
}
- llb.WriteTo(dt, os.Stdout)
+ return os.WriteFile(f, b, 0o644)
}
diff --git a/examples/kubernetes/README.md b/examples/kubernetes/README.md
index 66a36c102a9b..c8973dc5645d 100644
--- a/examples/kubernetes/README.md
+++ b/examples/kubernetes/README.md
@@ -7,8 +7,7 @@ This directory contains Kubernetes manifests for `Pod`, `Deployment` (with `Serv
* `Job`: good if you don't want to have daemon pods
Using Rootless mode (`*.rootless.yaml`) is recommended because Rootless mode image is executed as non-root user (UID 1000) and doesn't need `securityContext.privileged`.
-
-:warning: Rootless mode may not work on some host kernels. See [`../../docs/rootless.md`](../../docs/rootless.md).
+See [`../../docs/rootless.md`](../../docs/rootless.md).
See also ["Building Images Efficiently And Securely On Kubernetes With BuildKit" (KubeCon EU 2019)](https://kccnceu19.sched.com/event/MPX5).
diff --git a/examples/kubernetes/consistenthash/main.go b/examples/kubernetes/consistenthash/main.go
index ebe5f64f2840..4b100777ce3c 100644
--- a/examples/kubernetes/consistenthash/main.go
+++ b/examples/kubernetes/consistenthash/main.go
@@ -16,7 +16,7 @@ package main
import (
"fmt"
- "io/ioutil"
+ "io"
"os"
"strings"
@@ -38,7 +38,7 @@ func xmain() error {
return errors.New("should not reach here")
}
key := os.Args[1]
- stdin, err := ioutil.ReadAll(os.Stdin)
+ stdin, err := io.ReadAll(os.Stdin)
if err != nil {
return err
}
diff --git a/examples/kubernetes/deployment+service.rootless.yaml b/examples/kubernetes/deployment+service.rootless.yaml
index 00f89f7342f9..0b554096fde6 100644
--- a/examples/kubernetes/deployment+service.rootless.yaml
+++ b/examples/kubernetes/deployment+service.rootless.yaml
@@ -15,7 +15,6 @@ spec:
app: buildkitd
annotations:
container.apparmor.security.beta.kubernetes.io/buildkitd: unconfined
- container.seccomp.security.alpha.kubernetes.io/buildkitd: unconfined
# see buildkit/docs/rootless.md for caveats of rootless mode
spec:
containers:
@@ -52,6 +51,9 @@ spec:
initialDelaySeconds: 5
periodSeconds: 30
securityContext:
+ # Needs Kubernetes >= 1.19
+ seccompProfile:
+ type: Unconfined
# To change UID/GID, you need to rebuild the image
runAsUser: 1000
runAsGroup: 1000
@@ -61,11 +63,19 @@ spec:
- name: certs
readOnly: true
mountPath: /certs
+ # Dockerfile has `VOLUME /home/user/.local/share/buildkit` by default too,
+ # but the default VOLUME does not work with rootless on Google's Container-Optimized OS
+ # as it is mounted with `nosuid,nodev`.
+ # https://github.com/moby/buildkit/issues/879#issuecomment-1240347038
+ - mountPath: /home/user/.local/share/buildkit
+ name: buildkitd
volumes:
# buildkit-daemon-certs must contain ca.pem, cert.pem, and key.pem
- name: certs
secret:
secretName: buildkit-daemon-certs
+ - name: buildkitd
+ emptyDir: {}
---
apiVersion: v1
kind: Service
diff --git a/examples/kubernetes/job.rootless.yaml b/examples/kubernetes/job.rootless.yaml
index 7c9941d05434..06e608c6ab35 100644
--- a/examples/kubernetes/job.rootless.yaml
+++ b/examples/kubernetes/job.rootless.yaml
@@ -7,7 +7,6 @@ spec:
metadata:
annotations:
container.apparmor.security.beta.kubernetes.io/buildkit: unconfined
- container.seccomp.security.alpha.kubernetes.io/buildkit: unconfined
# see buildkit/docs/rootless.md for caveats of rootless mode
spec:
restartPolicy: Never
@@ -43,6 +42,9 @@ spec:
# To push the image to a registry, add
# `--output type=image,name=docker.io/username/image,push=true`
securityContext:
+ # Needs Kubernetes >= 1.19
+ seccompProfile:
+ type: Unconfined
# To change UID/GID, you need to rebuild the image
runAsUser: 1000
runAsGroup: 1000
@@ -50,8 +52,16 @@ spec:
- name: workspace
readOnly: true
mountPath: /workspace
+ # Dockerfile has `VOLUME /home/user/.local/share/buildkit` by default too,
+ # but the default VOLUME does not work with rootless on Google's Container-Optimized OS
+ # as it is mounted with `nosuid,nodev`.
+ # https://github.com/moby/buildkit/issues/879#issuecomment-1240347038
+ - mountPath: /home/user/.local/share/buildkit
+ name: buildkitd
# To push the image, you also need to create `~/.docker/config.json` secret
# and set $DOCKER_CONFIG to `/path/to/.docker` directory.
volumes:
- name: workspace
emptyDir: {}
+ - name: buildkitd
+ emptyDir: {}
diff --git a/examples/kubernetes/pod.rootless.yaml b/examples/kubernetes/pod.rootless.yaml
index ea63b35d177a..130ea43633fe 100644
--- a/examples/kubernetes/pod.rootless.yaml
+++ b/examples/kubernetes/pod.rootless.yaml
@@ -4,7 +4,6 @@ metadata:
name: buildkitd
annotations:
container.apparmor.security.beta.kubernetes.io/buildkitd: unconfined
- container.seccomp.security.alpha.kubernetes.io/buildkitd: unconfined
# see buildkit/docs/rootless.md for caveats of rootless mode
spec:
containers:
@@ -29,6 +28,19 @@ spec:
initialDelaySeconds: 5
periodSeconds: 30
securityContext:
+ # Needs Kubernetes >= 1.19
+ seccompProfile:
+ type: Unconfined
# To change UID/GID, you need to rebuild the image
runAsUser: 1000
runAsGroup: 1000
+ volumeMounts:
+ # Dockerfile has `VOLUME /home/user/.local/share/buildkit` by default too,
+ # but the default VOLUME does not work with rootless on Google's Container-Optimized OS
+ # as it is mounted with `nosuid,nodev`.
+ # https://github.com/moby/buildkit/issues/879#issuecomment-1240347038
+ - mountPath: /home/user/.local/share/buildkit
+ name: buildkitd
+ volumes:
+ - name: buildkitd
+ emptyDir: {}
diff --git a/examples/kubernetes/statefulset.rootless.yaml b/examples/kubernetes/statefulset.rootless.yaml
index e67c5a0bf246..0533d2a1004f 100644
--- a/examples/kubernetes/statefulset.rootless.yaml
+++ b/examples/kubernetes/statefulset.rootless.yaml
@@ -17,7 +17,6 @@ spec:
app: buildkitd
annotations:
container.apparmor.security.beta.kubernetes.io/buildkitd: unconfined
- container.seccomp.security.alpha.kubernetes.io/buildkitd: unconfined
# see buildkit/docs/rootless.md for caveats of rootless mode
spec:
containers:
@@ -42,6 +41,19 @@ spec:
initialDelaySeconds: 5
periodSeconds: 30
securityContext:
+ # Needs Kubernetes >= 1.19
+ seccompProfile:
+ type: Unconfined
# To change UID/GID, you need to rebuild the image
runAsUser: 1000
runAsGroup: 1000
+ volumeMounts:
+ # Dockerfile has `VOLUME /home/user/.local/share/buildkit` by default too,
+ # but the default VOLUME does not work with rootless on Google's Container-Optimized OS
+ # as it is mounted with `nosuid,nodev`.
+ # https://github.com/moby/buildkit/issues/879#issuecomment-1240347038
+ - mountPath: /home/user/.local/share/buildkit
+ name: buildkitd
+ volumes:
+ - name: buildkitd
+ emptyDir: {}
diff --git a/examples/nested-llb/main.go b/examples/nested-llb/main.go
index 74eb58864af1..93f9d41c2dcc 100644
--- a/examples/nested-llb/main.go
+++ b/examples/nested-llb/main.go
@@ -32,7 +32,7 @@ func main() {
}
func goBuildBase() llb.State {
- goAlpine := llb.Image("docker.io/library/golang:1.17-alpine")
+ goAlpine := llb.Image("docker.io/library/golang:1.19-alpine")
return goAlpine.
AddEnv("PATH", "/usr/local/go/bin:"+system.DefaultPathEnvUnix).
AddEnv("GOPATH", "/go").
diff --git a/executor/containerdexecutor/executor.go b/executor/containerdexecutor/executor.go
index 43a05cccefe9..ac195c431588 100644
--- a/executor/containerdexecutor/executor.go
+++ b/executor/containerdexecutor/executor.go
@@ -3,7 +3,6 @@ package containerdexecutor
import (
"context"
"io"
- "io/ioutil"
"os"
"path/filepath"
"sync"
@@ -41,12 +40,26 @@ type containerdExecutor struct {
running map[string]chan error
mu sync.Mutex
apparmorProfile string
+ selinux bool
traceSocket string
rootless bool
}
+// OnCreateRuntimer provides an alternative to OCI hooks for applying network
+// configuration to a container. If the [network.Provider] returns a
+// [network.Namespace] which also implements this interface, the containerd
+// executor will run the callback at the appropriate point in the container
+// lifecycle.
+type OnCreateRuntimer interface {
+ // OnCreateRuntime is analogous to the createRuntime OCI hook. The
+ // function is called after the container is created, before the user
+ // process has been executed. The argument is the container PID in the
+ // runtime namespace.
+ OnCreateRuntime(pid uint32) error
+}
+
// New creates a new executor backed by connection to containerd API
-func New(client *containerd.Client, root, cgroup string, networkProviders map[pb.NetMode]network.Provider, dnsConfig *oci.DNSConfig, apparmorProfile string, traceSocket string, rootless bool) executor.Executor {
+func New(client *containerd.Client, root, cgroup string, networkProviders map[pb.NetMode]network.Provider, dnsConfig *oci.DNSConfig, apparmorProfile string, selinux bool, traceSocket string, rootless bool) executor.Executor {
// clean up old hosts/resolv.conf file. ignore errors
os.RemoveAll(filepath.Join(root, "hosts"))
os.RemoveAll(filepath.Join(root, "resolv.conf"))
@@ -59,6 +72,7 @@ func New(client *containerd.Client, root, cgroup string, networkProviders map[pb
dnsConfig: dnsConfig,
running: make(map[string]chan error),
apparmorProfile: apparmorProfile,
+ selinux: selinux,
traceSocket: traceSocket,
rootless: rootless,
}
@@ -121,7 +135,7 @@ func (w *containerdExecutor) Run(ctx context.Context, id string, root executor.M
return err
}
defer lm.Unmount()
- defer executor.MountStubsCleaner(rootfsPath, mounts)()
+ defer executor.MountStubsCleaner(rootfsPath, mounts, meta.RemoveMountStubsRecursive)()
uid, gid, sgids, err := oci.GetUser(rootfsPath, meta.User)
if err != nil {
@@ -147,7 +161,7 @@ func (w *containerdExecutor) Run(ctx context.Context, id string, root executor.M
if !ok {
return errors.Errorf("unknown network mode %s", meta.NetMode)
}
- namespace, err := provider.New()
+ namespace, err := provider.New(ctx, meta.Hostname)
if err != nil {
return err
}
@@ -163,7 +177,7 @@ func (w *containerdExecutor) Run(ctx context.Context, id string, root executor.M
}
processMode := oci.ProcessSandbox // FIXME(AkihiroSuda)
- spec, cleanup, err := oci.GenerateSpec(ctx, meta, mounts, id, resolvConf, hostsFile, namespace, w.cgroupParent, processMode, nil, w.apparmorProfile, w.traceSocket, opts...)
+ spec, cleanup, err := oci.GenerateSpec(ctx, meta, mounts, id, resolvConf, hostsFile, namespace, w.cgroupParent, processMode, nil, w.apparmorProfile, w.selinux, w.traceSocket, opts...)
if err != nil {
return err
}
@@ -204,11 +218,17 @@ func (w *containerdExecutor) Run(ctx context.Context, id string, root executor.M
}
defer func() {
- if _, err1 := task.Delete(context.TODO()); err == nil && err1 != nil {
+ if _, err1 := task.Delete(context.TODO(), containerd.WithProcessKill); err == nil && err1 != nil {
err = errors.Wrapf(err1, "failed to delete task %s", id)
}
}()
+ if nn, ok := namespace.(OnCreateRuntimer); ok {
+ if err := nn.OnCreateRuntime(task.Pid()); err != nil {
+ return err
+ }
+ }
+
trace.SpanFromContext(ctx).AddEvent("Container created")
err = w.runProcess(ctx, task, process.Resize, process.Signal, func() {
startedOnce.Do(func() {
@@ -315,10 +335,10 @@ func fixProcessOutput(process *executor.ProcessInfo) {
// failed to start io pipe copy: unable to copy pipes: containerd-shim: opening file "" failed: open : no such file or directory: unknown
// So just stub out any missing output
if process.Stdout == nil {
- process.Stdout = &nopCloser{ioutil.Discard}
+ process.Stdout = &nopCloser{io.Discard}
}
if process.Stderr == nil {
- process.Stderr = &nopCloser{ioutil.Discard}
+ process.Stderr = &nopCloser{io.Discard}
}
}
diff --git a/executor/executor.go b/executor/executor.go
index 4727af4b03ef..a323bcc9cc94 100644
--- a/executor/executor.go
+++ b/executor/executor.go
@@ -23,6 +23,8 @@ type Meta struct {
CgroupParent string
NetMode pb.NetMode
SecurityMode pb.SecurityMode
+
+ RemoveMountStubsRecursive bool
}
type Mountable interface {
diff --git a/executor/oci/hosts.go b/executor/oci/hosts.go
index d0505c28ccd9..0d193555c941 100644
--- a/executor/oci/hosts.go
+++ b/executor/oci/hosts.go
@@ -4,7 +4,6 @@ import (
"bytes"
"context"
"fmt"
- "io/ioutil"
"os"
"path/filepath"
@@ -56,7 +55,7 @@ func makeHostsFile(stateDir string, extraHosts []executor.HostIP, idmap *idtools
}
tmpPath := p + ".tmp"
- if err := ioutil.WriteFile(tmpPath, b.Bytes(), 0644); err != nil {
+ if err := os.WriteFile(tmpPath, b.Bytes(), 0644); err != nil {
return "", nil, err
}
diff --git a/executor/oci/resolvconf.go b/executor/oci/resolvconf.go
index c510a1a1bc18..3ac0feda7aea 100644
--- a/executor/oci/resolvconf.go
+++ b/executor/oci/resolvconf.go
@@ -2,12 +2,10 @@ package oci
import (
"context"
- "io/ioutil"
"os"
"path/filepath"
"github.com/docker/docker/libnetwork/resolvconf"
- "github.com/docker/docker/libnetwork/types"
"github.com/docker/docker/pkg/idtools"
"github.com/moby/buildkit/util/flightcontrol"
"github.com/pkg/errors"
@@ -74,7 +72,7 @@ func GetResolvConf(ctx context.Context, stateDir string, idmap *idtools.Identity
if dns != nil {
var (
- dnsNameservers = resolvconf.GetNameservers(dt, types.IP)
+ dnsNameservers = resolvconf.GetNameservers(dt, resolvconf.IP)
dnsSearchDomains = resolvconf.GetSearchDomains(dt)
dnsOptions = resolvconf.GetOptions(dt)
)
@@ -101,7 +99,7 @@ func GetResolvConf(ctx context.Context, stateDir string, idmap *idtools.Identity
}
tmpPath := p + ".tmp"
- if err := ioutil.WriteFile(tmpPath, f.Content, 0644); err != nil {
+ if err := os.WriteFile(tmpPath, f.Content, 0644); err != nil {
return "", err
}
diff --git a/executor/oci/resolvconf_test.go b/executor/oci/resolvconf_test.go
index 316bcefda283..ec073885eb34 100644
--- a/executor/oci/resolvconf_test.go
+++ b/executor/oci/resolvconf_test.go
@@ -2,7 +2,6 @@ package oci
import (
"context"
- "io/ioutil"
"os"
"testing"
@@ -27,13 +26,10 @@ nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844`
- dir, err := ioutil.TempDir("", "buildkit-test")
- require.NoError(t, err)
- defer os.RemoveAll(dir)
ctx := context.Background()
- p, err := GetResolvConf(ctx, dir, nil, nil)
+ p, err := GetResolvConf(ctx, t.TempDir(), nil, nil)
require.NoError(t, err)
- b, err := ioutil.ReadFile(p)
+ b, err := os.ReadFile(p)
require.NoError(t, err)
require.Equal(t, string(b), defaultResolvConf)
}
diff --git a/executor/oci/spec.go b/executor/oci/spec.go
index ea8741995a11..94b48a7aa9ff 100644
--- a/executor/oci/spec.go
+++ b/executor/oci/spec.go
@@ -50,7 +50,7 @@ func (pm ProcessMode) String() string {
// GenerateSpec generates spec using containerd functionality.
// opts are ignored for s.Process, s.Hostname, and s.Mounts .
-func GenerateSpec(ctx context.Context, meta executor.Meta, mounts []executor.Mount, id, resolvConf, hostsFile string, namespace network.Namespace, cgroupParent string, processMode ProcessMode, idmap *idtools.IdentityMapping, apparmorProfile string, tracingSocket string, opts ...oci.SpecOpts) (*specs.Spec, func(), error) {
+func GenerateSpec(ctx context.Context, meta executor.Meta, mounts []executor.Mount, id, resolvConf, hostsFile string, namespace network.Namespace, cgroupParent string, processMode ProcessMode, idmap *idtools.IdentityMapping, apparmorProfile string, selinuxB bool, tracingSocket string, opts ...oci.SpecOpts) (*specs.Spec, func(), error) {
c := &containers.Container{
ID: id,
}
@@ -81,7 +81,7 @@ func GenerateSpec(ctx context.Context, meta executor.Meta, mounts []executor.Mou
return nil, nil, err
}
- if securityOpts, err := generateSecurityOpts(meta.SecurityMode, apparmorProfile); err == nil {
+ if securityOpts, err := generateSecurityOpts(meta.SecurityMode, apparmorProfile, selinuxB); err == nil {
opts = append(opts, securityOpts...)
} else {
return nil, nil, err
diff --git a/executor/oci/spec_unix.go b/executor/oci/spec_unix.go
index 5f4908ca6b6c..f906f79b6bac 100644
--- a/executor/oci/spec_unix.go
+++ b/executor/oci/spec_unix.go
@@ -16,7 +16,9 @@ import (
"github.com/moby/buildkit/solver/pb"
"github.com/moby/buildkit/util/entitlements/security"
specs "github.com/opencontainers/runtime-spec/specs-go"
+ selinux "github.com/opencontainers/selinux/go-selinux"
"github.com/opencontainers/selinux/go-selinux/label"
+ "github.com/pkg/errors"
)
func generateMountOpts(resolvConf, hostsFile string) ([]oci.SpecOpts, error) {
@@ -30,7 +32,10 @@ func generateMountOpts(resolvConf, hostsFile string) ([]oci.SpecOpts, error) {
}
// generateSecurityOpts may affect mounts, so must be called after generateMountOpts
-func generateSecurityOpts(mode pb.SecurityMode, apparmorProfile string) (opts []oci.SpecOpts, _ error) {
+func generateSecurityOpts(mode pb.SecurityMode, apparmorProfile string, selinuxB bool) (opts []oci.SpecOpts, _ error) {
+ if selinuxB && !selinux.GetEnabled() {
+ return nil, errors.New("selinux is not available")
+ }
switch mode {
case pb.SecurityMode_INSECURE:
return []oci.SpecOpts{
@@ -39,7 +44,9 @@ func generateSecurityOpts(mode pb.SecurityMode, apparmorProfile string) (opts []
oci.WithWriteableSysfs,
func(_ context.Context, _ oci.Client, _ *containers.Container, s *oci.Spec) error {
var err error
- s.Process.SelinuxLabel, s.Linux.MountLabel, err = label.InitLabels([]string{"disable"})
+ if selinuxB {
+ s.Process.SelinuxLabel, s.Linux.MountLabel, err = label.InitLabels([]string{"disable"})
+ }
return err
},
}, nil
@@ -52,7 +59,9 @@ func generateSecurityOpts(mode pb.SecurityMode, apparmorProfile string) (opts []
}
opts = append(opts, func(_ context.Context, _ oci.Client, _ *containers.Container, s *oci.Spec) error {
var err error
- s.Process.SelinuxLabel, s.Linux.MountLabel, err = label.InitLabels(nil)
+ if selinuxB {
+ s.Process.SelinuxLabel, s.Linux.MountLabel, err = label.InitLabels(nil)
+ }
return err
})
return opts, nil
@@ -77,7 +86,7 @@ func generateIDmapOpts(idmap *idtools.IdentityMapping) ([]oci.SpecOpts, error) {
return nil, nil
}
return []oci.SpecOpts{
- oci.WithUserNamespace(specMapping(idmap.UIDs()), specMapping(idmap.GIDs())),
+ oci.WithUserNamespace(specMapping(idmap.UIDMaps), specMapping(idmap.GIDMaps)),
}, nil
}
diff --git a/executor/oci/spec_windows.go b/executor/oci/spec_windows.go
index bc1a6261e284..48b0969e3922 100644
--- a/executor/oci/spec_windows.go
+++ b/executor/oci/spec_windows.go
@@ -15,7 +15,7 @@ func generateMountOpts(resolvConf, hostsFile string) ([]oci.SpecOpts, error) {
}
// generateSecurityOpts may affect mounts, so must be called after generateMountOpts
-func generateSecurityOpts(mode pb.SecurityMode, apparmorProfile string) ([]oci.SpecOpts, error) {
+func generateSecurityOpts(mode pb.SecurityMode, apparmorProfile string, selinuxB bool) ([]oci.SpecOpts, error) {
if mode == pb.SecurityMode_INSECURE {
return nil, errors.New("no support for running in insecure mode on Windows")
}
diff --git a/executor/oci/user.go b/executor/oci/user.go
index eb459f391fbe..bb58e834f634 100644
--- a/executor/oci/user.go
+++ b/executor/oci/user.go
@@ -91,6 +91,7 @@ func parseUID(str string) (uint32, error) {
// once the PR in containerd is merged we should remove this function.
func WithUIDGID(uid, gid uint32, sgids []uint32) containerdoci.SpecOpts {
return func(_ context.Context, _ containerdoci.Client, _ *containers.Container, s *containerdoci.Spec) error {
+ defer ensureAdditionalGids(s)
setProcess(s)
s.Process.User.UID = uid
s.Process.User.GID = gid
@@ -106,3 +107,15 @@ func setProcess(s *containerdoci.Spec) {
s.Process = &specs.Process{}
}
}
+
+// ensureAdditionalGids ensures that the primary GID is also included in the additional GID list.
+// From https://github.com/containerd/containerd/blob/v1.7.0-beta.4/oci/spec_opts.go#L124-L133
+func ensureAdditionalGids(s *containerdoci.Spec) {
+ setProcess(s)
+ for _, f := range s.Process.User.AdditionalGids {
+ if f == s.Process.User.GID {
+ return
+ }
+ }
+ s.Process.User.AdditionalGids = append([]uint32{s.Process.User.GID}, s.Process.User.AdditionalGids...)
+}
diff --git a/executor/runcexecutor/executor.go b/executor/runcexecutor/executor.go
index 702d513102b2..213ebb73665a 100644
--- a/executor/runcexecutor/executor.go
+++ b/executor/runcexecutor/executor.go
@@ -48,6 +48,7 @@ type Opt struct {
DNS *oci.DNSConfig
OOMScoreAdj *int
ApparmorProfile string
+ SELinux bool
TracingSocket string
}
@@ -67,6 +68,7 @@ type runcExecutor struct {
running map[string]chan error
mu sync.Mutex
apparmorProfile string
+ selinux bool
tracingSocket string
}
@@ -131,6 +133,7 @@ func New(opt Opt, networkProviders map[pb.NetMode]network.Provider) (executor.Ex
oomScoreAdj: opt.OOMScoreAdj,
running: make(map[string]chan error),
apparmorProfile: opt.ApparmorProfile,
+ selinux: opt.SELinux,
tracingSocket: opt.TracingSocket,
}
return w, nil
@@ -161,7 +164,7 @@ func (w *runcExecutor) Run(ctx context.Context, id string, root executor.Mount,
if !ok {
return errors.Errorf("unknown network mode %s", meta.NetMode)
}
- namespace, err := provider.New()
+ namespace, err := provider.New(ctx, meta.Hostname)
if err != nil {
return err
}
@@ -221,7 +224,7 @@ func (w *runcExecutor) Run(ctx context.Context, id string, root executor.Mount,
}
defer mount.Unmount(rootFSPath, 0)
- defer executor.MountStubsCleaner(rootFSPath, mounts)()
+ defer executor.MountStubsCleaner(rootFSPath, mounts, meta.RemoveMountStubsRecursive)()
uid, gid, sgids, err := oci.GetUser(rootFSPath, meta.User)
if err != nil {
@@ -251,7 +254,7 @@ func (w *runcExecutor) Run(ctx context.Context, id string, root executor.Mount,
}
}
- spec, cleanup, err := oci.GenerateSpec(ctx, meta, mounts, id, resolvConf, hostsFile, namespace, w.cgroupParent, w.processMode, w.idmap, w.apparmorProfile, w.tracingSocket, opts...)
+ spec, cleanup, err := oci.GenerateSpec(ctx, meta, mounts, id, resolvConf, hostsFile, namespace, w.cgroupParent, w.processMode, w.idmap, w.apparmorProfile, w.selinux, w.tracingSocket, opts...)
if err != nil {
return err
}
diff --git a/executor/stubs.go b/executor/stubs.go
index 2c13b13053a4..22a8ac1310c4 100644
--- a/executor/stubs.go
+++ b/executor/stubs.go
@@ -7,9 +7,11 @@ import (
"syscall"
"github.com/containerd/continuity/fs"
+ "github.com/moby/buildkit/util/system"
+ "github.com/sirupsen/logrus"
)
-func MountStubsCleaner(dir string, mounts []Mount) func() {
+func MountStubsCleaner(dir string, mounts []Mount, recursive bool) func() {
names := []string{"/etc/resolv.conf", "/etc/hosts"}
for _, m := range mounts {
@@ -28,9 +30,22 @@ func MountStubsCleaner(dir string, mounts []Mount) func() {
continue
}
- _, err = os.Lstat(realPath)
- if errors.Is(err, os.ErrNotExist) || errors.Is(err, syscall.ENOTDIR) {
+ for {
+ _, err = os.Lstat(realPath)
+ if !(errors.Is(err, os.ErrNotExist) || errors.Is(err, syscall.ENOTDIR)) {
+ break
+ }
paths = append(paths, realPath)
+
+ if !recursive {
+ break
+ }
+
+ realPathNext := filepath.Dir(realPath)
+ if realPath == realPathNext {
+ break
+ }
+ realPath = realPathNext
}
}
@@ -40,10 +55,41 @@ func MountStubsCleaner(dir string, mounts []Mount) func() {
if err != nil {
continue
}
- if st.Size() != 0 {
+ if st.IsDir() {
+ entries, err := os.ReadDir(p)
+ if err != nil {
+ continue
+ }
+ if len(entries) != 0 {
+ continue
+ }
+ } else if st.Size() != 0 {
continue
}
- os.Remove(p)
+
+ // Back up the timestamps of the dir for reproducible builds
+ // https://github.com/moby/buildkit/issues/3148
+ dir := filepath.Dir(p)
+ dirSt, err := os.Stat(dir)
+ if err != nil {
+ logrus.WithError(err).Warnf("Failed to stat %q (parent of mount stub %q)", dir, p)
+ continue
+ }
+ mtime := dirSt.ModTime()
+ atime, err := system.Atime(dirSt)
+ if err != nil {
+ logrus.WithError(err).Warnf("Failed to stat atime of %q (parent of mount stub %q)", dir, p)
+ atime = mtime
+ }
+
+ if err := os.Remove(p); err != nil {
+ logrus.WithError(err).Warnf("Failed to remove mount stub %q", p)
+ }
+
+ // Restore the timestamps of the dir
+ if err := os.Chtimes(dir, atime, mtime); err != nil {
+ logrus.WithError(err).Warnf("Failed to restore time time mount stub timestamp (os.Chtimes(%q, %v, %v))", dir, atime, mtime)
+ }
}
}
}
diff --git a/exporter/attestation/filter.go b/exporter/attestation/filter.go
new file mode 100644
index 000000000000..5abc234b875e
--- /dev/null
+++ b/exporter/attestation/filter.go
@@ -0,0 +1,45 @@
+package attestation
+
+import (
+ "bytes"
+
+ "github.com/moby/buildkit/exporter"
+)
+
+func Filter(attestations []exporter.Attestation, include map[string][]byte, exclude map[string][]byte) []exporter.Attestation {
+ if len(include) == 0 && len(exclude) == 0 {
+ return attestations
+ }
+
+ result := []exporter.Attestation{}
+ for _, att := range attestations {
+ meta := att.Metadata
+ if meta == nil {
+ meta = map[string][]byte{}
+ }
+
+ match := true
+ for k, v := range include {
+ if !bytes.Equal(meta[k], v) {
+ match = false
+ break
+ }
+ }
+ if !match {
+ continue
+ }
+
+ for k, v := range exclude {
+ if bytes.Equal(meta[k], v) {
+ match = false
+ break
+ }
+ }
+ if !match {
+ continue
+ }
+
+ result = append(result, att)
+ }
+ return result
+}
diff --git a/exporter/attestation/make.go b/exporter/attestation/make.go
new file mode 100644
index 000000000000..8ed910c1e8d3
--- /dev/null
+++ b/exporter/attestation/make.go
@@ -0,0 +1,138 @@
+package attestation
+
+import (
+ "context"
+ "encoding/json"
+ "os"
+
+ "github.com/containerd/continuity/fs"
+ intoto "github.com/in-toto/in-toto-golang/in_toto"
+ "github.com/moby/buildkit/exporter"
+ gatewaypb "github.com/moby/buildkit/frontend/gateway/pb"
+ "github.com/moby/buildkit/session"
+ "github.com/moby/buildkit/snapshot"
+ "github.com/moby/buildkit/solver/result"
+ "github.com/pkg/errors"
+ "golang.org/x/sync/errgroup"
+)
+
+// ReadAll reads the content of an attestation.
+func ReadAll(ctx context.Context, s session.Group, att exporter.Attestation) ([]byte, error) {
+ var content []byte
+ if att.ContentFunc != nil {
+ data, err := att.ContentFunc()
+ if err != nil {
+ return nil, err
+ }
+ content = data
+ } else if att.Ref != nil {
+ mount, err := att.Ref.Mount(ctx, true, s)
+ if err != nil {
+ return nil, err
+ }
+ lm := snapshot.LocalMounter(mount)
+ src, err := lm.Mount()
+ if err != nil {
+ return nil, err
+ }
+ defer lm.Unmount()
+
+ p, err := fs.RootPath(src, att.Path)
+ if err != nil {
+ return nil, err
+ }
+ content, err = os.ReadFile(p)
+ if err != nil {
+ return nil, errors.Wrap(err, "cannot read in-toto attestation")
+ }
+ } else {
+ return nil, errors.New("no available content for attestation")
+ }
+ if len(content) == 0 {
+ content = nil
+ }
+ return content, nil
+}
+
+// MakeInTotoStatements iterates over all provided result attestations and
+// generates intoto attestation statements.
+func MakeInTotoStatements(ctx context.Context, s session.Group, attestations []exporter.Attestation, defaultSubjects []intoto.Subject) ([]intoto.Statement, error) {
+ eg, ctx := errgroup.WithContext(ctx)
+ statements := make([]intoto.Statement, len(attestations))
+
+ for i, att := range attestations {
+ i, att := i, att
+ eg.Go(func() error {
+ content, err := ReadAll(ctx, s, att)
+ if err != nil {
+ return err
+ }
+
+ switch att.Kind {
+ case gatewaypb.AttestationKindInToto:
+ stmt, err := makeInTotoStatement(ctx, content, att, defaultSubjects)
+ if err != nil {
+ return err
+ }
+ statements[i] = *stmt
+ case gatewaypb.AttestationKindBundle:
+ return errors.New("bundle attestation kind must be un-bundled first")
+ }
+ return nil
+ })
+ }
+ if err := eg.Wait(); err != nil {
+ return nil, err
+ }
+ return statements, nil
+}
+
+func makeInTotoStatement(ctx context.Context, content []byte, attestation exporter.Attestation, defaultSubjects []intoto.Subject) (*intoto.Statement, error) {
+ if len(attestation.InToto.Subjects) == 0 {
+ attestation.InToto.Subjects = []result.InTotoSubject{{
+ Kind: gatewaypb.InTotoSubjectKindSelf,
+ }}
+ }
+ subjects := []intoto.Subject{}
+ for _, subject := range attestation.InToto.Subjects {
+ subjectName := "_"
+ if subject.Name != "" {
+ subjectName = subject.Name
+ }
+
+ switch subject.Kind {
+ case gatewaypb.InTotoSubjectKindSelf:
+ for _, defaultSubject := range defaultSubjects {
+ subjectNames := []string{}
+ subjectNames = append(subjectNames, defaultSubject.Name)
+ if subjectName != "_" {
+ subjectNames = append(subjectNames, subjectName)
+ }
+
+ for _, name := range subjectNames {
+ subjects = append(subjects, intoto.Subject{
+ Name: name,
+ Digest: defaultSubject.Digest,
+ })
+ }
+ }
+ case gatewaypb.InTotoSubjectKindRaw:
+ subjects = append(subjects, intoto.Subject{
+ Name: subjectName,
+ Digest: result.ToDigestMap(subject.Digest...),
+ })
+ default:
+ return nil, errors.Errorf("unknown attestation subject type %T", subject)
+ }
+ }
+
+ stmt := intoto.Statement{
+ StatementHeader: intoto.StatementHeader{
+ Type: intoto.StatementInTotoV01,
+ PredicateType: attestation.InToto.PredicateType,
+ Subject: subjects,
+ },
+ Predicate: json.RawMessage(content),
+ }
+ return &stmt, nil
+}
diff --git a/exporter/attestation/unbundle.go b/exporter/attestation/unbundle.go
new file mode 100644
index 000000000000..a2120d7975e1
--- /dev/null
+++ b/exporter/attestation/unbundle.go
@@ -0,0 +1,192 @@
+package attestation
+
+import (
+ "context"
+ "encoding/json"
+ "os"
+ "path"
+ "strings"
+
+ "github.com/containerd/continuity/fs"
+ intoto "github.com/in-toto/in-toto-golang/in_toto"
+ "github.com/moby/buildkit/exporter"
+ gatewaypb "github.com/moby/buildkit/frontend/gateway/pb"
+ "github.com/moby/buildkit/session"
+ "github.com/moby/buildkit/snapshot"
+ "github.com/moby/buildkit/solver/result"
+ "github.com/pkg/errors"
+ "golang.org/x/sync/errgroup"
+)
+
+// Unbundle iterates over all provided result attestations and un-bundles any
+// bundled attestations by loading them from the provided refs map.
+func Unbundle(ctx context.Context, s session.Group, bundled []exporter.Attestation) ([]exporter.Attestation, error) {
+ if err := Validate(bundled); err != nil {
+ return nil, err
+ }
+
+ eg, ctx := errgroup.WithContext(ctx)
+ unbundled := make([][]exporter.Attestation, len(bundled))
+
+ for i, att := range bundled {
+ i, att := i, att
+ eg.Go(func() error {
+ switch att.Kind {
+ case gatewaypb.AttestationKindInToto:
+ if strings.HasPrefix(att.InToto.PredicateType, "https://slsa.dev/provenance/") {
+ if att.ContentFunc == nil {
+ // provenance may only be set buildkit-side using ContentFunc
+ return errors.New("frontend may not set provenance attestations")
+ }
+ }
+ unbundled[i] = append(unbundled[i], att)
+ case gatewaypb.AttestationKindBundle:
+ if att.ContentFunc != nil {
+ return errors.New("attestation bundle cannot have callback")
+ }
+ if att.Ref == nil {
+ return errors.Errorf("no ref provided for attestation bundle")
+ }
+
+ mount, err := att.Ref.Mount(ctx, true, s)
+ if err != nil {
+ return err
+ }
+ lm := snapshot.LocalMounter(mount)
+ src, err := lm.Mount()
+ if err != nil {
+ return err
+ }
+ defer lm.Unmount()
+
+ atts, err := unbundle(ctx, src, att)
+ if err != nil {
+ return err
+ }
+ for _, att := range atts {
+ if strings.HasPrefix(att.InToto.PredicateType, "https://slsa.dev/provenance/") {
+ return errors.New("frontend may not bundle provenance attestations")
+ }
+ }
+ unbundled[i] = append(unbundled[i], atts...)
+ }
+ return nil
+ })
+ }
+ if err := eg.Wait(); err != nil {
+ return nil, err
+ }
+
+ var joined []exporter.Attestation
+ for _, atts := range unbundled {
+ joined = append(joined, atts...)
+ }
+ joined = sort(joined)
+
+ if err := Validate(joined); err != nil {
+ return nil, err
+ }
+ return joined, nil
+}
+
+func sort(atts []exporter.Attestation) []exporter.Attestation {
+ isCore := make([]bool, len(atts))
+ for i, att := range atts {
+ name, ok := att.Metadata[result.AttestationSBOMCore]
+ if !ok {
+ continue
+ }
+ if n, _, _ := strings.Cut(att.Path, "."); n != string(name) {
+ continue
+ }
+ isCore[i] = true
+ }
+
+ result := make([]exporter.Attestation, 0, len(atts))
+ for i, att := range atts {
+ if isCore[i] {
+ result = append(result, att)
+ }
+ }
+ for i, att := range atts {
+ if !isCore[i] {
+ result = append(result, att)
+ }
+ }
+ return result
+}
+
+func unbundle(ctx context.Context, root string, bundle exporter.Attestation) ([]exporter.Attestation, error) {
+ dir, err := fs.RootPath(root, bundle.Path)
+ if err != nil {
+ return nil, err
+ }
+ entries, err := os.ReadDir(dir)
+ if err != nil {
+ return nil, err
+ }
+
+ var unbundled []exporter.Attestation
+ for _, entry := range entries {
+ p, err := fs.RootPath(dir, entry.Name())
+ if err != nil {
+ return nil, err
+ }
+ f, err := os.Open(p)
+ if err != nil {
+ return nil, err
+ }
+ dec := json.NewDecoder(f)
+ var stmt intoto.Statement
+ if err := dec.Decode(&stmt); err != nil {
+ return nil, errors.Wrap(err, "cannot decode in-toto statement")
+ }
+ if bundle.InToto.PredicateType != "" && stmt.PredicateType != bundle.InToto.PredicateType {
+ return nil, errors.Errorf("bundle entry %s does not match required predicate type %s", stmt.PredicateType, bundle.InToto.PredicateType)
+ }
+
+ predicate, err := json.Marshal(stmt.Predicate)
+ if err != nil {
+ return nil, err
+ }
+
+ subjects := make([]result.InTotoSubject, len(stmt.Subject))
+ for i, subject := range stmt.Subject {
+ subjects[i] = result.InTotoSubject{
+ Kind: gatewaypb.InTotoSubjectKindRaw,
+ Name: subject.Name,
+ Digest: result.FromDigestMap(subject.Digest),
+ }
+ }
+ unbundled = append(unbundled, exporter.Attestation{
+ Kind: gatewaypb.AttestationKindInToto,
+ Metadata: bundle.Metadata,
+ Path: path.Join(bundle.Path, entry.Name()),
+ ContentFunc: func() ([]byte, error) { return predicate, nil },
+ InToto: result.InTotoAttestation{
+ PredicateType: stmt.PredicateType,
+ Subjects: subjects,
+ },
+ })
+ }
+ return unbundled, nil
+}
+
+func Validate(atts []exporter.Attestation) error {
+ for _, att := range atts {
+ if err := validate(att); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func validate(att exporter.Attestation) error {
+ if att.Kind != gatewaypb.AttestationKindBundle && att.Path == "" {
+ return errors.New("attestation does not have set path")
+ }
+ if att.Ref == nil && att.ContentFunc == nil {
+ return errors.New("attestation does not have available content")
+ }
+ return nil
+}
diff --git a/exporter/containerimage/annotations.go b/exporter/containerimage/annotations.go
new file mode 100644
index 000000000000..cdb5e945096c
--- /dev/null
+++ b/exporter/containerimage/annotations.go
@@ -0,0 +1,139 @@
+package containerimage
+
+import (
+ ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
+ "github.com/pkg/errors"
+
+ "github.com/containerd/containerd/platforms"
+ "github.com/moby/buildkit/exporter/containerimage/exptypes"
+)
+
+type Annotations struct {
+ Index map[string]string
+ IndexDescriptor map[string]string
+ Manifest map[string]string
+ ManifestDescriptor map[string]string
+}
+
+// AnnotationsGroup is a map of annotations keyed by the reference key
+type AnnotationsGroup map[string]*Annotations
+
+func ParseAnnotations(data map[string][]byte) (AnnotationsGroup, map[string][]byte, error) {
+ ag := make(AnnotationsGroup)
+ rest := make(map[string][]byte)
+
+ for k, v := range data {
+ a, ok, err := exptypes.ParseAnnotationKey(k)
+ if !ok {
+ rest[k] = v
+ continue
+ }
+ if err != nil {
+ return nil, nil, err
+ }
+
+ p := a.PlatformString()
+
+ if ag[p] == nil {
+ ag[p] = &Annotations{
+ IndexDescriptor: make(map[string]string),
+ Index: make(map[string]string),
+ Manifest: make(map[string]string),
+ ManifestDescriptor: make(map[string]string),
+ }
+ }
+
+ switch a.Type {
+ case exptypes.AnnotationIndex:
+ ag[p].Index[a.Key] = string(v)
+ case exptypes.AnnotationIndexDescriptor:
+ ag[p].IndexDescriptor[a.Key] = string(v)
+ case exptypes.AnnotationManifest:
+ ag[p].Manifest[a.Key] = string(v)
+ case exptypes.AnnotationManifestDescriptor:
+ ag[p].ManifestDescriptor[a.Key] = string(v)
+ default:
+ return nil, nil, errors.Errorf("unrecognized annotation type %s", a.Type)
+ }
+ }
+ return ag, rest, nil
+}
+
+func (ag AnnotationsGroup) Platform(p *ocispecs.Platform) *Annotations {
+ res := &Annotations{
+ IndexDescriptor: make(map[string]string),
+ Index: make(map[string]string),
+ Manifest: make(map[string]string),
+ ManifestDescriptor: make(map[string]string),
+ }
+
+ ps := []string{""}
+ if p != nil {
+ ps = append(ps, platforms.Format(*p))
+ }
+
+ for _, a := range ag {
+ for k, v := range a.Index {
+ res.Index[k] = v
+ }
+ for k, v := range a.IndexDescriptor {
+ res.IndexDescriptor[k] = v
+ }
+ }
+ for _, pk := range ps {
+ if _, ok := ag[pk]; !ok {
+ continue
+ }
+
+ for k, v := range ag[pk].Manifest {
+ res.Manifest[k] = v
+ }
+ for k, v := range ag[pk].ManifestDescriptor {
+ res.ManifestDescriptor[k] = v
+ }
+ }
+ return res
+}
+
+func (ag AnnotationsGroup) Merge(other AnnotationsGroup) AnnotationsGroup {
+ if other == nil {
+ return ag
+ }
+ if ag == nil {
+ ag = make(AnnotationsGroup)
+ }
+
+ for k, v := range other {
+ ag[k] = ag[k].merge(v)
+ }
+ return ag
+}
+
+func (a *Annotations) merge(other *Annotations) *Annotations {
+ if other == nil {
+ return a
+ }
+ if a == nil {
+ a = &Annotations{
+ IndexDescriptor: make(map[string]string),
+ Index: make(map[string]string),
+ Manifest: make(map[string]string),
+ ManifestDescriptor: make(map[string]string),
+ }
+ }
+
+ for k, v := range other.Index {
+ a.Index[k] = v
+ }
+ for k, v := range other.IndexDescriptor {
+ a.IndexDescriptor[k] = v
+ }
+ for k, v := range other.Manifest {
+ a.Manifest[k] = v
+ }
+ for k, v := range other.ManifestDescriptor {
+ a.ManifestDescriptor[k] = v
+ }
+
+ return a
+}
diff --git a/exporter/containerimage/attestations.go b/exporter/containerimage/attestations.go
new file mode 100644
index 000000000000..a41c6039f0ba
--- /dev/null
+++ b/exporter/containerimage/attestations.go
@@ -0,0 +1,220 @@
+package containerimage
+
+import (
+ "bytes"
+ "context"
+ "fmt"
+ "io/fs"
+ "path/filepath"
+ "strings"
+
+ intoto "github.com/in-toto/in-toto-golang/in_toto"
+ "github.com/moby/buildkit/cache"
+ "github.com/moby/buildkit/exporter"
+ "github.com/moby/buildkit/exporter/attestation"
+ gatewaypb "github.com/moby/buildkit/frontend/gateway/pb"
+ "github.com/moby/buildkit/session"
+ "github.com/moby/buildkit/solver"
+ "github.com/moby/buildkit/solver/result"
+ "github.com/moby/buildkit/version"
+ ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
+ "github.com/pkg/errors"
+ spdx_json "github.com/spdx/tools-golang/json"
+ "github.com/spdx/tools-golang/spdx/common"
+ spdx "github.com/spdx/tools-golang/spdx/v2_3"
+)
+
+var intotoPlatform ocispecs.Platform = ocispecs.Platform{
+ Architecture: "unknown",
+ OS: "unknown",
+}
+
+// supplementSBOM modifies SPDX attestations to include the file layers
+func supplementSBOM(ctx context.Context, s session.Group, target cache.ImmutableRef, targetRemote *solver.Remote, att exporter.Attestation) (exporter.Attestation, error) {
+ if target == nil {
+ return att, nil
+ }
+ if att.Kind != gatewaypb.AttestationKindInToto {
+ return att, nil
+ }
+ if att.InToto.PredicateType != intoto.PredicateSPDX {
+ return att, nil
+ }
+ name, ok := att.Metadata[result.AttestationSBOMCore]
+ if !ok {
+ return att, nil
+ }
+ if n, _, _ := strings.Cut(filepath.Base(att.Path), "."); n != string(name) {
+ return att, nil
+ }
+
+ content, err := attestation.ReadAll(ctx, s, att)
+ if err != nil {
+ return att, err
+ }
+
+ doc, err := decodeSPDX(content)
+ if err != nil {
+ // ignore decoding error
+ return att, nil
+ }
+
+ layers, err := newFileLayerFinder(target, targetRemote)
+ if err != nil {
+ return att, err
+ }
+ modifyFile := func(f *spdx.File) error {
+ if f == nil {
+ // Skip over nil entries - this is likely a bug in the SPDX parser,
+ // but we shouldn't accidentally panic if we encounter it.
+ return nil
+ }
+
+ if f.FileComment != "" {
+ // Skip over files that already have a comment - since the data is
+ // unstructured, we can't correctly overwrite this field without
+ // possibly breaking some scanner functionality.
+ return nil
+ }
+
+ _, desc, err := layers.find(ctx, s, f.FileName)
+ if err != nil {
+ if !errors.Is(err, fs.ErrNotExist) {
+ return err
+ }
+ return nil
+ }
+ f.FileComment = fmt.Sprintf("layerID: %s", desc.Digest.String())
+ return nil
+ }
+ for _, f := range doc.Files {
+ if err := modifyFile(f); err != nil {
+ return att, err
+ }
+ }
+ for _, p := range doc.Packages {
+ for _, f := range p.Files {
+ if err := modifyFile(f); err != nil {
+ return att, err
+ }
+ }
+ }
+
+ if doc.CreationInfo == nil {
+ doc.CreationInfo = &spdx.CreationInfo{}
+ }
+ doc.CreationInfo.Creators = append(doc.CreationInfo.Creators, common.Creator{
+ CreatorType: "Tool",
+ Creator: "buildkit-" + version.Version,
+ })
+
+ content, err = encodeSPDX(doc)
+ if err != nil {
+ return att, err
+ }
+
+ return exporter.Attestation{
+ Kind: att.Kind,
+ Path: att.Path,
+ ContentFunc: func() ([]byte, error) { return content, nil },
+ InToto: att.InToto,
+ }, nil
+}
+
+func decodeSPDX(dt []byte) (s *spdx.Document, err error) {
+ doc, err := spdx_json.Load2_3(bytes.NewReader(dt))
+ if err != nil {
+ return nil, errors.Wrap(err, "unable to decode spdx")
+ }
+ if doc == nil {
+ return nil, errors.New("decoding produced empty spdx document")
+ }
+ return doc, nil
+}
+
+func encodeSPDX(s *spdx.Document) (dt []byte, err error) {
+ w := bytes.NewBuffer(nil)
+ err = spdx_json.Save2_3(s, w)
+ if err != nil {
+ return nil, errors.Wrap(err, "unable to encode spdx")
+ }
+ return w.Bytes(), nil
+}
+
+// fileLayerFinder finds the layer that contains a file, with caching to avoid
+// repeated FileList lookups.
+type fileLayerFinder struct {
+ pending []fileLayerEntry
+ cache map[string]fileLayerEntry
+}
+
+type fileLayerEntry struct {
+ ref cache.ImmutableRef
+ desc ocispecs.Descriptor
+}
+
+func newFileLayerFinder(target cache.ImmutableRef, remote *solver.Remote) (fileLayerFinder, error) {
+ chain := target.LayerChain()
+ descs := remote.Descriptors
+ if len(chain) != len(descs) {
+ return fileLayerFinder{}, errors.New("layer chain and descriptor list are not the same length")
+ }
+
+ pending := make([]fileLayerEntry, len(chain))
+ for i, ref := range chain {
+ pending[i] = fileLayerEntry{ref: ref, desc: descs[i]}
+ }
+ return fileLayerFinder{
+ pending: pending,
+ cache: map[string]fileLayerEntry{},
+ }, nil
+}
+
+// find finds the layer that contains the file, returning the ImmutableRef and
+// descriptor for the layer. If the file searched for was deleted, find returns
+// the layer that created the file, not the one that deleted it.
+//
+// find is not concurrency-safe.
+func (c *fileLayerFinder) find(ctx context.Context, s session.Group, filename string) (cache.ImmutableRef, *ocispecs.Descriptor, error) {
+ filename = filepath.Join("/", filename)
+
+ // return immediately if we've already found the layer containing filename
+ if cache, ok := c.cache[filename]; ok {
+ return cache.ref, &cache.desc, nil
+ }
+
+ for len(c.pending) > 0 {
+ // pop the last entry off the pending list (we traverse the layers backwards)
+ pending := c.pending[len(c.pending)-1]
+ files, err := pending.ref.FileList(ctx, s)
+ if err != nil {
+ return nil, nil, err
+ }
+ c.pending = c.pending[:len(c.pending)-1]
+
+ found := false
+ for _, f := range files {
+ f = filepath.Join("/", f)
+
+ if strings.HasPrefix(filepath.Base(f), ".wh.") {
+ // skip whiteout files, we only care about file creations
+ continue
+ }
+
+ // add all files in this layer to the cache
+ if _, ok := c.cache[f]; ok {
+ continue
+ }
+ c.cache[f] = pending
+
+ // if we found the file, return the layer (but finish populating the cache first)
+ if f == filename {
+ found = true
+ }
+ }
+ if found {
+ return pending.ref, &pending.desc, nil
+ }
+ }
+ return nil, nil, fs.ErrNotExist
+}
diff --git a/exporter/containerimage/export.go b/exporter/containerimage/export.go
index 429a3ce6df3b..55eaf3ff5803 100644
--- a/exporter/containerimage/export.go
+++ b/exporter/containerimage/export.go
@@ -14,8 +14,10 @@ import (
"github.com/containerd/containerd/images"
"github.com/containerd/containerd/leases"
"github.com/containerd/containerd/platforms"
+ "github.com/containerd/containerd/remotes"
"github.com/containerd/containerd/remotes/docker"
"github.com/containerd/containerd/rootfs"
+ intoto "github.com/in-toto/in-toto-golang/in_toto"
"github.com/moby/buildkit/cache"
cacheconfig "github.com/moby/buildkit/cache/config"
"github.com/moby/buildkit/exporter"
@@ -25,32 +27,27 @@ import (
"github.com/moby/buildkit/util/compression"
"github.com/moby/buildkit/util/contentutil"
"github.com/moby/buildkit/util/leaseutil"
+ "github.com/moby/buildkit/util/progress"
"github.com/moby/buildkit/util/push"
digest "github.com/opencontainers/go-digest"
"github.com/opencontainers/image-spec/identity"
ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/pkg/errors"
- "github.com/sirupsen/logrus"
)
const (
- keyImageName = "name"
- keyPush = "push"
- keyPushByDigest = "push-by-digest"
- keyInsecure = "registry.insecure"
- keyUnpack = "unpack"
- keyDanglingPrefix = "dangling-name-prefix"
- keyNameCanonical = "name-canonical"
- keyLayerCompression = "compression"
- keyForceCompression = "force-compression"
- keyCompressionLevel = "compression-level"
- keyBuildInfo = "buildinfo"
- keyBuildInfoAttrs = "buildinfo-attrs"
- ociTypes = "oci-mediatypes"
- // preferNondistLayersKey is an exporter option which can be used to mark a layer as non-distributable if the layer reference was
- // already found to use a non-distributable media type.
- // When this option is not set, the exporter will change the media type of the layer to a distributable one.
- preferNondistLayersKey = "prefer-nondist-layers"
+ keyPush = "push"
+ keyPushByDigest = "push-by-digest"
+ keyInsecure = "registry.insecure"
+ keyUnpack = "unpack"
+ keyDanglingPrefix = "dangling-name-prefix"
+ keyNameCanonical = "name-canonical"
+ keyStore = "store"
+
+ // keyUnsafeInternalStoreAllowIncomplete should only be used for tests. This option allows exporting image to the image store
+ // as well as lacking some blobs in the content store. Some integration tests for lazyref behaviour depends on this option.
+ // Ignored when store=false.
+ keyUnsafeInternalStoreAllowIncomplete = "unsafe-internal-store-allow-incomplete"
)
type Opt struct {
@@ -76,16 +73,24 @@ func New(opt Opt) (exporter.Exporter, error) {
func (e *imageExporter) Resolve(ctx context.Context, opt map[string]string) (exporter.ExporterInstance, error) {
i := &imageExporterInstance{
- imageExporter: e,
- layerCompression: compression.Default,
- buildInfo: true,
+ imageExporter: e,
+ opts: ImageCommitOpts{
+ RefCfg: cacheconfig.RefConfig{
+ Compression: compression.New(compression.Default),
+ },
+ BuildInfo: true,
+ ForceInlineAttestations: true,
+ },
+ store: true,
+ }
+
+ opt, err := i.opts.Load(opt)
+ if err != nil {
+ return nil, err
}
- var esgz bool
for k, v := range opt {
switch k {
- case keyImageName:
- i.targetName = v
case keyPush:
if v == "" {
i.push = true
@@ -126,85 +131,38 @@ func (e *imageExporter) Resolve(ctx context.Context, opt map[string]string) (exp
return nil, errors.Wrapf(err, "non-bool value specified for %s", k)
}
i.unpack = b
- case ociTypes:
+ case keyStore:
if v == "" {
- i.ociTypes = true
+ i.store = true
continue
}
b, err := strconv.ParseBool(v)
if err != nil {
return nil, errors.Wrapf(err, "non-bool value specified for %s", k)
}
- i.ociTypes = b
- case keyDanglingPrefix:
- i.danglingPrefix = v
- case keyNameCanonical:
+ i.store = b
+ case keyUnsafeInternalStoreAllowIncomplete:
if v == "" {
- i.nameCanonical = true
+ i.storeAllowIncomplete = true
continue
}
b, err := strconv.ParseBool(v)
if err != nil {
return nil, errors.Wrapf(err, "non-bool value specified for %s", k)
}
- i.nameCanonical = b
- case keyLayerCompression:
- switch v {
- case "gzip":
- i.layerCompression = compression.Gzip
- case "estargz":
- i.layerCompression = compression.EStargz
- esgz = true
- case "zstd":
- i.layerCompression = compression.Zstd
- case "uncompressed":
- i.layerCompression = compression.Uncompressed
- default:
- return nil, errors.Errorf("unsupported layer compression type: %v", v)
- }
- case keyForceCompression:
- if v == "" {
- i.forceCompression = true
- continue
- }
- b, err := strconv.ParseBool(v)
- if err != nil {
- return nil, errors.Wrapf(err, "non-bool value %s specified for %s", v, k)
- }
- i.forceCompression = b
- case keyCompressionLevel:
- ii, err := strconv.ParseInt(v, 10, 64)
- if err != nil {
- return nil, errors.Wrapf(err, "non-integer value %s specified for %s", v, k)
- }
- v := int(ii)
- i.compressionLevel = &v
- case keyBuildInfo:
- if v == "" {
- i.buildInfo = true
- continue
- }
- b, err := strconv.ParseBool(v)
- if err != nil {
- return nil, errors.Wrapf(err, "non-bool value specified for %s", k)
- }
- i.buildInfo = b
- case keyBuildInfoAttrs:
+ i.storeAllowIncomplete = b
+ case keyDanglingPrefix:
+ i.danglingPrefix = v
+ case keyNameCanonical:
if v == "" {
- i.buildInfoAttrs = false
+ i.nameCanonical = true
continue
}
b, err := strconv.ParseBool(v)
if err != nil {
return nil, errors.Wrapf(err, "non-bool value specified for %s", k)
}
- i.buildInfoAttrs = b
- case preferNondistLayersKey:
- b, err := strconv.ParseBool(v)
- if err != nil {
- return nil, errors.Wrapf(err, "non-bool value %s specified for %s", v, k)
- }
- i.preferNondistLayers = b
+ i.nameCanonical = b
default:
if i.meta == nil {
i.meta = make(map[string][]byte)
@@ -212,51 +170,32 @@ func (e *imageExporter) Resolve(ctx context.Context, opt map[string]string) (exp
i.meta[k] = []byte(v)
}
}
- if esgz && !i.ociTypes {
- logrus.Warn("forcibly turning on oci-mediatype mode for estargz")
- i.ociTypes = true
- }
return i, nil
}
type imageExporterInstance struct {
*imageExporter
- targetName string
- push bool
- pushByDigest bool
- unpack bool
- insecure bool
- ociTypes bool
- nameCanonical bool
- danglingPrefix string
- layerCompression compression.Type
- forceCompression bool
- compressionLevel *int
- buildInfo bool
- buildInfoAttrs bool
- meta map[string][]byte
- preferNondistLayers bool
+ opts ImageCommitOpts
+ push bool
+ pushByDigest bool
+ unpack bool
+ store bool
+ storeAllowIncomplete bool
+ insecure bool
+ nameCanonical bool
+ danglingPrefix string
+ meta map[string][]byte
}
func (e *imageExporterInstance) Name() string {
return "exporting to image"
}
-func (e *imageExporterInstance) Config() exporter.Config {
- return exporter.Config{
- Compression: e.compression(),
- }
-}
-
-func (e *imageExporterInstance) compression() compression.Config {
- c := compression.New(e.layerCompression).SetForce(e.forceCompression)
- if e.compressionLevel != nil {
- c = c.SetLevel(*e.compressionLevel)
- }
- return c
+func (e *imageExporterInstance) Config() *exporter.Config {
+ return exporter.NewConfigWithCompression(e.opts.RefCfg.Compression)
}
-func (e *imageExporterInstance) Export(ctx context.Context, src exporter.Source, sessionID string) (map[string]string, error) {
+func (e *imageExporterInstance) Export(ctx context.Context, src *exporter.Source, sessionID string) (_ map[string]string, descref exporter.DescriptorReference, err error) {
if src.Metadata == nil {
src.Metadata = make(map[string][]byte)
}
@@ -264,39 +203,50 @@ func (e *imageExporterInstance) Export(ctx context.Context, src exporter.Source,
src.Metadata[k] = v
}
- ctx, done, err := leaseutil.WithLease(ctx, e.opt.LeaseManager, leaseutil.MakeTemporary)
+ opts := e.opts
+ as, _, err := ParseAnnotations(src.Metadata)
if err != nil {
- return nil, err
+ return nil, nil, err
}
- defer done(context.TODO())
+ opts.Annotations = opts.Annotations.Merge(as)
- refCfg := e.refCfg()
- desc, err := e.opt.ImageWriter.Commit(ctx, src, e.ociTypes, refCfg, e.buildInfo, e.buildInfoAttrs, sessionID)
+ ctx, done, err := leaseutil.WithLease(ctx, e.opt.LeaseManager, leaseutil.MakeTemporary)
if err != nil {
- return nil, err
+ return nil, nil, err
}
+ defer func() {
+ if descref == nil {
+ done(context.TODO())
+ }
+ }()
+ desc, err := e.opt.ImageWriter.Commit(ctx, src, sessionID, &opts)
+ if err != nil {
+ return nil, nil, err
+ }
defer func() {
- e.opt.ImageWriter.ContentStore().Delete(context.TODO(), desc.Digest)
+ if err == nil {
+ descref = NewDescriptorReference(*desc, done)
+ }
}()
resp := make(map[string]string)
- if n, ok := src.Metadata["image.name"]; e.targetName == "*" && ok {
- e.targetName = string(n)
+ if n, ok := src.Metadata["image.name"]; e.opts.ImageName == "*" && ok {
+ e.opts.ImageName = string(n)
}
nameCanonical := e.nameCanonical
- if e.targetName == "" && e.danglingPrefix != "" {
- e.targetName = e.danglingPrefix + "@" + desc.Digest.String()
+ if e.opts.ImageName == "" && e.danglingPrefix != "" {
+ e.opts.ImageName = e.danglingPrefix + "@" + desc.Digest.String()
nameCanonical = false
}
- if e.targetName != "" {
- targetNames := strings.Split(e.targetName, ",")
+ if e.opts.ImageName != "" {
+ targetNames := strings.Split(e.opts.ImageName, ",")
for _, targetName := range targetNames {
- if e.opt.Images != nil {
- tagDone := oneOffProgress(ctx, "naming to "+targetName)
+ if e.opt.Images != nil && e.store {
+ tagDone := progress.OneOff(ctx, "naming to "+targetName)
img := images.Image{
Target: *desc,
CreatedAt: time.Now(),
@@ -309,56 +259,59 @@ func (e *imageExporterInstance) Export(ctx context.Context, src exporter.Source,
img.Name = targetName + sfx
if _, err := e.opt.Images.Update(ctx, img); err != nil {
if !errors.Is(err, errdefs.ErrNotFound) {
- return nil, tagDone(err)
+ return nil, nil, tagDone(err)
}
if _, err := e.opt.Images.Create(ctx, img); err != nil {
- return nil, tagDone(err)
+ return nil, nil, tagDone(err)
}
}
}
tagDone(nil)
- if e.unpack {
+ if src.Ref != nil && e.unpack {
if err := e.unpackImage(ctx, img, src, session.NewGroup(sessionID)); err != nil {
- return nil, err
- }
- }
- }
- if e.push {
- annotations := map[digest.Digest]map[string]string{}
- mprovider := contentutil.NewMultiProvider(e.opt.ImageWriter.ContentStore())
- if src.Ref != nil {
- remotes, err := src.Ref.GetRemotes(ctx, false, refCfg, false, session.NewGroup(sessionID))
- if err != nil {
- return nil, err
- }
- remote := remotes[0]
- for _, desc := range remote.Descriptors {
- mprovider.Add(desc.Digest, remote.Provider)
- addAnnotations(annotations, desc)
+ return nil, nil, err
}
}
- if len(src.Refs) > 0 {
- for _, r := range src.Refs {
- remotes, err := r.GetRemotes(ctx, false, refCfg, false, session.NewGroup(sessionID))
+
+ if !e.storeAllowIncomplete {
+ if src.Ref != nil {
+ remotes, err := src.Ref.GetRemotes(ctx, false, e.opts.RefCfg, false, session.NewGroup(sessionID))
if err != nil {
- return nil, err
+ return nil, nil, err
}
remote := remotes[0]
- for _, desc := range remote.Descriptors {
- mprovider.Add(desc.Digest, remote.Provider)
- addAnnotations(annotations, desc)
+ if unlazier, ok := remote.Provider.(cache.Unlazier); ok {
+ if err := unlazier.Unlazy(ctx); err != nil {
+ return nil, nil, err
+ }
+ }
+ }
+ if len(src.Refs) > 0 {
+ for _, r := range src.Refs {
+ remotes, err := r.GetRemotes(ctx, false, e.opts.RefCfg, false, session.NewGroup(sessionID))
+ if err != nil {
+ return nil, nil, err
+ }
+ remote := remotes[0]
+ if unlazier, ok := remote.Provider.(cache.Unlazier); ok {
+ if err := unlazier.Unlazy(ctx); err != nil {
+ return nil, nil, err
+ }
+ }
}
}
}
-
- if err := push.Push(ctx, e.opt.SessionManager, sessionID, mprovider, e.opt.ImageWriter.ContentStore(), desc.Digest, targetName, e.insecure, e.opt.RegistryHosts, e.pushByDigest, annotations); err != nil {
- return nil, err
+ }
+ if e.push {
+ err := e.pushImage(ctx, src, sessionID, targetName, desc.Digest)
+ if err != nil {
+ return nil, nil, errors.Wrapf(err, "failed to push %v", targetName)
}
}
}
- resp["image.name"] = e.targetName
+ resp["image.name"] = e.opts.ImageName
}
resp[exptypes.ExporterImageDigestKey] = desc.Digest.String()
@@ -369,22 +322,47 @@ func (e *imageExporterInstance) Export(ctx context.Context, src exporter.Source,
dtdesc, err := json.Marshal(desc)
if err != nil {
- return nil, err
+ return nil, nil, err
}
resp[exptypes.ExporterImageDescriptorKey] = base64.StdEncoding.EncodeToString(dtdesc)
- return resp, nil
+ return resp, nil, nil
}
-func (e *imageExporterInstance) refCfg() cacheconfig.RefConfig {
- return cacheconfig.RefConfig{
- Compression: e.compression(),
- PreferNonDistributable: e.preferNondistLayers,
+func (e *imageExporterInstance) pushImage(ctx context.Context, src *exporter.Source, sessionID string, targetName string, dgst digest.Digest) error {
+ annotations := map[digest.Digest]map[string]string{}
+ mprovider := contentutil.NewMultiProvider(e.opt.ImageWriter.ContentStore())
+ if src.Ref != nil {
+ remotes, err := src.Ref.GetRemotes(ctx, false, e.opts.RefCfg, false, session.NewGroup(sessionID))
+ if err != nil {
+ return err
+ }
+ remote := remotes[0]
+ for _, desc := range remote.Descriptors {
+ mprovider.Add(desc.Digest, remote.Provider)
+ addAnnotations(annotations, desc)
+ }
+ }
+ if len(src.Refs) > 0 {
+ for _, r := range src.Refs {
+ remotes, err := r.GetRemotes(ctx, false, e.opts.RefCfg, false, session.NewGroup(sessionID))
+ if err != nil {
+ return err
+ }
+ remote := remotes[0]
+ for _, desc := range remote.Descriptors {
+ mprovider.Add(desc.Digest, remote.Provider)
+ addAnnotations(annotations, desc)
+ }
+ }
}
+
+ ctx = remotes.WithMediaTypeKeyPrefix(ctx, intoto.PayloadType, "intoto")
+ return push.Push(ctx, e.opt.SessionManager, sessionID, mprovider, e.opt.ImageWriter.ContentStore(), dgst, targetName, e.insecure, e.opt.RegistryHosts, e.pushByDigest, annotations)
}
-func (e *imageExporterInstance) unpackImage(ctx context.Context, img images.Image, src exporter.Source, s session.Group) (err0 error) {
- unpackDone := oneOffProgress(ctx, "unpacking to "+img.Name)
+func (e *imageExporterInstance) unpackImage(ctx context.Context, img images.Image, src *exporter.Source, s session.Group) (err0 error) {
+ unpackDone := progress.OneOff(ctx, "unpacking to "+img.Name)
defer func() {
unpackDone(err0)
}()
@@ -403,14 +381,14 @@ func (e *imageExporterInstance) unpackImage(ctx context.Context, img images.Imag
topLayerRef := src.Ref
if len(src.Refs) > 0 {
- if r, ok := src.Refs[platforms.DefaultString()]; ok {
+ if r, ok := src.Refs[defaultPlatform()]; ok {
topLayerRef = r
} else {
- return errors.Errorf("no reference for default platform %s", platforms.DefaultString())
+ return errors.Errorf("no reference for default platform %s", defaultPlatform())
}
}
- remotes, err := topLayerRef.GetRemotes(ctx, true, e.refCfg(), false, s)
+ remotes, err := topLayerRef.GetRemotes(ctx, true, e.opts.RefCfg, false, s)
if err != nil {
return err
}
@@ -482,3 +460,29 @@ func addAnnotations(m map[digest.Digest]map[string]string, desc ocispecs.Descrip
a[k] = v
}
}
+
+func defaultPlatform() string {
+ // Use normalized platform string to avoid the mismatch with platform options which
+ // are normalized using platforms.Normalize()
+ return platforms.Format(platforms.Normalize(platforms.DefaultSpec()))
+}
+
+func NewDescriptorReference(desc ocispecs.Descriptor, release func(context.Context) error) exporter.DescriptorReference {
+ return &descriptorReference{
+ desc: desc,
+ release: release,
+ }
+}
+
+type descriptorReference struct {
+ desc ocispecs.Descriptor
+ release func(context.Context) error
+}
+
+func (d *descriptorReference) Descriptor() ocispecs.Descriptor {
+ return d.desc
+}
+
+func (d *descriptorReference) Release() error {
+ return d.release(context.TODO())
+}
diff --git a/exporter/containerimage/exptypes/annotations.go b/exporter/containerimage/exptypes/annotations.go
new file mode 100644
index 000000000000..e7697d916ad2
--- /dev/null
+++ b/exporter/containerimage/exptypes/annotations.go
@@ -0,0 +1,115 @@
+package exptypes
+
+import (
+ "fmt"
+ "regexp"
+
+ "github.com/containerd/containerd/platforms"
+ ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
+ "github.com/pkg/errors"
+)
+
+const (
+ AnnotationIndex = "index"
+ AnnotationIndexDescriptor = "index-descriptor"
+ AnnotationManifest = "manifest"
+ AnnotationManifestDescriptor = "manifest-descriptor"
+)
+
+var (
+ keyAnnotationRegexp = regexp.MustCompile(`^annotation(?:-([a-z-]+))?(?:\[([A-Za-z0-9_/-]+)\])?\.(\S+)$`)
+)
+
+type AnnotationKey struct {
+ Type string
+ Platform *ocispecs.Platform
+ Key string
+}
+
+func (k AnnotationKey) String() string {
+ prefix := "annotation"
+
+ switch k.Type {
+ case "":
+ case AnnotationManifest, AnnotationManifestDescriptor:
+ prefix += fmt.Sprintf("-%s", k.Type)
+ if p := k.PlatformString(); p != "" {
+ prefix += fmt.Sprintf("[%s]", p)
+ }
+ case AnnotationIndex, AnnotationIndexDescriptor:
+ prefix += "-" + k.Type
+ default:
+ panic("unknown annotation type")
+ }
+
+ return fmt.Sprintf("%s.%s", prefix, k.Key)
+}
+
+func (k AnnotationKey) PlatformString() string {
+ if k.Platform == nil {
+ return ""
+ }
+ return platforms.Format(*k.Platform)
+}
+
+func AnnotationIndexKey(key string) string {
+ return AnnotationKey{
+ Type: AnnotationIndex,
+ Key: key,
+ }.String()
+}
+
+func AnnotationIndexDescriptorKey(key string) string {
+ return AnnotationKey{
+ Type: AnnotationIndexDescriptor,
+ Key: key,
+ }.String()
+}
+
+func AnnotationManifestKey(p *ocispecs.Platform, key string) string {
+ return AnnotationKey{
+ Type: AnnotationManifest,
+ Platform: p,
+ Key: key,
+ }.String()
+}
+
+func AnnotationManifestDescriptorKey(p *ocispecs.Platform, key string) string {
+ return AnnotationKey{
+ Type: AnnotationManifestDescriptor,
+ Platform: p,
+ Key: key,
+ }.String()
+}
+
+func ParseAnnotationKey(result string) (AnnotationKey, bool, error) {
+ groups := keyAnnotationRegexp.FindStringSubmatch(result)
+ if groups == nil {
+ return AnnotationKey{}, false, nil
+ }
+
+ tp, platform, key := groups[1], groups[2], groups[3]
+ switch tp {
+ case AnnotationIndex, AnnotationIndexDescriptor, AnnotationManifest, AnnotationManifestDescriptor:
+ case "":
+ tp = AnnotationManifest
+ default:
+ return AnnotationKey{}, true, errors.Errorf("unrecognized annotation type %s", tp)
+ }
+
+ var ociPlatform *ocispecs.Platform
+ if platform != "" {
+ p, err := platforms.Parse(platform)
+ if err != nil {
+ return AnnotationKey{}, true, err
+ }
+ ociPlatform = &p
+ }
+
+ annotation := AnnotationKey{
+ Type: tp,
+ Platform: ociPlatform,
+ Key: key,
+ }
+ return annotation, true, nil
+}
diff --git a/exporter/containerimage/exptypes/parse.go b/exporter/containerimage/exptypes/parse.go
new file mode 100644
index 000000000000..f77cd3f52565
--- /dev/null
+++ b/exporter/containerimage/exptypes/parse.go
@@ -0,0 +1,56 @@
+package exptypes
+
+import (
+ "encoding/json"
+ "fmt"
+
+ "github.com/containerd/containerd/platforms"
+ ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
+ "github.com/pkg/errors"
+)
+
+func ParsePlatforms(meta map[string][]byte) (Platforms, error) {
+ if platformsBytes, ok := meta[ExporterPlatformsKey]; ok {
+ var ps Platforms
+ if len(platformsBytes) > 0 {
+ if err := json.Unmarshal(platformsBytes, &ps); err != nil {
+ return Platforms{}, errors.Wrapf(err, "failed to parse platforms passed to provenance processor")
+ }
+ }
+ return ps, nil
+ }
+
+ p := platforms.DefaultSpec()
+ if imgConfig, ok := meta[ExporterImageConfigKey]; ok {
+ var img ocispecs.Image
+ err := json.Unmarshal(imgConfig, &img)
+ if err != nil {
+ return Platforms{}, err
+ }
+
+ if img.OS != "" && img.Architecture != "" {
+ p = ocispecs.Platform{
+ Architecture: img.Architecture,
+ OS: img.OS,
+ OSVersion: img.OSVersion,
+ OSFeatures: img.OSFeatures,
+ Variant: img.Variant,
+ }
+ }
+ }
+ p = platforms.Normalize(p)
+ pk := platforms.Format(p)
+ ps := Platforms{
+ Platforms: []Platform{{ID: pk, Platform: p}},
+ }
+ return ps, nil
+}
+
+func ParseKey(meta map[string][]byte, key string, p Platform) []byte {
+ if v, ok := meta[fmt.Sprintf("%s/%s", key, p.ID)]; ok {
+ return v
+ } else if v, ok := meta[key]; ok {
+ return v
+ }
+ return nil
+}
diff --git a/exporter/containerimage/exptypes/types.go b/exporter/containerimage/exptypes/types.go
index a18d660a5c4a..4531360afa80 100644
--- a/exporter/containerimage/exptypes/types.go
+++ b/exporter/containerimage/exptypes/types.go
@@ -11,10 +11,19 @@ const (
ExporterImageConfigDigestKey = "containerimage.config.digest"
ExporterImageDescriptorKey = "containerimage.descriptor"
ExporterInlineCache = "containerimage.inlinecache"
- ExporterBuildInfo = "containerimage.buildinfo"
+ ExporterBuildInfo = "containerimage.buildinfo" // Deprecated: Build information is deprecated: https://github.com/moby/buildkit/blob/master/docs/deprecated.md
ExporterPlatformsKey = "refs.platforms"
+ ExporterEpochKey = "source.date.epoch"
)
+// KnownRefMetadataKeys are the subset of exporter keys that can be suffixed by
+// a platform to become platform specific
+var KnownRefMetadataKeys = []string{
+ ExporterImageConfigKey,
+ ExporterInlineCache,
+ ExporterBuildInfo,
+}
+
type Platforms struct {
Platforms []Platform
}
diff --git a/exporter/containerimage/image/docker_image.go b/exporter/containerimage/image/docker_image.go
new file mode 100644
index 000000000000..a35d811d55cf
--- /dev/null
+++ b/exporter/containerimage/image/docker_image.go
@@ -0,0 +1,52 @@
+package image
+
+import (
+ "time"
+
+ "github.com/docker/docker/api/types/strslice"
+ ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
+)
+
+// HealthConfig holds configuration settings for the HEALTHCHECK feature.
+type HealthConfig struct {
+ // Test is the test to perform to check that the container is healthy.
+ // An empty slice means to inherit the default.
+ // The options are:
+ // {} : inherit healthcheck
+ // {"NONE"} : disable healthcheck
+ // {"CMD", args...} : exec arguments directly
+ // {"CMD-SHELL", command} : run command with system's default shell
+ Test []string `json:",omitempty"`
+
+ // Zero means to inherit. Durations are expressed as integer nanoseconds.
+ Interval time.Duration `json:",omitempty"` // Interval is the time to wait between checks.
+ Timeout time.Duration `json:",omitempty"` // Timeout is the time to wait before considering the check to have hung.
+ StartPeriod time.Duration `json:",omitempty"` // The start period for the container to initialize before the retries starts to count down.
+
+ // Retries is the number of consecutive failures needed to consider a container as unhealthy.
+ // Zero means inherit.
+ Retries int `json:",omitempty"`
+}
+
+// ImageConfig is a docker compatible config for an image
+type ImageConfig struct {
+ ocispecs.ImageConfig
+
+ Healthcheck *HealthConfig `json:",omitempty"` // Healthcheck describes how to check the container is healthy
+ ArgsEscaped bool `json:",omitempty"` // True if command is already escaped (Windows specific)
+
+ // NetworkDisabled bool `json:",omitempty"` // Is network disabled
+ // MacAddress string `json:",omitempty"` // Mac Address of the container
+ OnBuild []string // ONBUILD metadata that were defined on the image Dockerfile
+ StopTimeout *int `json:",omitempty"` // Timeout (in seconds) to stop a container
+ Shell strslice.StrSlice `json:",omitempty"` // Shell for shell-form of RUN, CMD, ENTRYPOINT
+}
+
+// Image is the JSON structure which describes some basic information about the image.
+// This provides the `application/vnd.oci.image.config.v1+json` mediatype when marshalled to JSON.
+type Image struct {
+ ocispecs.Image
+
+ // Config defines the execution parameters which should be used as a base when running a container using the image.
+ Config ImageConfig `json:"config,omitempty"`
+}
diff --git a/exporter/containerimage/opts.go b/exporter/containerimage/opts.go
new file mode 100644
index 000000000000..4948eaad2431
--- /dev/null
+++ b/exporter/containerimage/opts.go
@@ -0,0 +1,161 @@
+package containerimage
+
+import (
+ "strconv"
+ "time"
+
+ cacheconfig "github.com/moby/buildkit/cache/config"
+ "github.com/moby/buildkit/exporter/util/epoch"
+ "github.com/moby/buildkit/util/compression"
+ "github.com/pkg/errors"
+ "github.com/sirupsen/logrus"
+)
+
+const (
+ keyImageName = "name"
+ keyLayerCompression = "compression"
+ keyCompressionLevel = "compression-level"
+ keyForceCompression = "force-compression"
+ keyOCITypes = "oci-mediatypes"
+ keyBuildInfo = "buildinfo"
+ keyBuildInfoAttrs = "buildinfo-attrs"
+ keyForceInlineAttestations = "attestation-inline"
+
+ // preferNondistLayersKey is an exporter option which can be used to mark a layer as non-distributable if the layer reference was
+ // already found to use a non-distributable media type.
+ // When this option is not set, the exporter will change the media type of the layer to a distributable one.
+ keyPreferNondistLayers = "prefer-nondist-layers"
+)
+
+type ImageCommitOpts struct {
+ ImageName string
+ RefCfg cacheconfig.RefConfig
+ OCITypes bool
+ Annotations AnnotationsGroup
+ Epoch *time.Time
+
+ ForceInlineAttestations bool // force inline attestations to be attached
+
+ BuildInfo bool // Deprecated: Build information is deprecated: https://github.com/moby/buildkit/blob/master/docs/deprecated.md
+ BuildInfoAttrs bool // Deprecated: Build information is deprecated: https://github.com/moby/buildkit/blob/master/docs/deprecated.md
+}
+
+func (c *ImageCommitOpts) Load(opt map[string]string) (map[string]string, error) {
+ rest := make(map[string]string)
+
+ as, optb, err := ParseAnnotations(toBytesMap(opt))
+ if err != nil {
+ return nil, err
+ }
+ opt = toStringMap(optb)
+
+ c.Epoch, opt, err = epoch.ParseExporterAttrs(opt)
+ if err != nil {
+ return nil, err
+ }
+
+ for k, v := range opt {
+ var err error
+ switch k {
+ case keyImageName:
+ c.ImageName = v
+ case keyLayerCompression:
+ c.RefCfg.Compression.Type, err = compression.Parse(v)
+ case keyCompressionLevel:
+ ii, err2 := strconv.ParseInt(v, 10, 64)
+ if err != nil {
+ err = errors.Wrapf(err2, "non-int value %s specified for %s", v, k)
+ break
+ }
+ v := int(ii)
+ c.RefCfg.Compression.Level = &v
+ case keyForceCompression:
+ err = parseBoolWithDefault(&c.RefCfg.Compression.Force, k, v, true)
+ case keyOCITypes:
+ err = parseBoolWithDefault(&c.OCITypes, k, v, true)
+ case keyBuildInfo:
+ err = parseBoolWithDefault(&c.BuildInfo, k, v, true)
+ case keyBuildInfoAttrs:
+ err = parseBoolWithDefault(&c.BuildInfoAttrs, k, v, false)
+ case keyForceInlineAttestations:
+ err = parseBool(&c.ForceInlineAttestations, k, v)
+ case keyPreferNondistLayers:
+ err = parseBool(&c.RefCfg.PreferNonDistributable, k, v)
+ default:
+ rest[k] = v
+ }
+
+ if err != nil {
+ return nil, err
+ }
+ }
+
+ if c.RefCfg.Compression.Type.OnlySupportOCITypes() {
+ c.EnableOCITypes(c.RefCfg.Compression.Type.String())
+ }
+
+ if c.RefCfg.Compression.Type.NeedsForceCompression() {
+ c.EnableForceCompression(c.RefCfg.Compression.Type.String())
+ }
+
+ c.Annotations = c.Annotations.Merge(as)
+
+ return rest, nil
+}
+
+func (c *ImageCommitOpts) EnableOCITypes(reason string) {
+ if !c.OCITypes {
+ message := "forcibly turning on oci-mediatype mode"
+ if reason != "" {
+ message += " for " + reason
+ }
+ logrus.Warn(message)
+
+ c.OCITypes = true
+ }
+}
+
+func (c *ImageCommitOpts) EnableForceCompression(reason string) {
+ if !c.RefCfg.Compression.Force {
+ message := "forcibly turning on force-compression mode"
+ if reason != "" {
+ message += " for " + reason
+ }
+ logrus.Warn(message)
+
+ c.RefCfg.Compression.Force = true
+ }
+}
+
+func parseBool(dest *bool, key string, value string) error {
+ b, err := strconv.ParseBool(value)
+ if err != nil {
+ return errors.Wrapf(err, "non-bool value specified for %s", key)
+ }
+ *dest = b
+ return nil
+}
+
+func parseBoolWithDefault(dest *bool, key string, value string, defaultValue bool) error {
+ if value == "" {
+ *dest = defaultValue
+ return nil
+ }
+ return parseBool(dest, key, value)
+}
+
+func toBytesMap(m map[string]string) map[string][]byte {
+ result := make(map[string][]byte)
+ for k, v := range m {
+ result[k] = []byte(v)
+ }
+ return result
+}
+
+func toStringMap(m map[string][]byte) map[string]string {
+ result := make(map[string]string)
+ for k, v := range m {
+ result[k] = string(v)
+ }
+ return result
+}
diff --git a/exporter/containerimage/patch.go b/exporter/containerimage/patch.go
new file mode 100644
index 000000000000..93866b018bda
--- /dev/null
+++ b/exporter/containerimage/patch.go
@@ -0,0 +1,18 @@
+//go:build !nydus
+// +build !nydus
+
+package containerimage
+
+import (
+ "context"
+
+ "github.com/moby/buildkit/cache"
+ "github.com/moby/buildkit/session"
+ "github.com/moby/buildkit/solver"
+ ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
+)
+
+func patchImageLayers(ctx context.Context, remote *solver.Remote, history []ocispecs.History, ref cache.ImmutableRef, opts *ImageCommitOpts, sg session.Group) (*solver.Remote, []ocispecs.History, error) {
+ remote, history = normalizeLayersAndHistory(ctx, remote, history, ref, opts.OCITypes)
+ return remote, history, nil
+}
diff --git a/exporter/containerimage/patch_nydus.go b/exporter/containerimage/patch_nydus.go
new file mode 100644
index 000000000000..3a9336a66f64
--- /dev/null
+++ b/exporter/containerimage/patch_nydus.go
@@ -0,0 +1,35 @@
+//go:build nydus
+// +build nydus
+
+package containerimage
+
+import (
+ "context"
+
+ "github.com/moby/buildkit/cache"
+ "github.com/moby/buildkit/session"
+ "github.com/moby/buildkit/solver"
+ "github.com/moby/buildkit/util/compression"
+ ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
+ "github.com/pkg/errors"
+)
+
+// patchImageLayers appends an extra nydus bootstrap layer
+// to the manifest of nydus image, normalizes layers and
+// history. The nydus bootstrap layer represents the whole
+// metadata of filesystem view for the entire image.
+func patchImageLayers(ctx context.Context, remote *solver.Remote, history []ocispecs.History, ref cache.ImmutableRef, opts *ImageCommitOpts, sg session.Group) (*solver.Remote, []ocispecs.History, error) {
+ if opts.RefCfg.Compression.Type != compression.Nydus {
+ remote, history = normalizeLayersAndHistory(ctx, remote, history, ref, opts.OCITypes)
+ return remote, history, nil
+ }
+
+ desc, err := cache.MergeNydus(ctx, ref, opts.RefCfg.Compression, sg)
+ if err != nil {
+ return nil, nil, errors.Wrap(err, "merge nydus layer")
+ }
+ remote.Descriptors = append(remote.Descriptors, *desc)
+
+ remote, history = normalizeLayersAndHistory(ctx, remote, history, ref, opts.OCITypes)
+ return remote, history, nil
+}
diff --git a/exporter/containerimage/writer.go b/exporter/containerimage/writer.go
index e5ec1519803d..068d86958f8f 100644
--- a/exporter/containerimage/writer.go
+++ b/exporter/containerimage/writer.go
@@ -5,6 +5,7 @@ import (
"context"
"encoding/json"
"fmt"
+ "strconv"
"strings"
"time"
@@ -12,18 +13,24 @@ import (
"github.com/containerd/containerd/diff"
"github.com/containerd/containerd/images"
"github.com/containerd/containerd/platforms"
+ intoto "github.com/in-toto/in-toto-golang/in_toto"
"github.com/moby/buildkit/cache"
cacheconfig "github.com/moby/buildkit/cache/config"
"github.com/moby/buildkit/exporter"
+ "github.com/moby/buildkit/exporter/attestation"
"github.com/moby/buildkit/exporter/containerimage/exptypes"
+ "github.com/moby/buildkit/exporter/util/epoch"
"github.com/moby/buildkit/session"
"github.com/moby/buildkit/snapshot"
"github.com/moby/buildkit/solver"
+ "github.com/moby/buildkit/solver/result"
+ attestationTypes "github.com/moby/buildkit/util/attestation"
"github.com/moby/buildkit/util/bklog"
"github.com/moby/buildkit/util/buildinfo"
binfotypes "github.com/moby/buildkit/util/buildinfo/types"
"github.com/moby/buildkit/util/compression"
"github.com/moby/buildkit/util/progress"
+ "github.com/moby/buildkit/util/purl"
"github.com/moby/buildkit/util/system"
"github.com/moby/buildkit/util/tracing"
digest "github.com/opencontainers/go-digest"
@@ -50,57 +57,123 @@ type ImageWriter struct {
opt WriterOpt
}
-func (ic *ImageWriter) Commit(ctx context.Context, inp exporter.Source, oci bool, refCfg cacheconfig.RefConfig, buildInfo bool, buildInfoAttrs bool, sessionID string) (*ocispecs.Descriptor, error) {
- platformsBytes, ok := inp.Metadata[exptypes.ExporterPlatformsKey]
-
- if len(inp.Refs) > 0 && !ok {
+func (ic *ImageWriter) Commit(ctx context.Context, inp *exporter.Source, sessionID string, opts *ImageCommitOpts) (*ocispecs.Descriptor, error) {
+ if _, ok := inp.Metadata[exptypes.ExporterPlatformsKey]; len(inp.Refs) > 0 && !ok {
return nil, errors.Errorf("unable to export multiple refs, missing platforms mapping")
}
- if len(inp.Refs) == 0 {
- remotes, err := ic.exportLayers(ctx, refCfg, session.NewGroup(sessionID), inp.Ref)
+ isMap := len(inp.Refs) > 0
+
+ ps, err := exptypes.ParsePlatforms(inp.Metadata)
+ if err != nil {
+ return nil, err
+ }
+
+ if !isMap {
+ // enable index if we need to include attestations
+ for _, p := range ps.Platforms {
+ if atts, ok := inp.Attestations[p.ID]; ok {
+ if !opts.ForceInlineAttestations {
+ // if we don't need force inline attestations (for oci
+ // exporter), filter them out
+ atts = attestation.Filter(atts, nil, map[string][]byte{
+ result.AttestationInlineOnlyKey: []byte(strconv.FormatBool(true)),
+ })
+ }
+ if len(atts) > 0 {
+ isMap = true
+ break
+ }
+ }
+ }
+ }
+ if opts.Epoch == nil {
+ if tm, ok, err := epoch.ParseSource(inp); err != nil {
+ return nil, err
+ } else if ok {
+ opts.Epoch = tm
+ }
+ }
+
+ for pk, a := range opts.Annotations {
+ if pk != "" {
+ if _, ok := inp.FindRef(pk); !ok {
+ return nil, errors.Errorf("invalid annotation: no platform %s found in source", pk)
+ }
+ }
+ if len(a.Index)+len(a.IndexDescriptor)+len(a.ManifestDescriptor) > 0 {
+ opts.EnableOCITypes("annotations")
+ }
+ }
+
+ if !isMap {
+ if len(ps.Platforms) > 1 {
+ return nil, errors.Errorf("cannot export multiple platforms without multi-platform enabled")
+ }
+
+ var ref cache.ImmutableRef
+ var p exptypes.Platform
+ if len(ps.Platforms) > 0 {
+ p = ps.Platforms[0]
+ if r, ok := inp.FindRef(p.ID); ok {
+ ref = r
+ }
+ } else {
+ ref = inp.Ref
+ }
+
+ remotes, err := ic.exportLayers(ctx, opts.RefCfg, session.NewGroup(sessionID), ref)
if err != nil {
return nil, err
}
var dtbi []byte
- if buildInfo {
- if dtbi, err = buildinfo.Format(inp.Metadata[exptypes.ExporterBuildInfo], buildinfo.FormatOpts{
- RemoveAttrs: !buildInfoAttrs,
+ if opts.BuildInfo {
+ if dtbi, err = buildinfo.Format(exptypes.ParseKey(inp.Metadata, exptypes.ExporterBuildInfo, p), buildinfo.FormatOpts{
+ RemoveAttrs: !opts.BuildInfoAttrs,
}); err != nil {
return nil, err
}
}
- mfstDesc, configDesc, err := ic.commitDistributionManifest(ctx, inp.Ref, inp.Metadata[exptypes.ExporterImageConfigKey], &remotes[0], oci, inp.Metadata[exptypes.ExporterInlineCache], dtbi)
+ annotations := opts.Annotations.Platform(nil)
+ if len(annotations.Index) > 0 || len(annotations.IndexDescriptor) > 0 {
+ return nil, errors.Errorf("index annotations not supported for single platform export")
+ }
+
+ config := exptypes.ParseKey(inp.Metadata, exptypes.ExporterImageConfigKey, p)
+ inlineCache := exptypes.ParseKey(inp.Metadata, exptypes.ExporterInlineCache, p)
+ mfstDesc, configDesc, err := ic.commitDistributionManifest(ctx, opts, ref, config, &remotes[0], annotations, inlineCache, dtbi, opts.Epoch, session.NewGroup(sessionID))
if err != nil {
return nil, err
}
if mfstDesc.Annotations == nil {
mfstDesc.Annotations = make(map[string]string)
}
+ if len(ps.Platforms) == 1 {
+ mfstDesc.Platform = &ps.Platforms[0].Platform
+ }
mfstDesc.Annotations[exptypes.ExporterConfigDigestKey] = configDesc.Digest.String()
return mfstDesc, nil
}
- var p exptypes.Platforms
- if err := json.Unmarshal(platformsBytes, &p); err != nil {
- return nil, errors.Wrapf(err, "failed to parse platforms passed to exporter")
- }
-
- if len(p.Platforms) != len(inp.Refs) {
- return nil, errors.Errorf("number of platforms does not match references %d %d", len(p.Platforms), len(inp.Refs))
+ if len(inp.Attestations) > 0 {
+ opts.EnableOCITypes("attestations")
}
refs := make([]cache.ImmutableRef, 0, len(inp.Refs))
remotesMap := make(map[string]int, len(inp.Refs))
- for id, r := range inp.Refs {
- remotesMap[id] = len(refs)
+ for _, p := range ps.Platforms {
+ r, ok := inp.FindRef(p.ID)
+ if !ok {
+ return nil, errors.Errorf("failed to find ref for ID %s", p.ID)
+ }
+ remotesMap[p.ID] = len(refs)
refs = append(refs, r)
}
- remotes, err := ic.exportLayers(ctx, refCfg, session.NewGroup(sessionID), refs...)
+ remotes, err := ic.exportLayers(ctx, opts.RefCfg, session.NewGroup(sessionID), refs...)
if err != nil {
return nil, err
}
@@ -114,36 +187,46 @@ func (ic *ImageWriter) Commit(ctx context.Context, inp exporter.Source, oci bool
}{
MediaType: ocispecs.MediaTypeImageIndex,
Index: ocispecs.Index{
+ Annotations: opts.Annotations.Platform(nil).Index,
Versioned: specs.Versioned{
SchemaVersion: 2,
},
},
}
- if !oci {
+ if !opts.OCITypes {
idx.MediaType = images.MediaTypeDockerSchema2ManifestList
}
labels := map[string]string{}
- for i, p := range p.Platforms {
- r, ok := inp.Refs[p.ID]
+ var attestationManifests []ocispecs.Descriptor
+
+ for i, p := range ps.Platforms {
+ r, ok := inp.FindRef(p.ID)
if !ok {
return nil, errors.Errorf("failed to find ref for ID %s", p.ID)
}
- config := inp.Metadata[fmt.Sprintf("%s/%s", exptypes.ExporterImageConfigKey, p.ID)]
- inlineCache := inp.Metadata[fmt.Sprintf("%s/%s", exptypes.ExporterInlineCache, p.ID)]
+ config := exptypes.ParseKey(inp.Metadata, exptypes.ExporterImageConfigKey, p)
+ inlineCache := exptypes.ParseKey(inp.Metadata, exptypes.ExporterInlineCache, p)
var dtbi []byte
- if buildInfo {
- if dtbi, err = buildinfo.Format(inp.Metadata[fmt.Sprintf("%s/%s", exptypes.ExporterBuildInfo, p.ID)], buildinfo.FormatOpts{
- RemoveAttrs: !buildInfoAttrs,
+ if opts.BuildInfo {
+ if dtbi, err = buildinfo.Format(exptypes.ParseKey(inp.Metadata, exptypes.ExporterBuildInfo, p), buildinfo.FormatOpts{
+ RemoveAttrs: !opts.BuildInfoAttrs,
}); err != nil {
return nil, err
}
}
- desc, _, err := ic.commitDistributionManifest(ctx, r, config, &remotes[remotesMap[p.ID]], oci, inlineCache, dtbi)
+ remote := &remotes[remotesMap[p.ID]]
+ if remote == nil {
+ remote = &solver.Remote{
+ Provider: ic.opt.ContentStore,
+ }
+ }
+
+ desc, _, err := ic.commitDistributionManifest(ctx, opts, r, config, remote, opts.Annotations.Platform(&p.Platform), inlineCache, dtbi, opts.Epoch, session.NewGroup(sessionID))
if err != nil {
return nil, err
}
@@ -152,20 +235,75 @@ func (ic *ImageWriter) Commit(ctx context.Context, inp exporter.Source, oci bool
idx.Manifests = append(idx.Manifests, *desc)
labels[fmt.Sprintf("containerd.io/gc.ref.content.%d", i)] = desc.Digest.String()
+
+ if attestations, ok := inp.Attestations[p.ID]; ok {
+ attestations, err := attestation.Unbundle(ctx, session.NewGroup(sessionID), attestations)
+ if err != nil {
+ return nil, err
+ }
+
+ eg, ctx2 := errgroup.WithContext(ctx)
+ for i, att := range attestations {
+ i, att := i, att
+ eg.Go(func() error {
+ att, err := supplementSBOM(ctx2, session.NewGroup(sessionID), r, remote, att)
+ if err != nil {
+ return err
+ }
+ attestations[i] = att
+ return nil
+ })
+ }
+ if err := eg.Wait(); err != nil {
+ return nil, err
+ }
+
+ var defaultSubjects []intoto.Subject
+ for _, name := range strings.Split(opts.ImageName, ",") {
+ if name == "" {
+ continue
+ }
+ pl, err := purl.RefToPURL(name, &p.Platform)
+ if err != nil {
+ return nil, err
+ }
+ defaultSubjects = append(defaultSubjects, intoto.Subject{
+ Name: pl,
+ Digest: result.ToDigestMap(desc.Digest),
+ })
+ }
+ stmts, err := attestation.MakeInTotoStatements(ctx, session.NewGroup(sessionID), attestations, defaultSubjects)
+ if err != nil {
+ return nil, err
+ }
+
+ desc, err := ic.commitAttestationsManifest(ctx, opts, p, desc.Digest.String(), stmts)
+ if err != nil {
+ return nil, err
+ }
+ desc.Platform = &intotoPlatform
+ attestationManifests = append(attestationManifests, *desc)
+ }
+ }
+
+ for i, mfst := range attestationManifests {
+ idx.Manifests = append(idx.Manifests, mfst)
+ labels[fmt.Sprintf("containerd.io/gc.ref.content.%d", len(ps.Platforms)+i)] = mfst.Digest.String()
}
- idxBytes, err := json.MarshalIndent(idx, "", " ")
+ idxBytes, err := json.MarshalIndent(idx, "", " ")
if err != nil {
return nil, errors.Wrap(err, "failed to marshal index")
}
idxDigest := digest.FromBytes(idxBytes)
idxDesc := ocispecs.Descriptor{
- Digest: idxDigest,
- Size: int64(len(idxBytes)),
- MediaType: idx.MediaType,
+ Digest: idxDigest,
+ Size: int64(len(idxBytes)),
+ MediaType: idx.MediaType,
+ Annotations: opts.Annotations.Platform(nil).IndexDescriptor,
}
- idxDone := oneOffProgress(ctx, "exporting manifest list "+idxDigest.String())
+ idxDone := progress.OneOff(ctx, "exporting manifest list "+idxDigest.String())
if err := content.WriteBlob(ctx, ic.opt.ContentStore, idxDigest.String(), bytes.NewReader(idxBytes), idxDesc, content.WithLabels(labels)); err != nil {
return nil, idxDone(errors.Wrapf(err, "error writing manifest list blob %s", idxDigest))
@@ -186,7 +324,7 @@ func (ic *ImageWriter) exportLayers(ctx context.Context, refCfg cacheconfig.RefC
span, ctx := tracing.StartSpan(ctx, "export layers", trace.WithAttributes(attr...))
eg, ctx := errgroup.WithContext(ctx)
- layersDone := oneOffProgress(ctx, "exporting layers")
+ layersDone := progress.OneOff(ctx, "exporting layers")
out := make([]solver.Remote, len(refs))
@@ -212,29 +350,26 @@ func (ic *ImageWriter) exportLayers(ctx context.Context, refCfg cacheconfig.RefC
return out, err
}
-func (ic *ImageWriter) commitDistributionManifest(ctx context.Context, ref cache.ImmutableRef, config []byte, remote *solver.Remote, oci bool, inlineCache []byte, buildInfo []byte) (*ocispecs.Descriptor, *ocispecs.Descriptor, error) {
+func (ic *ImageWriter) commitDistributionManifest(ctx context.Context, opts *ImageCommitOpts, ref cache.ImmutableRef, config []byte, remote *solver.Remote, annotations *Annotations, inlineCache []byte, buildInfo []byte, epoch *time.Time, sg session.Group) (*ocispecs.Descriptor, *ocispecs.Descriptor, error) {
if len(config) == 0 {
var err error
- config, err = emptyImageConfig()
+ config, err = defaultImageConfig()
if err != nil {
return nil, nil, err
}
}
- if remote == nil {
- remote = &solver.Remote{
- Provider: ic.opt.ContentStore,
- }
- }
-
history, err := parseHistoryFromConfig(config)
if err != nil {
return nil, nil, err
}
- remote, history = normalizeLayersAndHistory(ctx, remote, history, ref, oci)
+ remote, history, err = patchImageLayers(ctx, remote, history, ref, opts, sg)
+ if err != nil {
+ return nil, nil, err
+ }
- config, err = patchImageConfig(config, remote.Descriptors, history, inlineCache, buildInfo)
+ config, err = patchImageConfig(config, remote.Descriptors, history, inlineCache, buildInfo, epoch)
if err != nil {
return nil, nil, err
}
@@ -246,7 +381,7 @@ func (ic *ImageWriter) commitDistributionManifest(ctx context.Context, ref cache
)
// Use docker media types for older Docker versions and registries
- if !oci {
+ if !opts.OCITypes {
manifestType = images.MediaTypeDockerSchema2Manifest
configType = images.MediaTypeDockerSchema2Config
}
@@ -260,6 +395,7 @@ func (ic *ImageWriter) commitDistributionManifest(ctx context.Context, ref cache
}{
MediaType: manifestType,
Manifest: ocispecs.Manifest{
+ Annotations: annotations.Manifest,
Versioned: specs.Versioned{
SchemaVersion: 2,
},
@@ -275,25 +411,12 @@ func (ic *ImageWriter) commitDistributionManifest(ctx context.Context, ref cache
"containerd.io/gc.ref.content.0": configDigest.String(),
}
- for i, desc := range remote.Descriptors {
- // oci supports annotations but don't export internal annotations
- if oci {
- delete(desc.Annotations, "containerd.io/uncompressed")
- delete(desc.Annotations, "buildkit/createdat")
- for k := range desc.Annotations {
- if strings.HasPrefix(k, "containerd.io/distribution.source.") {
- delete(desc.Annotations, k)
- }
- }
- } else {
- desc.Annotations = nil
- }
-
+ for _, desc := range remote.Descriptors {
+ desc.Annotations = RemoveInternalLayerAnnotations(desc.Annotations, opts.OCITypes)
mfst.Layers = append(mfst.Layers, desc)
- labels[fmt.Sprintf("containerd.io/gc.ref.content.%d", i+1)] = desc.Digest.String()
}
- mfstJSON, err := json.MarshalIndent(mfst, "", " ")
+ mfstJSON, err := json.MarshalIndent(mfst, "", " ")
if err != nil {
return nil, nil, errors.Wrap(err, "failed to marshal manifest")
}
@@ -303,7 +426,7 @@ func (ic *ImageWriter) commitDistributionManifest(ctx context.Context, ref cache
Digest: mfstDigest,
Size: int64(len(mfstJSON)),
}
- mfstDone := oneOffProgress(ctx, "exporting manifest "+mfstDigest.String())
+ mfstDone := progress.OneOff(ctx, "exporting manifest "+mfstDigest.String())
if err := content.WriteBlob(ctx, ic.opt.ContentStore, mfstDigest.String(), bytes.NewReader(mfstJSON), mfstDesc, content.WithLabels((labels))); err != nil {
return nil, nil, mfstDone(errors.Wrapf(err, "error writing manifest blob %s", mfstDigest))
@@ -315,18 +438,125 @@ func (ic *ImageWriter) commitDistributionManifest(ctx context.Context, ref cache
Size: int64(len(config)),
MediaType: configType,
}
- configDone := oneOffProgress(ctx, "exporting config "+configDigest.String())
+ configDone := progress.OneOff(ctx, "exporting config "+configDigest.String())
if err := content.WriteBlob(ctx, ic.opt.ContentStore, configDigest.String(), bytes.NewReader(config), configDesc); err != nil {
return nil, nil, configDone(errors.Wrap(err, "error writing config blob"))
}
configDone(nil)
+ return &ocispecs.Descriptor{
+ Annotations: annotations.ManifestDescriptor,
+ Digest: mfstDigest,
+ Size: int64(len(mfstJSON)),
+ MediaType: manifestType,
+ }, &configDesc, nil
+}
+
+func (ic *ImageWriter) commitAttestationsManifest(ctx context.Context, opts *ImageCommitOpts, p exptypes.Platform, target string, statements []intoto.Statement) (*ocispecs.Descriptor, error) {
+ var (
+ manifestType = ocispecs.MediaTypeImageManifest
+ configType = ocispecs.MediaTypeImageConfig
+ )
+ if !opts.OCITypes {
+ manifestType = images.MediaTypeDockerSchema2Manifest
+ configType = images.MediaTypeDockerSchema2Config
+ }
+
+ layers := make([]ocispecs.Descriptor, len(statements))
+ for i, statement := range statements {
+ i, statement := i, statement
+
+ data, err := json.Marshal(statement)
+ if err != nil {
+ return nil, errors.Wrap(err, "failed to marshal attestation")
+ }
+ digest := digest.FromBytes(data)
+ desc := ocispecs.Descriptor{
+ MediaType: attestationTypes.MediaTypeDockerSchema2AttestationType,
+ Digest: digest,
+ Size: int64(len(data)),
+ Annotations: map[string]string{
+ "containerd.io/uncompressed": digest.String(),
+ "in-toto.io/predicate-type": statement.PredicateType,
+ },
+ }
+
+ if err := content.WriteBlob(ctx, ic.opt.ContentStore, digest.String(), bytes.NewReader(data), desc); err != nil {
+ return nil, errors.Wrapf(err, "error writing data blob %s", digest)
+ }
+ layers[i] = desc
+ }
+
+ config, err := attestationsConfig(layers)
+ if err != nil {
+ return nil, err
+ }
+ configDigest := digest.FromBytes(config)
+ configDesc := ocispecs.Descriptor{
+ Digest: configDigest,
+ Size: int64(len(config)),
+ MediaType: configType,
+ }
+
+ mfst := struct {
+ // MediaType is reserved in the OCI spec but
+ // excluded from go types.
+ MediaType string `json:"mediaType,omitempty"`
+
+ ocispecs.Manifest
+ }{
+ MediaType: manifestType,
+ Manifest: ocispecs.Manifest{
+ Versioned: specs.Versioned{
+ SchemaVersion: 2,
+ },
+ Config: ocispecs.Descriptor{
+ Digest: configDigest,
+ Size: int64(len(config)),
+ MediaType: configType,
+ },
+ },
+ }
+
+ labels := map[string]string{
+ "containerd.io/gc.ref.content.0": configDigest.String(),
+ }
+ for i, desc := range layers {
+ desc.Annotations = RemoveInternalLayerAnnotations(desc.Annotations, opts.OCITypes)
+ mfst.Layers = append(mfst.Layers, desc)
+ labels[fmt.Sprintf("containerd.io/gc.ref.content.%d", i+1)] = desc.Digest.String()
+ }
+
+ mfstJSON, err := json.MarshalIndent(mfst, "", " ")
+ if err != nil {
+ return nil, errors.Wrap(err, "failed to marshal manifest")
+ }
+
+ mfstDigest := digest.FromBytes(mfstJSON)
+ mfstDesc := ocispecs.Descriptor{
+ Digest: mfstDigest,
+ Size: int64(len(mfstJSON)),
+ }
+
+ done := progress.OneOff(ctx, "exporting attestation manifest "+mfstDigest.String())
+ if err := content.WriteBlob(ctx, ic.opt.ContentStore, mfstDigest.String(), bytes.NewReader(mfstJSON), mfstDesc, content.WithLabels((labels))); err != nil {
+ return nil, done(errors.Wrapf(err, "error writing manifest blob %s", mfstDigest))
+ }
+ if err := content.WriteBlob(ctx, ic.opt.ContentStore, configDigest.String(), bytes.NewReader(config), configDesc); err != nil {
+ return nil, done(errors.Wrap(err, "error writing config blob"))
+ }
+ done(nil)
+
return &ocispecs.Descriptor{
Digest: mfstDigest,
Size: int64(len(mfstJSON)),
MediaType: manifestType,
- }, &configDesc, nil
+ Annotations: map[string]string{
+ attestationTypes.DockerAnnotationReferenceType: attestationTypes.DockerAnnotationReferenceTypeDefault,
+ attestationTypes.DockerAnnotationReferenceDigest: target,
+ },
+ }, nil
}
func (ic *ImageWriter) ContentStore() content.Store {
@@ -341,22 +571,13 @@ func (ic *ImageWriter) Applier() diff.Applier {
return ic.opt.Applier
}
-func emptyImageConfig() ([]byte, error) {
+func defaultImageConfig() ([]byte, error) {
pl := platforms.Normalize(platforms.DefaultSpec())
- type image struct {
- ocispecs.Image
-
- // Variant defines platform variant. To be added to OCI.
- Variant string `json:"variant,omitempty"`
- }
-
- img := image{
- Image: ocispecs.Image{
- Architecture: pl.Architecture,
- OS: pl.OS,
- },
- Variant: pl.Variant,
+ img := ocispecs.Image{
+ Architecture: pl.Architecture,
+ OS: pl.OS,
+ Variant: pl.Variant,
}
img.RootFS.Type = "layers"
img.Config.WorkingDir = "/"
@@ -365,6 +586,22 @@ func emptyImageConfig() ([]byte, error) {
return dt, errors.Wrap(err, "failed to create empty image config")
}
+func attestationsConfig(layers []ocispecs.Descriptor) ([]byte, error) {
+ img := ocispecs.Image{
+ Architecture: intotoPlatform.Architecture,
+ OS: intotoPlatform.OS,
+ OSVersion: intotoPlatform.OSVersion,
+ OSFeatures: intotoPlatform.OSFeatures,
+ Variant: intotoPlatform.Variant,
+ }
+ img.RootFS.Type = "layers"
+ for _, layer := range layers {
+ img.RootFS.DiffIDs = append(img.RootFS.DiffIDs, digest.Digest(layer.Annotations["containerd.io/uncompressed"]))
+ }
+ dt, err := json.Marshal(img)
+ return dt, errors.Wrap(err, "failed to create attestations image config")
+}
+
func parseHistoryFromConfig(dt []byte) ([]ocispecs.History, error) {
var config struct {
History []ocispecs.History
@@ -375,7 +612,7 @@ func parseHistoryFromConfig(dt []byte) ([]ocispecs.History, error) {
return config.History, nil
}
-func patchImageConfig(dt []byte, descs []ocispecs.Descriptor, history []ocispecs.History, cache []byte, buildInfo []byte) ([]byte, error) {
+func patchImageConfig(dt []byte, descs []ocispecs.Descriptor, history []ocispecs.History, cache []byte, buildInfo []byte, epoch *time.Time) ([]byte, error) {
m := map[string]json.RawMessage{}
if err := json.Unmarshal(dt, &m); err != nil {
return nil, errors.Wrap(err, "failed to parse image config for patch")
@@ -392,12 +629,35 @@ func patchImageConfig(dt []byte, descs []ocispecs.Descriptor, history []ocispecs
}
m["rootfs"] = dt
+ if epoch != nil {
+ for i, h := range history {
+ if h.Created == nil || h.Created.After(*epoch) {
+ history[i].Created = epoch
+ }
+ }
+ }
+
dt, err = json.Marshal(history)
if err != nil {
return nil, errors.Wrap(err, "failed to marshal history")
}
m["history"] = dt
+ // if epoch is set then clamp creation time
+ if v, ok := m["created"]; ok && epoch != nil {
+ var tm time.Time
+ if err := json.Unmarshal(v, &tm); err != nil {
+ return nil, errors.Wrapf(err, "failed to unmarshal creation time %q", m["created"])
+ }
+ if tm.After(*epoch) {
+ dt, err = json.Marshal(&epoch)
+ if err != nil {
+ return nil, errors.Wrap(err, "failed to marshal creation time")
+ }
+ m["created"] = dt
+ }
+ }
+
if _, ok := m["created"]; !ok {
var tm *time.Time
for _, h := range history {
@@ -426,7 +686,7 @@ func patchImageConfig(dt []byte, descs []ocispecs.Descriptor, history []ocispecs
return nil, err
}
m[binfotypes.ImageConfigField] = dt
- } else if _, ok := m[binfotypes.ImageConfigField]; ok {
+ } else {
delete(m, binfotypes.ImageConfigField)
}
@@ -521,6 +781,26 @@ func normalizeLayersAndHistory(ctx context.Context, remote *solver.Remote, histo
return remote, history
}
+func RemoveInternalLayerAnnotations(in map[string]string, oci bool) map[string]string {
+ if len(in) == 0 || !oci {
+ return nil
+ }
+ m := make(map[string]string, len(in))
+ for k, v := range in {
+ // oci supports annotations but don't export internal annotations
+ switch k {
+ case "containerd.io/uncompressed", "buildkit/createdat":
+ continue
+ default:
+ if strings.HasPrefix(k, "containerd.io/distribution.source.") {
+ continue
+ }
+ m[k] = v
+ }
+ }
+ return m
+}
+
type refMetadata struct {
description string
createdAt *time.Time
@@ -553,20 +833,3 @@ func getRefMetadata(ref cache.ImmutableRef, limit int) []refMetadata {
}
return metas
}
-
-func oneOffProgress(ctx context.Context, id string) func(err error) error {
- pw, _, _ := progress.NewFromContext(ctx)
- now := time.Now()
- st := progress.Status{
- Started: &now,
- }
- pw.Write(id, st)
- return func(err error) error {
- // TODO: set error on status
- now := time.Now()
- st.Completed = &now
- pw.Write(id, st)
- pw.Close()
- return err
- }
-}
diff --git a/exporter/exporter.go b/exporter/exporter.go
index 610481b710f6..0e7d8d14f280 100644
--- a/exporter/exporter.go
+++ b/exporter/exporter.go
@@ -4,25 +4,49 @@ import (
"context"
"github.com/moby/buildkit/cache"
+ "github.com/moby/buildkit/solver/result"
"github.com/moby/buildkit/util/compression"
+ ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
)
+type Source = result.Result[cache.ImmutableRef]
+
+type Attestation = result.Attestation[cache.ImmutableRef]
+
type Exporter interface {
Resolve(context.Context, map[string]string) (ExporterInstance, error)
}
type ExporterInstance interface {
Name() string
- Config() Config
- Export(ctx context.Context, src Source, sessionID string) (map[string]string, error)
+ Config() *Config
+ Export(ctx context.Context, src *Source, sessionID string) (map[string]string, DescriptorReference, error)
}
-type Source struct {
- Ref cache.ImmutableRef
- Refs map[string]cache.ImmutableRef
- Metadata map[string][]byte
+type DescriptorReference interface {
+ Release() error
+ Descriptor() ocispecs.Descriptor
}
type Config struct {
- Compression compression.Config
+ // Make the field private in case it is initialized with nil compression.Type
+ compression compression.Config
+}
+
+func NewConfig() *Config {
+ return &Config{
+ compression: compression.Config{
+ Type: compression.Default,
+ },
+ }
+}
+
+func NewConfigWithCompression(comp compression.Config) *Config {
+ return &Config{
+ compression: comp,
+ }
+}
+
+func (c *Config) Compression() compression.Config {
+ return c.compression
}
diff --git a/exporter/local/export.go b/exporter/local/export.go
index 5daa4aa4268c..7d08b172e019 100644
--- a/exporter/local/export.go
+++ b/exporter/local/export.go
@@ -2,24 +2,28 @@ package local
import (
"context"
- "io/ioutil"
"os"
"strings"
"time"
- "github.com/docker/docker/pkg/idtools"
"github.com/moby/buildkit/cache"
"github.com/moby/buildkit/exporter"
+ "github.com/moby/buildkit/exporter/containerimage/exptypes"
+ "github.com/moby/buildkit/exporter/util/epoch"
"github.com/moby/buildkit/session"
"github.com/moby/buildkit/session/filesync"
- "github.com/moby/buildkit/snapshot"
"github.com/moby/buildkit/util/progress"
+ "github.com/pkg/errors"
"github.com/tonistiigi/fsutil"
fstypes "github.com/tonistiigi/fsutil/types"
"golang.org/x/sync/errgroup"
"golang.org/x/time/rate"
)
+const (
+ keyAttestationPrefix = "attestation-prefix"
+)
+
type Opt struct {
SessionManager *session.Manager
}
@@ -35,93 +39,103 @@ func New(opt Opt) (exporter.Exporter, error) {
}
func (e *localExporter) Resolve(ctx context.Context, opt map[string]string) (exporter.ExporterInstance, error) {
- return &localExporterInstance{localExporter: e}, nil
+ tm, _, err := epoch.ParseExporterAttrs(opt)
+ if err != nil {
+ return nil, err
+ }
+
+ i := &localExporterInstance{
+ localExporter: e,
+ opts: CreateFSOpts{
+ Epoch: tm,
+ },
+ }
+
+ for k, v := range opt {
+ switch k {
+ case keyAttestationPrefix:
+ i.opts.AttestationPrefix = v
+ }
+ }
+
+ return i, nil
}
type localExporterInstance struct {
*localExporter
+ opts CreateFSOpts
}
func (e *localExporterInstance) Name() string {
- return "exporting to client"
+ return "exporting to client directory"
}
-func (e *localExporter) Config() exporter.Config {
- return exporter.Config{}
+func (e *localExporter) Config() *exporter.Config {
+ return exporter.NewConfig()
}
-func (e *localExporterInstance) Export(ctx context.Context, inp exporter.Source, sessionID string) (map[string]string, error) {
+func (e *localExporterInstance) Export(ctx context.Context, inp *exporter.Source, sessionID string) (map[string]string, exporter.DescriptorReference, error) {
timeoutCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
+ if e.opts.Epoch == nil {
+ if tm, ok, err := epoch.ParseSource(inp); err != nil {
+ return nil, nil, err
+ } else if ok {
+ e.opts.Epoch = tm
+ }
+ }
+
caller, err := e.opt.SessionManager.Get(timeoutCtx, sessionID, false)
if err != nil {
- return nil, err
+ return nil, nil, err
}
isMap := len(inp.Refs) > 0
- export := func(ctx context.Context, k string, ref cache.ImmutableRef) func() error {
- return func() error {
- var src string
- var err error
- var idmap *idtools.IdentityMapping
- if ref == nil {
- src, err = ioutil.TempDir("", "buildkit")
- if err != nil {
- return err
- }
- defer os.RemoveAll(src)
- } else {
- mount, err := ref.Mount(ctx, true, session.NewGroup(sessionID))
- if err != nil {
- return err
- }
-
- lm := snapshot.LocalMounter(mount)
+ if _, ok := inp.Metadata[exptypes.ExporterPlatformsKey]; isMap && !ok {
+ return nil, nil, errors.Errorf("unable to export multiple refs, missing platforms mapping")
+ }
+ p, err := exptypes.ParsePlatforms(inp.Metadata)
+ if err != nil {
+ return nil, nil, err
+ }
- src, err = lm.Mount()
- if err != nil {
- return err
- }
+ if !isMap && len(p.Platforms) > 1 {
+ return nil, nil, errors.Errorf("unable to export multiple platforms without map")
+ }
- idmap = mount.IdentityMapping()
+ now := time.Now().Truncate(time.Second)
- defer lm.Unmount()
+ export := func(ctx context.Context, k string, ref cache.ImmutableRef, attestations []exporter.Attestation) func() error {
+ return func() error {
+ outputFS, cleanup, err := CreateFS(ctx, sessionID, k, ref, attestations, now, e.opts)
+ if err != nil {
+ return err
}
-
- walkOpt := &fsutil.WalkOpt{}
-
- if idmap != nil {
- walkOpt.Map = func(p string, st *fstypes.Stat) bool {
- uid, gid, err := idmap.ToContainer(idtools.Identity{
- UID: int(st.Uid),
- GID: int(st.Gid),
- })
- if err != nil {
- return false
- }
- st.Uid = uint32(uid)
- st.Gid = uint32(gid)
- return true
- }
+ if cleanup != nil {
+ defer cleanup()
}
- fs := fsutil.NewFS(src, walkOpt)
lbl := "copying files"
if isMap {
lbl += " " + k
- fs, err = fsutil.SubDirFS([]fsutil.Dir{{FS: fs, Stat: fstypes.Stat{
+ st := fstypes.Stat{
Mode: uint32(os.ModeDir | 0755),
Path: strings.Replace(k, "/", "_", -1),
- }}})
+ }
+ if e.opts.Epoch != nil {
+ st.ModTime = e.opts.Epoch.UnixNano()
+ }
+
+ outputFS, err = fsutil.SubDirFS([]fsutil.Dir{{FS: outputFS, Stat: st}})
if err != nil {
return err
}
}
- progress := newProgressHandler(ctx, lbl)
- if err := filesync.CopyToCaller(ctx, fs, caller, progress); err != nil {
+ progress := NewProgressHandler(ctx, lbl)
+ if err := filesync.CopyToCaller(ctx, outputFS, caller, progress); err != nil {
return err
}
return nil
@@ -130,21 +144,25 @@ func (e *localExporterInstance) Export(ctx context.Context, inp exporter.Source,
eg, ctx := errgroup.WithContext(ctx)
- if isMap {
- for k, ref := range inp.Refs {
- eg.Go(export(ctx, k, ref))
+ if len(p.Platforms) > 0 {
+ for _, p := range p.Platforms {
+ r, ok := inp.FindRef(p.ID)
+ if !ok {
+ return nil, nil, errors.Errorf("failed to find ref for ID %s", p.ID)
+ }
+ eg.Go(export(ctx, p.ID, r, inp.Attestations[p.ID]))
}
} else {
- eg.Go(export(ctx, "", inp.Ref))
+ eg.Go(export(ctx, "", inp.Ref, nil))
}
if err := eg.Wait(); err != nil {
- return nil, err
+ return nil, nil, err
}
- return nil, nil
+ return nil, nil, nil
}
-func newProgressHandler(ctx context.Context, id string) func(int, bool) {
+func NewProgressHandler(ctx context.Context, id string) func(int, bool) {
limiter := rate.NewLimiter(rate.Every(100*time.Millisecond), 1)
pw, _, _ := progress.NewFromContext(ctx)
now := time.Now()
diff --git a/exporter/local/fs.go b/exporter/local/fs.go
new file mode 100644
index 000000000000..c5a524aae32f
--- /dev/null
+++ b/exporter/local/fs.go
@@ -0,0 +1,161 @@
+package local
+
+import (
+ "context"
+ "encoding/json"
+ "io"
+ "io/fs"
+ "os"
+ "path"
+ "strconv"
+ "time"
+
+ "github.com/docker/docker/pkg/idtools"
+ intoto "github.com/in-toto/in-toto-golang/in_toto"
+ "github.com/moby/buildkit/cache"
+ "github.com/moby/buildkit/exporter"
+ "github.com/moby/buildkit/exporter/attestation"
+ "github.com/moby/buildkit/session"
+ "github.com/moby/buildkit/snapshot"
+ "github.com/moby/buildkit/solver/result"
+ "github.com/moby/buildkit/util/staticfs"
+ digest "github.com/opencontainers/go-digest"
+ "github.com/pkg/errors"
+ "github.com/tonistiigi/fsutil"
+ fstypes "github.com/tonistiigi/fsutil/types"
+)
+
+type CreateFSOpts struct {
+ Epoch *time.Time
+ AttestationPrefix string
+}
+
+func CreateFS(ctx context.Context, sessionID string, k string, ref cache.ImmutableRef, attestations []exporter.Attestation, defaultTime time.Time, opt CreateFSOpts) (fsutil.FS, func() error, error) {
+ var cleanup func() error
+ var src string
+ var err error
+ var idmap *idtools.IdentityMapping
+ if ref == nil {
+ src, err = os.MkdirTemp("", "buildkit")
+ if err != nil {
+ return nil, nil, err
+ }
+ cleanup = func() error { return os.RemoveAll(src) }
+ } else {
+ mount, err := ref.Mount(ctx, true, session.NewGroup(sessionID))
+ if err != nil {
+ return nil, nil, err
+ }
+
+ lm := snapshot.LocalMounter(mount)
+
+ src, err = lm.Mount()
+ if err != nil {
+ return nil, nil, err
+ }
+
+ idmap = mount.IdentityMapping()
+
+ cleanup = lm.Unmount
+ }
+
+ walkOpt := &fsutil.WalkOpt{}
+ var idMapFunc func(p string, st *fstypes.Stat) fsutil.MapResult
+
+ if idmap != nil {
+ idMapFunc = func(p string, st *fstypes.Stat) fsutil.MapResult {
+ uid, gid, err := idmap.ToContainer(idtools.Identity{
+ UID: int(st.Uid),
+ GID: int(st.Gid),
+ })
+ if err != nil {
+ return fsutil.MapResultExclude
+ }
+ st.Uid = uint32(uid)
+ st.Gid = uint32(gid)
+ return fsutil.MapResultKeep
+ }
+ }
+
+ walkOpt.Map = func(p string, st *fstypes.Stat) fsutil.MapResult {
+ res := fsutil.MapResultKeep
+ if idMapFunc != nil {
+ res = idMapFunc(p, st)
+ }
+ if opt.Epoch != nil {
+ st.ModTime = opt.Epoch.UnixNano()
+ }
+ return res
+ }
+
+ outputFS := fsutil.NewFS(src, walkOpt)
+ attestations = attestation.Filter(attestations, nil, map[string][]byte{
+ result.AttestationInlineOnlyKey: []byte(strconv.FormatBool(true)),
+ })
+ attestations, err = attestation.Unbundle(ctx, session.NewGroup(sessionID), attestations)
+ if err != nil {
+ return nil, nil, err
+ }
+ if len(attestations) > 0 {
+ subjects := []intoto.Subject{}
+ err = outputFS.Walk(ctx, func(path string, info fs.FileInfo, err error) error {
+ if err != nil {
+ return err
+ }
+ if !info.Mode().IsRegular() {
+ return nil
+ }
+ f, err := outputFS.Open(path)
+ if err != nil {
+ return err
+ }
+ defer f.Close()
+ d := digest.Canonical.Digester()
+ if _, err := io.Copy(d.Hash(), f); err != nil {
+ return err
+ }
+ subjects = append(subjects, intoto.Subject{
+ Name: path,
+ Digest: result.ToDigestMap(d.Digest()),
+ })
+ return nil
+ })
+ if err != nil {
+ return nil, nil, err
+ }
+
+ stmts, err := attestation.MakeInTotoStatements(ctx, session.NewGroup(sessionID), attestations, subjects)
+ if err != nil {
+ return nil, nil, err
+ }
+ stmtFS := staticfs.NewFS()
+
+ names := map[string]struct{}{}
+ for i, stmt := range stmts {
+ dt, err := json.MarshalIndent(stmt, "", " ")
+ if err != nil {
+ return nil, nil, errors.Wrap(err, "failed to marshal attestation")
+ }
+
+ name := opt.AttestationPrefix + path.Base(attestations[i].Path)
+ if _, ok := names[name]; ok {
+ return nil, nil, errors.Errorf("duplicate attestation path name %s", name)
+ }
+ names[name] = struct{}{}
+
+ st := fstypes.Stat{
+ Mode: 0600,
+ Path: name,
+ ModTime: defaultTime.UnixNano(),
+ }
+ if opt.Epoch != nil {
+ st.ModTime = opt.Epoch.UnixNano()
+ }
+ stmtFS.Add(name, st, dt)
+ }
+
+ outputFS = staticfs.NewMergeFS(outputFS, stmtFS)
+ }
+
+ return outputFS, cleanup, nil
+}
diff --git a/exporter/oci/export.go b/exporter/oci/export.go
index 153211c9b709..60982f4daf3c 100644
--- a/exporter/oci/export.go
+++ b/exporter/oci/export.go
@@ -4,19 +4,23 @@ import (
"context"
"encoding/base64"
"encoding/json"
+ "fmt"
"strconv"
"strings"
"time"
archiveexporter "github.com/containerd/containerd/images/archive"
"github.com/containerd/containerd/leases"
+ "github.com/containerd/containerd/remotes"
"github.com/docker/distribution/reference"
+ intoto "github.com/in-toto/in-toto-golang/in_toto"
"github.com/moby/buildkit/cache"
cacheconfig "github.com/moby/buildkit/cache/config"
"github.com/moby/buildkit/exporter"
"github.com/moby/buildkit/exporter/containerimage"
"github.com/moby/buildkit/exporter/containerimage/exptypes"
"github.com/moby/buildkit/session"
+ sessioncontent "github.com/moby/buildkit/session/content"
"github.com/moby/buildkit/session/filesync"
"github.com/moby/buildkit/util/compression"
"github.com/moby/buildkit/util/contentutil"
@@ -25,26 +29,18 @@ import (
"github.com/moby/buildkit/util/progress"
ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/pkg/errors"
- "github.com/sirupsen/logrus"
"google.golang.org/grpc/codes"
)
type ExporterVariant string
const (
- keyImageName = "name"
- keyLayerCompression = "compression"
- VariantOCI = "oci"
- VariantDocker = "docker"
- ociTypes = "oci-mediatypes"
- keyForceCompression = "force-compression"
- keyCompressionLevel = "compression-level"
- keyBuildInfo = "buildinfo"
- keyBuildInfoAttrs = "buildinfo-attrs"
- // preferNondistLayersKey is an exporter option which can be used to mark a layer as non-distributable if the layer reference was
- // already found to use a non-distributable media type.
- // When this option is not set, the exporter will change the media type of the layer to a distributable one.
- preferNondistLayersKey = "prefer-nondist-layers"
+ VariantOCI = "oci"
+ VariantDocker = "docker"
+)
+
+const (
+ keyTar = "tar"
)
type Opt struct {
@@ -64,85 +60,35 @@ func New(opt Opt) (exporter.Exporter, error) {
}
func (e *imageExporter) Resolve(ctx context.Context, opt map[string]string) (exporter.ExporterInstance, error) {
- var ot *bool
i := &imageExporterInstance{
- imageExporter: e,
- layerCompression: compression.Default,
- buildInfo: true,
+ imageExporter: e,
+ tar: true,
+ opts: containerimage.ImageCommitOpts{
+ RefCfg: cacheconfig.RefConfig{
+ Compression: compression.New(compression.Default),
+ },
+ BuildInfo: true,
+ OCITypes: e.opt.Variant == VariantOCI,
+ },
+ }
+
+ opt, err := i.opts.Load(opt)
+ if err != nil {
+ return nil, err
}
- var esgz bool
+
for k, v := range opt {
switch k {
- case keyImageName:
- i.name = v
- case keyLayerCompression:
- switch v {
- case "gzip":
- i.layerCompression = compression.Gzip
- case "estargz":
- i.layerCompression = compression.EStargz
- esgz = true
- case "zstd":
- i.layerCompression = compression.Zstd
- case "uncompressed":
- i.layerCompression = compression.Uncompressed
- default:
- return nil, errors.Errorf("unsupported layer compression type: %v", v)
- }
- case keyForceCompression:
- if v == "" {
- i.forceCompression = true
- continue
- }
- b, err := strconv.ParseBool(v)
- if err != nil {
- return nil, errors.Wrapf(err, "non-bool value %v specified for %s", v, k)
- }
- i.forceCompression = b
- case keyCompressionLevel:
- ii, err := strconv.ParseInt(v, 10, 64)
- if err != nil {
- return nil, errors.Wrapf(err, "non-int value %s specified for %s", v, k)
- }
- v := int(ii)
- i.compressionLevel = &v
- case ociTypes:
- ot = new(bool)
- if v == "" {
- *ot = true
- continue
- }
- b, err := strconv.ParseBool(v)
- if err != nil {
- return nil, errors.Wrapf(err, "non-bool value specified for %s", k)
- }
- *ot = b
- case keyBuildInfo:
- if v == "" {
- i.buildInfo = true
- continue
- }
- b, err := strconv.ParseBool(v)
- if err != nil {
- return nil, errors.Wrapf(err, "non-bool value specified for %s", k)
- }
- i.buildInfo = b
- case keyBuildInfoAttrs:
+ case keyTar:
if v == "" {
- i.buildInfoAttrs = false
+ i.tar = true
continue
}
b, err := strconv.ParseBool(v)
if err != nil {
return nil, errors.Wrapf(err, "non-bool value specified for %s", k)
}
- i.buildInfoAttrs = b
- case preferNondistLayersKey:
- b, err := strconv.ParseBool(v)
- if err != nil {
- return nil, errors.Wrapf(err, "non-bool value specified for %s", k)
- }
- i.preferNonDist = b
+ i.tar = b
default:
if i.meta == nil {
i.meta = make(map[string][]byte)
@@ -150,59 +96,27 @@ func (e *imageExporter) Resolve(ctx context.Context, opt map[string]string) (exp
i.meta[k] = []byte(v)
}
}
- if ot == nil {
- i.ociTypes = e.opt.Variant == VariantOCI
- } else {
- i.ociTypes = *ot
- }
- if esgz && !i.ociTypes {
- logrus.Warn("forcibly turning on oci-mediatype mode for estargz")
- i.ociTypes = true
- }
return i, nil
}
type imageExporterInstance struct {
*imageExporter
- meta map[string][]byte
- name string
- ociTypes bool
- layerCompression compression.Type
- forceCompression bool
- compressionLevel *int
- buildInfo bool
- buildInfoAttrs bool
- preferNonDist bool
+ opts containerimage.ImageCommitOpts
+ tar bool
+ meta map[string][]byte
}
func (e *imageExporterInstance) Name() string {
- return "exporting to oci image format"
+ return fmt.Sprintf("exporting to %s image format", e.opt.Variant)
}
-func (e *imageExporterInstance) Config() exporter.Config {
- return exporter.Config{
- Compression: e.compression(),
- }
+func (e *imageExporterInstance) Config() *exporter.Config {
+ return exporter.NewConfigWithCompression(e.opts.RefCfg.Compression)
}
-func (e *imageExporterInstance) compression() compression.Config {
- c := compression.New(e.layerCompression).SetForce(e.forceCompression)
- if e.compressionLevel != nil {
- c = c.SetLevel(*e.compressionLevel)
- }
- return c
-}
-
-func (e *imageExporterInstance) refCfg() cacheconfig.RefConfig {
- return cacheconfig.RefConfig{
- Compression: e.compression(),
- PreferNonDistributable: e.preferNonDist,
- }
-}
-
-func (e *imageExporterInstance) Export(ctx context.Context, src exporter.Source, sessionID string) (map[string]string, error) {
+func (e *imageExporterInstance) Export(ctx context.Context, src *exporter.Source, sessionID string) (_ map[string]string, descref exporter.DescriptorReference, err error) {
if e.opt.Variant == VariantDocker && len(src.Refs) > 0 {
- return nil, errors.Errorf("docker exporter does not currently support exporting manifest lists")
+ return nil, nil, errors.Errorf("docker exporter does not currently support exporting manifest lists")
}
if src.Metadata == nil {
@@ -212,24 +126,43 @@ func (e *imageExporterInstance) Export(ctx context.Context, src exporter.Source,
src.Metadata[k] = v
}
+ opts := e.opts
+ as, _, err := containerimage.ParseAnnotations(src.Metadata)
+ if err != nil {
+ return nil, nil, err
+ }
+ opts.Annotations = opts.Annotations.Merge(as)
+
ctx, done, err := leaseutil.WithLease(ctx, e.opt.LeaseManager, leaseutil.MakeTemporary)
if err != nil {
- return nil, err
+ return nil, nil, err
}
- defer done(context.TODO())
+ defer func() {
+ if descref == nil {
+ done(context.TODO())
+ }
+ }()
- desc, err := e.opt.ImageWriter.Commit(ctx, src, e.ociTypes, e.refCfg(), e.buildInfo, e.buildInfoAttrs, sessionID)
+ desc, err := e.opt.ImageWriter.Commit(ctx, src, sessionID, &opts)
if err != nil {
- return nil, err
+ return nil, nil, err
}
defer func() {
- e.opt.ImageWriter.ContentStore().Delete(context.TODO(), desc.Digest)
+ if err == nil {
+ descref = containerimage.NewDescriptorReference(*desc, done)
+ }
}()
if desc.Annotations == nil {
desc.Annotations = map[string]string{}
}
- desc.Annotations[ocispecs.AnnotationCreated] = time.Now().UTC().Format(time.RFC3339)
+ if _, ok := desc.Annotations[ocispecs.AnnotationCreated]; !ok {
+ tm := time.Now()
+ if opts.Epoch != nil {
+ tm = *opts.Epoch
+ }
+ desc.Annotations[ocispecs.AnnotationCreated] = tm.UTC().Format(time.RFC3339)
+ }
resp := make(map[string]string)
@@ -241,17 +174,17 @@ func (e *imageExporterInstance) Export(ctx context.Context, src exporter.Source,
dtdesc, err := json.Marshal(desc)
if err != nil {
- return nil, err
+ return nil, nil, err
}
resp[exptypes.ExporterImageDescriptorKey] = base64.StdEncoding.EncodeToString(dtdesc)
- if n, ok := src.Metadata["image.name"]; e.name == "*" && ok {
- e.name = string(n)
+ if n, ok := src.Metadata["image.name"]; e.opts.ImageName == "*" && ok {
+ e.opts.ImageName = string(n)
}
- names, err := normalizedNames(e.name)
+ names, err := normalizedNames(e.opts.ImageName)
if err != nil {
- return nil, err
+ return nil, nil, err
}
if len(names) != 0 {
@@ -264,7 +197,7 @@ func (e *imageExporterInstance) Export(ctx context.Context, src exporter.Source,
expOpts = append(expOpts, archiveexporter.WithAllPlatforms(), archiveexporter.WithSkipDockerManifest())
case VariantDocker:
default:
- return nil, errors.Errorf("invalid variant %q", e.opt.Variant)
+ return nil, nil, errors.Errorf("invalid variant %q", e.opt.Variant)
}
timeoutCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
@@ -272,26 +205,21 @@ func (e *imageExporterInstance) Export(ctx context.Context, src exporter.Source,
caller, err := e.opt.SessionManager.Get(timeoutCtx, sessionID, false)
if err != nil {
- return nil, err
- }
-
- w, err := filesync.CopyFileWriter(ctx, resp, caller)
- if err != nil {
- return nil, err
+ return nil, nil, err
}
mprovider := contentutil.NewMultiProvider(e.opt.ImageWriter.ContentStore())
if src.Ref != nil {
- remotes, err := src.Ref.GetRemotes(ctx, false, e.refCfg(), false, session.NewGroup(sessionID))
+ remotes, err := src.Ref.GetRemotes(ctx, false, e.opts.RefCfg, false, session.NewGroup(sessionID))
if err != nil {
- return nil, err
+ return nil, nil, err
}
remote := remotes[0]
// unlazy before tar export as the tar writer does not handle
// layer blobs in parallel (whereas unlazy does)
if unlazier, ok := remote.Provider.(cache.Unlazier); ok {
if err := unlazier.Unlazy(ctx); err != nil {
- return nil, err
+ return nil, nil, err
}
}
for _, desc := range remote.Descriptors {
@@ -300,14 +228,14 @@ func (e *imageExporterInstance) Export(ctx context.Context, src exporter.Source,
}
if len(src.Refs) > 0 {
for _, r := range src.Refs {
- remotes, err := r.GetRemotes(ctx, false, e.refCfg(), false, session.NewGroup(sessionID))
+ remotes, err := r.GetRemotes(ctx, false, e.opts.RefCfg, false, session.NewGroup(sessionID))
if err != nil {
- return nil, err
+ return nil, nil, err
}
remote := remotes[0]
if unlazier, ok := remote.Provider.(cache.Unlazier); ok {
if err := unlazier.Unlazy(ctx); err != nil {
- return nil, err
+ return nil, nil, err
}
}
for _, desc := range remote.Descriptors {
@@ -316,36 +244,41 @@ func (e *imageExporterInstance) Export(ctx context.Context, src exporter.Source,
}
}
- report := oneOffProgress(ctx, "sending tarball")
- if err := archiveexporter.Export(ctx, mprovider, w, expOpts...); err != nil {
- w.Close()
+ if e.tar {
+ w, err := filesync.CopyFileWriter(ctx, resp, caller)
+ if err != nil {
+ return nil, nil, err
+ }
+
+ report := progress.OneOff(ctx, "sending tarball")
+ if err := archiveexporter.Export(ctx, mprovider, w, expOpts...); err != nil {
+ w.Close()
+ if grpcerrors.Code(err) == codes.AlreadyExists {
+ return resp, nil, report(nil)
+ }
+ return nil, nil, report(err)
+ }
+ err = w.Close()
if grpcerrors.Code(err) == codes.AlreadyExists {
- return resp, report(nil)
+ return resp, nil, report(nil)
+ }
+ if err != nil {
+ return nil, nil, report(err)
+ }
+ report(nil)
+ } else {
+ ctx = remotes.WithMediaTypeKeyPrefix(ctx, intoto.PayloadType, "intoto")
+ store := sessioncontent.NewCallerStore(caller, "export")
+ if err != nil {
+ return nil, nil, err
+ }
+ err := contentutil.CopyChain(ctx, store, mprovider, *desc)
+ if err != nil {
+ return nil, nil, err
}
- return nil, report(err)
- }
- err = w.Close()
- if grpcerrors.Code(err) == codes.AlreadyExists {
- return resp, report(nil)
}
- return resp, report(err)
-}
-func oneOffProgress(ctx context.Context, id string) func(err error) error {
- pw, _, _ := progress.NewFromContext(ctx)
- now := time.Now()
- st := progress.Status{
- Started: &now,
- }
- pw.Write(id, st)
- return func(err error) error {
- // TODO: set error on status
- now := time.Now()
- st.Completed = &now
- pw.Write(id, st)
- pw.Close()
- return err
- }
+ return resp, nil, nil
}
func normalizedNames(name string) ([]string, error) {
diff --git a/exporter/tar/export.go b/exporter/tar/export.go
index 0febefd0b023..4d136c89c1ca 100644
--- a/exporter/tar/export.go
+++ b/exporter/tar/export.go
@@ -2,18 +2,18 @@ package local
import (
"context"
- "io/ioutil"
"os"
"strconv"
"strings"
"time"
- "github.com/docker/docker/pkg/idtools"
"github.com/moby/buildkit/cache"
"github.com/moby/buildkit/exporter"
+ "github.com/moby/buildkit/exporter/containerimage/exptypes"
+ "github.com/moby/buildkit/exporter/local"
+ "github.com/moby/buildkit/exporter/util/epoch"
"github.com/moby/buildkit/session"
"github.com/moby/buildkit/session/filesync"
- "github.com/moby/buildkit/snapshot"
"github.com/moby/buildkit/util/progress"
"github.com/pkg/errors"
"github.com/tonistiigi/fsutil"
@@ -21,6 +21,8 @@ import (
)
const (
+ attestationPrefixKey = "attestation-prefix"
+
// preferNondistLayersKey is an exporter option which can be used to mark a layer as non-distributable if the layer reference was
// already found to use a non-distributable media type.
// When this option is not set, the exporter will change the media type of the layer to a distributable one.
@@ -44,13 +46,23 @@ func New(opt Opt) (exporter.Exporter, error) {
func (e *localExporter) Resolve(ctx context.Context, opt map[string]string) (exporter.ExporterInstance, error) {
li := &localExporterInstance{localExporter: e}
- v, ok := opt[preferNondistLayersKey]
- if ok {
- b, err := strconv.ParseBool(v)
- if err != nil {
- return nil, errors.Wrapf(err, "non-bool value for %s: %s", preferNondistLayersKey, v)
+ tm, opt, err := epoch.ParseExporterAttrs(opt)
+ if err != nil {
+ return nil, err
+ }
+ li.opts.Epoch = tm
+
+ for k, v := range opt {
+ switch k {
+ case preferNondistLayersKey:
+ b, err := strconv.ParseBool(v)
+ if err != nil {
+ return nil, errors.Wrapf(err, "non-bool value for %s: %s", preferNondistLayersKey, v)
+ }
+ li.preferNonDist = b
+ case attestationPrefixKey:
+ li.opts.AttestationPrefix = v
}
- li.preferNonDist = b
}
return li, nil
@@ -58,19 +70,20 @@ func (e *localExporter) Resolve(ctx context.Context, opt map[string]string) (exp
type localExporterInstance struct {
*localExporter
+ opts local.CreateFSOpts
preferNonDist bool
}
func (e *localExporterInstance) Name() string {
- return "exporting to client"
+ return "exporting to client tarball"
}
-func (e *localExporterInstance) Config() exporter.Config {
- return exporter.Config{}
+func (e *localExporterInstance) Config() *exporter.Config {
+ return exporter.NewConfig()
}
-func (e *localExporterInstance) Export(ctx context.Context, inp exporter.Source, sessionID string) (map[string]string, error) {
- var defers []func()
+func (e *localExporterInstance) Export(ctx context.Context, inp *exporter.Source, sessionID string) (map[string]string, exporter.DescriptorReference, error) {
+ var defers []func() error
defer func() {
for i := len(defers) - 1; i >= 0; i-- {
@@ -78,80 +91,79 @@ func (e *localExporterInstance) Export(ctx context.Context, inp exporter.Source,
}
}()
- getDir := func(ctx context.Context, k string, ref cache.ImmutableRef) (*fsutil.Dir, error) {
- var src string
- var err error
- var idmap *idtools.IdentityMapping
- if ref == nil {
- src, err = ioutil.TempDir("", "buildkit")
- if err != nil {
- return nil, err
- }
- defers = append(defers, func() { os.RemoveAll(src) })
- } else {
- mount, err := ref.Mount(ctx, true, session.NewGroup(sessionID))
- if err != nil {
- return nil, err
- }
-
- lm := snapshot.LocalMounter(mount)
-
- src, err = lm.Mount()
- if err != nil {
- return nil, err
- }
+ if e.opts.Epoch == nil {
+ if tm, ok, err := epoch.ParseSource(inp); err != nil {
+ return nil, nil, err
+ } else if ok {
+ e.opts.Epoch = tm
+ }
+ }
- idmap = mount.IdentityMapping()
+ now := time.Now().Truncate(time.Second)
- defers = append(defers, func() { lm.Unmount() })
+ getDir := func(ctx context.Context, k string, ref cache.ImmutableRef, attestations []exporter.Attestation) (*fsutil.Dir, error) {
+ outputFS, cleanup, err := local.CreateFS(ctx, sessionID, k, ref, attestations, now, e.opts)
+ if err != nil {
+ return nil, err
+ }
+ if cleanup != nil {
+ defers = append(defers, cleanup)
}
- walkOpt := &fsutil.WalkOpt{}
-
- if idmap != nil {
- walkOpt.Map = func(p string, st *fstypes.Stat) bool {
- uid, gid, err := idmap.ToContainer(idtools.Identity{
- UID: int(st.Uid),
- GID: int(st.Gid),
- })
- if err != nil {
- return false
- }
- st.Uid = uint32(uid)
- st.Gid = uint32(gid)
- return true
- }
+ st := fstypes.Stat{
+ Mode: uint32(os.ModeDir | 0755),
+ Path: strings.Replace(k, "/", "_", -1),
+ }
+ if e.opts.Epoch != nil {
+ st.ModTime = e.opts.Epoch.UnixNano()
}
return &fsutil.Dir{
- FS: fsutil.NewFS(src, walkOpt),
- Stat: fstypes.Stat{
- Mode: uint32(os.ModeDir | 0755),
- Path: strings.Replace(k, "/", "_", -1),
- },
+ FS: outputFS,
+ Stat: st,
}, nil
}
+ isMap := len(inp.Refs) > 0
+ if _, ok := inp.Metadata[exptypes.ExporterPlatformsKey]; isMap && !ok {
+ return nil, nil, errors.Errorf("unable to export multiple refs, missing platforms mapping")
+ }
+ p, err := exptypes.ParsePlatforms(inp.Metadata)
+ if err != nil {
+ return nil, nil, err
+ }
+ if !isMap && len(p.Platforms) > 1 {
+ return nil, nil, errors.Errorf("unable to export multiple platforms without map")
+ }
+
var fs fsutil.FS
- if len(inp.Refs) > 0 {
- dirs := make([]fsutil.Dir, 0, len(inp.Refs))
- for k, ref := range inp.Refs {
- d, err := getDir(ctx, k, ref)
+ if len(p.Platforms) > 0 {
+ dirs := make([]fsutil.Dir, 0, len(p.Platforms))
+ for _, p := range p.Platforms {
+ r, ok := inp.FindRef(p.ID)
+ if !ok {
+ return nil, nil, errors.Errorf("failed to find ref for ID %s", p.ID)
+ }
+ d, err := getDir(ctx, p.ID, r, inp.Attestations[p.ID])
if err != nil {
- return nil, err
+ return nil, nil, err
}
dirs = append(dirs, *d)
}
- var err error
- fs, err = fsutil.SubDirFS(dirs)
- if err != nil {
- return nil, err
+ if isMap {
+ var err error
+ fs, err = fsutil.SubDirFS(dirs)
+ if err != nil {
+ return nil, nil, err
+ }
+ } else {
+ fs = dirs[0].FS
}
} else {
- d, err := getDir(ctx, "", inp.Ref)
+ d, err := getDir(ctx, "", inp.Ref, nil)
if err != nil {
- return nil, err
+ return nil, nil, err
}
fs = d.FS
}
@@ -161,34 +173,17 @@ func (e *localExporterInstance) Export(ctx context.Context, inp exporter.Source,
caller, err := e.opt.SessionManager.Get(timeoutCtx, sessionID, false)
if err != nil {
- return nil, err
+ return nil, nil, err
}
w, err := filesync.CopyFileWriter(ctx, nil, caller)
if err != nil {
- return nil, err
+ return nil, nil, err
}
- report := oneOffProgress(ctx, "sending tarball")
+ report := progress.OneOff(ctx, "sending tarball")
if err := fsutil.WriteTar(ctx, fs, w); err != nil {
w.Close()
- return nil, report(err)
- }
- return nil, report(w.Close())
-}
-
-func oneOffProgress(ctx context.Context, id string) func(err error) error {
- pw, _, _ := progress.NewFromContext(ctx)
- now := time.Now()
- st := progress.Status{
- Started: &now,
- }
- pw.Write(id, st)
- return func(err error) error {
- // TODO: set error on status
- now := time.Now()
- st.Completed = &now
- pw.Write(id, st)
- pw.Close()
- return err
+ return nil, nil, report(err)
}
+ return nil, nil, report(w.Close())
}
diff --git a/exporter/util/epoch/parse.go b/exporter/util/epoch/parse.go
new file mode 100644
index 000000000000..63f806e1b76a
--- /dev/null
+++ b/exporter/util/epoch/parse.go
@@ -0,0 +1,65 @@
+package epoch
+
+import (
+ "strconv"
+ "time"
+
+ "github.com/moby/buildkit/exporter"
+ "github.com/moby/buildkit/exporter/containerimage/exptypes"
+ "github.com/pkg/errors"
+)
+
+const (
+ frontendSourceDateEpochArg = "build-arg:SOURCE_DATE_EPOCH"
+
+ KeySourceDateEpoch = "source-date-epoch"
+)
+
+func ParseBuildArgs(opt map[string]string) (string, bool) {
+ v, ok := opt[frontendSourceDateEpochArg]
+ return v, ok
+}
+
+func ParseExporterAttrs(opt map[string]string) (*time.Time, map[string]string, error) {
+ rest := make(map[string]string, len(opt))
+
+ var tm *time.Time
+
+ for k, v := range opt {
+ switch k {
+ case KeySourceDateEpoch:
+ var err error
+ tm, err = parseTime(k, v)
+ if err != nil {
+ return nil, nil, err
+ }
+ default:
+ rest[k] = v
+ }
+ }
+
+ return tm, rest, nil
+}
+
+func ParseSource(inp *exporter.Source) (*time.Time, bool, error) {
+ if v, ok := inp.Metadata[exptypes.ExporterEpochKey]; ok {
+ epoch, err := parseTime("", string(v))
+ if err != nil {
+ return nil, false, errors.Wrapf(err, "invalid SOURCE_DATE_EPOCH from frontend: %q", v)
+ }
+ return epoch, true, nil
+ }
+ return nil, false, nil
+}
+
+func parseTime(key, value string) (*time.Time, error) {
+ if value == "" {
+ return nil, nil
+ }
+ sde, err := strconv.ParseInt(value, 10, 64)
+ if err != nil {
+ return nil, errors.Wrapf(err, "invalid %s: %s", key, err)
+ }
+ tm := time.Unix(sde, 0).UTC()
+ return &tm, nil
+}
diff --git a/frontend/attestations/parse.go b/frontend/attestations/parse.go
new file mode 100644
index 000000000000..00de649fdefe
--- /dev/null
+++ b/frontend/attestations/parse.go
@@ -0,0 +1,81 @@
+package attestations
+
+import (
+ "encoding/csv"
+ "strings"
+
+ "github.com/pkg/errors"
+)
+
+const (
+ KeyTypeSbom = "sbom"
+ KeyTypeProvenance = "provenance"
+)
+
+const (
+ defaultSBOMGenerator = "docker/buildkit-syft-scanner:stable-1"
+)
+
+func Filter(v map[string]string) map[string]string {
+ attests := make(map[string]string)
+ for k, v := range v {
+ if strings.HasPrefix(k, "attest:") {
+ attests[k] = v
+ continue
+ }
+ if strings.HasPrefix(k, "build-arg:BUILDKIT_ATTEST_") {
+ attests[k] = v
+ continue
+ }
+ }
+ return attests
+}
+
+func Validate(values map[string]map[string]string) (map[string]map[string]string, error) {
+ for k := range values {
+ if k != KeyTypeSbom && k != KeyTypeProvenance {
+ return nil, errors.Errorf("unknown attestation type %q", k)
+ }
+ }
+ return values, nil
+}
+
+func Parse(values map[string]string) (map[string]map[string]string, error) {
+ attests := make(map[string]string)
+ for k, v := range values {
+ if strings.HasPrefix(k, "attest:") {
+ attests[strings.ToLower(strings.TrimPrefix(k, "attest:"))] = v
+ continue
+ }
+ if strings.HasPrefix(k, "build-arg:BUILDKIT_ATTEST_") {
+ attests[strings.ToLower(strings.TrimPrefix(k, "build-arg:BUILDKIT_ATTEST_"))] = v
+ continue
+ }
+ }
+
+ out := make(map[string]map[string]string)
+ for k, v := range attests {
+ attrs := make(map[string]string)
+ out[k] = attrs
+ if k == KeyTypeSbom {
+ attrs["generator"] = defaultSBOMGenerator
+ }
+ if v == "" {
+ continue
+ }
+ csvReader := csv.NewReader(strings.NewReader(v))
+ fields, err := csvReader.Read()
+ if err != nil {
+ return nil, errors.Wrapf(err, "failed to parse %s", k)
+ }
+ for _, field := range fields {
+ parts := strings.SplitN(field, "=", 2)
+ if len(parts) != 2 {
+ parts = append(parts, "")
+ }
+ attrs[parts[0]] = parts[1]
+ }
+ }
+
+ return Validate(out)
+}
diff --git a/frontend/attestations/sbom/sbom.go b/frontend/attestations/sbom/sbom.go
new file mode 100644
index 000000000000..113797b2139c
--- /dev/null
+++ b/frontend/attestations/sbom/sbom.go
@@ -0,0 +1,112 @@
+package sbom
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "path"
+ "strings"
+
+ intoto "github.com/in-toto/in-toto-golang/in_toto"
+ "github.com/moby/buildkit/client/llb"
+ gatewaypb "github.com/moby/buildkit/frontend/gateway/pb"
+ "github.com/moby/buildkit/solver/result"
+ ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
+ "github.com/pkg/errors"
+)
+
+const (
+ CoreSBOMName = "sbom"
+ ExtraSBOMPrefix = CoreSBOMName + "-"
+
+ srcDir = "/run/src/"
+ outDir = "/run/out/"
+)
+
+// Scanner is a function type for scanning the contents of a state and
+// returning a new attestation and state representing the scan results.
+//
+// A scanner is designed a scan a single state, however, additional states can
+// also be attached, for attaching additional information, such as scans of
+// build-contexts or multi-stage builds. Handling these separately allows the
+// scanner to optionally ignore these or to mark them as such in the
+// attestation.
+type Scanner func(ctx context.Context, name string, ref llb.State, extras map[string]llb.State, opts ...llb.ConstraintsOpt) (result.Attestation[llb.State], error)
+
+func CreateSBOMScanner(ctx context.Context, resolver llb.ImageMetaResolver, scanner string) (Scanner, error) {
+ if scanner == "" {
+ return nil, nil
+ }
+
+ _, dt, err := resolver.ResolveImageConfig(ctx, scanner, llb.ResolveImageConfigOpt{})
+ if err != nil {
+ return nil, err
+ }
+
+ var cfg ocispecs.Image
+ if err := json.Unmarshal(dt, &cfg); err != nil {
+ return nil, err
+ }
+
+ var args []string
+ args = append(args, cfg.Config.Entrypoint...)
+ args = append(args, cfg.Config.Cmd...)
+ if len(args) == 0 {
+ return nil, errors.Errorf("scanner %s does not have cmd", scanner)
+ }
+
+ return func(ctx context.Context, name string, ref llb.State, extras map[string]llb.State, opts ...llb.ConstraintsOpt) (result.Attestation[llb.State], error) {
+ var env []string
+ env = append(env, cfg.Config.Env...)
+ env = append(env, "BUILDKIT_SCAN_DESTINATION="+outDir)
+ env = append(env, "BUILDKIT_SCAN_SOURCE="+path.Join(srcDir, "core", CoreSBOMName))
+ if len(extras) > 0 {
+ env = append(env, "BUILDKIT_SCAN_SOURCE_EXTRAS="+path.Join(srcDir, "extras/"))
+ }
+
+ runOpts := []llb.RunOption{
+ llb.WithCustomName(fmt.Sprintf("[%s] generating sbom using %s", name, scanner)),
+ }
+ for _, opt := range opts {
+ runOpts = append(runOpts, opt)
+ }
+ runOpts = append(runOpts, llb.Dir(cfg.Config.WorkingDir))
+ runOpts = append(runOpts, llb.Args(args))
+ for _, e := range env {
+ k, v, _ := strings.Cut(e, "=")
+ runOpts = append(runOpts, llb.AddEnv(k, v))
+ }
+
+ runscan := llb.Image(scanner).Run(runOpts...)
+ runscan.AddMount("/tmp", llb.Scratch(), llb.Tmpfs())
+
+ runscan.AddMount(path.Join(srcDir, "core", CoreSBOMName), ref, llb.Readonly)
+ for k, extra := range extras {
+ runscan.AddMount(path.Join(srcDir, "extras", ExtraSBOMPrefix+k), extra, llb.Readonly)
+ }
+
+ stsbom := runscan.AddMount(outDir, llb.Scratch())
+ return result.Attestation[llb.State]{
+ Kind: gatewaypb.AttestationKindBundle,
+ Ref: stsbom,
+ Metadata: map[string][]byte{
+ result.AttestationReasonKey: []byte(result.AttestationReasonSBOM),
+ result.AttestationSBOMCore: []byte(CoreSBOMName),
+ },
+ InToto: result.InTotoAttestation{
+ PredicateType: intoto.PredicateSPDX,
+ },
+ }, nil
+ }, nil
+}
+
+func HasSBOM[T any](res *result.Result[T]) bool {
+ for _, as := range res.Attestations {
+ for _, a := range as {
+ if a.InToto.PredicateType == intoto.PredicateSPDX {
+ return true
+ }
+ }
+ }
+ return false
+}
diff --git a/frontend/dockerfile/builder/build.go b/frontend/dockerfile/builder/build.go
index e0a1806901bc..aafd9c9a73f3 100644
--- a/frontend/dockerfile/builder/build.go
+++ b/frontend/dockerfile/builder/build.go
@@ -12,6 +12,7 @@ import (
"regexp"
"strconv"
"strings"
+ "time"
"github.com/containerd/containerd/platforms"
"github.com/docker/distribution/reference"
@@ -19,15 +20,20 @@ import (
controlapi "github.com/moby/buildkit/api/services/control"
"github.com/moby/buildkit/client/llb"
"github.com/moby/buildkit/exporter/containerimage/exptypes"
+ "github.com/moby/buildkit/frontend"
+ "github.com/moby/buildkit/frontend/attestations"
+ "github.com/moby/buildkit/frontend/attestations/sbom"
"github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb"
"github.com/moby/buildkit/frontend/dockerfile/dockerignore"
"github.com/moby/buildkit/frontend/dockerfile/parser"
"github.com/moby/buildkit/frontend/gateway/client"
gwpb "github.com/moby/buildkit/frontend/gateway/pb"
+ "github.com/moby/buildkit/frontend/subrequests/outline"
+ "github.com/moby/buildkit/frontend/subrequests/targets"
"github.com/moby/buildkit/solver/errdefs"
"github.com/moby/buildkit/solver/pb"
- "github.com/moby/buildkit/util/apicaps"
- binfotypes "github.com/moby/buildkit/util/buildinfo/types"
+ "github.com/moby/buildkit/solver/result"
+ "github.com/moby/buildkit/util/gitutil"
ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/pkg/errors"
"golang.org/x/sync/errgroup"
@@ -39,45 +45,59 @@ const (
defaultDockerfileName = "Dockerfile"
dockerignoreFilename = ".dockerignore"
- buildArgPrefix = "build-arg:"
- labelPrefix = "label:"
-
- keyTarget = "target"
- keyFilename = "filename"
- keyCacheFrom = "cache-from" // for registry only. deprecated in favor of keyCacheImports
- keyCacheImports = "cache-imports" // JSON representation of []CacheOptionsEntry
- keyCgroupParent = "cgroup-parent"
- keyContextSubDir = "contextsubdir"
- keyForceNetwork = "force-network-mode"
- keyGlobalAddHosts = "add-hosts"
- keyHostname = "hostname"
- keyImageResolveMode = "image-resolve-mode"
- keyMultiPlatform = "multi-platform"
- keyNameContext = "contextkey"
- keyNameDockerfile = "dockerfilekey"
- keyNoCache = "no-cache"
- keyOverrideCopyImage = "override-copy-image" // remove after CopyOp implemented
- keyShmSize = "shm-size"
- keyTargetPlatform = "platform"
- keyUlimit = "ulimit"
+ buildArgPrefix = "build-arg:"
+ labelPrefix = "label:"
+ contextPrefix = "context:"
+ inputMetadataPrefix = "input-metadata:"
+
+ keyTarget = "target"
+ keyFilename = "filename"
+ keyCacheFrom = "cache-from" // for registry only. deprecated in favor of keyCacheImports
+ keyCacheImports = "cache-imports" // JSON representation of []CacheOptionsEntry
+ keyCgroupParent = "cgroup-parent"
+ keyContextSubDir = "contextsubdir"
+ keyForceNetwork = "force-network-mode"
+ keyGlobalAddHosts = "add-hosts"
+ keyHostname = "hostname"
+ keyImageResolveMode = "image-resolve-mode"
+ keyMultiPlatform = "multi-platform"
+ keyNameContext = "contextkey"
+ keyNameDockerfile = "dockerfilekey"
+ keyNoCache = "no-cache"
+ keyShmSize = "shm-size"
+ keyTargetPlatform = "platform"
+ keyUlimit = "ulimit"
+ keyRequestID = "requestid"
// Don't forget to update frontend documentation if you add
- // a new build-arg: frontend/dockerfile/docs/syntax.md
+ // a new build-arg: frontend/dockerfile/docs/reference.md
keyCacheNSArg = "build-arg:BUILDKIT_CACHE_MOUNT_NS"
keyContextKeepGitDirArg = "build-arg:BUILDKIT_CONTEXT_KEEP_GIT_DIR"
keyHostnameArg = "build-arg:BUILDKIT_SANDBOX_HOSTNAME"
keyMultiPlatformArg = "build-arg:BUILDKIT_MULTI_PLATFORM"
keySyntaxArg = "build-arg:BUILDKIT_SYNTAX"
+ keySourceDateEpoch = "build-arg:SOURCE_DATE_EPOCH"
)
var httpPrefix = regexp.MustCompile(`^https?://`)
-var gitURLPathWithFragmentSuffix = regexp.MustCompile(`\.git(?:#.+)?$`)
-func Build(ctx context.Context, c client.Client) (*client.Result, error) {
+func Build(ctx context.Context, c client.Client) (_ *client.Result, err error) {
opts := c.BuildOpts().Opts
caps := c.BuildOpts().LLBCaps
gwcaps := c.BuildOpts().Caps
+ if err := caps.Supports(pb.CapFileBase); err != nil {
+ return nil, errors.Wrap(err, "needs BuildKit 0.5 or later")
+ }
+ if opts["override-copy-image"] != "" {
+ return nil, errors.New("support for \"override-copy-image\" was removed in BuildKit 0.11")
+ }
+ if v, ok := opts["build-arg:BUILDKIT_DISABLE_FILEOP"]; ok {
+ if b, err := strconv.ParseBool(v); err == nil && b {
+ return nil, errors.New("support for \"build-arg:BUILDKIT_DISABLE_FILEOP\" was removed in BuildKit 0.11")
+ }
+ }
+
allowForward, capsError := validateCaps(opts["frontend.caps"])
if !allowForward && capsError != nil {
return nil, capsError
@@ -168,11 +188,13 @@ func Build(ctx context.Context, c client.Client) (*client.Result, error) {
llb.Differ(llb.DiffNone, false),
)
- fileop := useFileOp(opts, &caps)
-
var buildContext *llb.State
isNotLocalContext := false
- if st, ok := detectGitContext(opts[localNameContext], opts[keyContextKeepGitDirArg]); ok {
+ keepGit := false
+ if v, err := strconv.ParseBool(opts[keyContextKeepGitDirArg]); err == nil {
+ keepGit = v
+ }
+ if st, ok := detectGitContext(opts[localNameContext], keepGit); ok {
if !forceLocalDockerfile {
src = *st
}
@@ -205,28 +227,13 @@ func Build(ctx context.Context, c client.Client) (*client.Result, error) {
return nil, errors.Wrapf(err, "failed to read downloaded context")
}
if isArchive(dt) {
- if fileop {
- bc := llb.Scratch().File(llb.Copy(httpContext, "/context", "/", &llb.CopyInfo{
- AttemptUnpack: true,
- }))
- if !forceLocalDockerfile {
- src = bc
- }
- buildContext = &bc
- } else {
- copyImage := opts[keyOverrideCopyImage]
- if copyImage == "" {
- copyImage = dockerfile2llb.DefaultCopyImage
- }
- unpack := llb.Image(copyImage, dockerfile2llb.WithInternalName("helper image for file operations")).
- Run(llb.Shlex("copy --unpack /src/context /out/"), llb.ReadonlyRootFS(), dockerfile2llb.WithInternalName("extracting build context"))
- unpack.AddMount("/src", httpContext, llb.Readonly)
- bc := unpack.AddMount("/out", llb.Scratch())
- if !forceLocalDockerfile {
- src = bc
- }
- buildContext = &bc
+ bc := llb.Scratch().File(llb.Copy(httpContext, "/context", "/", &llb.CopyInfo{
+ AttemptUnpack: true,
+ }))
+ if !forceLocalDockerfile {
+ src = bc
}
+ buildContext = &bc
} else {
filename = "context"
if !forceLocalDockerfile {
@@ -257,7 +264,7 @@ func Build(ctx context.Context, c client.Client) (*client.Result, error) {
if buildContext != nil {
if sub, ok := opts[keyContextSubDir]; ok {
- buildContext = scopeToSubDir(buildContext, fileop, sub)
+ buildContext = scopeToSubDir(buildContext, sub)
}
}
@@ -380,7 +387,7 @@ func Build(ctx context.Context, c client.Client) (*client.Result, error) {
return nil, errors.Wrapf(err, "failed with %s = %s", keySyntaxArg, cmdline)
}
return res, err
- } else if ref, cmdline, loc, ok := dockerfile2llb.DetectSyntax(bytes.NewBuffer(dtDockerfile)); ok {
+ } else if ref, cmdline, loc, ok := parser.DetectSyntax(dtDockerfile); ok {
res, err := forwardGateway(ctx, c, ref, cmdline)
if err != nil && len(errdefs.Sources(err)) == 0 {
return nil, wrapSource(err, sourceMap, loc)
@@ -408,7 +415,7 @@ func Build(ctx context.Context, c client.Client) (*client.Result, error) {
return nil, errors.Errorf("invalid boolean value %s", v)
}
if !b && exportMap {
- return nil, errors.Errorf("returning multiple target plaforms is not allowed")
+ return nil, errors.Errorf("returning multiple target platforms is not allowed")
}
exportMap = b
}
@@ -422,55 +429,107 @@ func Build(ctx context.Context, c client.Client) (*client.Result, error) {
opts[keyHostname] = v
}
- eg, ctx = errgroup.WithContext(ctx)
+ epoch, err := parseSourceDateEpoch(opts[keySourceDateEpoch])
+ if err != nil {
+ return nil, err
+ }
+
+ target := opts[keyTarget]
+ convertOpt := dockerfile2llb.ConvertOpt{
+ Target: target,
+ MetaResolver: c,
+ BuildArgs: filter(opts, buildArgPrefix),
+ Labels: filter(opts, labelPrefix),
+ CacheIDNamespace: opts[keyCacheNSArg],
+ SessionID: c.BuildOpts().SessionID,
+ BuildContext: buildContext,
+ Excludes: excludes,
+ IgnoreCache: ignoreCache,
+ TargetPlatform: targetPlatforms[0],
+ BuildPlatforms: buildPlatforms,
+ ImageResolveMode: resolveMode,
+ PrefixPlatform: exportMap,
+ ExtraHosts: extraHosts,
+ ShmSize: shmSize,
+ Ulimit: ulimit,
+ CgroupParent: opts[keyCgroupParent],
+ ForceNetMode: defaultNetMode,
+ LLBCaps: &caps,
+ SourceMap: sourceMap,
+ Hostname: opts[keyHostname],
+ SourceDateEpoch: epoch,
+ Warn: func(msg, url string, detail [][]byte, location *parser.Range) {
+ c.Warn(ctx, defVtx, msg, warnOpts(sourceMap, location, detail, url))
+ },
+ ContextByName: contextByNameFunc(c, c.BuildOpts().SessionID),
+ }
+
+ defer func() {
+ var el *parser.ErrorLocation
+ if errors.As(err, &el) {
+ err = wrapSource(err, sourceMap, el.Location)
+ }
+ }()
+
+ if req, ok := opts[keyRequestID]; ok {
+ switch req {
+ case outline.SubrequestsOutlineDefinition.Name:
+ o, err := dockerfile2llb.Dockefile2Outline(ctx, dtDockerfile, convertOpt)
+ if err != nil {
+ return nil, err
+ }
+ return o.ToResult()
+ case targets.SubrequestsTargetsDefinition.Name:
+ targets, err := dockerfile2llb.ListTargets(ctx, dtDockerfile)
+ if err != nil {
+ return nil, err
+ }
+ return targets.ToResult()
+ default:
+ return nil, errdefs.NewUnsupportedSubrequestError(req)
+ }
+ }
+
+ var scanner sbom.Scanner
+ attests, err := attestations.Parse(opts)
+ if err != nil {
+ return nil, err
+ }
+ if attrs, ok := attests[attestations.KeyTypeSbom]; ok {
+ src, ok := attrs["generator"]
+ if !ok {
+ return nil, errors.Errorf("sbom scanner cannot be empty")
+ }
+ ref, err := reference.ParseNormalizedNamed(src)
+ if err != nil {
+ return nil, errors.Wrapf(err, "failed to parse sbom scanner %s", src)
+ }
+ ref = reference.TagNameOnly(ref)
+
+ scanner, err = sbom.CreateSBOMScanner(ctx, c, ref.String())
+ if err != nil {
+ return nil, err
+ }
+ }
+ scanTargets := make([]*dockerfile2llb.SBOMTargets, len(targetPlatforms))
+
+ eg, ctx2 = errgroup.WithContext(ctx)
for i, tp := range targetPlatforms {
func(i int, tp *ocispecs.Platform) {
eg.Go(func() (err error) {
- defer func() {
- var el *parser.ErrorLocation
- if errors.As(err, &el) {
- err = wrapSource(err, sourceMap, el.Location)
- }
- }()
-
- st, img, bi, err := dockerfile2llb.Dockerfile2LLB(ctx, dtDockerfile, dockerfile2llb.ConvertOpt{
- Target: opts[keyTarget],
- MetaResolver: c,
- BuildArgs: filter(opts, buildArgPrefix),
- Labels: filter(opts, labelPrefix),
- CacheIDNamespace: opts[keyCacheNSArg],
- SessionID: c.BuildOpts().SessionID,
- BuildContext: buildContext,
- Excludes: excludes,
- IgnoreCache: ignoreCache,
- TargetPlatform: tp,
- BuildPlatforms: buildPlatforms,
- ImageResolveMode: resolveMode,
- PrefixPlatform: exportMap,
- ExtraHosts: extraHosts,
- ShmSize: shmSize,
- Ulimit: ulimit,
- CgroupParent: opts[keyCgroupParent],
- ForceNetMode: defaultNetMode,
- OverrideCopyImage: opts[keyOverrideCopyImage],
- LLBCaps: &caps,
- SourceMap: sourceMap,
- Hostname: opts[keyHostname],
- Warn: func(msg, url string, detail [][]byte, location *parser.Range) {
- if i != 0 {
- return
- }
- c.Warn(ctx, defVtx, msg, warnOpts(sourceMap, location, detail, url))
- },
- ContextByName: contextByNameFunc(c, tp),
- })
-
+ opt := convertOpt
+ opt.TargetPlatform = tp
+ if i != 0 {
+ opt.Warn = nil
+ }
+ opt.ContextByName = contextByNameFunc(c, c.BuildOpts().SessionID)
+ st, img, scanTarget, err := dockerfile2llb.Dockerfile2LLB(ctx2, dtDockerfile, opt)
if err != nil {
return err
}
- def, err := st.Marshal(ctx)
+ def, err := st.Marshal(ctx2)
if err != nil {
return errors.Wrapf(err, "failed to marshal LLB definition")
}
@@ -506,7 +565,7 @@ func Build(ctx context.Context, c client.Client) (*client.Result, error) {
}
}
- r, err := c.Solve(ctx, client.SolveRequest{
+ r, err := c.Solve(ctx2, client.SolveRequest{
Definition: def.ToPB(),
CacheImports: cacheImports,
})
@@ -519,30 +578,30 @@ func Build(ctx context.Context, c client.Client) (*client.Result, error) {
return err
}
- buildinfo, err := json.Marshal(bi)
- if err != nil {
- return errors.Wrapf(err, "failed to marshal build info")
+ p := platforms.DefaultSpec()
+ if tp != nil {
+ p = *tp
}
+ p = platforms.Normalize(p)
+ k := platforms.Format(p)
if !exportMap {
res.AddMeta(exptypes.ExporterImageConfigKey, config)
- res.AddMeta(exptypes.ExporterBuildInfo, buildinfo)
res.SetRef(ref)
- } else {
- p := platforms.DefaultSpec()
- if tp != nil {
- p = *tp
- }
- k := platforms.Format(p)
+ expPlatforms.Platforms[i] = exptypes.Platform{
+ ID: k,
+ Platform: p,
+ }
+ } else {
res.AddMeta(fmt.Sprintf("%s/%s", exptypes.ExporterImageConfigKey, k), config)
- res.AddMeta(fmt.Sprintf("%s/%s", exptypes.ExporterBuildInfo, k), buildinfo)
res.AddRef(k, ref)
expPlatforms.Platforms[i] = exptypes.Platform{
ID: k,
Platform: p,
}
}
+ scanTargets[i] = scanTarget
return nil
})
}(i, tp)
@@ -552,14 +611,45 @@ func Build(ctx context.Context, c client.Client) (*client.Result, error) {
return nil, err
}
- if exportMap {
- dt, err := json.Marshal(expPlatforms)
- if err != nil {
- return nil, err
+ if scanner != nil {
+ for i, p := range expPlatforms.Platforms {
+ target := scanTargets[i]
+
+ var opts []llb.ConstraintsOpt
+ if target.IgnoreCache {
+ opts = append(opts, llb.IgnoreCache)
+ }
+ att, err := scanner(ctx, p.ID, target.Core, target.Extras, opts...)
+ if err != nil {
+ return nil, err
+ }
+
+ attSolve, err := result.ConvertAttestation(&att, func(st llb.State) (client.Reference, error) {
+ def, err := st.Marshal(ctx)
+ if err != nil {
+ return nil, err
+ }
+ r, err := c.Solve(ctx, frontend.SolveRequest{
+ Definition: def.ToPB(),
+ })
+ if err != nil {
+ return nil, err
+ }
+ return r.Ref, nil
+ })
+ if err != nil {
+ return nil, err
+ }
+ res.AddAttestation(p.ID, *attSolve)
}
- res.AddMeta(exptypes.ExporterPlatformsKey, dt)
}
+ dt, err := json.Marshal(expPlatforms)
+ if err != nil {
+ return nil, err
+ }
+ res.AddMeta(exptypes.ExporterPlatformsKey, dt)
+
return res, nil
}
@@ -606,40 +696,21 @@ func filter(opt map[string]string, key string) map[string]string {
return m
}
-func detectGitContext(ref, gitContext string) (*llb.State, bool) {
- found := false
- if httpPrefix.MatchString(ref) && gitURLPathWithFragmentSuffix.MatchString(ref) {
- found = true
- }
-
- keepGit := false
- if gitContext != "" {
- if v, err := strconv.ParseBool(gitContext); err == nil {
- keepGit = v
- }
- }
-
- for _, prefix := range []string{"git://", "github.com/", "git@"} {
- if strings.HasPrefix(ref, prefix) {
- found = true
- break
- }
- }
- if !found {
+func detectGitContext(ref string, keepGit bool) (*llb.State, bool) {
+ g, err := gitutil.ParseGitRef(ref)
+ if err != nil {
return nil, false
}
-
- parts := strings.SplitN(ref, "#", 2)
- branch := ""
- if len(parts) > 1 {
- branch = parts[1]
+ commit := g.Commit
+ if g.SubDir != "" {
+ commit += ":" + g.SubDir
}
gitOpts := []llb.GitOption{dockerfile2llb.WithInternalName("load git source " + ref)}
if keepGit {
gitOpts = append(gitOpts, llb.KeepGitDir())
}
- st := llb.Git(parts[0], branch, gitOpts...)
+ st := llb.Git(g.Remote, commit, gitOpts...)
return &st, true
}
@@ -765,27 +836,10 @@ func parseNetMode(v string) (pb.NetMode, error) {
}
}
-func useFileOp(args map[string]string, caps *apicaps.CapSet) bool {
- enabled := true
- if v, ok := args["build-arg:BUILDKIT_DISABLE_FILEOP"]; ok {
- if b, err := strconv.ParseBool(v); err == nil {
- enabled = !b
- }
- }
- return enabled && caps != nil && caps.Supports(pb.CapFileBase) == nil
-}
-
-func scopeToSubDir(c *llb.State, fileop bool, dir string) *llb.State {
- if fileop {
- bc := llb.Scratch().File(llb.Copy(*c, dir, "/", &llb.CopyInfo{
- CopyDirContentsOnly: true,
- }))
- return &bc
- }
- unpack := llb.Image(dockerfile2llb.DefaultCopyImage, dockerfile2llb.WithInternalName("helper image for file operations")).
- Run(llb.Shlexf("copy %s/. /out/", path.Join("/src", dir)), llb.ReadonlyRootFS(), dockerfile2llb.WithInternalName("filtering build context"))
- unpack.AddMount("/src", *c, llb.Readonly)
- bc := unpack.AddMount("/out", llb.Scratch())
+func scopeToSubDir(c *llb.State, dir string) *llb.State {
+ bc := llb.Scratch().File(llb.Copy(*c, dir, "/", &llb.CopyInfo{
+ CopyDirContentsOnly: true,
+ }))
return &bc
}
@@ -812,11 +866,11 @@ func warnOpts(sm *llb.SourceMap, r *parser.Range, detail [][]byte, url string) c
return opts
}
-func contextByNameFunc(c client.Client, p *ocispecs.Platform) func(context.Context, string) (*llb.State, *dockerfile2llb.Image, *binfotypes.BuildInfo, error) {
- return func(ctx context.Context, name string) (*llb.State, *dockerfile2llb.Image, *binfotypes.BuildInfo, error) {
+func contextByNameFunc(c client.Client, sessionID string) func(context.Context, string, string, *ocispecs.Platform) (*llb.State, *dockerfile2llb.Image, error) {
+ return func(ctx context.Context, name, resolveMode string, p *ocispecs.Platform) (*llb.State, *dockerfile2llb.Image, error) {
named, err := reference.ParseNormalizedNamed(name)
if err != nil {
- return nil, nil, nil, errors.Wrapf(err, "invalid context name %s", name)
+ return nil, nil, errors.Wrapf(err, "invalid context name %s", name)
}
name = strings.TrimSuffix(reference.FamiliarString(named), ":latest")
@@ -825,77 +879,174 @@ func contextByNameFunc(c client.Client, p *ocispecs.Platform) func(context.Conte
p = &pp
}
if p != nil {
- name := name + "::" + platforms.Format(platforms.Normalize(*p))
- st, img, bi, err := contextByName(ctx, c, name, p)
+ pname := name + "::" + platforms.Format(platforms.Normalize(*p))
+ st, img, err := contextByName(ctx, c, sessionID, name, pname, p, resolveMode)
if err != nil {
- return nil, nil, nil, err
+ return nil, nil, err
}
if st != nil {
- return st, img, bi, nil
+ return st, img, nil
}
}
- return contextByName(ctx, c, name, p)
+ return contextByName(ctx, c, sessionID, name, name, p, resolveMode)
}
}
-func contextByName(ctx context.Context, c client.Client, name string, platform *ocispecs.Platform) (*llb.State, *dockerfile2llb.Image, *binfotypes.BuildInfo, error) {
+func contextByName(ctx context.Context, c client.Client, sessionID, name string, pname string, platform *ocispecs.Platform, resolveMode string) (*llb.State, *dockerfile2llb.Image, error) {
opts := c.BuildOpts().Opts
- v, ok := opts["context:"+name]
+ v, ok := opts[contextPrefix+pname]
if !ok {
- return nil, nil, nil, nil
+ return nil, nil, nil
}
vv := strings.SplitN(v, ":", 2)
if len(vv) != 2 {
- return nil, nil, nil, errors.Errorf("invalid context specifier %s for %s", v, name)
+ return nil, nil, errors.Errorf("invalid context specifier %s for %s", v, pname)
+ }
+ // allow git@ without protocol for SSH URLs for backwards compatibility
+ if strings.HasPrefix(vv[0], "git@") {
+ vv[0] = "git"
}
switch vv[0] {
case "docker-image":
ref := strings.TrimPrefix(vv[1], "//")
+ if ref == "scratch" {
+ st := llb.Scratch()
+ return &st, nil, nil
+ }
+
imgOpt := []llb.ImageOption{
- llb.WithCustomName("[context " + name + "] " + ref),
- llb.WithMetaResolver(c),
+ llb.WithCustomName("[context " + pname + "] " + ref),
}
if platform != nil {
imgOpt = append(imgOpt, llb.Platform(*platform))
}
+
+ named, err := reference.ParseNormalizedNamed(ref)
+ if err != nil {
+ return nil, nil, err
+ }
+
+ named = reference.TagNameOnly(named)
+
+ _, data, err := c.ResolveImageConfig(ctx, named.String(), llb.ResolveImageConfigOpt{
+ Platform: platform,
+ ResolveMode: resolveMode,
+ LogName: fmt.Sprintf("[context %s] load metadata for %s", pname, ref),
+ ResolverType: llb.ResolverTypeRegistry,
+ })
+ if err != nil {
+ return nil, nil, err
+ }
+
+ var img dockerfile2llb.Image
+ if err := json.Unmarshal(data, &img); err != nil {
+ return nil, nil, err
+ }
+ img.Created = nil
+
st := llb.Image(ref, imgOpt...)
- return &st, nil, nil, nil
+ st, err = st.WithImageConfig(data)
+ if err != nil {
+ return nil, nil, err
+ }
+ return &st, &img, nil
case "git":
- st, ok := detectGitContext(v, "1")
+ st, ok := detectGitContext(v, true)
if !ok {
- return nil, nil, nil, errors.Errorf("invalid git context %s", v)
+ return nil, nil, errors.Errorf("invalid git context %s", v)
}
- return st, nil, nil, nil
+ return st, nil, nil
case "http", "https":
- st, ok := detectGitContext(v, "1")
+ st, ok := detectGitContext(v, true)
if !ok {
- httpst := llb.HTTP(v, llb.WithCustomName("[context "+name+"] "+v))
+ httpst := llb.HTTP(v, llb.WithCustomName("[context "+pname+"] "+v))
st = &httpst
}
- return st, nil, nil, nil
+ return st, nil, nil
+ case "oci-layout":
+ refSpec := strings.TrimPrefix(vv[1], "//")
+ ref, err := reference.Parse(refSpec)
+ if err != nil {
+ return nil, nil, errors.Wrapf(err, "could not parse oci-layout reference %q", refSpec)
+ }
+ named, ok := ref.(reference.Named)
+ if !ok {
+ return nil, nil, errors.Errorf("oci-layout reference %q has no name", ref.String())
+ }
+ dgstd, ok := named.(reference.Digested)
+ if !ok {
+ return nil, nil, errors.Errorf("oci-layout reference %q has no digest", named.String())
+ }
+
+ // for the dummy ref primarily used in log messages, we can use the
+ // original name, since the store key may not be significant
+ dummyRef, err := reference.ParseNormalizedNamed(name)
+ if err != nil {
+ return nil, nil, errors.Wrapf(err, "could not parse oci-layout reference %q", name)
+ }
+ dummyRef, err = reference.WithDigest(dummyRef, dgstd.Digest())
+ if err != nil {
+ return nil, nil, errors.Wrapf(err, "could not wrap %q with digest", name)
+ }
+
+ _, data, err := c.ResolveImageConfig(ctx, dummyRef.String(), llb.ResolveImageConfigOpt{
+ Platform: platform,
+ ResolveMode: resolveMode,
+ LogName: fmt.Sprintf("[context %s] load metadata for %s", pname, dummyRef.String()),
+ ResolverType: llb.ResolverTypeOCILayout,
+ Store: llb.ResolveImageConfigOptStore{
+ SessionID: sessionID,
+ StoreID: named.Name(),
+ },
+ })
+ if err != nil {
+ return nil, nil, err
+ }
+
+ var img dockerfile2llb.Image
+ if err := json.Unmarshal(data, &img); err != nil {
+ return nil, nil, errors.Wrap(err, "could not parse oci-layout image config")
+ }
+
+ ociOpt := []llb.OCILayoutOption{
+ llb.WithCustomName("[context " + pname + "] OCI load from client"),
+ llb.OCIStore(c.BuildOpts().SessionID, named.Name()),
+ }
+ if platform != nil {
+ ociOpt = append(ociOpt, llb.Platform(*platform))
+ }
+ st := llb.OCILayout(
+ dummyRef.String(),
+ ociOpt...,
+ )
+ st, err = st.WithImageConfig(data)
+ if err != nil {
+ return nil, nil, err
+ }
+ return &st, &img, nil
case "local":
st := llb.Local(vv[1],
llb.SessionID(c.BuildOpts().SessionID),
llb.FollowPaths([]string{dockerignoreFilename}),
- llb.SharedKeyHint("context:"+name+"-"+dockerignoreFilename),
- llb.WithCustomName("[context "+name+"] load "+dockerignoreFilename),
+ llb.SharedKeyHint("context:"+pname+"-"+dockerignoreFilename),
+ llb.WithCustomName("[context "+pname+"] load "+dockerignoreFilename),
llb.Differ(llb.DiffNone, false),
)
def, err := st.Marshal(ctx)
if err != nil {
- return nil, nil, nil, err
+ return nil, nil, err
}
res, err := c.Solve(ctx, client.SolveRequest{
Evaluate: true,
Definition: def.ToPB(),
})
if err != nil {
- return nil, nil, nil, err
+ return nil, nil, err
}
ref, err := res.SingleRef()
if err != nil {
- return nil, nil, nil, err
+ return nil, nil, err
}
dt, _ := ref.ReadFile(ctx, client.ReadRequest{
Filename: dockerignoreFilename,
@@ -904,58 +1055,46 @@ func contextByName(ctx context.Context, c client.Client, name string, platform *
if len(dt) != 0 {
excludes, err = dockerignore.ReadAll(bytes.NewBuffer(dt))
if err != nil {
- return nil, nil, nil, err
+ return nil, nil, err
}
}
st = llb.Local(vv[1],
- llb.WithCustomName("[context "+name+"] load from client"),
+ llb.WithCustomName("[context "+pname+"] load from client"),
llb.SessionID(c.BuildOpts().SessionID),
- llb.SharedKeyHint("context:"+name),
+ llb.SharedKeyHint("context:"+pname),
llb.ExcludePatterns(excludes),
)
- return &st, nil, nil, nil
+ return &st, nil, nil
case "input":
inputs, err := c.Inputs(ctx)
if err != nil {
- return nil, nil, nil, err
+ return nil, nil, err
}
st, ok := inputs[vv[1]]
if !ok {
- return nil, nil, nil, errors.Errorf("invalid input %s for %s", vv[1], name)
+ return nil, nil, errors.Errorf("invalid input %s for %s", vv[1], pname)
}
- md, ok := opts["input-metadata:"+vv[1]]
+ md, ok := opts[inputMetadataPrefix+vv[1]]
if ok {
m := make(map[string][]byte)
if err := json.Unmarshal([]byte(md), &m); err != nil {
- return nil, nil, nil, errors.Wrapf(err, "failed to parse input metadata %s", md)
- }
- var bi *binfotypes.BuildInfo
- if dtbi, ok := m[exptypes.ExporterBuildInfo]; ok {
- var depbi binfotypes.BuildInfo
- if err := json.Unmarshal(dtbi, &depbi); err != nil {
- return nil, nil, nil, errors.Wrapf(err, "failed to parse buildinfo for %s", name)
- }
- bi = &binfotypes.BuildInfo{
- Deps: map[string]binfotypes.BuildInfo{
- strings.SplitN(vv[1], "::", 2)[0]: depbi,
- },
- }
+ return nil, nil, errors.Wrapf(err, "failed to parse input metadata %s", md)
}
var img *dockerfile2llb.Image
if dtic, ok := m[exptypes.ExporterImageConfigKey]; ok {
st, err = st.WithImageConfig(dtic)
if err != nil {
- return nil, nil, nil, err
+ return nil, nil, err
}
if err := json.Unmarshal(dtic, &img); err != nil {
- return nil, nil, nil, errors.Wrapf(err, "failed to parse image config for %s", name)
+ return nil, nil, errors.Wrapf(err, "failed to parse image config for %s", pname)
}
}
- return &st, img, bi, nil
+ return &st, img, nil
}
- return &st, nil, nil, nil
+ return &st, nil, nil
default:
- return nil, nil, nil, errors.Errorf("unsupported context source %s for %s", vv[0], name)
+ return nil, nil, errors.Errorf("unsupported context source %s for %s", vv[0], pname)
}
}
@@ -985,3 +1124,15 @@ func wrapSource(err error, sm *llb.SourceMap, ranges []parser.Range) error {
}
return errdefs.WithSource(err, s)
}
+
+func parseSourceDateEpoch(v string) (*time.Time, error) {
+ if v == "" {
+ return nil, nil
+ }
+ sde, err := strconv.ParseInt(v, 10, 64)
+ if err != nil {
+ return nil, errors.Wrapf(err, "invalid SOURCE_DATE_EPOCH: %s", v)
+ }
+ tm := time.Unix(sde, 0).UTC()
+ return &tm, nil
+}
diff --git a/frontend/dockerfile/builder/subrequests.go b/frontend/dockerfile/builder/subrequests.go
index 6d30b7b8cc1c..844953023822 100644
--- a/frontend/dockerfile/builder/subrequests.go
+++ b/frontend/dockerfile/builder/subrequests.go
@@ -1,16 +1,19 @@
package builder
import (
+ "bytes"
"context"
"encoding/json"
"github.com/moby/buildkit/frontend/gateway/client"
"github.com/moby/buildkit/frontend/subrequests"
+ "github.com/moby/buildkit/frontend/subrequests/outline"
+ "github.com/moby/buildkit/frontend/subrequests/targets"
"github.com/moby/buildkit/solver/errdefs"
)
func checkSubRequest(ctx context.Context, opts map[string]string) (*client.Result, bool, error) {
- req, ok := opts["requestid"]
+ req, ok := opts[keyRequestID]
if !ok {
return nil, false, nil
}
@@ -18,6 +21,8 @@ func checkSubRequest(ctx context.Context, opts map[string]string) (*client.Resul
case subrequests.RequestSubrequestsDescribe:
res, err := describe()
return res, true, err
+ case outline.RequestSubrequestsOutline, targets.RequestTargets: // handled later
+ return nil, false, nil
default:
return nil, true, errdefs.NewUnsupportedSubrequestError(req)
}
@@ -25,15 +30,25 @@ func checkSubRequest(ctx context.Context, opts map[string]string) (*client.Resul
func describe() (*client.Result, error) {
all := []subrequests.Request{
+ outline.SubrequestsOutlineDefinition,
+ targets.SubrequestsTargetsDefinition,
subrequests.SubrequestsDescribeDefinition,
}
- dt, err := json.MarshalIndent(all, " ", "")
+ dt, err := json.MarshalIndent(all, "", " ")
if err != nil {
return nil, err
}
+
+ b := bytes.NewBuffer(nil)
+ if err := subrequests.PrintDescribe(dt, b); err != nil {
+ return nil, err
+ }
+
res := client.NewResult()
res.Metadata = map[string][]byte{
"result.json": dt,
+ "result.txt": b.Bytes(),
+ "version": []byte(subrequests.SubrequestsDescribeDefinition.Version),
}
return res, nil
}
diff --git a/frontend/dockerfile/cmd/dockerfile-frontend/Dockerfile b/frontend/dockerfile/cmd/dockerfile-frontend/Dockerfile
index 7f181948816b..23b1fd288fa0 100644
--- a/frontend/dockerfile/cmd/dockerfile-frontend/Dockerfile
+++ b/frontend/dockerfile/cmd/dockerfile-frontend/Dockerfile
@@ -1,9 +1,9 @@
# syntax=docker/dockerfile-upstream:master
# xx is a helper for cross-compilation
-FROM --platform=$BUILDPLATFORM tonistiigi/xx:golang@sha256:810dc54d5144f133a218e88e319184bf8b9ce01d37d46ddb37573e90decd9eef AS xx
+FROM --platform=$BUILDPLATFORM tonistiigi/xx:master@sha256:d4254d9739ce2de9fb88e09bdc716aa0c65f0446a2a2143399f991d71136a3d4 AS xx
-FROM --platform=$BUILDPLATFORM golang:1.17-alpine AS base
+FROM --platform=$BUILDPLATFORM golang:1.19-alpine AS base
RUN apk add git bash
COPY --from=xx / /
WORKDIR /src
diff --git a/frontend/dockerfile/cmd/dockerfile-frontend/hack/release b/frontend/dockerfile/cmd/dockerfile-frontend/hack/release
index b572bda1756b..7a7909c5fca3 100755
--- a/frontend/dockerfile/cmd/dockerfile-frontend/hack/release
+++ b/frontend/dockerfile/cmd/dockerfile-frontend/hack/release
@@ -1,10 +1,11 @@
#!/usr/bin/env bash
. $(dirname $0)/../../../../../hack/util
-set -e
+set -eu
-: ${PLATFORMS=}
-: ${DAILY_TARGETS=}
+: "${RELEASE=false}"
+: "${PLATFORMS=}"
+: "${DAILY_TARGETS=}"
usage() {
echo "$0 (master|tag|daily) (tag|channel) [push]"
@@ -23,7 +24,7 @@ parseTag() {
fi
local suffix=$(echo $1 | awk -F- '{print $NF}')
local tagf=./frontend/dockerfile/release/$suffix/tags
- if [ "$sufffix" == "$1" ] || [ ! -f $tagf ]; then
+ if [ "$suffix" == "$1" ] || [ ! -f $tagf ]; then
suffix="mainline"
fi
@@ -70,6 +71,11 @@ if [ "$PUSH" = "push" ]; then
pushFlag="push=true"
fi
+nocacheFilterFlag=""
+if [[ "$RELEASE" = "true" ]] && [[ "$GITHUB_ACTIONS" = "true" ]]; then
+ nocacheFilterFlag="--no-cache-filter=base"
+fi
+
case $TYP in
"master")
tagf=./frontend/dockerfile/release/$TAG/tags
@@ -84,10 +90,10 @@ case $TYP in
pushTag=${pushTag}-$TAG
fi
- buildxCmd build $platformFlag $cacheFromFlags $cacheToFlags \
+ buildxCmd build $platformFlag $cacheFromFlags $cacheToFlags $nocacheFilterFlag $(buildAttestFlags) \
--build-arg "CHANNEL=$TAG" \
--build-arg "BUILDTAGS=$buildTags" \
- --output "type=image,name=$REPO:$pushTag,buildinfo-attrs=true,$pushFlag" \
+ --output "type=image,name=$REPO:$pushTag,$pushFlag" \
--file "./frontend/dockerfile/cmd/dockerfile-frontend/Dockerfile" \
$currentcontext
;;
@@ -101,10 +107,10 @@ case $TYP in
fi
buildTags=$(cat $tagf)
- buildxCmd build $platformFlag $cacheFromFlags $cacheToFlags \
+ buildxCmd build $platformFlag $cacheFromFlags $cacheToFlags $nocacheFilterFlag $(buildAttestFlags) \
--build-arg "CHANNEL=$TAG" \
--build-arg "BUILDTAGS=$buildTags" \
- --output "type=image,\"name=$publishedNames\",buildinfo-attrs=true,$pushFlag" \
+ --output "type=image,\"name=$publishedNames\",$pushFlag" \
--file "./frontend/dockerfile/cmd/dockerfile-frontend/Dockerfile" \
$currentcontext
;;
@@ -127,7 +133,7 @@ case $TYP in
tmp=$(mktemp -d -t buildid.XXXXXXXXXX)
dt=$(date +%Y%m%d)
- buildxCmd build $platformFlag $cacheFromFlags $cacheToFlags \
+ buildxCmd build $platformFlag $cacheFromFlags $cacheToFlags $nocacheFilterFlag \
--target "buildid" \
--build-arg "CHANNEL=$TAG" \
--build-arg "BUILDTAGS=$buildTags" \
@@ -141,10 +147,10 @@ case $TYP in
buildid=$(cat $tmp/buildid)
echo "buildid: $buildid"
- buildxCmd build $platformFlag $cacheFromFlags $cacheToFlags \
+ buildxCmd build $platformFlag $cacheFromFlags $cacheToFlags $nocacheFilterFlag $(buildAttestFlags) \
--build-arg "CHANNEL=$TAG" \
--build-arg "BUILDTAGS=$buildTags" \
- --output "type=image,name=$REPO:$dt-$TAG,buildinfo-attrs=true,$pushFlag" \
+ --output "type=image,name=$REPO:$dt-$TAG,$pushFlag" \
--file "./frontend/dockerfile/cmd/dockerfile-frontend/Dockerfile" \
$currentcontext
rm $tmp/buildid
diff --git a/frontend/dockerfile/dockerfile2llb/convert.go b/frontend/dockerfile/dockerfile2llb/convert.go
index 7ac6b9bdd774..6476267e2d32 100644
--- a/frontend/dockerfile/dockerfile2llb/convert.go
+++ b/frontend/dockerfile/dockerfile2llb/convert.go
@@ -13,22 +13,28 @@ import (
"sort"
"strconv"
"strings"
+ "time"
"github.com/containerd/containerd/platforms"
"github.com/docker/distribution/reference"
"github.com/docker/go-connections/nat"
"github.com/moby/buildkit/client/llb"
"github.com/moby/buildkit/client/llb/imagemetaresolver"
+ "github.com/moby/buildkit/exporter/containerimage/image"
"github.com/moby/buildkit/frontend/dockerfile/instructions"
"github.com/moby/buildkit/frontend/dockerfile/parser"
"github.com/moby/buildkit/frontend/dockerfile/shell"
+ "github.com/moby/buildkit/frontend/subrequests/outline"
+ "github.com/moby/buildkit/frontend/subrequests/targets"
"github.com/moby/buildkit/identity"
"github.com/moby/buildkit/solver/pb"
"github.com/moby/buildkit/util/apicaps"
binfotypes "github.com/moby/buildkit/util/buildinfo/types"
+ "github.com/moby/buildkit/util/gitutil"
"github.com/moby/buildkit/util/suggest"
"github.com/moby/buildkit/util/system"
"github.com/moby/sys/signal"
+ digest "github.com/opencontainers/go-digest"
ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/pkg/errors"
"golang.org/x/sync/errgroup"
@@ -39,9 +45,15 @@ const (
defaultContextLocalName = "context"
historyComment = "buildkit.dockerfile.v0"
- DefaultCopyImage = "docker/dockerfile-copy:v0.1.9@sha256:e8f159d3f00786604b93c675ee2783f8dc194bb565e61ca5788f6a6e9d304061"
+ sbomScanContext = "BUILDKIT_SBOM_SCAN_CONTEXT"
+ sbomScanStage = "BUILDKIT_SBOM_SCAN_STAGE"
)
+var nonEnvArgs = map[string]struct{}{
+ sbomScanContext: {},
+ sbomScanStage: {},
+}
+
type ConvertOpt struct {
Target string
MetaResolver llb.ImageMetaResolver
@@ -54,57 +66,127 @@ type ConvertOpt struct {
// Empty slice means ignore cache for all stages. Nil doesn't disable cache.
IgnoreCache []string
// CacheIDNamespace scopes the IDs for different cache mounts
- CacheIDNamespace string
- ImageResolveMode llb.ResolveMode
- TargetPlatform *ocispecs.Platform
- BuildPlatforms []ocispecs.Platform
- PrefixPlatform bool
- ExtraHosts []llb.HostIP
- ShmSize int64
- Ulimit []pb.Ulimit
- CgroupParent string
- ForceNetMode pb.NetMode
- OverrideCopyImage string
- LLBCaps *apicaps.CapSet
- ContextLocalName string
- SourceMap *llb.SourceMap
- Hostname string
- Warn func(short, url string, detail [][]byte, location *parser.Range)
- ContextByName func(context.Context, string) (*llb.State, *Image, *binfotypes.BuildInfo, error)
-}
-
-func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State, *Image, *binfotypes.BuildInfo, error) {
- buildInfo := &binfotypes.BuildInfo{}
+ CacheIDNamespace string
+ ImageResolveMode llb.ResolveMode
+ TargetPlatform *ocispecs.Platform
+ BuildPlatforms []ocispecs.Platform
+ PrefixPlatform bool
+ ExtraHosts []llb.HostIP
+ ShmSize int64
+ Ulimit []pb.Ulimit
+ CgroupParent string
+ ForceNetMode pb.NetMode
+ LLBCaps *apicaps.CapSet
+ ContextLocalName string
+ SourceMap *llb.SourceMap
+ Hostname string
+ SourceDateEpoch *time.Time
+ Warn func(short, url string, detail [][]byte, location *parser.Range)
+ ContextByName func(ctx context.Context, name, resolveMode string, p *ocispecs.Platform) (*llb.State, *Image, error)
+}
+
+type SBOMTargets struct {
+ Core llb.State
+ Extras map[string]llb.State
+
+ IgnoreCache bool
+}
+
+func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State, *Image, *SBOMTargets, error) {
+ ds, err := toDispatchState(ctx, dt, opt)
+ if err != nil {
+ return nil, nil, nil, err
+ }
+
+ sbom := SBOMTargets{
+ Core: ds.state,
+ Extras: map[string]llb.State{},
+ }
+ if ds.scanContext {
+ sbom.Extras["context"] = ds.opt.buildContext
+ }
+ if ds.ignoreCache {
+ sbom.IgnoreCache = true
+ }
+ for _, dsi := range findReachable(ds) {
+ if ds != dsi && dsi.scanStage {
+ sbom.Extras[dsi.stageName] = dsi.state
+ if dsi.ignoreCache {
+ sbom.IgnoreCache = true
+ }
+ }
+ }
+
+ return &ds.state, &ds.image, &sbom, nil
+}
+
+func Dockefile2Outline(ctx context.Context, dt []byte, opt ConvertOpt) (*outline.Outline, error) {
+ ds, err := toDispatchState(ctx, dt, opt)
+ if err != nil {
+ return nil, err
+ }
+ o := ds.Outline(dt)
+ return &o, nil
+}
+
+func ListTargets(ctx context.Context, dt []byte) (*targets.List, error) {
+ dockerfile, err := parser.Parse(bytes.NewReader(dt))
+ if err != nil {
+ return nil, err
+ }
+ stages, _, err := instructions.Parse(dockerfile.AST)
+ if err != nil {
+ return nil, err
+ }
+
+ l := &targets.List{
+ Sources: [][]byte{dt},
+ }
+
+ for i, s := range stages {
+ t := targets.Target{
+ Name: s.Name,
+ Description: s.Comment,
+ Default: i == len(stages)-1,
+ Base: s.BaseName,
+ Platform: s.Platform,
+ Location: toSourceLocation(s.Location),
+ }
+ l.Targets = append(l.Targets, t)
+ }
+ return l, nil
+}
+
+func toDispatchState(ctx context.Context, dt []byte, opt ConvertOpt) (*dispatchState, error) {
contextByName := opt.ContextByName
- opt.ContextByName = func(ctx context.Context, name string) (*llb.State, *Image, *binfotypes.BuildInfo, error) {
+ opt.ContextByName = func(ctx context.Context, name, resolveMode string, p *ocispecs.Platform) (*llb.State, *Image, error) {
if !strings.EqualFold(name, "scratch") && !strings.EqualFold(name, "context") {
if contextByName != nil {
- st, img, bi, err := contextByName(ctx, name)
- if err != nil {
- return nil, nil, nil, err
+ if p == nil {
+ p = opt.TargetPlatform
}
- if bi != nil && bi.Deps != nil {
- for k := range bi.Deps {
- if buildInfo.Deps == nil {
- buildInfo.Deps = make(map[string]binfotypes.BuildInfo)
- }
- buildInfo.Deps[k] = bi.Deps[k]
- }
+ st, img, err := contextByName(ctx, name, resolveMode, p)
+ if err != nil {
+ return nil, nil, err
}
- return st, img, bi, nil
+ return st, img, nil
}
}
- return nil, nil, nil, nil
+ return nil, nil, nil
}
if len(dt) == 0 {
- return nil, nil, nil, errors.Errorf("the Dockerfile cannot be empty")
+ return nil, errors.Errorf("the Dockerfile cannot be empty")
}
if opt.ContextLocalName == "" {
opt.ContextLocalName = defaultContextLocalName
}
+ if opt.Warn == nil {
+ opt.Warn = func(string, string, [][]byte, *parser.Range) {}
+ }
+
platformOpt := buildPlatformOpt(&opt)
optMetaArgs := getPlatformArgs(platformOpt)
@@ -114,7 +196,7 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
dockerfile, err := parser.Parse(bytes.NewReader(dt))
if err != nil {
- return nil, nil, nil, err
+ return nil, err
}
for _, w := range dockerfile.Warnings {
@@ -125,17 +207,27 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
stages, metaArgs, err := instructions.Parse(dockerfile.AST)
if err != nil {
- return nil, nil, nil, err
+ return nil, err
}
shlex := shell.NewLex(dockerfile.EscapeToken)
+ outline := newOutlineCapture()
for _, cmd := range metaArgs {
for _, metaArg := range cmd.Args {
+ info := argInfo{definition: metaArg, location: cmd.Location()}
+ if v, ok := opt.BuildArgs[metaArg.Key]; !ok {
+ if metaArg.Value != nil {
+ *metaArg.Value, info.deps, _ = shlex.ProcessWordWithMatches(*metaArg.Value, metaArgsToMap(optMetaArgs))
+ }
+ } else {
+ metaArg.Value = &v
+ }
+ optMetaArgs = append(optMetaArgs, metaArg)
if metaArg.Value != nil {
- *metaArg.Value, _ = shlex.ProcessWordWithMap(*metaArg.Value, metaArgsToMap(optMetaArgs))
+ info.value = *metaArg.Value
}
- optMetaArgs = append(optMetaArgs, setKVValue(metaArg, opt.BuildArgs))
+ outline.allArgs[metaArg.Key] = info
}
}
@@ -148,60 +240,73 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
// set base state for every image
for i, st := range stages {
- name, err := shlex.ProcessWordWithMap(st.BaseName, metaArgsToMap(optMetaArgs))
+ name, used, err := shlex.ProcessWordWithMatches(st.BaseName, metaArgsToMap(optMetaArgs))
if err != nil {
- return nil, nil, nil, parser.WithLocation(err, st.Location)
+ return nil, parser.WithLocation(err, st.Location)
}
if name == "" {
- return nil, nil, nil, parser.WithLocation(errors.Errorf("base name (%s) should not be blank", st.BaseName), st.Location)
+ return nil, parser.WithLocation(errors.Errorf("base name (%s) should not be blank", st.BaseName), st.Location)
}
st.BaseName = name
ds := &dispatchState{
+ stage: st,
deps: make(map[*dispatchState]struct{}),
ctxPaths: make(map[string]struct{}),
stageName: st.Name,
prefixPlatform: opt.PrefixPlatform,
+ outline: outline.clone(),
+ epoch: opt.SourceDateEpoch,
+ }
+
+ if v := st.Platform; v != "" {
+ v, u, err := shlex.ProcessWordWithMatches(v, metaArgsToMap(optMetaArgs))
+ if err != nil {
+ return nil, parser.WithLocation(errors.Wrapf(err, "failed to process arguments for platform %s", v), st.Location)
+ }
+
+ p, err := platforms.Parse(v)
+ if err != nil {
+ return nil, parser.WithLocation(errors.Wrapf(err, "failed to parse platform %s", v), st.Location)
+ }
+ for k := range u {
+ used[k] = struct{}{}
+ }
+ ds.platform = &p
}
if st.Name != "" {
- s, img, bi, err := opt.ContextByName(ctx, st.Name)
+ s, img, err := opt.ContextByName(ctx, st.Name, opt.ImageResolveMode.String(), ds.platform)
if err != nil {
- return nil, nil, nil, err
+ return nil, err
}
if s != nil {
ds.noinit = true
ds.state = *s
if img != nil {
- ds.image = *img
- }
- if bi != nil {
- ds.buildInfo = *bi
+ ds.image = clampTimes(*img, opt.SourceDateEpoch)
+ if img.Architecture != "" && img.OS != "" {
+ ds.platform = &ocispecs.Platform{
+ OS: img.OS,
+ Architecture: img.Architecture,
+ Variant: img.Variant,
+ }
+ }
}
allDispatchStates.addState(ds)
continue
}
}
- ds.stage = st
-
if st.Name == "" {
ds.stageName = fmt.Sprintf("stage-%d", i)
}
- if v := st.Platform; v != "" {
- v, err := shlex.ProcessWordWithMap(v, metaArgsToMap(optMetaArgs))
- if err != nil {
- return nil, nil, nil, parser.WithLocation(errors.Wrapf(err, "failed to process arguments for platform %s", v), st.Location)
- }
+ allDispatchStates.addState(ds)
- p, err := platforms.Parse(v)
- if err != nil {
- return nil, nil, nil, parser.WithLocation(errors.Wrapf(err, "failed to parse platform %s", v), st.Location)
- }
- ds.platform = &p
+ for k := range used {
+ ds.outline.usedArgs[k] = struct{}{}
}
- allDispatchStates.addState(ds)
total := 0
if ds.stage.BaseName != emptyImageName && ds.base == nil {
@@ -212,9 +317,7 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
case *instructions.AddCommand, *instructions.CopyCommand, *instructions.RunCommand:
total++
case *instructions.WorkdirCommand:
- if useFileOp(opt.BuildArgs, opt.LLBCaps) {
- total++
- }
+ total++
}
}
ds.cmdTotal = total
@@ -239,7 +342,7 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
var ok bool
target, ok = allDispatchStates.findStateByName(opt.Target)
if !ok {
- return nil, nil, nil, errors.Errorf("target stage %s could not be found", opt.Target)
+ return nil, errors.Errorf("target stage %s could not be found", opt.Target)
}
}
@@ -249,7 +352,7 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
for i, cmd := range d.stage.Commands {
newCmd, err := toCommand(cmd, allDispatchStates)
if err != nil {
- return nil, nil, nil, err
+ return nil, err
}
d.commands[i] = newCmd
for _, src := range newCmd.sources {
@@ -264,7 +367,7 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
}
if has, state := hasCircularDependency(allDispatchStates.states); has {
- return nil, nil, nil, errors.Errorf("circular dependency detected on stage: %s", state.stageName)
+ return nil, errors.Errorf("circular dependency detected on stage: %s", state.stageName)
}
if len(allDispatchStates.states) == 1 {
@@ -307,7 +410,7 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
d.stage.BaseName = reference.TagNameOnly(ref).String()
var isScratch bool
- st, img, bi, err := opt.ContextByName(ctx, d.stage.BaseName)
+ st, img, err := opt.ContextByName(ctx, d.stage.BaseName, opt.ImageResolveMode.String(), platform)
if err != nil {
return err
}
@@ -317,10 +420,7 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
} else {
d.image = emptyImage(platformOpt.targetPlatform)
}
- if bi != nil {
- d.buildInfo = *bi
- }
- d.state = *st
+ d.state = st.Platform(*platform)
d.platform = platform
return nil
}
@@ -331,9 +431,10 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
}
prefix += "internal]"
dgst, dt, err := metaResolver.ResolveImageConfig(ctx, d.stage.BaseName, llb.ResolveImageConfigOpt{
- Platform: platform,
- ResolveMode: opt.ImageResolveMode.String(),
- LogName: fmt.Sprintf("%s load metadata for %s", prefix, d.stage.BaseName),
+ Platform: platform,
+ ResolveMode: opt.ImageResolveMode.String(),
+ LogName: fmt.Sprintf("%s load metadata for %s", prefix, d.stage.BaseName),
+ ResolverType: llb.ResolverTypeRegistry,
})
if err != nil {
return suggest.WrapError(errors.Wrap(err, origName), origName, append(allStageNames, commonImageNames()...), true)
@@ -396,30 +497,17 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
}
if err := eg.Wait(); err != nil {
- return nil, nil, nil, err
+ return nil, err
}
buildContext := &mutableOutput{}
ctxPaths := map[string]struct{}{}
for _, d := range allDispatchStates.states {
- if !isReachable(target, d) {
+ if !isReachable(target, d) || d.noinit {
continue
}
- // collect build sources and dependencies
- if len(d.buildInfo.Sources) > 0 {
- buildInfo.Sources = append(buildInfo.Sources, d.buildInfo.Sources...)
- }
- if d.buildInfo.Deps != nil {
- for name, bi := range d.buildInfo.Deps {
- if buildInfo.Deps == nil {
- buildInfo.Deps = make(map[string]binfotypes.BuildInfo)
- }
- buildInfo.Deps[name] = bi
- }
- }
-
if d.base != nil {
d.state = d.base.state
d.platform = d.base.platform
@@ -428,11 +516,11 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
// make sure that PATH is always set
if _, ok := shell.BuildEnvs(d.image.Config.Env)["PATH"]; !ok {
- var os string
+ var pathOS string
if d.platform != nil {
- os = d.platform.OS
+ pathOS = d.platform.OS
}
- d.image.Config.Env = append(d.image.Config.Env, "PATH="+system.DefaultPathEnv(os))
+ d.image.Config.Env = append(d.image.Config.Env, "PATH="+system.DefaultPathEnv(pathOS))
}
// initialize base metadata from image conf
@@ -445,12 +533,12 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
}
if d.image.Config.WorkingDir != "" {
if err = dispatchWorkdir(d, &instructions.WorkdirCommand{Path: d.image.Config.WorkingDir}, false, nil); err != nil {
- return nil, nil, nil, parser.WithLocation(err, d.stage.Location)
+ return nil, parser.WithLocation(err, d.stage.Location)
}
}
if d.image.Config.User != "" {
if err = dispatchUser(d, &instructions.UserCommand{User: d.image.Config.User}, false); err != nil {
- return nil, nil, nil, parser.WithLocation(err, d.stage.Location)
+ return nil, parser.WithLocation(err, d.stage.Location)
}
}
d.state = d.state.Network(opt.ForceNetMode)
@@ -470,35 +558,37 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
shmSize: opt.ShmSize,
ulimit: opt.Ulimit,
cgroupParent: opt.CgroupParent,
- copyImage: opt.OverrideCopyImage,
llbCaps: opt.LLBCaps,
sourceMap: opt.SourceMap,
}
- if opt.copyImage == "" {
- opt.copyImage = DefaultCopyImage
- }
if err = dispatchOnBuildTriggers(d, d.image.Config.OnBuild, opt); err != nil {
- return nil, nil, nil, parser.WithLocation(err, d.stage.Location)
+ return nil, parser.WithLocation(err, d.stage.Location)
}
d.image.Config.OnBuild = nil
for _, cmd := range d.commands {
if err := dispatch(d, cmd, opt); err != nil {
- return nil, nil, nil, parser.WithLocation(err, cmd.Location())
+ return nil, parser.WithLocation(err, cmd.Location())
}
}
+ d.opt = opt
for p := range d.ctxPaths {
ctxPaths[p] = struct{}{}
}
- }
- // sort build sources
- if len(buildInfo.Sources) > 0 {
- sort.Slice(buildInfo.Sources, func(i, j int) bool {
- return buildInfo.Sources[i].Ref < buildInfo.Sources[j].Ref
- })
+ locals := []instructions.KeyValuePairOptional{}
+ locals = append(locals, d.opt.metaArgs...)
+ locals = append(locals, d.buildArgs...)
+ for _, a := range locals {
+ switch a.Key {
+ case sbomScanStage:
+ d.scanStage = isEnabledForStage(d.stageName, a.ValueString())
+ case sbomScanContext:
+ d.scanContext = isEnabledForStage(d.stageName, a.ValueString())
+ }
+ }
}
if len(opt.Labels) != 0 && target.image.Config.Labels == nil {
@@ -530,7 +620,7 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
if opt.LLBCaps != nil {
defaults = append(defaults, llb.WithCaps(*opt.LLBCaps))
}
- st := target.state.SetMarshalDefaults(defaults...)
+ target.state = target.state.SetMarshalDefaults(defaults...)
if !platformOpt.implicitTarget {
target.image.OS = platformOpt.targetPlatform.OS
@@ -538,7 +628,7 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
target.image.Variant = platformOpt.targetPlatform.Variant
}
- return &st, &target.image, buildInfo, nil
+ return target, nil
}
func metaArgsToMap(metaArgs []instructions.KeyValuePairOptional) map[string]string {
@@ -598,7 +688,6 @@ type dispatchOpt struct {
shmSize int64
ulimit []pb.Ulimit
cgroupParent string
- copyImage string
llbCaps *apicaps.CapSet
sourceMap *llb.SourceMap
}
@@ -643,17 +732,25 @@ func dispatch(d *dispatchState, cmd command, opt dispatchOpt) error {
case *instructions.WorkdirCommand:
err = dispatchWorkdir(d, c, true, &opt)
case *instructions.AddCommand:
- err = dispatchCopy(d, copyConfig{
- params: c.SourcesAndDest,
- source: opt.buildContext,
- isAddCommand: true,
- cmdToPrint: c,
- chown: c.Chown,
- chmod: c.Chmod,
- link: c.Link,
- location: c.Location(),
- opt: opt,
- })
+ var checksum digest.Digest
+ if c.Checksum != "" {
+ checksum, err = digest.Parse(c.Checksum)
+ }
+ if err == nil {
+ err = dispatchCopy(d, copyConfig{
+ params: c.SourcesAndDest,
+ source: opt.buildContext,
+ isAddCommand: true,
+ cmdToPrint: c,
+ chown: c.Chown,
+ chmod: c.Chmod,
+ link: c.Link,
+ keepGitDir: c.KeepGitDir,
+ checksum: checksum,
+ location: c.Location(),
+ opt: opt,
+ })
+ }
if err == nil {
for _, src := range c.SourcePaths {
if !strings.HasPrefix(src, "http://") && !strings.HasPrefix(src, "https://") {
@@ -710,6 +807,7 @@ func dispatch(d *dispatchState, cmd command, opt dispatchOpt) error {
}
type dispatchState struct {
+ opt dispatchOpt
state llb.State
image Image
platform *ocispecs.Platform
@@ -728,6 +826,10 @@ type dispatchState struct {
cmdTotal int
prefixPlatform bool
buildInfo binfotypes.BuildInfo
+ outline outlineCapture
+ epoch *time.Time
+ scanStage bool
+ scanContext bool
}
type dispatchStates struct {
@@ -744,6 +846,7 @@ func (dss *dispatchStates) addState(ds *dispatchState) {
if d, ok := dss.statesByName[ds.stage.BaseName]; ok {
ds.base = d
+ ds.outline = d.outline.clone()
}
if ds.stage.Name != "" {
dss.statesByName[strings.ToLower(ds.stage.Name)] = ds
@@ -803,7 +906,7 @@ func dispatchEnv(d *dispatchState, c *instructions.EnvCommand) error {
d.state = d.state.AddEnv(e.Key, e.Value)
d.image.Config.Env = addEnv(d.image.Config.Env, e.Key, e.Value)
}
- return commitToHistory(&d.image, commitMessage.String(), false, nil)
+ return commitToHistory(&d.image, commitMessage.String(), false, nil, d.epoch)
}
func dispatchRun(d *dispatchState, c *instructions.RunCommand, proxy *llb.ProxyEnv, sources []*dispatchState, dopt dispatchOpt) error {
@@ -814,7 +917,7 @@ func dispatchRun(d *dispatchState, c *instructions.RunCommand, proxy *llb.ProxyE
var args []string = c.CmdLine
if len(c.Files) > 0 {
if len(args) != 1 || !c.PrependShell {
- return fmt.Errorf("parsing produced an invalid run command: %v", args)
+ return errors.Errorf("parsing produced an invalid run command: %v", args)
}
if heredoc := parser.MustParseHeredoc(args[0]); heredoc != nil {
@@ -933,7 +1036,7 @@ func dispatchRun(d *dispatchState, c *instructions.RunCommand, proxy *llb.ProxyE
}
d.state = d.state.Run(opt...).Root()
- return commitToHistory(&d.image, "RUN "+runCommandString(args, d.buildArgs, shell.BuildEnvs(env)), true, &d.state)
+ return commitToHistory(&d.image, "RUN "+runCommandString(args, d.buildArgs, shell.BuildEnvs(env)), true, &d.state, d.epoch)
}
func dispatchWorkdir(d *dispatchState, c *instructions.WorkdirCommand, commit bool, opt *dispatchOpt) error {
@@ -945,7 +1048,7 @@ func dispatchWorkdir(d *dispatchState, c *instructions.WorkdirCommand, commit bo
d.image.Config.WorkingDir = wd
if commit {
withLayer := false
- if wd != "/" && opt != nil && useFileOp(opt.buildArgValues, opt.llbCaps) {
+ if wd != "/" {
mkdirOpt := []llb.MkdirOption{llb.WithParents(true)}
if user := d.image.Config.User; user != "" {
mkdirOpt = append(mkdirOpt, llb.WithUser(user))
@@ -964,12 +1067,12 @@ func dispatchWorkdir(d *dispatchState, c *instructions.WorkdirCommand, commit bo
)
withLayer = true
}
- return commitToHistory(&d.image, "WORKDIR "+wd, withLayer, nil)
+ return commitToHistory(&d.image, "WORKDIR "+wd, withLayer, nil, d.epoch)
}
return nil
}
-func dispatchCopyFileOp(d *dispatchState, cfg copyConfig) error {
+func dispatchCopy(d *dispatchState, cfg copyConfig) error {
pp, err := pathRelativeToWorkingDir(d.state, cfg.params.DestPath)
if err != nil {
return err
@@ -994,6 +1097,21 @@ func dispatchCopyFileOp(d *dispatchState, cfg copyConfig) error {
}
}
+ if cfg.checksum != "" {
+ if !cfg.isAddCommand {
+ return errors.New("checksum can't be specified for COPY")
+ }
+ if !addChecksumEnabled {
+ return errors.New("instruction 'ADD --checksum=' requires the labs channel")
+ }
+ if len(cfg.params.SourcePaths) != 1 {
+ return errors.New("checksum can't be specified for multiple sources")
+ }
+ if !isHTTPSource(cfg.params.SourcePaths[0]) {
+ return errors.New("checksum can't be specified for non-HTTP sources")
+ }
+ }
+
commitMessage := bytes.NewBufferString("")
if cfg.isAddCommand {
commitMessage.WriteString("ADD")
@@ -1005,7 +1123,34 @@ func dispatchCopyFileOp(d *dispatchState, cfg copyConfig) error {
for _, src := range cfg.params.SourcePaths {
commitMessage.WriteString(" " + src)
- if strings.HasPrefix(src, "http://") || strings.HasPrefix(src, "https://") {
+ gitRef, gitRefErr := gitutil.ParseGitRef(src)
+ if gitRefErr == nil && !gitRef.IndistinguishableFromLocal {
+ if !cfg.isAddCommand {
+ return errors.New("source can't be a git ref for COPY")
+ }
+ if !addGitEnabled {
+ return errors.New("instruction ADD requires the labs channel")
+ }
+ // TODO: print a warning (not an error) if gitRef.UnencryptedTCP is true
+ commit := gitRef.Commit
+ if gitRef.SubDir != "" {
+ commit += ":" + gitRef.SubDir
+ }
+ var gitOptions []llb.GitOption
+ if cfg.keepGitDir {
+ gitOptions = append(gitOptions, llb.KeepGitDir())
+ }
+ st := llb.Git(gitRef.Remote, commit, gitOptions...)
+ opts := append([]llb.CopyOption{&llb.CopyInfo{
+ Mode: mode,
+ CreateDestPath: true,
+ }}, copyOpt...)
+ if a == nil {
+ a = llb.Copy(st, "/", dest, opts...)
+ } else {
+ a = a.Copy(st, "/", dest, opts...)
+ }
+ } else if isHTTPSource(src) {
if !cfg.isAddCommand {
return errors.New("source can't be a URL for COPY")
}
@@ -1023,7 +1168,7 @@ func dispatchCopyFileOp(d *dispatchState, cfg copyConfig) error {
}
}
- st := llb.HTTP(src, llb.Filename(f), dfCmd(cfg.params))
+ st := llb.HTTP(src, llb.Filename(f), llb.Checksum(cfg.checksum), dfCmd(cfg.params))
opts := append([]llb.CopyOption{&llb.CopyInfo{
Mode: mode,
@@ -1097,7 +1242,8 @@ func dispatchCopyFileOp(d *dispatchState, cfg copyConfig) error {
fileOpt = append(fileOpt, llb.IgnoreCache)
}
- if cfg.opt.llbCaps.Supports(pb.CapMergeOp) == nil && cfg.link && cfg.chmod == "" {
+ // cfg.opt.llbCaps can be nil in unit tests
+ if cfg.opt.llbCaps != nil && cfg.opt.llbCaps.Supports(pb.CapMergeOp) == nil && cfg.link && cfg.chmod == "" {
pgID := identity.NewID()
d.cmdIndex-- // prefixCommand increases it
pgName := prefixCommand(d, name, d.prefixPlatform, &platform, env)
@@ -1116,7 +1262,7 @@ func dispatchCopyFileOp(d *dispatchState, cfg copyConfig) error {
d.state = d.state.File(a, fileOpt...)
}
- return commitToHistory(&d.image, commitMessage.String(), true, &d.state)
+ return commitToHistory(&d.image, commitMessage.String(), true, &d.state, d.epoch)
}
type copyConfig struct {
@@ -1127,136 +1273,15 @@ type copyConfig struct {
chown string
chmod string
link bool
+ keepGitDir bool
+ checksum digest.Digest
location []parser.Range
opt dispatchOpt
}
-func dispatchCopy(d *dispatchState, cfg copyConfig) error {
- if useFileOp(cfg.opt.buildArgValues, cfg.opt.llbCaps) {
- return dispatchCopyFileOp(d, cfg)
- }
-
- if len(cfg.params.SourceContents) > 0 {
- return errors.New("inline content copy is not supported")
- }
-
- if cfg.chmod != "" {
- if cfg.opt.llbCaps != nil && cfg.opt.llbCaps.Supports(pb.CapFileBase) != nil {
- return errors.Wrap(cfg.opt.llbCaps.Supports(pb.CapFileBase), "chmod is not supported")
- }
- return errors.New("chmod is not supported")
- }
-
- img := llb.Image(cfg.opt.copyImage, llb.MarkImageInternal, llb.Platform(cfg.opt.buildPlatforms[0]), WithInternalName("helper image for file operations"))
- pp, err := pathRelativeToWorkingDir(d.state, cfg.params.DestPath)
- if err != nil {
- return err
- }
- dest := path.Join(".", pp)
- if cfg.params.DestPath == "." || cfg.params.DestPath == "" || cfg.params.DestPath[len(cfg.params.DestPath)-1] == filepath.Separator {
- dest += string(filepath.Separator)
- }
- args := []string{"copy"}
- unpack := cfg.isAddCommand
-
- mounts := make([]llb.RunOption, 0, len(cfg.params.SourcePaths))
- if cfg.chown != "" {
- args = append(args, fmt.Sprintf("--chown=%s", cfg.chown))
- _, _, err := parseUser(cfg.chown)
- if err != nil {
- mounts = append(mounts, llb.AddMount("/etc/passwd", d.state, llb.SourcePath("/etc/passwd"), llb.Readonly))
- mounts = append(mounts, llb.AddMount("/etc/group", d.state, llb.SourcePath("/etc/group"), llb.Readonly))
- }
- }
-
- commitMessage := bytes.NewBufferString("")
- if cfg.isAddCommand {
- commitMessage.WriteString("ADD")
- } else {
- commitMessage.WriteString("COPY")
- }
-
- for i, src := range cfg.params.SourcePaths {
- commitMessage.WriteString(" " + src)
- if strings.HasPrefix(src, "http://") || strings.HasPrefix(src, "https://") {
- if !cfg.isAddCommand {
- return errors.New("source can't be a URL for COPY")
- }
-
- // Resources from remote URLs are not decompressed.
- // https://docs.docker.com/engine/reference/builder/#add
- //
- // Note: mixing up remote archives and local archives in a single ADD instruction
- // would result in undefined behavior: https://github.com/moby/buildkit/pull/387#discussion_r189494717
- unpack = false
- u, err := url.Parse(src)
- f := "__unnamed__"
- if err == nil {
- if base := path.Base(u.Path); base != "." && base != "/" {
- f = base
- }
- }
- target := path.Join(fmt.Sprintf("/src-%d", i), f)
- args = append(args, target)
- mounts = append(mounts, llb.AddMount(path.Dir(target), llb.HTTP(src, llb.Filename(f), dfCmd(cfg.params)), llb.Readonly))
- } else {
- d, f := splitWildcards(src)
- targetCmd := fmt.Sprintf("/src-%d", i)
- targetMount := targetCmd
- if f == "" {
- f = path.Base(src)
- targetMount = path.Join(targetMount, f)
- }
- targetCmd = path.Join(targetCmd, f)
- args = append(args, targetCmd)
- mounts = append(mounts, llb.AddMount(targetMount, cfg.source, llb.SourcePath(d), llb.Readonly))
- }
- }
-
- commitMessage.WriteString(" " + cfg.params.DestPath)
-
- args = append(args, dest)
- if unpack {
- args = append(args[:1], append([]string{"--unpack"}, args[1:]...)...)
- }
-
- platform := cfg.opt.targetPlatform
- if d.platform != nil {
- platform = *d.platform
- }
-
- env, err := d.state.Env(context.TODO())
- if err != nil {
- return err
- }
-
- runOpt := []llb.RunOption{
- llb.Args(args),
- llb.Dir("/dest"),
- llb.ReadonlyRootFS(),
- dfCmd(cfg.cmdToPrint),
- llb.WithCustomName(prefixCommand(d, uppercaseCmd(processCmdEnv(cfg.opt.shlex, cfg.cmdToPrint.String(), env)), d.prefixPlatform, &platform, env)),
- location(cfg.opt.sourceMap, cfg.location),
- }
- if d.ignoreCache {
- runOpt = append(runOpt, llb.IgnoreCache)
- }
-
- if cfg.opt.llbCaps != nil {
- if err := cfg.opt.llbCaps.Supports(pb.CapExecMetaNetwork); err == nil {
- runOpt = append(runOpt, llb.Network(llb.NetModeNone))
- }
- }
-
- run := img.Run(append(runOpt, mounts...)...)
- d.state = run.AddMount("/dest", d.state).Platform(platform)
-
- return commitToHistory(&d.image, commitMessage.String(), true, &d.state)
-}
-
func dispatchMaintainer(d *dispatchState, c *instructions.MaintainerCommand) error {
d.image.Author = c.Maintainer
- return commitToHistory(&d.image, fmt.Sprintf("MAINTAINER %v", c.Maintainer), false, nil)
+ return commitToHistory(&d.image, fmt.Sprintf("MAINTAINER %v", c.Maintainer), false, nil, d.epoch)
}
func dispatchLabel(d *dispatchState, c *instructions.LabelCommand) error {
@@ -1268,7 +1293,7 @@ func dispatchLabel(d *dispatchState, c *instructions.LabelCommand) error {
d.image.Config.Labels[v.Key] = v.Value
commitMessage.WriteString(" " + v.String())
}
- return commitToHistory(&d.image, commitMessage.String(), false, nil)
+ return commitToHistory(&d.image, commitMessage.String(), false, nil, d.epoch)
}
func dispatchOnbuild(d *dispatchState, c *instructions.OnbuildCommand) error {
@@ -1284,7 +1309,7 @@ func dispatchCmd(d *dispatchState, c *instructions.CmdCommand) error {
d.image.Config.Cmd = args
d.image.Config.ArgsEscaped = true
d.cmdSet = true
- return commitToHistory(&d.image, fmt.Sprintf("CMD %q", args), false, nil)
+ return commitToHistory(&d.image, fmt.Sprintf("CMD %q", args), false, nil, d.epoch)
}
func dispatchEntrypoint(d *dispatchState, c *instructions.EntrypointCommand) error {
@@ -1296,18 +1321,18 @@ func dispatchEntrypoint(d *dispatchState, c *instructions.EntrypointCommand) err
if !d.cmdSet {
d.image.Config.Cmd = nil
}
- return commitToHistory(&d.image, fmt.Sprintf("ENTRYPOINT %q", args), false, nil)
+ return commitToHistory(&d.image, fmt.Sprintf("ENTRYPOINT %q", args), false, nil, d.epoch)
}
func dispatchHealthcheck(d *dispatchState, c *instructions.HealthCheckCommand) error {
- d.image.Config.Healthcheck = &HealthConfig{
+ d.image.Config.Healthcheck = &image.HealthConfig{
Test: c.Health.Test,
Interval: c.Health.Interval,
Timeout: c.Health.Timeout,
StartPeriod: c.Health.StartPeriod,
Retries: c.Health.Retries,
}
- return commitToHistory(&d.image, fmt.Sprintf("HEALTHCHECK %q", d.image.Config.Healthcheck), false, nil)
+ return commitToHistory(&d.image, fmt.Sprintf("HEALTHCHECK %q", d.image.Config.Healthcheck), false, nil, d.epoch)
}
func dispatchExpose(d *dispatchState, c *instructions.ExposeCommand, shlex *shell.Lex) error {
@@ -1337,14 +1362,14 @@ func dispatchExpose(d *dispatchState, c *instructions.ExposeCommand, shlex *shel
d.image.Config.ExposedPorts[string(p)] = struct{}{}
}
- return commitToHistory(&d.image, fmt.Sprintf("EXPOSE %v", ps), false, nil)
+ return commitToHistory(&d.image, fmt.Sprintf("EXPOSE %v", ps), false, nil, d.epoch)
}
func dispatchUser(d *dispatchState, c *instructions.UserCommand, commit bool) error {
d.state = d.state.User(c.User)
d.image.Config.User = c.User
if commit {
- return commitToHistory(&d.image, fmt.Sprintf("USER %v", c.User), false, nil)
+ return commitToHistory(&d.image, fmt.Sprintf("USER %v", c.User), false, nil, d.epoch)
}
return nil
}
@@ -1359,7 +1384,7 @@ func dispatchVolume(d *dispatchState, c *instructions.VolumeCommand) error {
}
d.image.Config.Volumes[v] = struct{}{}
}
- return commitToHistory(&d.image, fmt.Sprintf("VOLUME %v", c.Volumes), false, nil)
+ return commitToHistory(&d.image, fmt.Sprintf("VOLUME %v", c.Volumes), false, nil, d.epoch)
}
func dispatchStopSignal(d *dispatchState, c *instructions.StopSignalCommand) error {
@@ -1367,12 +1392,12 @@ func dispatchStopSignal(d *dispatchState, c *instructions.StopSignalCommand) err
return err
}
d.image.Config.StopSignal = c.Signal
- return commitToHistory(&d.image, fmt.Sprintf("STOPSIGNAL %v", c.Signal), false, nil)
+ return commitToHistory(&d.image, fmt.Sprintf("STOPSIGNAL %v", c.Signal), false, nil, d.epoch)
}
func dispatchShell(d *dispatchState, c *instructions.ShellCommand) error {
d.image.Config.Shell = c.Shell
- return commitToHistory(&d.image, fmt.Sprintf("SHELL %v", c.Shell), false, nil)
+ return commitToHistory(&d.image, fmt.Sprintf("SHELL %v", c.Shell), false, nil, d.epoch)
}
func dispatchArg(d *dispatchState, c *instructions.ArgCommand, metaArgs []instructions.KeyValuePairOptional, buildArgValues map[string]string) error {
@@ -1385,21 +1410,34 @@ func dispatchArg(d *dispatchState, c *instructions.ArgCommand, metaArgs []instru
commitStr += "=" + *arg.Value
}
commitStrs = append(commitStrs, commitStr)
+
+ skipArgInfo := false // skip the arg info if the arg is inherited from global scope
if buildArg.Value == nil {
for _, ma := range metaArgs {
if ma.Key == buildArg.Key {
buildArg.Value = ma.Value
+ skipArgInfo = true
}
}
}
+ ai := argInfo{definition: arg, location: c.Location()}
+
if buildArg.Value != nil {
- d.state = d.state.AddEnv(buildArg.Key, *buildArg.Value)
+ if _, ok := nonEnvArgs[buildArg.Key]; !ok {
+ d.state = d.state.AddEnv(buildArg.Key, *buildArg.Value)
+ }
+ ai.value = *buildArg.Value
+ }
+
+ if !skipArgInfo {
+ d.outline.allArgs[arg.Key] = ai
}
+ d.outline.usedArgs[arg.Key] = struct{}{}
d.buildArgs = append(d.buildArgs, buildArg)
}
- return commitToHistory(&d.image, "ARG "+strings.Join(commitStrs, " "), false, nil)
+ return commitToHistory(&d.image, "ARG "+strings.Join(commitStrs, " "), false, nil, d.epoch)
}
func pathRelativeToWorkingDir(s llb.State, p string) (string, error) {
@@ -1413,27 +1451,6 @@ func pathRelativeToWorkingDir(s llb.State, p string) (string, error) {
return path.Join(dir, p), nil
}
-func splitWildcards(name string) (string, string) {
- i := 0
- for ; i < len(name); i++ {
- ch := name[i]
- if ch == '\\' {
- i++
- } else if ch == '*' || ch == '?' || ch == '[' {
- break
- }
- }
- if i == len(name) {
- return name, ""
- }
-
- base := path.Base(name[:i])
- if name[:i] == "" || strings.HasSuffix(name[:i], string(filepath.Separator)) {
- base = ""
- }
- return path.Dir(name[:i]), base + name[i:]
-}
-
func addEnv(env []string, k, v string) []string {
gotOne := false
for i, envVar := range env {
@@ -1497,7 +1514,7 @@ func runCommandString(args []string, buildArgs []instructions.KeyValuePairOption
return strings.Join(append(tmpBuildEnv, args...), " ")
}
-func commitToHistory(img *Image, msg string, withLayer bool, st *llb.State) error {
+func commitToHistory(img *Image, msg string, withLayer bool, st *llb.State, tm *time.Time) error {
if st != nil {
msg += " # buildkit"
}
@@ -1506,6 +1523,7 @@ func commitToHistory(img *Image, msg string, withLayer bool, st *llb.State) erro
CreatedBy: msg,
Comment: historyComment,
EmptyLayer: !withLayer,
+ Created: tm,
})
return nil
}
@@ -1525,6 +1543,20 @@ func isReachable(from, to *dispatchState) (ret bool) {
return false
}
+func findReachable(from *dispatchState) (ret []*dispatchState) {
+ if from == nil {
+ return nil
+ }
+ ret = append(ret, from)
+ if from.base != nil {
+ ret = append(ret, findReachable(from.base)...)
+ }
+ for d := range from.deps {
+ ret = append(ret, findReachable(d)...)
+ }
+ return ret
+}
+
func hasCircularDependency(states []*dispatchState) (bool, *dispatchState) {
var visit func(state *dispatchState) bool
if states == nil {
@@ -1560,42 +1592,6 @@ func hasCircularDependency(states []*dispatchState) (bool, *dispatchState) {
return false, nil
}
-func parseUser(str string) (uid uint32, gid uint32, err error) {
- if str == "" {
- return 0, 0, nil
- }
- parts := strings.SplitN(str, ":", 2)
- for i, v := range parts {
- switch i {
- case 0:
- uid, err = parseUID(v)
- if err != nil {
- return 0, 0, err
- }
- if len(parts) == 1 {
- gid = uid
- }
- case 1:
- gid, err = parseUID(v)
- if err != nil {
- return 0, 0, err
- }
- }
- }
- return
-}
-
-func parseUID(str string) (uint32, error) {
- if str == "root" {
- return 0, nil
- }
- uid, err := strconv.ParseUint(str, 10, 32)
- if err != nil {
- return 0, err
- }
- return uint32(uid), nil
-}
-
func normalizeContextPaths(paths map[string]struct{}) []string {
pathSlice := make([]string, 0, len(paths))
for p := range paths {
@@ -1760,16 +1756,6 @@ func platformFromEnv(env []string) *ocispecs.Platform {
return &p
}
-func useFileOp(args map[string]string, caps *apicaps.CapSet) bool {
- enabled := true
- if v, ok := args["BUILDKIT_DISABLE_FILEOP"]; ok {
- if b, err := strconv.ParseBool(v); err == nil {
- enabled = !b
- }
- }
- return enabled && caps != nil && caps.Supports(pb.CapFileBase) == nil
-}
-
func location(sm *llb.SourceMap, locations []parser.Range) llb.ConstraintsOpt {
loc := make([]*pb.Range, 0, len(locations))
for _, l := range locations {
@@ -1807,3 +1793,36 @@ func commonImageNames() []string {
}
return out
}
+
+func clampTimes(img Image, tm *time.Time) Image {
+ if tm == nil {
+ return img
+ }
+ for i, h := range img.History {
+ if h.Created == nil || h.Created.After(*tm) {
+ img.History[i].Created = tm
+ }
+ }
+ if img.Created != nil && img.Created.After(*tm) {
+ img.Created = tm
+ }
+ return img
+}
+
+func isHTTPSource(src string) bool {
+ return strings.HasPrefix(src, "http://") || strings.HasPrefix(src, "https://")
+}
+
+func isEnabledForStage(stage string, value string) bool {
+ if enabled, err := strconv.ParseBool(value); err == nil {
+ return enabled
+ }
+
+ vv := strings.Split(value, ",")
+ for _, v := range vv {
+ if v == stage {
+ return true
+ }
+ }
+ return false
+}
diff --git a/frontend/dockerfile/dockerfile2llb/convert_addchecksum.go b/frontend/dockerfile/dockerfile2llb/convert_addchecksum.go
new file mode 100644
index 000000000000..4506baeb8ba8
--- /dev/null
+++ b/frontend/dockerfile/dockerfile2llb/convert_addchecksum.go
@@ -0,0 +1,6 @@
+//go:build dfaddchecksum
+// +build dfaddchecksum
+
+package dockerfile2llb
+
+const addChecksumEnabled = true
diff --git a/frontend/dockerfile/dockerfile2llb/convert_addgit.go b/frontend/dockerfile/dockerfile2llb/convert_addgit.go
new file mode 100644
index 000000000000..9ccb7a20e840
--- /dev/null
+++ b/frontend/dockerfile/dockerfile2llb/convert_addgit.go
@@ -0,0 +1,6 @@
+//go:build dfaddgit
+// +build dfaddgit
+
+package dockerfile2llb
+
+const addGitEnabled = true
diff --git a/frontend/dockerfile/dockerfile2llb/convert_noaddchecksum.go b/frontend/dockerfile/dockerfile2llb/convert_noaddchecksum.go
new file mode 100644
index 000000000000..8de035297c1b
--- /dev/null
+++ b/frontend/dockerfile/dockerfile2llb/convert_noaddchecksum.go
@@ -0,0 +1,6 @@
+//go:build !dfaddchecksum
+// +build !dfaddchecksum
+
+package dockerfile2llb
+
+const addChecksumEnabled = false
diff --git a/frontend/dockerfile/dockerfile2llb/convert_noaddgit.go b/frontend/dockerfile/dockerfile2llb/convert_noaddgit.go
new file mode 100644
index 000000000000..119bb32c8895
--- /dev/null
+++ b/frontend/dockerfile/dockerfile2llb/convert_noaddgit.go
@@ -0,0 +1,6 @@
+//go:build !dfaddgit
+// +build !dfaddgit
+
+package dockerfile2llb
+
+const addGitEnabled = false
diff --git a/frontend/dockerfile/dockerfile2llb/convert_runmount.go b/frontend/dockerfile/dockerfile2llb/convert_runmount.go
index 7777fba91ac9..1015590a0dc6 100644
--- a/frontend/dockerfile/dockerfile2llb/convert_runmount.go
+++ b/frontend/dockerfile/dockerfile2llb/convert_runmount.go
@@ -2,12 +2,9 @@ package dockerfile2llb
import (
"context"
- "fmt"
"os"
"path"
"path/filepath"
- "strconv"
- "strings"
"github.com/moby/buildkit/client/llb"
"github.com/moby/buildkit/frontend/dockerfile/instructions"
@@ -46,7 +43,7 @@ func detectRunMount(cmd *command, allDispatchStates *dispatchStates) bool {
return false
}
-func setCacheUIDGIDFileOp(m *instructions.Mount, st llb.State) llb.State {
+func setCacheUIDGID(m *instructions.Mount, st llb.State) llb.State {
uid := 0
gid := 0
mode := os.FileMode(0755)
@@ -62,24 +59,6 @@ func setCacheUIDGIDFileOp(m *instructions.Mount, st llb.State) llb.State {
return st.File(llb.Mkdir("/cache", mode, llb.WithUIDGID(uid, gid)), llb.WithCustomName("[internal] settings cache mount permissions"))
}
-func setCacheUIDGID(m *instructions.Mount, st llb.State, fileop bool) llb.State {
- if fileop {
- return setCacheUIDGIDFileOp(m, st)
- }
-
- var b strings.Builder
- if m.UID != nil {
- b.WriteString(fmt.Sprintf("chown %d /mnt/cache;", *m.UID))
- }
- if m.GID != nil {
- b.WriteString(fmt.Sprintf("chown :%d /mnt/cache;", *m.GID))
- }
- if m.Mode != nil {
- b.WriteString(fmt.Sprintf("chmod %s /mnt/cache;", strconv.FormatUint(*m.Mode, 8)))
- }
- return llb.Image("busybox").Run(llb.Shlex(fmt.Sprintf("sh -c 'mkdir -p /mnt/cache;%s'", b.String())), llb.WithCustomName("[internal] settings cache mount permissions")).AddMount("/mnt", st)
-}
-
func dispatchRunMounts(d *dispatchState, c *instructions.RunCommand, sources []*dispatchState, opt dispatchOpt) ([]llb.RunOption, error) {
var out []llb.RunOption
mounts := instructions.GetMounts(c)
@@ -100,7 +79,7 @@ func dispatchRunMounts(d *dispatchState, c *instructions.RunCommand, sources []*
))
}
if mount.Type == instructions.MountTypeSecret {
- secret, err := dispatchSecret(mount)
+ secret, err := dispatchSecret(d, mount, c.Location())
if err != nil {
return nil, err
}
@@ -108,7 +87,7 @@ func dispatchRunMounts(d *dispatchState, c *instructions.RunCommand, sources []*
continue
}
if mount.Type == instructions.MountTypeSSH {
- ssh, err := dispatchSSH(mount)
+ ssh, err := dispatchSSH(d, mount, c.Location())
if err != nil {
return nil, err
}
@@ -148,7 +127,7 @@ func dispatchRunMounts(d *dispatchState, c *instructions.RunCommand, sources []*
mountOpts = append(mountOpts, llb.SourcePath(src))
} else {
if mount.UID != nil || mount.GID != nil || mount.Mode != nil {
- st = setCacheUIDGID(mount, st, useFileOp(opt.buildArgValues, opt.llbCaps))
+ st = setCacheUIDGID(mount, st)
mountOpts = append(mountOpts, llb.SourcePath("/cache"))
}
}
diff --git a/frontend/dockerfile/dockerfile2llb/convert_secrets.go b/frontend/dockerfile/dockerfile2llb/convert_secrets.go
index 2c88a5e4f7e7..ced2bff1b070 100644
--- a/frontend/dockerfile/dockerfile2llb/convert_secrets.go
+++ b/frontend/dockerfile/dockerfile2llb/convert_secrets.go
@@ -5,10 +5,11 @@ import (
"github.com/moby/buildkit/client/llb"
"github.com/moby/buildkit/frontend/dockerfile/instructions"
+ "github.com/moby/buildkit/frontend/dockerfile/parser"
"github.com/pkg/errors"
)
-func dispatchSecret(m *instructions.Mount) (llb.RunOption, error) {
+func dispatchSecret(d *dispatchState, m *instructions.Mount, loc []parser.Range) (llb.RunOption, error) {
id := m.CacheID
if m.Source != "" {
id = m.Source
@@ -26,6 +27,13 @@ func dispatchSecret(m *instructions.Mount) (llb.RunOption, error) {
target = "/run/secrets/" + path.Base(id)
}
+ if _, ok := d.outline.secrets[id]; !ok {
+ d.outline.secrets[id] = secretInfo{
+ location: loc,
+ required: m.Required,
+ }
+ }
+
opts := []llb.SecretOption{llb.SecretID(id)}
if !m.Required {
diff --git a/frontend/dockerfile/dockerfile2llb/convert_ssh.go b/frontend/dockerfile/dockerfile2llb/convert_ssh.go
index b55659d97883..ab7aaa60127f 100644
--- a/frontend/dockerfile/dockerfile2llb/convert_ssh.go
+++ b/frontend/dockerfile/dockerfile2llb/convert_ssh.go
@@ -3,13 +3,26 @@ package dockerfile2llb
import (
"github.com/moby/buildkit/client/llb"
"github.com/moby/buildkit/frontend/dockerfile/instructions"
+ "github.com/moby/buildkit/frontend/dockerfile/parser"
"github.com/pkg/errors"
)
-func dispatchSSH(m *instructions.Mount) (llb.RunOption, error) {
+func dispatchSSH(d *dispatchState, m *instructions.Mount, loc []parser.Range) (llb.RunOption, error) {
if m.Source != "" {
return nil, errors.Errorf("ssh does not support source")
}
+
+ id := m.CacheID
+ if id == "" {
+ id = "default"
+ }
+ if _, ok := d.outline.ssh[id]; !ok {
+ d.outline.ssh[id] = sshInfo{
+ location: loc,
+ required: m.Required,
+ }
+ }
+
opts := []llb.SSHOption{llb.SSHID(m.CacheID)}
if m.Target != "" {
diff --git a/frontend/dockerfile/dockerfile2llb/convert_test.go b/frontend/dockerfile/dockerfile2llb/convert_test.go
index 5c1817addf91..ffcfee86fc15 100644
--- a/frontend/dockerfile/dockerfile2llb/convert_test.go
+++ b/frontend/dockerfile/dockerfile2llb/convert_test.go
@@ -1,16 +1,12 @@
package dockerfile2llb
import (
- "strings"
"testing"
"github.com/moby/buildkit/frontend/dockerfile/instructions"
"github.com/moby/buildkit/frontend/dockerfile/shell"
"github.com/moby/buildkit/util/appcontext"
- binfotypes "github.com/moby/buildkit/util/buildinfo/types"
- ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/stretchr/testify/assert"
- "github.com/stretchr/testify/require"
)
func toEnvMap(args []instructions.KeyValuePairOptional, env []string) map[string]string {
@@ -192,30 +188,3 @@ COPY --from=stage1 f2 /sub/
_, _, _, err = Dockerfile2LLB(appcontext.Context(), []byte(df), ConvertOpt{})
assert.EqualError(t, err, "circular dependency detected on stage: stage0")
}
-
-// moby/buildkit#2311
-func TestTargetBuildInfo(t *testing.T) {
- df := `
-FROM busybox
-ADD https://raw.githubusercontent.com/moby/buildkit/master/README.md /
-`
- _, _, bi, err := Dockerfile2LLB(appcontext.Context(), []byte(df), ConvertOpt{
- TargetPlatform: &ocispecs.Platform{
- Architecture: "amd64",
- OS: "linux",
- },
- BuildPlatforms: []ocispecs.Platform{
- {
- Architecture: "amd64",
- OS: "linux",
- },
- },
- })
- require.NoError(t, err)
-
- require.Equal(t, 1, len(bi.Sources))
- assert.Equal(t, binfotypes.SourceTypeDockerImage, bi.Sources[0].Type)
- assert.Equal(t, "busybox", bi.Sources[0].Ref)
- assert.True(t, strings.HasPrefix(bi.Sources[0].Alias, "docker.io/library/busybox@"))
- assert.NotEmpty(t, bi.Sources[0].Pin)
-}
diff --git a/frontend/dockerfile/dockerfile2llb/directives.go b/frontend/dockerfile/dockerfile2llb/directives.go
deleted file mode 100644
index 3cf982b9a9b3..000000000000
--- a/frontend/dockerfile/dockerfile2llb/directives.go
+++ /dev/null
@@ -1,55 +0,0 @@
-package dockerfile2llb
-
-import (
- "bufio"
- "io"
- "regexp"
- "strings"
-
- "github.com/moby/buildkit/frontend/dockerfile/parser"
-)
-
-const keySyntax = "syntax"
-
-var reDirective = regexp.MustCompile(`^#\s*([a-zA-Z][a-zA-Z0-9]*)\s*=\s*(.+?)\s*$`)
-
-type Directive struct {
- Name string
- Value string
- Location []parser.Range
-}
-
-func DetectSyntax(r io.Reader) (string, string, []parser.Range, bool) {
- directives := ParseDirectives(r)
- if len(directives) == 0 {
- return "", "", nil, false
- }
- v, ok := directives[keySyntax]
- if !ok {
- return "", "", nil, false
- }
- p := strings.SplitN(v.Value, " ", 2)
- return p[0], v.Value, v.Location, true
-}
-
-func ParseDirectives(r io.Reader) map[string]Directive {
- m := map[string]Directive{}
- s := bufio.NewScanner(r)
- var l int
- for s.Scan() {
- l++
- match := reDirective.FindStringSubmatch(s.Text())
- if len(match) == 0 {
- return m
- }
- m[strings.ToLower(match[1])] = Directive{
- Name: match[1],
- Value: match[2],
- Location: []parser.Range{{
- Start: parser.Position{Line: l},
- End: parser.Position{Line: l},
- }},
- }
- }
- return m
-}
diff --git a/frontend/dockerfile/dockerfile2llb/directives_test.go b/frontend/dockerfile/dockerfile2llb/directives_test.go
deleted file mode 100644
index 6f45b2903111..000000000000
--- a/frontend/dockerfile/dockerfile2llb/directives_test.go
+++ /dev/null
@@ -1,71 +0,0 @@
-package dockerfile2llb
-
-import (
- "bytes"
- "fmt"
- "testing"
-
- "github.com/stretchr/testify/require"
-)
-
-func TestDirectives(t *testing.T) {
- t.Parallel()
-
- dt := `#escape=\
-# key = FOO bar
-
-# smth
-`
-
- d := ParseDirectives(bytes.NewBuffer([]byte(dt)))
- require.Equal(t, len(d), 2, fmt.Sprintf("%+v", d))
-
- v, ok := d["escape"]
- require.True(t, ok)
- require.Equal(t, v.Value, "\\")
-
- v, ok = d["key"]
- require.True(t, ok)
- require.Equal(t, v.Value, "FOO bar")
-
- // for some reason Moby implementation in case insensitive for escape
- dt = `# EScape=\
-# KEY = FOO bar
-
-# smth
-`
-
- d = ParseDirectives(bytes.NewBuffer([]byte(dt)))
- require.Equal(t, len(d), 2, fmt.Sprintf("%+v", d))
-
- v, ok = d["escape"]
- require.True(t, ok)
- require.Equal(t, v.Value, "\\")
-
- v, ok = d["key"]
- require.True(t, ok)
- require.Equal(t, v.Value, "FOO bar")
-}
-
-func TestSyntaxDirective(t *testing.T) {
- t.Parallel()
-
- dt := `# syntax = dockerfile:experimental // opts
-FROM busybox
-`
-
- ref, cmdline, loc, ok := DetectSyntax(bytes.NewBuffer([]byte(dt)))
- require.True(t, ok)
- require.Equal(t, ref, "dockerfile:experimental")
- require.Equal(t, cmdline, "dockerfile:experimental // opts")
- require.Equal(t, 1, loc[0].Start.Line)
- require.Equal(t, 1, loc[0].End.Line)
-
- dt = `FROM busybox
-RUN ls
-`
- ref, cmdline, _, ok = DetectSyntax(bytes.NewBuffer([]byte(dt)))
- require.False(t, ok)
- require.Equal(t, ref, "")
- require.Equal(t, cmdline, "")
-}
diff --git a/frontend/dockerfile/dockerfile2llb/image.go b/frontend/dockerfile/dockerfile2llb/image.go
index d4c82700e3e4..36b27aa28aba 100644
--- a/frontend/dockerfile/dockerfile2llb/image.go
+++ b/frontend/dockerfile/dockerfile2llb/image.go
@@ -1,59 +1,14 @@
package dockerfile2llb
import (
- "time"
-
- "github.com/docker/docker/api/types/strslice"
+ "github.com/moby/buildkit/exporter/containerimage/image"
"github.com/moby/buildkit/util/system"
ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
)
-// HealthConfig holds configuration settings for the HEALTHCHECK feature.
-type HealthConfig struct {
- // Test is the test to perform to check that the container is healthy.
- // An empty slice means to inherit the default.
- // The options are:
- // {} : inherit healthcheck
- // {"NONE"} : disable healthcheck
- // {"CMD", args...} : exec arguments directly
- // {"CMD-SHELL", command} : run command with system's default shell
- Test []string `json:",omitempty"`
-
- // Zero means to inherit. Durations are expressed as integer nanoseconds.
- Interval time.Duration `json:",omitempty"` // Interval is the time to wait between checks.
- Timeout time.Duration `json:",omitempty"` // Timeout is the time to wait before considering the check to have hung.
- StartPeriod time.Duration `json:",omitempty"` // The start period for the container to initialize before the retries starts to count down.
-
- // Retries is the number of consecutive failures needed to consider a container as unhealthy.
- // Zero means inherit.
- Retries int `json:",omitempty"`
-}
-
-// ImageConfig is a docker compatible config for an image
-type ImageConfig struct {
- ocispecs.ImageConfig
-
- Healthcheck *HealthConfig `json:",omitempty"` // Healthcheck describes how to check the container is healthy
- ArgsEscaped bool `json:",omitempty"` // True if command is already escaped (Windows specific)
-
- // NetworkDisabled bool `json:",omitempty"` // Is network disabled
- // MacAddress string `json:",omitempty"` // Mac Address of the container
- OnBuild []string // ONBUILD metadata that were defined on the image Dockerfile
- StopTimeout *int `json:",omitempty"` // Timeout (in seconds) to stop a container
- Shell strslice.StrSlice `json:",omitempty"` // Shell for shell-form of RUN, CMD, ENTRYPOINT
-}
-
// Image is the JSON structure which describes some basic information about the image.
// This provides the `application/vnd.oci.image.config.v1+json` mediatype when marshalled to JSON.
-type Image struct {
- ocispecs.Image
-
- // Config defines the execution parameters which should be used as a base when running a container using the image.
- Config ImageConfig `json:"config,omitempty"`
-
- // Variant defines platform variant. To be added to OCI.
- Variant string `json:"variant,omitempty"`
-}
+type Image image.Image
func clone(src Image) Image {
img := src
@@ -69,8 +24,8 @@ func emptyImage(platform ocispecs.Platform) Image {
Image: ocispecs.Image{
Architecture: platform.Architecture,
OS: platform.OS,
+ Variant: platform.Variant,
},
- Variant: platform.Variant,
}
img.RootFS.Type = "layers"
img.Config.WorkingDir = "/"
diff --git a/frontend/dockerfile/dockerfile2llb/outline.go b/frontend/dockerfile/dockerfile2llb/outline.go
new file mode 100644
index 000000000000..f93c8961b2ec
--- /dev/null
+++ b/frontend/dockerfile/dockerfile2llb/outline.go
@@ -0,0 +1,210 @@
+package dockerfile2llb
+
+import (
+ "sort"
+
+ "github.com/moby/buildkit/frontend/dockerfile/instructions"
+ "github.com/moby/buildkit/frontend/dockerfile/parser"
+ "github.com/moby/buildkit/frontend/subrequests/outline"
+ pb "github.com/moby/buildkit/solver/pb"
+)
+
+type outlineCapture struct {
+ allArgs map[string]argInfo
+ usedArgs map[string]struct{}
+ secrets map[string]secretInfo
+ ssh map[string]sshInfo
+}
+
+type argInfo struct {
+ value string
+ definition instructions.KeyValuePairOptional
+ deps map[string]struct{}
+ location []parser.Range
+}
+
+type secretInfo struct {
+ required bool
+ location []parser.Range
+}
+
+type sshInfo struct {
+ required bool
+ location []parser.Range
+}
+
+func newOutlineCapture() outlineCapture {
+ return outlineCapture{
+ allArgs: map[string]argInfo{},
+ usedArgs: map[string]struct{}{},
+ secrets: map[string]secretInfo{},
+ ssh: map[string]sshInfo{},
+ }
+}
+
+func (o outlineCapture) clone() outlineCapture {
+ allArgs := map[string]argInfo{}
+ for k, v := range o.allArgs {
+ allArgs[k] = v
+ }
+ usedArgs := map[string]struct{}{}
+ for k := range o.usedArgs {
+ usedArgs[k] = struct{}{}
+ }
+ secrets := map[string]secretInfo{}
+ for k, v := range o.secrets {
+ secrets[k] = v
+ }
+ ssh := map[string]sshInfo{}
+ for k, v := range o.ssh {
+ ssh[k] = v
+ }
+ return outlineCapture{
+ allArgs: allArgs,
+ usedArgs: usedArgs,
+ secrets: secrets,
+ ssh: ssh,
+ }
+}
+
+func (o outlineCapture) markAllUsed(in map[string]struct{}) {
+ for k := range in {
+ if a, ok := o.allArgs[k]; ok {
+ o.markAllUsed(a.deps)
+ }
+ o.usedArgs[k] = struct{}{}
+ }
+}
+
+func (ds *dispatchState) args(visited map[string]struct{}) []outline.Arg {
+ ds.outline.markAllUsed(ds.outline.usedArgs)
+
+ args := make([]outline.Arg, 0, len(ds.outline.usedArgs))
+ for k := range ds.outline.usedArgs {
+ if a, ok := ds.outline.allArgs[k]; ok {
+ if _, ok := visited[k]; !ok {
+ args = append(args, outline.Arg{
+ Name: a.definition.Key,
+ Value: a.value,
+ Description: a.definition.Comment,
+ Location: toSourceLocation(a.location),
+ })
+ visited[k] = struct{}{}
+ }
+ }
+ }
+
+ if ds.base != nil {
+ args = append(args, ds.base.args(visited)...)
+ }
+ for d := range ds.deps {
+ args = append(args, d.args(visited)...)
+ }
+
+ return args
+}
+
+func (ds *dispatchState) secrets(visited map[string]struct{}) []outline.Secret {
+ secrets := make([]outline.Secret, 0, len(ds.outline.secrets))
+ for k, v := range ds.outline.secrets {
+ if _, ok := visited[k]; !ok {
+ secrets = append(secrets, outline.Secret{
+ Name: k,
+ Required: v.required,
+ Location: toSourceLocation(v.location),
+ })
+ visited[k] = struct{}{}
+ }
+ }
+ if ds.base != nil {
+ secrets = append(secrets, ds.base.secrets(visited)...)
+ }
+ for d := range ds.deps {
+ secrets = append(secrets, d.secrets(visited)...)
+ }
+ return secrets
+}
+
+func (ds *dispatchState) ssh(visited map[string]struct{}) []outline.SSH {
+ ssh := make([]outline.SSH, 0, len(ds.outline.secrets))
+ for k, v := range ds.outline.ssh {
+ if _, ok := visited[k]; !ok {
+ ssh = append(ssh, outline.SSH{
+ Name: k,
+ Required: v.required,
+ Location: toSourceLocation(v.location),
+ })
+ visited[k] = struct{}{}
+ }
+ }
+ if ds.base != nil {
+ ssh = append(ssh, ds.base.ssh(visited)...)
+ }
+ for d := range ds.deps {
+ ssh = append(ssh, d.ssh(visited)...)
+ }
+ return ssh
+}
+
+func (ds *dispatchState) Outline(dt []byte) outline.Outline {
+ args := ds.args(map[string]struct{}{})
+ sort.Slice(args, func(i, j int) bool {
+ return compLocation(args[i].Location, args[j].Location)
+ })
+
+ secrets := ds.secrets(map[string]struct{}{})
+ sort.Slice(secrets, func(i, j int) bool {
+ return compLocation(secrets[i].Location, secrets[j].Location)
+ })
+
+ ssh := ds.ssh(map[string]struct{}{})
+ sort.Slice(ssh, func(i, j int) bool {
+ return compLocation(ssh[i].Location, ssh[j].Location)
+ })
+
+ out := outline.Outline{
+ Name: ds.stage.Name,
+ Description: ds.stage.Comment,
+ Sources: [][]byte{dt},
+ Args: args,
+ Secrets: secrets,
+ SSH: ssh,
+ }
+
+ return out
+}
+
+func toSourceLocation(r []parser.Range) *pb.Location {
+ if len(r) == 0 {
+ return nil
+ }
+ arr := make([]*pb.Range, len(r))
+ for i, r := range r {
+ arr[i] = &pb.Range{
+ Start: pb.Position{
+ Line: int32(r.Start.Line),
+ Character: int32(r.Start.Character),
+ },
+ End: pb.Position{
+ Line: int32(r.End.Line),
+ Character: int32(r.End.Character),
+ },
+ }
+ }
+ return &pb.Location{Ranges: arr}
+}
+
+func compLocation(a, b *pb.Location) bool {
+ if a.SourceIndex != b.SourceIndex {
+ return a.SourceIndex < b.SourceIndex
+ }
+ linea := 0
+ lineb := 0
+ if len(a.Ranges) > 0 {
+ linea = int(a.Ranges[0].Start.Line)
+ }
+ if len(b.Ranges) > 0 {
+ lineb = int(b.Ranges[0].Start.Line)
+ }
+ return linea < lineb
+}
diff --git a/frontend/dockerfile/dockerfile_addchecksum_test.go b/frontend/dockerfile/dockerfile_addchecksum_test.go
new file mode 100644
index 000000000000..f34cf31a5038
--- /dev/null
+++ b/frontend/dockerfile/dockerfile_addchecksum_test.go
@@ -0,0 +1,175 @@
+//go:build dfaddchecksum
+// +build dfaddchecksum
+
+package dockerfile
+
+import (
+ "fmt"
+ "testing"
+
+ "github.com/containerd/continuity/fs/fstest"
+ "github.com/moby/buildkit/client"
+ "github.com/moby/buildkit/frontend/dockerfile/builder"
+ "github.com/moby/buildkit/identity"
+ "github.com/moby/buildkit/util/testutil/httpserver"
+ "github.com/moby/buildkit/util/testutil/integration"
+ digest "github.com/opencontainers/go-digest"
+ "github.com/stretchr/testify/require"
+)
+
+var addChecksumTests = integration.TestFuncs(
+ testAddChecksum,
+)
+
+func init() {
+ allTests = append(allTests, addChecksumTests...)
+}
+
+func testAddChecksum(t *testing.T, sb integration.Sandbox) {
+ f := getFrontend(t, sb)
+ f.RequiresBuildctl(t)
+
+ resp := httpserver.Response{
+ Etag: identity.NewID(),
+ Content: []byte("content1"),
+ }
+ server := httpserver.NewTestServer(map[string]httpserver.Response{
+ "/foo": resp,
+ })
+ defer server.Close()
+
+ c, err := client.New(sb.Context(), sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ t.Run("Valid", func(t *testing.T) {
+ dockerfile := []byte(fmt.Sprintf(`
+FROM scratch
+ADD --checksum=%s %s /tmp/foo
+`, digest.FromBytes(resp.Content).String(), server.URL+"/foo"))
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ }, nil)
+ require.NoError(t, err)
+ })
+ t.Run("DigestFromEnv", func(t *testing.T) {
+ dockerfile := []byte(fmt.Sprintf(`
+FROM scratch
+ENV DIGEST=%s
+ENV LINK=%s
+ADD --checksum=${DIGEST} ${LINK} /tmp/foo
+`, digest.FromBytes(resp.Content).String(), server.URL+"/foo"))
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ }, nil)
+ require.NoError(t, err)
+ })
+ t.Run("DigestMismatch", func(t *testing.T) {
+ dockerfile := []byte(fmt.Sprintf(`
+FROM scratch
+ADD --checksum=%s %s /tmp/foo
+`, digest.FromBytes(nil).String(), server.URL+"/foo"))
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ }, nil)
+ require.Error(t, err, "digest mismatch")
+ })
+ t.Run("DigestWithKnownButUnsupportedAlgoName", func(t *testing.T) {
+ dockerfile := []byte(fmt.Sprintf(`
+FROM scratch
+ADD --checksum=md5:7e55db001d319a94b0b713529a756623 %s /tmp/foo
+`, server.URL+"/foo"))
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ }, nil)
+ require.Error(t, err, "unsupported digest algorithm")
+ })
+ t.Run("DigestWithUnknownAlgoName", func(t *testing.T) {
+ dockerfile := []byte(fmt.Sprintf(`
+FROM scratch
+ADD --checksum=unknown:%s %s /tmp/foo
+`, digest.FromBytes(resp.Content).Encoded(), server.URL+"/foo"))
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ }, nil)
+ require.Error(t, err, "unsupported digest algorithm")
+ })
+ t.Run("DigestWithoutAlgoName", func(t *testing.T) {
+ dockerfile := []byte(fmt.Sprintf(`
+FROM scratch
+ADD --checksum=%s %s /tmp/foo
+`, digest.FromBytes(resp.Content).Encoded(), server.URL+"/foo"))
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ }, nil)
+ require.Error(t, err, "invalid checksum digest format")
+ })
+ t.Run("NonHTTPSource", func(t *testing.T) {
+ foo := []byte("local file")
+ dockerfile := []byte(fmt.Sprintf(`
+FROM scratch
+ADD --checksum=%s foo /tmp/foo
+`, digest.FromBytes(foo).String()))
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("foo", foo, 0600),
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ }, nil)
+ require.Error(t, err, "checksum can't be specified for non-HTTP sources")
+ })
+}
diff --git a/frontend/dockerfile/dockerfile_addgit_test.go b/frontend/dockerfile/dockerfile_addgit_test.go
new file mode 100644
index 000000000000..fa99dea5648c
--- /dev/null
+++ b/frontend/dockerfile/dockerfile_addgit_test.go
@@ -0,0 +1,115 @@
+//go:build dfaddgit
+// +build dfaddgit
+
+package dockerfile
+
+import (
+ "bytes"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "testing"
+ "text/template"
+
+ "github.com/containerd/continuity/fs/fstest"
+ "github.com/moby/buildkit/client"
+ "github.com/moby/buildkit/frontend/dockerfile/builder"
+ "github.com/moby/buildkit/util/testutil/integration"
+ "github.com/stretchr/testify/require"
+)
+
+var addGitTests = integration.TestFuncs(
+ testAddGit,
+)
+
+func init() {
+ allTests = append(allTests, addGitTests...)
+}
+
+func testAddGit(t *testing.T, sb integration.Sandbox) {
+ f := getFrontend(t, sb)
+
+ gitDir, err := os.MkdirTemp("", "buildkit")
+ require.NoError(t, err)
+ defer os.RemoveAll(gitDir)
+ gitCommands := []string{
+ "git init",
+ "git config --local user.email test",
+ "git config --local user.name test",
+ }
+ makeCommit := func(tag string) []string {
+ return []string{
+ "echo foo of " + tag + " >foo",
+ "git add foo",
+ "git commit -m " + tag,
+ "git tag " + tag,
+ }
+ }
+ gitCommands = append(gitCommands, makeCommit("v0.0.1")...)
+ gitCommands = append(gitCommands, makeCommit("v0.0.2")...)
+ gitCommands = append(gitCommands, makeCommit("v0.0.3")...)
+ gitCommands = append(gitCommands, "git update-server-info")
+ err = runShell(gitDir, gitCommands...)
+ require.NoError(t, err)
+
+ server := httptest.NewServer(http.FileServer(http.Dir(filepath.Join(gitDir))))
+ defer server.Close()
+ serverURL := server.URL
+ t.Logf("serverURL=%q", serverURL)
+
+ dockerfile, err := applyTemplate(`
+FROM alpine
+
+# Basic case
+ADD {{.ServerURL}}/.git#v0.0.1 /x
+RUN cd /x && \
+ [ "$(cat foo)" = "foo of v0.0.1" ]
+
+# Complicated case
+ARG REPO="{{.ServerURL}}/.git"
+ARG TAG="v0.0.2"
+ADD --keep-git-dir=true --chown=4242:8484 ${REPO}#${TAG} /buildkit-chowned
+RUN apk add git
+USER 4242
+RUN cd /buildkit-chowned && \
+ [ "$(cat foo)" = "foo of v0.0.2" ] && \
+ [ "$(stat -c %u foo)" = "4242" ] && \
+ [ "$(stat -c %g foo)" = "8484" ] && \
+ [ -z "$(git status -s)" ]
+`, map[string]string{
+ "ServerURL": serverURL,
+ })
+ require.NoError(t, err)
+ t.Logf("dockerfile=%s", dockerfile)
+
+ dir, err := integration.Tmpdir(t,
+ fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
+ )
+ require.NoError(t, err)
+ defer os.RemoveAll(dir)
+
+ c, err := client.New(sb.Context(), sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ }, nil)
+ require.NoError(t, err)
+}
+
+func applyTemplate(tmpl string, x interface{}) (string, error) {
+ var buf bytes.Buffer
+ parsed, err := template.New("").Parse(tmpl)
+ if err != nil {
+ return "", err
+ }
+ if err := parsed.Execute(&buf, x); err != nil {
+ return "", err
+ }
+ return buf.String(), nil
+}
diff --git a/frontend/dockerfile/dockerfile_buildinfo_test.go b/frontend/dockerfile/dockerfile_buildinfo_test.go
index 84e0c24622d5..ea3705920972 100644
--- a/frontend/dockerfile/dockerfile_buildinfo_test.go
+++ b/frontend/dockerfile/dockerfile_buildinfo_test.go
@@ -5,7 +5,7 @@ import (
"encoding/base64"
"encoding/json"
"fmt"
- "io/ioutil"
+ "io"
"net/http"
"net/http/httptest"
"os"
@@ -47,18 +47,16 @@ func testBuildInfoSources(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
f.RequiresBuildctl(t)
- gitDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(gitDir)
+ gitDir := t.TempDir()
dockerfile := `
FROM alpine:latest@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300 AS alpine
FROM busybox:latest
-ADD https://raw.githubusercontent.com/moby/moby/master/README.md /
+ADD https://user2:pw2@raw.githubusercontent.com/moby/moby/v20.10.21/README.md /
COPY --from=alpine /bin/busybox /alpine-busybox
`
- err = ioutil.WriteFile(filepath.Join(gitDir, "Dockerfile"), []byte(dockerfile), 0600)
+ err := os.WriteFile(filepath.Join(gitDir, "Dockerfile"), []byte(dockerfile), 0600)
require.NoError(t, err)
err = runShell(gitDir,
@@ -75,27 +73,35 @@ COPY --from=alpine /bin/busybox /alpine-busybox
server := httptest.NewServer(http.FileServer(http.Dir(filepath.Join(gitDir))))
defer server.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
-
- out := filepath.Join(destDir, "out.tar")
- outW, err := os.Create(out)
- require.NoError(t, err)
-
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- res, err := f.Solve(sb.Context(), c, client.SolveOpt{
- Exports: []client.ExportEntry{
- {
- Type: client.ExporterOCI,
- Output: fixedWriteCloser(outW),
+ var exports []client.ExportEntry
+ if integration.IsTestDockerdMoby(sb) {
+ exports = []client.ExportEntry{{
+ Type: "moby",
+ Attrs: map[string]string{
+ "name": "reg.dummy:5000/buildkit/test:latest",
},
- },
+ }}
+ } else {
+ exports = []client.ExportEntry{{
+ Type: client.ExporterOCI,
+ Attrs: map[string]string{},
+ Output: fixedWriteCloser(nopWriteCloser{io.Discard}),
+ }}
+ }
+
+ expectedURL := strings.Replace(server.URL, "http://", "http://xxxxx:xxxxx@", 1)
+ require.NotEqual(t, expectedURL, server.URL)
+ server.URL = strings.Replace(server.URL, "http://", "http://user:pass@", 1)
+
+ res, err := f.Solve(sb.Context(), c, client.SolveOpt{
+ Exports: exports,
FrontendAttrs: map[string]string{
- builder.DefaultLocalNameContext: server.URL + "/.git#buildinfo",
+ builder.DefaultLocalNameContext: server.URL + "/.git#buildinfo",
+ builder.DefaultLocalNameContext + ":foo": "https://foo:bar@example.invalid/foo.html",
},
}, nil)
require.NoError(t, err)
@@ -109,11 +115,20 @@ COPY --from=alpine /bin/busybox /alpine-busybox
require.NoError(t, err)
require.Contains(t, bi.Attrs, "context")
- require.Equal(t, server.URL+"/.git#buildinfo", *bi.Attrs["context"])
+ require.Equal(t, expectedURL+"/.git#buildinfo", *bi.Attrs["context"])
- sources := bi.Sources
- require.Equal(t, 3, len(sources))
+ require.Equal(t, "https://xxxxx:xxxxx@example.invalid/foo.html", *bi.Attrs["context:foo"])
+
+ _, isGateway := f.(*gatewayFrontend)
+ sources := bi.Sources
+ if isGateway {
+ require.Equal(t, 5, len(sources), "%+v", sources)
+ assert.Equal(t, binfotypes.SourceTypeDockerImage, sources[0].Type)
+ assert.Contains(t, sources[0].Ref, "buildkit_test")
+ sources = sources[1:]
+ }
+ require.Equal(t, 4, len(sources), "%+v", sources)
assert.Equal(t, binfotypes.SourceTypeDockerImage, sources[0].Type)
assert.Equal(t, "docker.io/library/alpine:latest@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300", sources[0].Ref)
assert.Equal(t, "sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300", sources[0].Pin)
@@ -122,9 +137,13 @@ COPY --from=alpine /bin/busybox /alpine-busybox
assert.Equal(t, "docker.io/library/busybox:latest", sources[1].Ref)
assert.NotEmpty(t, sources[1].Pin)
- assert.Equal(t, binfotypes.SourceTypeHTTP, sources[2].Type)
- assert.Equal(t, "https://raw.githubusercontent.com/moby/moby/master/README.md", sources[2].Ref)
- assert.Equal(t, "sha256:419455202b0ef97e480d7f8199b26a721a417818bc0e2d106975f74323f25e6c", sources[2].Pin)
+ assert.Equal(t, binfotypes.SourceTypeGit, sources[2].Type)
+ assert.Equal(t, expectedURL+"/.git#buildinfo", sources[2].Ref)
+ assert.NotEmpty(t, sources[2].Pin)
+
+ assert.Equal(t, binfotypes.SourceTypeHTTP, sources[3].Type)
+ assert.Equal(t, "https://xxxxx:xxxxx@raw.githubusercontent.com/moby/moby/v20.10.21/README.md", sources[3].Ref)
+ assert.Equal(t, "sha256:419455202b0ef97e480d7f8199b26a721a417818bc0e2d106975f74323f25e6c", sources[3].Pin)
}
func testBuildInfoSourcesNoop(t *testing.T, sb integration.Sandbox) {
@@ -135,31 +154,34 @@ func testBuildInfoSourcesNoop(t *testing.T, sb integration.Sandbox) {
FROM busybox:latest
`
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
-
- out := filepath.Join(destDir, "out.tar")
- outW, err := os.Create(out)
- require.NoError(t, err)
+ var exports []client.ExportEntry
+ if integration.IsTestDockerdMoby(sb) {
+ exports = []client.ExportEntry{{
+ Type: "moby",
+ Attrs: map[string]string{
+ "name": "reg.dummy:5000/buildkit/test:latest",
+ },
+ }}
+ } else {
+ exports = []client.ExportEntry{{
+ Type: client.ExporterOCI,
+ Attrs: map[string]string{},
+ Output: fixedWriteCloser(nopWriteCloser{io.Discard}),
+ }}
+ }
res, err := f.Solve(sb.Context(), c, client.SolveOpt{
- Exports: []client.ExportEntry{
- {
- Type: client.ExporterOCI,
- Output: fixedWriteCloser(outW),
- },
- },
+ Exports: exports,
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -176,8 +198,12 @@ FROM busybox:latest
require.NoError(t, err)
sources := bi.Sources
- require.Equal(t, 1, len(sources))
+ if _, isGateway := f.(*gatewayFrontend); isGateway {
+ require.Equal(t, 2, len(sources), "%+v", sources)
+ sources = sources[1:]
+ }
+ require.Equal(t, 1, len(sources))
assert.Equal(t, binfotypes.SourceTypeDockerImage, sources[0].Type)
assert.Equal(t, "docker.io/library/busybox:latest", sources[0].Ref)
assert.NotEmpty(t, sources[0].Pin)
@@ -194,34 +220,37 @@ ARG foo
RUN echo $foo
`
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
-
- out := filepath.Join(destDir, "out.tar")
- outW, err := os.Create(out)
- require.NoError(t, err)
+ var exports []client.ExportEntry
+ if integration.IsTestDockerdMoby(sb) {
+ exports = []client.ExportEntry{{
+ Type: "moby",
+ Attrs: map[string]string{
+ "name": "reg.dummy:5000/buildkit/test:latest",
+ },
+ }}
+ } else {
+ exports = []client.ExportEntry{{
+ Type: client.ExporterOCI,
+ Attrs: map[string]string{},
+ Output: fixedWriteCloser(nopWriteCloser{io.Discard}),
+ }}
+ }
res, err := f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
"build-arg:foo": "bar",
},
- Exports: []client.ExportEntry{
- {
- Type: client.ExporterOCI,
- Output: fixedWriteCloser(outW),
- },
- },
+ Exports: exports,
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -243,6 +272,7 @@ RUN echo $foo
// moby/buildkit#2476
func testBuildInfoMultiPlatform(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureOCIExporter, integration.FeatureMultiPlatform)
f := getFrontend(t, sb)
f.RequiresBuildctl(t)
@@ -250,27 +280,19 @@ func testBuildInfoMultiPlatform(t *testing.T, sb integration.Sandbox) {
FROM busybox:latest
ARG foo
RUN echo $foo
-ADD https://raw.githubusercontent.com/moby/moby/master/README.md /
+ADD https://raw.githubusercontent.com/moby/moby/v20.10.21/README.md /
`
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
-
- out := filepath.Join(destDir, "out.tar")
- outW, err := os.Create(out)
- require.NoError(t, err)
-
platforms := []string{"linux/amd64", "linux/arm64"}
res, err := f.Solve(sb.Context(), c, client.SolveOpt{
@@ -281,7 +303,7 @@ ADD https://raw.githubusercontent.com/moby/moby/master/README.md /
Exports: []client.ExportEntry{
{
Type: client.ExporterOCI,
- Output: fixedWriteCloser(outW),
+ Output: fixedWriteCloser(nopWriteCloser{io.Discard}),
},
},
LocalDirs: map[string]string{
@@ -303,15 +325,21 @@ ADD https://raw.githubusercontent.com/moby/moby/master/README.md /
require.Contains(t, bi.Attrs, "build-arg:foo")
require.Equal(t, "bar", *bi.Attrs["build-arg:foo"])
+ _, isGateway := f.(*gatewayFrontend)
+
sources := bi.Sources
- require.Equal(t, 2, len(sources))
+ if isGateway {
+ require.Equal(t, 3, len(sources), "%+v", sources)
+ sources = sources[1:]
+ }
+ require.Equal(t, 2, len(sources), "%+v", sources)
assert.Equal(t, binfotypes.SourceTypeDockerImage, sources[0].Type)
assert.Equal(t, "docker.io/library/busybox:latest", sources[0].Ref)
assert.NotEmpty(t, sources[0].Pin)
assert.Equal(t, binfotypes.SourceTypeHTTP, sources[1].Type)
- assert.Equal(t, "https://raw.githubusercontent.com/moby/moby/master/README.md", sources[1].Ref)
+ assert.Equal(t, "https://raw.githubusercontent.com/moby/moby/v20.10.21/README.md", sources[1].Ref)
assert.Equal(t, "sha256:419455202b0ef97e480d7f8199b26a721a417818bc0e2d106975f74323f25e6c", sources[1].Pin)
}
}
@@ -327,35 +355,38 @@ FROM scratch
COPY --from=base /out /
`
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
-
- out := filepath.Join(destDir, "out.tar")
- outW, err := os.Create(out)
- require.NoError(t, err)
+ var exports []client.ExportEntry
+ if integration.IsTestDockerdMoby(sb) {
+ exports = []client.ExportEntry{{
+ Type: "moby",
+ Attrs: map[string]string{
+ "name": "reg.dummy:5000/buildkit/test:latest",
+ },
+ }}
+ } else {
+ exports = []client.ExportEntry{{
+ Type: client.ExporterOCI,
+ Attrs: map[string]string{},
+ Output: fixedWriteCloser(nopWriteCloser{io.Discard}),
+ }}
+ }
res, err := f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
"build-arg:foo": "bar",
"context:busybox": "docker-image://alpine",
},
- Exports: []client.ExportEntry{
- {
- Type: client.ExporterOCI,
- Output: fixedWriteCloser(outW),
- },
- },
+ Exports: exports,
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -376,8 +407,15 @@ COPY --from=base /out /
require.Contains(t, bi.Attrs, "build-arg:foo")
require.Equal(t, "bar", *bi.Attrs["build-arg:foo"])
+ _, isGateway := f.(*gatewayFrontend)
+
sources := bi.Sources
- require.Equal(t, 1, len(sources))
+ if isGateway {
+ require.Equal(t, 2, len(sources), "%+v", sources)
+ sources = sources[1:]
+ } else {
+ require.Equal(t, 1, len(sources))
+ }
assert.Equal(t, binfotypes.SourceTypeDockerImage, sources[0].Type)
assert.Equal(t, "docker.io/library/alpine:latest", sources[0].Ref)
assert.NotEmpty(t, sources[0].Pin)
@@ -394,45 +432,48 @@ FROM scratch
COPY --from=base /o* /
`
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
-
- out := filepath.Join(destDir, "out.tar")
- outW, err := os.Create(out)
- require.NoError(t, err)
-
outf := []byte(`dummy-result`)
- dir2, err := tmpdir(
+ dir2, err := integration.Tmpdir(
+ t,
fstest.CreateFile("out", outf, 0600),
fstest.CreateFile("out2", outf, 0600),
fstest.CreateFile(".dockerignore", []byte("out2\n"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir2)
+
+ var exports []client.ExportEntry
+ if integration.IsTestDockerdMoby(sb) {
+ exports = []client.ExportEntry{{
+ Type: "moby",
+ Attrs: map[string]string{
+ "name": "reg.dummy:5000/buildkit/test:latest",
+ },
+ }}
+ } else {
+ exports = []client.ExportEntry{{
+ Type: client.ExporterOCI,
+ Attrs: map[string]string{},
+ Output: fixedWriteCloser(nopWriteCloser{io.Discard}),
+ }}
+ }
res, err := f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
"build-arg:foo": "bar",
"context:base": "local:basedir",
},
- Exports: []client.ExportEntry{
- {
- Type: client.ExporterOCI,
- Output: fixedWriteCloser(outW),
- },
- },
+ Exports: exports,
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -454,10 +495,17 @@ COPY --from=base /o* /
require.Contains(t, bi.Attrs, "build-arg:foo")
require.Equal(t, "bar", *bi.Attrs["build-arg:foo"])
- require.Equal(t, 0, len(bi.Sources))
+ _, isGateway := f.(*gatewayFrontend)
+ if isGateway {
+ require.Equal(t, 1, len(bi.Sources))
+ } else {
+ require.Equal(t, 0, len(bi.Sources))
+ }
}
func testBuildInfoDeps(t *testing.T, sb integration.Sandbox) {
+ t.Skip("deps temporarily disabled with SLSA provenance support")
+
ctx := sb.Context()
f := getFrontend(t, sb)
f.RequiresBuildctl(t)
@@ -469,15 +517,15 @@ func testBuildInfoDeps(t *testing.T, sb integration.Sandbox) {
dockerfile := []byte(`
FROM alpine
ENV FOO=bar
-ADD https://raw.githubusercontent.com/moby/moby/master/README.md /
+ADD https://raw.githubusercontent.com/moby/moby/v20.10.21/README.md /
RUN echo first > /out
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
dockerfile2 := []byte(`
FROM base AS build
@@ -486,11 +534,11 @@ FROM busybox
COPY --from=build /foo /out /
`)
- dir2, err := tmpdir(
+ dir2, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile2, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
b := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
res, err := f.SolveGateway(ctx, c, gateway.SolveRequest{})
@@ -545,9 +593,7 @@ COPY --from=build /foo /out /
return res, nil
}
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
res, err := c.Build(ctx, client.SolveOpt{
LocalDirs: map[string]string{
@@ -582,7 +628,7 @@ COPY --from=build /foo /out /
assert.NotEmpty(t, bi.Sources[0].Pin)
assert.Equal(t, binfotypes.SourceTypeHTTP, bi.Sources[1].Type)
- assert.Equal(t, "https://raw.githubusercontent.com/moby/moby/master/README.md", bi.Sources[1].Ref)
+ assert.Equal(t, "https://raw.githubusercontent.com/moby/moby/v20.10.21/README.md", bi.Sources[1].Ref)
assert.Equal(t, "sha256:419455202b0ef97e480d7f8199b26a721a417818bc0e2d106975f74323f25e6c", bi.Sources[1].Pin)
require.Contains(t, bi.Deps, "base")
@@ -594,6 +640,8 @@ COPY --from=build /foo /out /
}
func testBuildInfoDepsMultiPlatform(t *testing.T, sb integration.Sandbox) {
+ t.Skip("deps temporarily disabled with SLSA provenance support")
+
ctx := sb.Context()
f := getFrontend(t, sb)
f.RequiresBuildctl(t)
@@ -611,11 +659,11 @@ ENV FOO=bar-$TARGETARCH
RUN echo "foo $TARGETARCH" > /out
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
dockerfile2 := []byte(`
FROM base AS build
@@ -624,11 +672,11 @@ FROM busybox
COPY --from=build /foo /out /
`)
- dir2, err := tmpdir(
+ dir2, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile2, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
b := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
res, err := f.SolveGateway(ctx, c, gateway.SolveRequest{
@@ -691,9 +739,7 @@ COPY --from=build /foo /out /
return res, nil
}
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
res, err := c.Build(ctx, client.SolveOpt{
LocalDirs: map[string]string{
@@ -737,6 +783,8 @@ COPY --from=build /foo /out /
}
func testBuildInfoDepsMainNoSource(t *testing.T, sb integration.Sandbox) {
+ t.Skip("deps temporarily disabled with SLSA provenance support")
+
ctx := sb.Context()
f := getFrontend(t, sb)
f.RequiresBuildctl(t)
@@ -748,26 +796,26 @@ func testBuildInfoDepsMainNoSource(t *testing.T, sb integration.Sandbox) {
dockerfile := []byte(`
FROM alpine
ENV FOO=bar
-ADD https://raw.githubusercontent.com/moby/moby/master/README.md /
+ADD https://raw.githubusercontent.com/moby/moby/v20.10.21/README.md /
RUN echo first > /out
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
dockerfile2 := []byte(`
FROM base AS build
RUN echo "foo is $FOO" > /foo
`)
- dir2, err := tmpdir(
+ dir2, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile2, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
b := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
res, err := f.SolveGateway(ctx, c, gateway.SolveRequest{})
@@ -822,9 +870,7 @@ RUN echo "foo is $FOO" > /foo
return res, nil
}
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
res, err := c.Build(ctx, client.SolveOpt{
LocalDirs: map[string]string{
@@ -855,7 +901,7 @@ RUN echo "foo is $FOO" > /foo
require.Equal(t, 1, len(bi.Sources))
assert.Equal(t, binfotypes.SourceTypeHTTP, bi.Sources[0].Type)
- assert.Equal(t, "https://raw.githubusercontent.com/moby/moby/master/README.md", bi.Sources[0].Ref)
+ assert.Equal(t, "https://raw.githubusercontent.com/moby/moby/v20.10.21/README.md", bi.Sources[0].Ref)
assert.Equal(t, "sha256:419455202b0ef97e480d7f8199b26a721a417818bc0e2d106975f74323f25e6c", bi.Sources[0].Pin)
require.Contains(t, bi.Deps, "base")
diff --git a/frontend/dockerfile/dockerfile_heredoc_test.go b/frontend/dockerfile/dockerfile_heredoc_test.go
index 4be68738e982..cbb386b33e1c 100644
--- a/frontend/dockerfile/dockerfile_heredoc_test.go
+++ b/frontend/dockerfile/dockerfile_heredoc_test.go
@@ -2,7 +2,6 @@ package dockerfile
import (
"fmt"
- "io/ioutil"
"os"
"path/filepath"
"testing"
@@ -68,19 +67,17 @@ FROM scratch
COPY --from=build /dest /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -104,7 +101,7 @@ COPY --from=build /dest /
}
for name, content := range contents {
- dt, err := ioutil.ReadFile(filepath.Join(destDir, name))
+ dt, err := os.ReadFile(filepath.Join(destDir, name))
require.NoError(t, err)
require.Equal(t, content, string(dt))
}
@@ -141,19 +138,17 @@ COPY <<"EOF" rawslashfile3
EOF
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -169,31 +164,31 @@ EOF
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "quotefile"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "quotefile"))
require.NoError(t, err)
require.Equal(t, "\"quotes in file\"\n", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "slashfile1"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "slashfile1"))
require.NoError(t, err)
require.Equal(t, "\n", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "slashfile2"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "slashfile2"))
require.NoError(t, err)
require.Equal(t, "\\\n", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "slashfile3"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "slashfile3"))
require.NoError(t, err)
require.Equal(t, "$\n", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "rawslashfile1"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "rawslashfile1"))
require.NoError(t, err)
require.Equal(t, "\\\n", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "rawslashfile2"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "rawslashfile2"))
require.NoError(t, err)
require.Equal(t, "\\\\\n", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "rawslashfile3"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "rawslashfile3"))
require.NoError(t, err)
require.Equal(t, "\\$\n", string(dt))
}
@@ -213,19 +208,17 @@ FROM scratch
COPY --from=build /dest /dest
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -241,7 +234,7 @@ COPY --from=build /dest /dest
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "dest"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "dest"))
require.NoError(t, err)
require.Equal(t, "i am\nroot\n", string(dt))
}
@@ -263,19 +256,17 @@ FROM scratch
COPY --from=build /dest /dest
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -291,7 +282,7 @@ COPY --from=build /dest /dest
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "dest"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "dest"))
require.NoError(t, err)
require.Equal(t, "foo\n", string(dt))
}
@@ -314,19 +305,17 @@ FROM scratch
COPY --from=build /dest /dest
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -342,7 +331,7 @@ COPY --from=build /dest /dest
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "dest"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "dest"))
require.NoError(t, err)
require.Equal(t, "hello\nworld\n", string(dt))
}
@@ -379,19 +368,17 @@ FROM scratch
COPY --from=build /dest /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -415,7 +402,7 @@ COPY --from=build /dest /
}
for name, content := range contents {
- dt, err := ioutil.ReadFile(filepath.Join(destDir, name))
+ dt, err := os.ReadFile(filepath.Join(destDir, name))
require.NoError(t, err)
require.Equal(t, content, string(dt))
}
@@ -471,19 +458,17 @@ FROM scratch
COPY --from=build /dest /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -511,7 +496,7 @@ COPY --from=build /dest /
}
for name, content := range contents {
- dt, err := ioutil.ReadFile(filepath.Join(destDir, name))
+ dt, err := os.ReadFile(filepath.Join(destDir, name))
require.NoError(t, err)
require.Equal(t, content, string(dt))
}
@@ -567,19 +552,17 @@ FROM scratch
COPY --from=build /dest /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -610,13 +593,14 @@ COPY --from=build /dest /
}
for name, content := range contents {
- dt, err := ioutil.ReadFile(filepath.Join(destDir, name))
+ dt, err := os.ReadFile(filepath.Join(destDir, name))
require.NoError(t, err)
require.Equal(t, content, string(dt))
}
}
func testOnBuildHeredoc(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureDirectPush)
f := getFrontend(t, sb)
registry, err := sb.NewRegistry()
@@ -632,11 +616,11 @@ echo "hello world" >> /dest
EOF
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -660,31 +644,19 @@ EOF
}, nil)
require.NoError(t, err)
- dockerfile = []byte(fmt.Sprintf(`
- FROM %s
- `, target))
-
- dir, err = tmpdir(
- fstest.CreateFile("Dockerfile", dockerfile, 0600),
- )
- require.NoError(t, err)
- defer os.RemoveAll(dir)
-
dockerfile = []byte(fmt.Sprintf(`
FROM %s AS base
FROM scratch
COPY --from=base /dest /dest
`, target))
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -700,7 +672,7 @@ EOF
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "dest"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "dest"))
require.NoError(t, err)
require.Equal(t, "hello world\n", string(dt))
}
diff --git a/frontend/dockerfile/dockerfile_mount_test.go b/frontend/dockerfile/dockerfile_mount_test.go
index 731d5f2d78dc..3a9ec3875119 100644
--- a/frontend/dockerfile/dockerfile_mount_test.go
+++ b/frontend/dockerfile/dockerfile_mount_test.go
@@ -1,10 +1,8 @@
package dockerfile
import (
- "io/ioutil"
"os"
"path/filepath"
- "strconv"
"testing"
"github.com/containerd/continuity/fs/fstest"
@@ -26,12 +24,11 @@ var mountTests = integration.TestFuncs(
testMountFromError,
testMountInvalid,
testMountTmpfsSize,
+ testCacheMountUser,
)
func init() {
allTests = append(allTests, mountTests...)
-
- fileOpTests = append(fileOpTests, integration.TestFuncs(testCacheMountUser)...)
}
func testMountContext(t *testing.T, sb integration.Sandbox) {
@@ -42,12 +39,12 @@ FROM busybox
RUN --mount=target=/context [ "$(cat /context/testfile)" == "contents0" ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("testfile", []byte("contents0"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -71,11 +68,11 @@ RUN --mount=target=/mytmp,type=tmpfs touch /mytmp/foo
RUN [ ! -f /mytmp/foo ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -98,11 +95,11 @@ FROM scratch
RUN --mont=target=/mytmp,type=tmpfs /bin/true
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -123,11 +120,11 @@ RUN --mont=target=/mytmp,type=tmpfs /bin/true
RUN --mount=typ=tmpfs /bin/true
`)
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
LocalDirs: map[string]string{
@@ -144,11 +141,11 @@ RUN --mont=target=/mytmp,type=tmpfs /bin/true
RUN --mount=type=tmp /bin/true
`)
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
LocalDirs: map[string]string{
@@ -176,20 +173,18 @@ from scratch
COPY --from=second /unique /unique
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("cachebust", []byte("0"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -205,20 +200,18 @@ COPY --from=second /unique /unique
}, nil)
require.NoError(t, err)
- dt1, err := ioutil.ReadFile(filepath.Join(destDir, "unique"))
+ dt1, err := os.ReadFile(filepath.Join(destDir, "unique"))
require.NoError(t, err)
// repeat with changed file that should be still cached by content
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("cachebust", []byte("1"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
- destDir, err = ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -234,34 +227,30 @@ COPY --from=second /unique /unique
}, nil)
require.NoError(t, err)
- dt2, err := ioutil.ReadFile(filepath.Join(destDir, "unique"))
+ dt2, err := os.ReadFile(filepath.Join(destDir, "unique"))
require.NoError(t, err)
require.Equal(t, dt1, dt2)
}
func testCacheMountUser(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM busybox
RUN --mount=type=cache,target=/mycache,uid=1001,gid=1002,mode=0751 [ "$(stat -c "%u %g %f" /mycache)" == "1001 1002 41e9" ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -280,11 +269,11 @@ RUN --mount=type=cache,target=/mycache2 [ ! -f /mycache2/foo ]
RUN --mount=type=cache,target=/mycache [ -f /mycache/foo ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -309,11 +298,11 @@ RUN --mount=type=cache,target=/mycache touch /mycache/foo
RUN --mount=type=cache,target=$SOME_PATH [ -f $SOME_PATH/foo ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -338,11 +327,11 @@ RUN --mount=type=$MNT_TYPE,target=/mycache2 touch /mycache2/foo
RUN --mount=type=cache,target=/mycache2 [ -f /mycache2/foo ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -372,11 +361,11 @@ FROM stage1
RUN --mount=type=$MNT_TYPE2,id=$MNT_ID,target=/whatever [ -f /whatever/foo ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -403,11 +392,11 @@ RUN --mount=type=cache,id=mycache,target=/tmp/meta touch /tmp/meta/foo
RUN --mount=type=cache,id=mycache,target=$META_PATH [ -f /tmp/meta/foo ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -434,11 +423,11 @@ ENV ttt=test
RUN --mount=from=$ttt,type=cache,target=/tmp ls
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -464,19 +453,17 @@ FROM scratch
COPY --from=base /tmpfssize /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -492,7 +479,7 @@ COPY --from=base /tmpfssize /
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "tmpfssize"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "tmpfssize"))
require.NoError(t, err)
require.Contains(t, string(dt), `size=131072k`)
}
diff --git a/frontend/dockerfile/dockerfile_outline_test.go b/frontend/dockerfile/dockerfile_outline_test.go
new file mode 100644
index 000000000000..346fde14dff7
--- /dev/null
+++ b/frontend/dockerfile/dockerfile_outline_test.go
@@ -0,0 +1,309 @@
+package dockerfile
+
+import (
+ "context"
+ "encoding/json"
+ "os"
+ "testing"
+
+ "github.com/containerd/continuity/fs/fstest"
+ "github.com/moby/buildkit/client"
+ "github.com/moby/buildkit/frontend/dockerfile/builder"
+ gateway "github.com/moby/buildkit/frontend/gateway/client"
+ "github.com/moby/buildkit/frontend/subrequests"
+ "github.com/moby/buildkit/frontend/subrequests/outline"
+ "github.com/moby/buildkit/util/testutil/integration"
+ "github.com/pkg/errors"
+ "github.com/stretchr/testify/require"
+)
+
+var outlineTests = integration.TestFuncs(
+ testOutlineArgs,
+ testOutlineSecrets,
+ testOutlineDescribeDefinition,
+)
+
+func testOutlineArgs(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureFrontendOutline)
+ f := getFrontend(t, sb)
+ if _, ok := f.(*clientFrontend); !ok {
+ t.Skip("only test with client frontend")
+ }
+
+ dockerfile := []byte(`ARG inherited=box
+ARG inherited2=box2
+ARG unused=abc${inherited2}
+# sfx is a suffix
+ARG sfx="usy${inherited}"
+
+FROM b${sfx} AS first
+# this is not assigned to anything
+ARG FOO=123
+# BAR is a number
+ARG BAR=456
+RUN true
+
+FROM alpine${unused} AS second
+ARG BAZ
+RUN true
+
+FROM scratch AS third
+ARG ABC=a
+
+# target defines build target
+FROM third AS target
+COPY --from=first /etc/passwd /
+
+FROM second
+`)
+
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
+ )
+ require.NoError(t, err)
+ defer os.RemoveAll(dir)
+
+ c, err := client.New(sb.Context(), sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ destDir, err := os.MkdirTemp("", "buildkit")
+ require.NoError(t, err)
+ defer os.RemoveAll(destDir)
+
+ called := false
+ frontend := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
+ res, err := c.Solve(ctx, gateway.SolveRequest{
+ FrontendOpt: map[string]string{
+ "frontend.caps": "moby.buildkit.frontend.subrequests",
+ "requestid": "frontend.outline",
+ "build-arg:BAR": "678",
+ "target": "target",
+ },
+ Frontend: "dockerfile.v0",
+ })
+ require.NoError(t, err)
+
+ outline, err := unmarshalOutline(res)
+ require.NoError(t, err)
+
+ require.Equal(t, "target", outline.Name)
+ require.Equal(t, "defines build target", outline.Description)
+
+ require.Equal(t, 1, len(outline.Sources))
+ require.Equal(t, dockerfile, outline.Sources[0])
+
+ require.Equal(t, 5, len(outline.Args))
+
+ arg := outline.Args[0]
+ require.Equal(t, "inherited", arg.Name)
+ require.Equal(t, "box", arg.Value)
+ require.Equal(t, "", arg.Description)
+ require.Equal(t, int32(0), arg.Location.SourceIndex)
+ require.Equal(t, int32(1), arg.Location.Ranges[0].Start.Line)
+
+ arg = outline.Args[1]
+ require.Equal(t, "sfx", arg.Name)
+ require.Equal(t, "usybox", arg.Value)
+ require.Equal(t, "is a suffix", arg.Description)
+ require.Equal(t, int32(5), arg.Location.Ranges[0].Start.Line)
+
+ arg = outline.Args[2]
+ require.Equal(t, "FOO", arg.Name)
+ require.Equal(t, "123", arg.Value)
+ require.Equal(t, "", arg.Description)
+ require.Equal(t, int32(9), arg.Location.Ranges[0].Start.Line)
+
+ arg = outline.Args[3]
+ require.Equal(t, "BAR", arg.Name)
+ require.Equal(t, "678", arg.Value)
+ require.Equal(t, "is a number", arg.Description)
+
+ arg = outline.Args[4]
+ require.Equal(t, "ABC", arg.Name)
+ require.Equal(t, "a", arg.Value)
+
+ called = true
+ return nil, nil
+ }
+
+ _, err = c.Build(sb.Context(), client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ },
+ }, "", frontend, nil)
+ require.NoError(t, err)
+
+ require.True(t, called)
+}
+
+func testOutlineSecrets(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureFrontendOutline)
+ f := getFrontend(t, sb)
+ if _, ok := f.(*clientFrontend); !ok {
+ t.Skip("only test with client frontend")
+ }
+
+ dockerfile := []byte(`
+FROM busybox AS first
+RUN --mount=type=secret,target=/etc/passwd,required=true --mount=type=ssh true
+
+FROM alpine AS second
+RUN --mount=type=secret,id=unused --mount=type=ssh,id=ssh2 true
+
+FROM scratch AS third
+ARG BAR
+RUN --mount=type=secret,id=second${BAR} true
+
+FROM third AS target
+COPY --from=first /foo /
+RUN --mount=type=ssh,id=ssh3,required true
+
+FROM second
+`)
+
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
+ )
+ require.NoError(t, err)
+ defer os.RemoveAll(dir)
+
+ c, err := client.New(sb.Context(), sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ destDir, err := os.MkdirTemp("", "buildkit")
+ require.NoError(t, err)
+ defer os.RemoveAll(destDir)
+
+ called := false
+ frontend := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
+ res, err := c.Solve(ctx, gateway.SolveRequest{
+ FrontendOpt: map[string]string{
+ "frontend.caps": "moby.buildkit.frontend.subrequests",
+ "requestid": "frontend.outline",
+ "build-arg:BAR": "678",
+ "target": "target",
+ },
+ Frontend: "dockerfile.v0",
+ })
+ require.NoError(t, err)
+
+ outline, err := unmarshalOutline(res)
+ require.NoError(t, err)
+
+ require.Equal(t, 1, len(outline.Sources))
+ require.Equal(t, dockerfile, outline.Sources[0])
+
+ require.Equal(t, 2, len(outline.Secrets))
+
+ secret := outline.Secrets[0]
+ require.Equal(t, "passwd", secret.Name)
+ require.Equal(t, true, secret.Required)
+ require.Equal(t, int32(0), secret.Location.SourceIndex)
+ require.Equal(t, int32(3), secret.Location.Ranges[0].Start.Line)
+
+ secret = outline.Secrets[1]
+ require.Equal(t, "second678", secret.Name)
+ require.Equal(t, false, secret.Required)
+ require.Equal(t, int32(0), secret.Location.SourceIndex)
+ require.Equal(t, int32(10), secret.Location.Ranges[0].Start.Line)
+
+ require.Equal(t, 2, len(outline.SSH))
+
+ ssh := outline.SSH[0]
+ require.Equal(t, "default", ssh.Name)
+ require.Equal(t, false, ssh.Required)
+ require.Equal(t, int32(0), ssh.Location.SourceIndex)
+ require.Equal(t, int32(3), ssh.Location.Ranges[0].Start.Line)
+
+ ssh = outline.SSH[1]
+ require.Equal(t, "ssh3", ssh.Name)
+ require.Equal(t, true, ssh.Required)
+ require.Equal(t, int32(0), ssh.Location.SourceIndex)
+ require.Equal(t, int32(14), ssh.Location.Ranges[0].Start.Line)
+
+ called = true
+ return nil, nil
+ }
+
+ _, err = c.Build(sb.Context(), client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ },
+ }, "", frontend, nil)
+ require.NoError(t, err)
+
+ require.True(t, called)
+}
+
+func testOutlineDescribeDefinition(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureFrontendOutline)
+ f := getFrontend(t, sb)
+ if _, ok := f.(*clientFrontend); !ok {
+ t.Skip("only test with client frontend")
+ }
+
+ c, err := client.New(sb.Context(), sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ dockerfile := []byte(`
+FROM scratch
+COPY Dockerfile Dockerfile
+`)
+
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+ defer os.RemoveAll(dir)
+
+ called := false
+
+ frontend := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
+ reqs, err := subrequests.Describe(ctx, c)
+ require.NoError(t, err)
+
+ require.True(t, len(reqs) > 0)
+
+ hasOutline := false
+
+ for _, req := range reqs {
+ if req.Name != "frontend.outline" {
+ continue
+ }
+ hasOutline = true
+ require.Equal(t, subrequests.RequestType("rpc"), req.Type)
+ require.NotEqual(t, req.Version, "")
+ }
+ require.True(t, hasOutline)
+
+ called = true
+ return nil, nil
+ }
+
+ _, err = c.Build(sb.Context(), client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ },
+ }, "", frontend, nil)
+ require.NoError(t, err)
+
+ require.True(t, called)
+}
+
+func unmarshalOutline(res *gateway.Result) (*outline.Outline, error) {
+ dt, ok := res.Metadata["result.json"]
+ if !ok {
+ return nil, errors.Errorf("missing frontend.outline")
+ }
+ var o outline.Outline
+ if err := json.Unmarshal(dt, &o); err != nil {
+ return nil, err
+ }
+ return &o, nil
+}
diff --git a/frontend/dockerfile/dockerfile_provenance_test.go b/frontend/dockerfile/dockerfile_provenance_test.go
new file mode 100644
index 000000000000..835f72a52b86
--- /dev/null
+++ b/frontend/dockerfile/dockerfile_provenance_test.go
@@ -0,0 +1,909 @@
+package dockerfile
+
+import (
+ "context"
+ "encoding/json"
+ "net/http"
+ "net/http/httptest"
+ "net/url"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/containerd/containerd/platforms"
+ "github.com/containerd/continuity/fs/fstest"
+ intoto "github.com/in-toto/in-toto-golang/in_toto"
+ "github.com/moby/buildkit/client"
+ "github.com/moby/buildkit/client/llb"
+ "github.com/moby/buildkit/exporter/containerimage/exptypes"
+ "github.com/moby/buildkit/frontend/dockerfile/builder"
+ gateway "github.com/moby/buildkit/frontend/gateway/client"
+ "github.com/moby/buildkit/solver/llbsolver/provenance"
+ "github.com/moby/buildkit/util/contentutil"
+ "github.com/moby/buildkit/util/testutil"
+ "github.com/moby/buildkit/util/testutil/integration"
+ ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
+ "github.com/pkg/errors"
+ "github.com/stretchr/testify/require"
+)
+
+func testProvenanceAttestation(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureDirectPush, integration.FeatureProvenance)
+ ctx := sb.Context()
+
+ c, err := client.New(ctx, sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ registry, err := sb.NewRegistry()
+ if errors.Is(err, integration.ErrRequirements) {
+ t.Skip(err.Error())
+ }
+ require.NoError(t, err)
+
+ f := getFrontend(t, sb)
+
+ dockerfile := []byte(`
+FROM busybox:latest
+RUN echo "ok" > /foo
+`)
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ for _, mode := range []string{"", "min", "max"} {
+ t.Run(mode, func(t *testing.T) {
+ var target string
+ if target == "" {
+ target = registry + "/buildkit/testwithprovenance:none"
+ } else {
+ target = registry + "/buildkit/testwithprovenance:" + mode
+ }
+
+ provReq := ""
+ if mode != "" {
+ provReq = "mode=" + mode
+ }
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ FrontendAttrs: map[string]string{
+ "attest:provenance": provReq,
+ "build-arg:FOO": "bar",
+ "label:lbl": "abc",
+ "vcs:source": "https://user:pass@example.invalid/repo.git",
+ "vcs:revision": "123456",
+ "filename": "Dockerfile",
+ builder.DefaultLocalNameContext + ":foo": "https://foo:bar@example.invalid/foo.html",
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterImage,
+ Attrs: map[string]string{
+ "name": target,
+ "push": "true",
+ },
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+
+ desc, provider, err := contentutil.ProviderFromRef(target)
+ require.NoError(t, err)
+ imgs, err := testutil.ReadImages(sb.Context(), provider, desc)
+ require.NoError(t, err)
+ require.Equal(t, 2, len(imgs.Images))
+
+ img := imgs.Find(platforms.Format(platforms.Normalize(platforms.DefaultSpec())))
+ require.NotNil(t, img)
+ require.Equal(t, []byte("ok\n"), img.Layers[1]["foo"].Data)
+
+ att := imgs.Find("unknown/unknown")
+ require.NotNil(t, att)
+ require.Equal(t, att.Desc.Annotations["vnd.docker.reference.digest"], string(img.Desc.Digest))
+ require.Equal(t, att.Desc.Annotations["vnd.docker.reference.type"], "attestation-manifest")
+ var attest intoto.Statement
+ require.NoError(t, json.Unmarshal(att.LayersRaw[0], &attest))
+ require.Equal(t, "https://in-toto.io/Statement/v0.1", attest.Type)
+ require.Equal(t, "https://slsa.dev/provenance/v0.2", attest.PredicateType) // intentionally not const
+
+ type stmtT struct {
+ Predicate provenance.ProvenancePredicate `json:"predicate"`
+ }
+ var stmt stmtT
+ require.NoError(t, json.Unmarshal(att.LayersRaw[0], &stmt))
+ pred := stmt.Predicate
+
+ require.Equal(t, "https://mobyproject.org/buildkit@v1", pred.BuildType)
+ require.Equal(t, "", pred.Builder.ID)
+
+ require.Equal(t, "", pred.Invocation.ConfigSource.URI)
+
+ _, isClient := f.(*clientFrontend)
+ _, isGateway := f.(*gatewayFrontend)
+
+ args := pred.Invocation.Parameters.Args
+ if isClient {
+ require.Equal(t, "", pred.Invocation.Parameters.Frontend)
+ require.Equal(t, 0, len(args), "%v", args)
+ require.False(t, pred.Metadata.Completeness.Parameters)
+ require.Equal(t, "", pred.Invocation.ConfigSource.EntryPoint)
+ } else if isGateway {
+ require.Equal(t, "gateway.v0", pred.Invocation.Parameters.Frontend)
+
+ if mode == "max" || mode == "" {
+ require.Equal(t, 4, len(args), "%v", args)
+ require.True(t, pred.Metadata.Completeness.Parameters)
+
+ require.Equal(t, "bar", args["build-arg:FOO"])
+ require.Equal(t, "abc", args["label:lbl"])
+ require.Contains(t, args["source"], "buildkit_test/")
+ } else {
+ require.False(t, pred.Metadata.Completeness.Parameters)
+ require.Equal(t, 2, len(args), "%v", args)
+ require.Contains(t, args["source"], "buildkit_test/")
+ }
+ require.Equal(t, "https://xxxxx:xxxxx@example.invalid/foo.html", args["context:foo"])
+ } else {
+ require.Equal(t, "dockerfile.v0", pred.Invocation.Parameters.Frontend)
+
+ if mode == "max" || mode == "" {
+ require.Equal(t, 3, len(args))
+ require.True(t, pred.Metadata.Completeness.Parameters)
+
+ require.Equal(t, "bar", args["build-arg:FOO"])
+ require.Equal(t, "abc", args["label:lbl"])
+ } else {
+ require.False(t, pred.Metadata.Completeness.Parameters)
+ require.Equal(t, 1, len(args), "%v", args)
+ }
+ require.Equal(t, "https://xxxxx:xxxxx@example.invalid/foo.html", args["context:foo"])
+ }
+
+ expectedBase := "pkg:docker/busybox@latest?platform=" + url.PathEscape(platforms.Format(platforms.Normalize(platforms.DefaultSpec())))
+ if isGateway {
+ require.Equal(t, 2, len(pred.Materials), "%+v", pred.Materials)
+ require.Contains(t, pred.Materials[0].URI, "docker/buildkit_test")
+ require.Equal(t, expectedBase, pred.Materials[1].URI)
+ require.NotEmpty(t, pred.Materials[1].Digest["sha256"])
+ } else {
+ require.Equal(t, 1, len(pred.Materials), "%+v", pred.Materials)
+ require.Equal(t, expectedBase, pred.Materials[0].URI)
+ require.NotEmpty(t, pred.Materials[0].Digest["sha256"])
+ }
+
+ if !isClient {
+ require.Equal(t, "Dockerfile", pred.Invocation.ConfigSource.EntryPoint)
+ require.Equal(t, "https://xxxxx:xxxxx@example.invalid/repo.git", pred.Metadata.BuildKitMetadata.VCS["source"])
+ require.Equal(t, "123456", pred.Metadata.BuildKitMetadata.VCS["revision"])
+ }
+
+ require.NotEmpty(t, pred.Metadata.BuildInvocationID)
+
+ require.Equal(t, 2, len(pred.Invocation.Parameters.Locals), "%+v", pred.Invocation.Parameters.Locals)
+ require.Equal(t, "context", pred.Invocation.Parameters.Locals[0].Name)
+ require.Equal(t, "dockerfile", pred.Invocation.Parameters.Locals[1].Name)
+
+ require.NotNil(t, pred.Metadata.BuildFinishedOn)
+ require.True(t, time.Since(*pred.Metadata.BuildFinishedOn) < 5*time.Minute)
+ require.NotNil(t, pred.Metadata.BuildStartedOn)
+ require.True(t, time.Since(*pred.Metadata.BuildStartedOn) < 5*time.Minute)
+ require.True(t, pred.Metadata.BuildStartedOn.Before(*pred.Metadata.BuildFinishedOn))
+
+ require.True(t, pred.Metadata.Completeness.Environment)
+ require.Equal(t, platforms.Format(platforms.Normalize(platforms.DefaultSpec())), pred.Invocation.Environment.Platform)
+
+ require.False(t, pred.Metadata.Completeness.Materials)
+ require.False(t, pred.Metadata.Reproducible)
+ require.False(t, pred.Metadata.Hermetic)
+
+ if mode == "max" || mode == "" {
+ require.Equal(t, 2, len(pred.Metadata.BuildKitMetadata.Layers))
+ require.NotNil(t, pred.Metadata.BuildKitMetadata.Source)
+ require.Equal(t, "Dockerfile", pred.Metadata.BuildKitMetadata.Source.Infos[0].Filename)
+ require.Equal(t, dockerfile, pred.Metadata.BuildKitMetadata.Source.Infos[0].Data)
+ require.NotNil(t, pred.BuildConfig)
+
+ require.Equal(t, 3, len(pred.BuildConfig.Definition))
+ } else {
+ require.Equal(t, 0, len(pred.Metadata.BuildKitMetadata.Layers))
+ require.Nil(t, pred.Metadata.BuildKitMetadata.Source)
+ require.Nil(t, pred.BuildConfig)
+ }
+ })
+ }
+}
+
+func testGitProvenanceAttestation(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureDirectPush, integration.FeatureProvenance)
+ ctx := sb.Context()
+
+ c, err := client.New(ctx, sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ registry, err := sb.NewRegistry()
+ if errors.Is(err, integration.ErrRequirements) {
+ t.Skip(err.Error())
+ }
+ require.NoError(t, err)
+
+ f := getFrontend(t, sb)
+
+ dockerfile := []byte(`
+FROM busybox:latest
+RUN --network=none echo "git" > /foo
+COPY myapp.Dockerfile /
+`)
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("myapp.Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ err = runShell(dir,
+ "git init",
+ "git config --local user.email test",
+ "git config --local user.name test",
+ "git add myapp.Dockerfile",
+ "git commit -m initial",
+ "git branch v1",
+ "git update-server-info",
+ )
+ require.NoError(t, err)
+
+ cmd := exec.Command("git", "rev-parse", "v1")
+ cmd.Dir = dir
+ expectedGitSHA, err := cmd.Output()
+ require.NoError(t, err)
+
+ server := httptest.NewServer(http.FileServer(http.Dir(filepath.Join(dir))))
+ defer server.Close()
+
+ target := registry + "/buildkit/testwithprovenance:git"
+
+ // inject dummy credentials to test that they are masked
+ expectedURL := strings.Replace(server.URL, "http://", "http://xxxxx:xxxxx@", 1)
+ require.NotEqual(t, expectedURL, server.URL)
+ server.URL = strings.Replace(server.URL, "http://", "http://user:pass@", 1)
+
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ FrontendAttrs: map[string]string{
+ "context": server.URL + "/.git#v1",
+ "attest:provenance": "",
+ "filename": "myapp.Dockerfile",
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterImage,
+ Attrs: map[string]string{
+ "name": target,
+ "push": "true",
+ },
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+
+ desc, provider, err := contentutil.ProviderFromRef(target)
+ require.NoError(t, err)
+ imgs, err := testutil.ReadImages(sb.Context(), provider, desc)
+ require.NoError(t, err)
+ require.Equal(t, 2, len(imgs.Images))
+
+ img := imgs.Find(platforms.Format(platforms.Normalize(platforms.DefaultSpec())))
+ require.NotNil(t, img)
+ require.Equal(t, []byte("git\n"), img.Layers[1]["foo"].Data)
+
+ att := imgs.Find("unknown/unknown")
+ require.NotNil(t, att)
+ require.Equal(t, att.Desc.Annotations["vnd.docker.reference.digest"], string(img.Desc.Digest))
+ require.Equal(t, att.Desc.Annotations["vnd.docker.reference.type"], "attestation-manifest")
+ var attest intoto.Statement
+ require.NoError(t, json.Unmarshal(att.LayersRaw[0], &attest))
+ require.Equal(t, "https://in-toto.io/Statement/v0.1", attest.Type)
+ require.Equal(t, "https://slsa.dev/provenance/v0.2", attest.PredicateType) // intentionally not const
+
+ type stmtT struct {
+ Predicate provenance.ProvenancePredicate `json:"predicate"`
+ }
+ var stmt stmtT
+ require.NoError(t, json.Unmarshal(att.LayersRaw[0], &stmt))
+ pred := stmt.Predicate
+
+ _, isClient := f.(*clientFrontend)
+ _, isGateway := f.(*gatewayFrontend)
+
+ if isClient {
+ require.Empty(t, pred.Invocation.Parameters.Frontend)
+ require.Equal(t, "", pred.Invocation.ConfigSource.URI)
+ require.Equal(t, "", pred.Invocation.ConfigSource.EntryPoint)
+ } else {
+ require.NotEmpty(t, pred.Invocation.Parameters.Frontend)
+ require.Equal(t, expectedURL+"/.git#v1", pred.Invocation.ConfigSource.URI)
+ require.Equal(t, "myapp.Dockerfile", pred.Invocation.ConfigSource.EntryPoint)
+ }
+
+ expBase := "pkg:docker/busybox@latest?platform=" + url.PathEscape(platforms.Format(platforms.Normalize(platforms.DefaultSpec())))
+ if isGateway {
+ require.Equal(t, 3, len(pred.Materials), "%+v", pred.Materials)
+
+ require.Contains(t, pred.Materials[0].URI, "pkg:docker/buildkit_test/")
+ require.NotEmpty(t, pred.Materials[0].Digest)
+
+ require.Equal(t, expBase, pred.Materials[1].URI)
+ require.NotEmpty(t, pred.Materials[1].Digest["sha256"])
+
+ require.Equal(t, expectedURL+"/.git#v1", pred.Materials[2].URI)
+ require.Equal(t, strings.TrimSpace(string(expectedGitSHA)), pred.Materials[2].Digest["sha1"])
+ } else {
+ require.Equal(t, 2, len(pred.Materials), "%+v", pred.Materials)
+
+ require.Equal(t, expBase, pred.Materials[0].URI)
+ require.NotEmpty(t, pred.Materials[0].Digest["sha256"])
+
+ require.Equal(t, expectedURL+"/.git#v1", pred.Materials[1].URI)
+ require.Equal(t, strings.TrimSpace(string(expectedGitSHA)), pred.Materials[1].Digest["sha1"])
+ }
+
+ require.Equal(t, 0, len(pred.Invocation.Parameters.Locals))
+
+ require.True(t, pred.Metadata.Completeness.Materials)
+ require.True(t, pred.Metadata.Completeness.Environment)
+ require.True(t, pred.Metadata.Hermetic)
+
+ if isClient {
+ require.False(t, pred.Metadata.Completeness.Parameters)
+ } else {
+ require.True(t, pred.Metadata.Completeness.Parameters)
+ }
+ require.False(t, pred.Metadata.Reproducible)
+
+ require.Equal(t, 0, len(pred.Metadata.BuildKitMetadata.VCS), "%+v", pred.Metadata.BuildKitMetadata.VCS)
+}
+
+func testMultiPlatformProvenance(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureDirectPush, integration.FeatureMultiPlatform, integration.FeatureProvenance)
+ ctx := sb.Context()
+
+ c, err := client.New(ctx, sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ registry, err := sb.NewRegistry()
+ if errors.Is(err, integration.ErrRequirements) {
+ t.Skip(err.Error())
+ }
+ require.NoError(t, err)
+
+ f := getFrontend(t, sb)
+
+ dockerfile := []byte(`
+FROM busybox:latest
+ARG TARGETARCH
+RUN echo "ok-$TARGETARCH" > /foo
+`)
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ target := registry + "/buildkit/testmultiprovenance:latest"
+
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ FrontendAttrs: map[string]string{
+ "attest:provenance": "mode=max",
+ "build-arg:FOO": "bar",
+ "label:lbl": "abc",
+ "platform": "linux/amd64,linux/arm64",
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterImage,
+ Attrs: map[string]string{
+ "name": target,
+ "push": "true",
+ },
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+
+ desc, provider, err := contentutil.ProviderFromRef(target)
+ require.NoError(t, err)
+ imgs, err := testutil.ReadImages(sb.Context(), provider, desc)
+ require.NoError(t, err)
+ require.Equal(t, 4, len(imgs.Images))
+
+ _, isClient := f.(*clientFrontend)
+ _, isGateway := f.(*gatewayFrontend)
+
+ for _, p := range []string{"linux/amd64", "linux/arm64"} {
+ img := imgs.Find(p)
+ require.NotNil(t, img)
+ if p == "linux/amd64" {
+ require.Equal(t, []byte("ok-amd64\n"), img.Layers[1]["foo"].Data)
+ } else {
+ require.Equal(t, []byte("ok-arm64\n"), img.Layers[1]["foo"].Data)
+ }
+
+ att := imgs.FindAttestation(p)
+ require.NotNil(t, att)
+ require.Equal(t, att.Desc.Annotations["vnd.docker.reference.type"], "attestation-manifest")
+ var attest intoto.Statement
+ require.NoError(t, json.Unmarshal(att.LayersRaw[0], &attest))
+ require.Equal(t, "https://in-toto.io/Statement/v0.1", attest.Type)
+ require.Equal(t, "https://slsa.dev/provenance/v0.2", attest.PredicateType) // intentionally not const
+
+ type stmtT struct {
+ Predicate provenance.ProvenancePredicate `json:"predicate"`
+ }
+ var stmt stmtT
+ require.NoError(t, json.Unmarshal(att.LayersRaw[0], &stmt))
+ pred := stmt.Predicate
+
+ require.Equal(t, "https://mobyproject.org/buildkit@v1", pred.BuildType)
+ require.Equal(t, "", pred.Builder.ID)
+ require.Equal(t, "", pred.Invocation.ConfigSource.URI)
+
+ if isGateway {
+ require.Equal(t, 2, len(pred.Materials), "%+v", pred.Materials)
+ require.Contains(t, pred.Materials[0].URI, "buildkit_test")
+ require.Contains(t, pred.Materials[1].URI, "pkg:docker/busybox@latest")
+ require.Contains(t, pred.Materials[1].URI, url.PathEscape(p))
+ } else {
+ require.Equal(t, 1, len(pred.Materials), "%+v", pred.Materials)
+ require.Contains(t, pred.Materials[0].URI, "pkg:docker/busybox@latest")
+ require.Contains(t, pred.Materials[0].URI, url.PathEscape(p))
+ }
+
+ args := pred.Invocation.Parameters.Args
+ if isClient {
+ require.Equal(t, 0, len(args), "%+v", args)
+ } else if isGateway {
+ require.Equal(t, 3, len(args), "%+v", args)
+ require.Equal(t, "bar", args["build-arg:FOO"])
+ require.Equal(t, "abc", args["label:lbl"])
+ require.Contains(t, args["source"], "buildkit_test/")
+ } else {
+ require.Equal(t, 2, len(args), "%+v", args)
+ require.Equal(t, "bar", args["build-arg:FOO"])
+ require.Equal(t, "abc", args["label:lbl"])
+ }
+ }
+}
+
+func testClientFrontendProvenance(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureDirectPush, integration.FeatureProvenance)
+ // Building with client frontend does not capture frontend provenance
+ // because frontend runs in client, not in BuildKit.
+ // This test builds Dockerfile inside a client frontend ensuring that
+ // in that case frontend provenance is captured.
+ ctx := sb.Context()
+
+ c, err := client.New(ctx, sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ registry, err := sb.NewRegistry()
+ if errors.Is(err, integration.ErrRequirements) {
+ t.Skip(err.Error())
+ }
+ require.NoError(t, err)
+
+ target := registry + "/buildkit/clientprovenance:latest"
+
+ f := getFrontend(t, sb)
+
+ _, isClient := f.(*clientFrontend)
+ if !isClient {
+ t.Skip("not a client frontend")
+ }
+
+ dockerfile := []byte(`
+ FROM alpine as x86target
+ RUN echo "alpine" > /foo
+
+ FROM busybox:latest AS armtarget
+ RUN --network=none echo "bbox" > /foo
+ `)
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ frontend := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
+ st := llb.HTTP("https://raw.githubusercontent.com/moby/moby/v20.10.21/README.md")
+ def, err := st.Marshal(ctx)
+ if err != nil {
+ return nil, err
+ }
+ // This does not show up in provenance
+ res0, err := c.Solve(ctx, gateway.SolveRequest{
+ Definition: def.ToPB(),
+ })
+ if err != nil {
+ return nil, err
+ }
+ dt, err := res0.Ref.ReadFile(ctx, gateway.ReadRequest{
+ Filename: "README.md",
+ })
+ if err != nil {
+ return nil, err
+ }
+
+ res1, err := c.Solve(ctx, gateway.SolveRequest{
+ Frontend: "dockerfile.v0",
+ FrontendOpt: map[string]string{
+ "build-arg:FOO": string(dt[:3]),
+ "target": "armtarget",
+ },
+ })
+ if err != nil {
+ return nil, err
+ }
+
+ res2, err := c.Solve(ctx, gateway.SolveRequest{
+ Frontend: "dockerfile.v0",
+ FrontendOpt: map[string]string{
+ "build-arg:FOO": string(dt[4:8]),
+ "target": "x86target",
+ },
+ })
+ if err != nil {
+ return nil, err
+ }
+
+ res := gateway.NewResult()
+ res.AddRef("linux/arm64", res1.Ref)
+ res.AddRef("linux/amd64", res2.Ref)
+
+ pl, err := json.Marshal(exptypes.Platforms{
+ Platforms: []exptypes.Platform{
+ {
+ ID: "linux/arm64",
+ Platform: ocispecs.Platform{OS: "linux", Architecture: "arm64"},
+ },
+ {
+ ID: "linux/amd64",
+ Platform: ocispecs.Platform{OS: "linux", Architecture: "amd64"},
+ },
+ },
+ })
+ if err != nil {
+ return nil, err
+ }
+ res.AddMeta(exptypes.ExporterPlatformsKey, pl)
+
+ return res, nil
+ }
+
+ _, err = c.Build(sb.Context(), client.SolveOpt{
+ FrontendAttrs: map[string]string{
+ "attest:provenance": "mode=full",
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterImage,
+ Attrs: map[string]string{
+ "name": target,
+ "push": "true",
+ },
+ },
+ },
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ }, "", frontend, nil)
+ require.NoError(t, err)
+
+ desc, provider, err := contentutil.ProviderFromRef(target)
+ require.NoError(t, err)
+ imgs, err := testutil.ReadImages(sb.Context(), provider, desc)
+ require.NoError(t, err)
+ require.Equal(t, 4, len(imgs.Images))
+
+ img := imgs.Find("linux/arm64")
+ require.NotNil(t, img)
+ require.Equal(t, []byte("bbox\n"), img.Layers[1]["foo"].Data)
+
+ att := imgs.FindAttestation("linux/arm64")
+ require.NotNil(t, att)
+ require.Equal(t, att.Desc.Annotations["vnd.docker.reference.type"], "attestation-manifest")
+ var attest intoto.Statement
+ require.NoError(t, json.Unmarshal(att.LayersRaw[0], &attest))
+ require.Equal(t, "https://in-toto.io/Statement/v0.1", attest.Type)
+ require.Equal(t, "https://slsa.dev/provenance/v0.2", attest.PredicateType) // intentionally not const
+
+ type stmtT struct {
+ Predicate provenance.ProvenancePredicate `json:"predicate"`
+ }
+ var stmt stmtT
+ require.NoError(t, json.Unmarshal(att.LayersRaw[0], &stmt))
+ pred := stmt.Predicate
+
+ require.Equal(t, "https://mobyproject.org/buildkit@v1", pred.BuildType)
+ require.Equal(t, "", pred.Builder.ID)
+ require.Equal(t, "", pred.Invocation.ConfigSource.URI)
+
+ args := pred.Invocation.Parameters.Args
+ require.Equal(t, 2, len(args), "%+v", args)
+ require.Equal(t, "The", args["build-arg:FOO"])
+ require.Equal(t, "armtarget", args["target"])
+
+ require.Equal(t, 2, len(pred.Invocation.Parameters.Locals))
+ require.Equal(t, 1, len(pred.Materials))
+ require.Contains(t, pred.Materials[0].URI, "docker/busybox")
+
+ // amd64
+ img = imgs.Find("linux/amd64")
+ require.NotNil(t, img)
+ require.Equal(t, []byte("alpine\n"), img.Layers[1]["foo"].Data)
+
+ att = imgs.FindAttestation("linux/amd64")
+ require.NotNil(t, att)
+ require.Equal(t, att.Desc.Annotations["vnd.docker.reference.type"], "attestation-manifest")
+ attest = intoto.Statement{}
+ require.NoError(t, json.Unmarshal(att.LayersRaw[0], &attest))
+ require.Equal(t, "https://in-toto.io/Statement/v0.1", attest.Type)
+ require.Equal(t, "https://slsa.dev/provenance/v0.2", attest.PredicateType) // intentionally not const
+
+ stmt = stmtT{}
+ require.NoError(t, json.Unmarshal(att.LayersRaw[0], &stmt))
+ pred = stmt.Predicate
+
+ require.Equal(t, "https://mobyproject.org/buildkit@v1", pred.BuildType)
+ require.Equal(t, "", pred.Builder.ID)
+ require.Equal(t, "", pred.Invocation.ConfigSource.URI)
+
+ args = pred.Invocation.Parameters.Args
+ require.Equal(t, 2, len(args), "%+v", args)
+ require.Equal(t, "Moby", args["build-arg:FOO"])
+ require.Equal(t, "x86target", args["target"])
+
+ require.Equal(t, 2, len(pred.Invocation.Parameters.Locals))
+ require.Equal(t, 1, len(pred.Materials))
+ require.Contains(t, pred.Materials[0].URI, "docker/alpine")
+}
+
+func testClientLLBProvenance(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureDirectPush, integration.FeatureProvenance)
+ ctx := sb.Context()
+
+ c, err := client.New(ctx, sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ registry, err := sb.NewRegistry()
+ if errors.Is(err, integration.ErrRequirements) {
+ t.Skip(err.Error())
+ }
+ require.NoError(t, err)
+
+ target := registry + "/buildkit/clientprovenance:llb"
+
+ f := getFrontend(t, sb)
+
+ _, isClient := f.(*clientFrontend)
+ if !isClient {
+ t.Skip("not a client frontend")
+ }
+
+ frontend := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
+ st := llb.HTTP("https://raw.githubusercontent.com/moby/moby/v20.10.21/README.md")
+ def, err := st.Marshal(ctx)
+ if err != nil {
+ return nil, err
+ }
+ // this also shows up in the provenance
+ res0, err := c.Solve(ctx, gateway.SolveRequest{
+ Definition: def.ToPB(),
+ })
+ if err != nil {
+ return nil, err
+ }
+ dt, err := res0.Ref.ReadFile(ctx, gateway.ReadRequest{
+ Filename: "README.md",
+ })
+ if err != nil {
+ return nil, err
+ }
+
+ st = llb.Image("alpine").File(llb.Mkfile("/foo", 0600, dt))
+ def, err = st.Marshal(ctx)
+ if err != nil {
+ return nil, err
+ }
+ res1, err := c.Solve(ctx, gateway.SolveRequest{
+ Definition: def.ToPB(),
+ })
+ if err != nil {
+ return nil, err
+ }
+ return res1, nil
+ }
+
+ _, err = c.Build(sb.Context(), client.SolveOpt{
+ FrontendAttrs: map[string]string{
+ "attest:provenance": "mode=full",
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterImage,
+ Attrs: map[string]string{
+ "name": target,
+ "push": "true",
+ },
+ },
+ },
+ LocalDirs: map[string]string{},
+ }, "", frontend, nil)
+ require.NoError(t, err)
+
+ desc, provider, err := contentutil.ProviderFromRef(target)
+ require.NoError(t, err)
+ imgs, err := testutil.ReadImages(sb.Context(), provider, desc)
+ require.NoError(t, err)
+ require.Equal(t, 2, len(imgs.Images))
+
+ nativePlatform := platforms.Format(platforms.Normalize(platforms.DefaultSpec()))
+
+ img := imgs.Find(nativePlatform)
+ require.NotNil(t, img)
+ require.Contains(t, string(img.Layers[1]["foo"].Data), "The Moby Project")
+
+ att := imgs.FindAttestation(nativePlatform)
+ require.NotNil(t, att)
+ require.Equal(t, att.Desc.Annotations["vnd.docker.reference.type"], "attestation-manifest")
+ var attest intoto.Statement
+ require.NoError(t, json.Unmarshal(att.LayersRaw[0], &attest))
+ require.Equal(t, "https://in-toto.io/Statement/v0.1", attest.Type)
+ require.Equal(t, "https://slsa.dev/provenance/v0.2", attest.PredicateType) // intentionally not const
+
+ type stmtT struct {
+ Predicate provenance.ProvenancePredicate `json:"predicate"`
+ }
+ var stmt stmtT
+ require.NoError(t, json.Unmarshal(att.LayersRaw[0], &stmt))
+ pred := stmt.Predicate
+
+ require.Equal(t, "https://mobyproject.org/buildkit@v1", pred.BuildType)
+ require.Equal(t, "", pred.Builder.ID)
+ require.Equal(t, "", pred.Invocation.ConfigSource.URI)
+
+ args := pred.Invocation.Parameters.Args
+ require.Equal(t, 0, len(args), "%+v", args)
+ require.Equal(t, 0, len(pred.Invocation.Parameters.Locals))
+
+ require.Equal(t, 2, len(pred.Materials), "%+v", pred.Materials)
+ require.Contains(t, pred.Materials[0].URI, "docker/alpine")
+ require.Contains(t, pred.Materials[1].URI, "README.md")
+}
+
+func testSecretSSHProvenance(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureDirectPush, integration.FeatureProvenance)
+ ctx := sb.Context()
+
+ c, err := client.New(ctx, sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ registry, err := sb.NewRegistry()
+ if errors.Is(err, integration.ErrRequirements) {
+ t.Skip(err.Error())
+ }
+ require.NoError(t, err)
+
+ f := getFrontend(t, sb)
+
+ dockerfile := []byte(`
+FROM busybox:latest
+RUN --mount=type=secret,id=mysecret --mount=type=secret,id=othersecret --mount=type=ssh echo "ok" > /foo
+`)
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ target := registry + "/buildkit/testsecretprovenance:latest"
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ FrontendAttrs: map[string]string{
+ "attest:provenance": "mode=max",
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterImage,
+ Attrs: map[string]string{
+ "name": target,
+ "push": "true",
+ },
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+
+ desc, provider, err := contentutil.ProviderFromRef(target)
+ require.NoError(t, err)
+ imgs, err := testutil.ReadImages(sb.Context(), provider, desc)
+ require.NoError(t, err)
+ require.Equal(t, 2, len(imgs.Images))
+
+ expPlatform := platforms.Format(platforms.Normalize(platforms.DefaultSpec()))
+
+ img := imgs.Find(expPlatform)
+ require.NotNil(t, img)
+ require.Equal(t, []byte("ok\n"), img.Layers[1]["foo"].Data)
+
+ att := imgs.FindAttestation(expPlatform)
+ type stmtT struct {
+ Predicate provenance.ProvenancePredicate `json:"predicate"`
+ }
+ var stmt stmtT
+ require.NoError(t, json.Unmarshal(att.LayersRaw[0], &stmt))
+ pred := stmt.Predicate
+
+ require.Equal(t, 2, len(pred.Invocation.Parameters.Secrets), "%+v", pred.Invocation.Parameters.Secrets)
+ require.Equal(t, "mysecret", pred.Invocation.Parameters.Secrets[0].ID)
+ require.True(t, pred.Invocation.Parameters.Secrets[0].Optional)
+ require.Equal(t, "othersecret", pred.Invocation.Parameters.Secrets[1].ID)
+ require.True(t, pred.Invocation.Parameters.Secrets[1].Optional)
+
+ require.Equal(t, 1, len(pred.Invocation.Parameters.SSH), "%+v", pred.Invocation.Parameters.SSH)
+ require.Equal(t, "default", pred.Invocation.Parameters.SSH[0].ID)
+ require.True(t, pred.Invocation.Parameters.SSH[0].Optional)
+}
+
+func testNilProvenance(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureProvenance)
+ ctx := sb.Context()
+
+ c, err := client.New(ctx, sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ f := getFrontend(t, sb)
+
+ dockerfile := []byte(`
+FROM scratch
+ENV FOO=bar
+`)
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ FrontendAttrs: map[string]string{
+ "attest:provenance": "mode=max",
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterImage,
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+}
diff --git a/frontend/dockerfile/dockerfile_runnetwork_test.go b/frontend/dockerfile/dockerfile_runnetwork_test.go
index 8244d48f5716..2d02110e383c 100644
--- a/frontend/dockerfile/dockerfile_runnetwork_test.go
+++ b/frontend/dockerfile/dockerfile_runnetwork_test.go
@@ -41,11 +41,11 @@ FROM busybox
RUN ip link show eth0
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -77,11 +77,11 @@ RUN --network=none ! ip link show eth0
dockerfile += "RUN ip link show eth0"
}
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -118,11 +118,11 @@ RUN --network=host nc 127.0.0.1 %s | grep foo
dockerfile += fmt.Sprintf(`RUN ! nc 127.0.0.1 %s | grep foo`, port)
}
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -141,8 +141,12 @@ RUN --network=host nc 127.0.0.1 %s | grep foo
case networkHostGranted:
require.NoError(t, err)
case networkHostDenied:
- require.Error(t, err)
- require.Contains(t, err.Error(), "entitlement network.host is not allowed")
+ if !integration.IsTestDockerd() {
+ require.Error(t, err)
+ require.Contains(t, err.Error(), "entitlement network.host is not allowed")
+ } else {
+ require.NoError(t, err)
+ }
default:
require.Fail(t, "unexpected network.host mode %q", hostAllowed)
}
@@ -162,11 +166,11 @@ RUN nc 127.0.0.1 %s | grep foo
RUN --network=none ! nc -z 127.0.0.1 %s
`, port, port)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -188,8 +192,12 @@ RUN --network=none ! nc -z 127.0.0.1 %s
case networkHostGranted:
require.NoError(t, err)
case networkHostDenied:
- require.Error(t, err)
- require.Contains(t, err.Error(), "entitlement network.host is not allowed")
+ if !integration.IsTestDockerd() {
+ require.Error(t, err)
+ require.Contains(t, err.Error(), "entitlement network.host is not allowed")
+ } else {
+ require.NoError(t, err)
+ }
default:
require.Fail(t, "unexpected network.host mode %q", hostAllowed)
}
diff --git a/frontend/dockerfile/dockerfile_runsecurity_test.go b/frontend/dockerfile/dockerfile_runsecurity_test.go
index 4726182a9d71..99e3b3c49d38 100644
--- a/frontend/dockerfile/dockerfile_runsecurity_test.go
+++ b/frontend/dockerfile/dockerfile_runsecurity_test.go
@@ -4,7 +4,6 @@
package dockerfile
import (
- "os"
"testing"
"github.com/containerd/continuity/fs/fstest"
@@ -58,11 +57,11 @@ RUN --security=insecure ls -l /dev && dd if=/dev/zero of=disk.img bs=20M count=1
rm disk.img
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -97,11 +96,11 @@ RUN --security=insecure [ "$(printf '%x' $(( $(cat /proc/self/status | grep CapB
RUN [ "$(cat /proc/self/status | grep CapBnd)" == "CapBnd: 00000000a80425fb" ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -135,11 +134,11 @@ FROM busybox
RUN --security=sandbox [ "$(cat /proc/self/status | grep CapBnd)" == "CapBnd: 00000000a80425fb" ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -163,11 +162,11 @@ FROM busybox
RUN [ "$(cat /proc/self/status | grep CapBnd)" == "CapBnd: 00000000a80425fb" ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
diff --git a/frontend/dockerfile/dockerfile_secrets_test.go b/frontend/dockerfile/dockerfile_secrets_test.go
index ae00fac07df6..984bfacfe487 100644
--- a/frontend/dockerfile/dockerfile_secrets_test.go
+++ b/frontend/dockerfile/dockerfile_secrets_test.go
@@ -1,7 +1,6 @@
package dockerfile
import (
- "os"
"testing"
"github.com/containerd/continuity/fs/fstest"
@@ -31,11 +30,11 @@ RUN --mount=type=secret,required=false,mode=741,uid=100,gid=102,target=/mysecret
RUN [ ! -f /mysecret ] # check no stub left behind
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -61,11 +60,11 @@ FROM busybox
RUN --mount=type=secret,required,id=mysecret foo
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
diff --git a/frontend/dockerfile/dockerfile_ssh_test.go b/frontend/dockerfile/dockerfile_ssh_test.go
index 9515aad12882..0714578e1bcb 100644
--- a/frontend/dockerfile/dockerfile_ssh_test.go
+++ b/frontend/dockerfile/dockerfile_ssh_test.go
@@ -5,7 +5,6 @@ import (
"crypto/rsa"
"crypto/x509"
"encoding/pem"
- "io/ioutil"
"os"
"path/filepath"
"testing"
@@ -35,17 +34,17 @@ FROM busybox
RUN --mount=type=ssh,mode=741,uid=100,gid=102 [ "$(stat -c "%u %g %f" $SSH_AUTH_SOCK)" = "100 102 c1e1" ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- k, err := rsa.GenerateKey(rand.Reader, 1024)
+ k, err := rsa.GenerateKey(rand.Reader, 2048)
require.NoError(t, err)
dt := pem.EncodeToMemory(
@@ -55,11 +54,9 @@ RUN --mount=type=ssh,mode=741,uid=100,gid=102 [ "$(stat -c "%u %g %f" $SSH_AUTH_
},
)
- tmpDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(tmpDir)
+ tmpDir := t.TempDir()
- err = ioutil.WriteFile(filepath.Join(tmpDir, "key"), dt, 0600)
+ err = os.WriteFile(filepath.Join(tmpDir, "key"), dt, 0600)
require.NoError(t, err)
ssh, err := sshprovider.NewSSHAgentProvider([]sshprovider.AgentConfig{{
diff --git a/frontend/dockerfile/dockerfile_targets_test.go b/frontend/dockerfile/dockerfile_targets_test.go
new file mode 100644
index 000000000000..43e473c40da7
--- /dev/null
+++ b/frontend/dockerfile/dockerfile_targets_test.go
@@ -0,0 +1,193 @@
+package dockerfile
+
+import (
+ "context"
+ "encoding/json"
+ "os"
+ "testing"
+
+ "github.com/containerd/continuity/fs/fstest"
+ "github.com/moby/buildkit/client"
+ "github.com/moby/buildkit/frontend/dockerfile/builder"
+ gateway "github.com/moby/buildkit/frontend/gateway/client"
+ "github.com/moby/buildkit/frontend/subrequests"
+ "github.com/moby/buildkit/frontend/subrequests/targets"
+ "github.com/moby/buildkit/util/testutil/integration"
+ "github.com/pkg/errors"
+ "github.com/stretchr/testify/require"
+)
+
+var targetsTests = integration.TestFuncs(
+ testTargetsList,
+ testTargetsDescribeDefinition,
+)
+
+func testTargetsList(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureFrontendTargets)
+ f := getFrontend(t, sb)
+ if _, ok := f.(*clientFrontend); !ok {
+ t.Skip("only test with client frontend")
+ }
+
+ dockerfile := []byte(`
+# build defines stage for compiling the binary
+FROM alpine AS build
+RUN true
+
+FROM busybox as second
+RUN false
+
+FROM alpine
+RUN false
+
+# binary returns the compiled binary
+FROM second AS binary
+`)
+
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", []byte(dockerfile), 0600),
+ )
+ require.NoError(t, err)
+ defer os.RemoveAll(dir)
+
+ c, err := client.New(sb.Context(), sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ destDir, err := os.MkdirTemp("", "buildkit")
+ require.NoError(t, err)
+ defer os.RemoveAll(destDir)
+
+ called := false
+ frontend := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
+ res, err := c.Solve(ctx, gateway.SolveRequest{
+ FrontendOpt: map[string]string{
+ "frontend.caps": "moby.buildkit.frontend.subrequests",
+ "requestid": "frontend.targets",
+ },
+ Frontend: "dockerfile.v0",
+ })
+ require.NoError(t, err)
+
+ list, err := unmarshalTargets(res)
+ require.NoError(t, err)
+
+ require.Equal(t, 1, len(list.Sources))
+ require.Equal(t, dockerfile, list.Sources[0])
+
+ require.Equal(t, 4, len(list.Targets))
+
+ target := list.Targets[0]
+ require.Equal(t, "build", target.Name)
+ require.Equal(t, "alpine", target.Base)
+ require.Equal(t, "defines stage for compiling the binary", target.Description)
+ require.Equal(t, false, target.Default)
+ require.Equal(t, int32(0), target.Location.SourceIndex)
+ require.Equal(t, int32(3), target.Location.Ranges[0].Start.Line)
+
+ target = list.Targets[1]
+ require.Equal(t, "second", target.Name)
+ require.Equal(t, "", target.Description)
+ require.Equal(t, "busybox", target.Base)
+ require.Equal(t, false, target.Default)
+ require.Equal(t, int32(0), target.Location.SourceIndex)
+ require.Equal(t, int32(6), target.Location.Ranges[0].Start.Line)
+
+ target = list.Targets[2]
+ require.Equal(t, "", target.Name)
+ require.Equal(t, "", target.Description)
+ require.Equal(t, "alpine", target.Base)
+ require.Equal(t, false, target.Default)
+ require.Equal(t, int32(0), target.Location.SourceIndex)
+ require.Equal(t, int32(9), target.Location.Ranges[0].Start.Line)
+
+ target = list.Targets[3]
+ require.Equal(t, "binary", target.Name)
+ require.Equal(t, "returns the compiled binary", target.Description)
+ require.Equal(t, true, target.Default)
+ require.Equal(t, int32(0), target.Location.SourceIndex)
+ require.Equal(t, int32(13), target.Location.Ranges[0].Start.Line)
+
+ called = true
+ return nil, nil
+ }
+
+ _, err = c.Build(sb.Context(), client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ },
+ }, "", frontend, nil)
+ require.NoError(t, err)
+
+ require.True(t, called)
+}
+
+func testTargetsDescribeDefinition(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureFrontendTargets)
+ f := getFrontend(t, sb)
+ if _, ok := f.(*clientFrontend); !ok {
+ t.Skip("only test with client frontend")
+ }
+
+ c, err := client.New(sb.Context(), sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ dockerfile := []byte(`
+FROM scratch
+COPY Dockerfile Dockerfile
+`)
+
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+ defer os.RemoveAll(dir)
+
+ called := false
+
+ frontend := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
+ reqs, err := subrequests.Describe(ctx, c)
+ require.NoError(t, err)
+
+ require.True(t, len(reqs) > 0)
+
+ hasTargets := false
+
+ for _, req := range reqs {
+ if req.Name != "frontend.targets" {
+ continue
+ }
+ hasTargets = true
+ require.Equal(t, subrequests.RequestType("rpc"), req.Type)
+ require.NotEqual(t, req.Version, "")
+ }
+ require.True(t, hasTargets)
+
+ called = true
+ return nil, nil
+ }
+
+ _, err = c.Build(sb.Context(), client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ },
+ }, "", frontend, nil)
+ require.NoError(t, err)
+
+ require.True(t, called)
+}
+
+func unmarshalTargets(res *gateway.Result) (*targets.List, error) {
+ dt, ok := res.Metadata["result.json"]
+ if !ok {
+ return nil, errors.Errorf("missing frontend.outline")
+ }
+ var l targets.List
+ if err := json.Unmarshal(dt, &l); err != nil {
+ return nil, err
+ }
+ return &l, nil
+}
diff --git a/frontend/dockerfile/dockerfile_test.go b/frontend/dockerfile/dockerfile_test.go
index a4ca3578c7dd..ed49f9c8502a 100644
--- a/frontend/dockerfile/dockerfile_test.go
+++ b/frontend/dockerfile/dockerfile_test.go
@@ -8,29 +8,30 @@ import (
"encoding/json"
"fmt"
"io"
- "io/ioutil"
"net/http"
"net/http/httptest"
"os"
"os/exec"
+ "path"
"path/filepath"
"runtime"
"sort"
- "strconv"
"strings"
"testing"
"time"
"github.com/containerd/containerd"
"github.com/containerd/containerd/content"
+ "github.com/containerd/containerd/content/local"
"github.com/containerd/containerd/namespaces"
"github.com/containerd/containerd/platforms"
"github.com/containerd/containerd/snapshots"
"github.com/containerd/continuity/fs/fstest"
+ intoto "github.com/in-toto/in-toto-golang/in_toto"
+ controlapi "github.com/moby/buildkit/api/services/control"
"github.com/moby/buildkit/client"
"github.com/moby/buildkit/client/llb"
"github.com/moby/buildkit/frontend/dockerfile/builder"
- "github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb"
gateway "github.com/moby/buildkit/frontend/gateway/client"
"github.com/moby/buildkit/frontend/subrequests"
"github.com/moby/buildkit/identity"
@@ -39,16 +40,18 @@ import (
"github.com/moby/buildkit/solver/errdefs"
"github.com/moby/buildkit/solver/pb"
"github.com/moby/buildkit/util/contentutil"
+ "github.com/moby/buildkit/util/iohelper"
"github.com/moby/buildkit/util/testutil"
"github.com/moby/buildkit/util/testutil/httpserver"
"github.com/moby/buildkit/util/testutil/integration"
+ digest "github.com/opencontainers/go-digest"
ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/pkg/errors"
"github.com/stretchr/testify/require"
)
func init() {
- if os.Getenv("TEST_DOCKERD") == "1" {
+ if integration.IsTestDockerd() {
integration.InitDockerdWorker()
} else {
integration.InitOCIWorker()
@@ -67,6 +70,7 @@ var allTests = integration.TestFuncs(
testExportedHistory,
testExposeExpansion,
testUser,
+ testUserAdditionalGids,
testCacheReleased,
testDockerignore,
testDockerignoreInvalid,
@@ -89,7 +93,6 @@ var allTests = integration.TestFuncs(
testQuotedMetaArgs,
testIgnoreEntrypoint,
testSymlinkedDockerfile,
- testDockerfileAddArchiveWildcard,
testEmptyWildcard,
testWorkdirCreatesDir,
testDockerfileAddArchiveWildcard,
@@ -116,12 +119,14 @@ var allTests = integration.TestFuncs(
testUlimit,
testCgroupParent,
testNamedImageContext,
+ testNamedImageContextPlatform,
+ testNamedImageContextTimestamps,
+ testNamedImageContextScratch,
testNamedLocalContext,
+ testNamedOCILayoutContext,
+ testNamedOCILayoutContextExport,
testNamedInputContext,
testNamedMultiplatformInputContext,
-)
-
-var fileOpTests = integration.TestFuncs(
testEmptyDestDir,
testCopyChownCreateDest,
testCopyThroughSymlinkContext,
@@ -144,6 +149,18 @@ var fileOpTests = integration.TestFuncs(
testWorkdirCopyIgnoreRelative,
testCopyFollowAllSymlinks,
testDockerfileAddChownExpand,
+ testSourceDateEpochWithoutExporter,
+ testSBOMScannerImage,
+ testProvenanceAttestation,
+ testGitProvenanceAttestation,
+ testMultiPlatformProvenance,
+ testClientFrontendProvenance,
+ testClientLLBProvenance,
+ testSecretSSHProvenance,
+ testNilProvenance,
+ testSBOMScannerArgs,
+ testMultiPlatformWarnings,
+ testNilContextInSolveGateway,
)
// Tests that depend on the `security.*` entitlements
@@ -155,6 +172,11 @@ var networkTests = []integration.Test{}
// Tests that depend on heredoc support
var heredocTests = []integration.Test{}
+// Tests that depend on reproducible env
+var reproTests = integration.TestFuncs(
+ testReproSourceDateEpoch,
+)
+
var opts []integration.TestOpt
var securityOpts []integration.TestOpt
@@ -170,9 +192,6 @@ func init() {
opts = []integration.TestOpt{
integration.WithMirroredImages(integration.OfficialImages("busybox:latest")),
- integration.WithMirroredImages(map[string]string{
- "docker/dockerfile-copy:v0.1.9": "docker.io/" + dockerfile2llb.DefaultCopyImage,
- }),
integration.WithMatrix("frontend", frontends),
}
@@ -194,10 +213,6 @@ func init() {
func TestIntegration(t *testing.T) {
integration.Run(t, allTests, opts...)
- integration.Run(t, fileOpTests, append(opts, integration.WithMatrix("fileop", map[string]interface{}{
- "true": true,
- "false": false,
- }))...)
integration.Run(t, securityTests, append(append(opts, securityOpts...),
integration.WithMatrix("security.insecure", map[string]interface{}{
"granted": securityInsecureGranted,
@@ -209,6 +224,12 @@ func TestIntegration(t *testing.T) {
"denied": networkHostDenied,
}))...)
integration.Run(t, heredocTests, opts...)
+ integration.Run(t, outlineTests, opts...)
+ integration.Run(t, targetsTests, opts...)
+
+ integration.Run(t, reproTests, append(opts,
+ // Only use the amd64 digest, regardless to the host platform
+ integration.WithMirroredImages(integration.OfficialImages("debian:bullseye-20230109-slim@sha256:1acb06a0c31fb467eb8327ad361f1091ab265e0bf26d452dea45dcb0c0ea5e75")))...)
}
func testDefaultEnvWithArgs(t *testing.T, sb integration.Sandbox) {
@@ -229,20 +250,18 @@ COPY --from=build /out /out
echo -n $my_arg $1 > /out
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("myscript.sh", script, 0700),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
for _, x := range []struct {
name string
@@ -269,7 +288,7 @@ echo -n $my_arg $1 > /out
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "out"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "out"))
require.NoError(t, err)
require.Equal(t, x.expected, string(dt))
})
@@ -285,19 +304,17 @@ ENV myenv foo%sbar
RUN [ "$myenv" = 'foo%sbar' ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -336,7 +353,8 @@ RUN [ ! -f foo ] && [ -f bar ]
foo
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("Dockerfile.dockerignore", ignore, 0600),
fstest.CreateFile("Dockerfile2", dockerfile2, 0600),
@@ -345,7 +363,6 @@ foo
fstest.CreateFile("bar", []byte("contents0"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -373,7 +390,6 @@ foo
func testEmptyDestDir(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM busybox
@@ -382,21 +398,18 @@ COPY testfile $empty
RUN [ "$(cat testfile)" == "contents0" ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("testfile", []byte("contents0"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -406,6 +419,7 @@ RUN [ "$(cat testfile)" == "contents0" ]
}
func testExportCacheLoop(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureCacheExport)
f := getFrontend(t, sb)
dockerfile := []byte(`
@@ -424,16 +438,14 @@ FROM scratch
COPY --from=base2 /foo /f
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("hello.txt", []byte("hello"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
- cacheDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(cacheDir)
+ cacheDir := t.TempDir()
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -497,13 +509,13 @@ COPY bar fordarwin
FROM stage-$TARGETOS
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte("data"), 0600),
fstest.CreateFile("bar", []byte("data2"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -571,19 +583,17 @@ WORKDIR /foo
WORKDIR /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -611,11 +621,11 @@ func testCacheReleased(t *testing.T, sb integration.Sandbox) {
FROM busybox
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -648,12 +658,12 @@ FROM scratch
ENV foo bar
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile.web", dockerfile, 0600),
fstest.Symlink("Dockerfile.web", "Dockerfile"),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -731,12 +741,12 @@ RUN e="300:400"; p="/file" ; a=` + "`" + `stat -c "%u:%g
&& e="300:400"; p="/existingdir/subdir/nestedfile"; a=` + "`" + `stat -c "%u:%g" "$p"` + "`" + `; if [ "$a" != "$e" ]; then echo "incorrect ownership on $p. expected $e, got $a"; exit 1; fi
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile.web", dockerfile, 0600),
fstest.Symlink("Dockerfile.web", "Dockerfile"),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -774,22 +784,20 @@ FROM scratch
COPY --from=base unique /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo1", []byte("foo1-data"), 0600),
fstest.CreateFile("foo2", []byte("foo2-data"), 0600),
fstest.CreateFile("bar", []byte("bar-data"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -805,15 +813,13 @@ COPY --from=base unique /
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "unique"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "unique"))
require.NoError(t, err)
- err = ioutil.WriteFile(filepath.Join(dir, "bar"), []byte("bar-data-mod"), 0600)
+ err = os.WriteFile(filepath.Join(dir, "bar"), []byte("bar-data-mod"), 0600)
require.NoError(t, err)
- destDir, err = ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -829,16 +835,14 @@ COPY --from=base unique /
}, nil)
require.NoError(t, err)
- dt2, err := ioutil.ReadFile(filepath.Join(destDir, "unique"))
+ dt2, err := os.ReadFile(filepath.Join(destDir, "unique"))
require.NoError(t, err)
require.Equal(t, string(dt), string(dt2))
- err = ioutil.WriteFile(filepath.Join(dir, "foo2"), []byte("foo2-data-mod"), 0600)
+ err = os.WriteFile(filepath.Join(dir, "foo2"), []byte("foo2-data-mod"), 0600)
require.NoError(t, err)
- destDir, err = ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -854,7 +858,7 @@ COPY --from=base unique /
}, nil)
require.NoError(t, err)
- dt2, err = ioutil.ReadFile(filepath.Join(destDir, "unique"))
+ dt2, err = os.ReadFile(filepath.Join(destDir, "unique"))
require.NoError(t, err)
require.NotEqual(t, string(dt), string(dt2))
}
@@ -867,20 +871,18 @@ FROM scratch
COPY foo nomatch* /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte("contents0"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -896,14 +898,13 @@ COPY foo nomatch* /
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "foo"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "foo"))
require.NoError(t, err)
require.Equal(t, "contents0", string(dt))
}
func testWorkdirUser(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM busybox
@@ -913,20 +914,17 @@ WORKDIR /mydir
RUN [ "$(stat -c "%U %G" /mydir)" == "user user" ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -937,7 +935,6 @@ RUN [ "$(stat -c "%U %G" /mydir)" == "user user" ]
func testWorkdirCopyIgnoreRelative(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM scratch AS base
@@ -948,20 +945,17 @@ FROM scratch
COPY --from=base Dockerfile .
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -972,7 +966,6 @@ COPY --from=base Dockerfile .
func testWorkdirExists(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM busybox
@@ -982,20 +975,17 @@ WORKDIR /mydir
RUN [ "$(stat -c "%U %G" /mydir)" == "user user" ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -1006,7 +996,6 @@ RUN [ "$(stat -c "%U %G" /mydir)" == "user user" ]
func testCopyChownCreateDest(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM busybox
@@ -1020,11 +1009,11 @@ RUN [ "$(stat -c "%U %G" /dest)" == "user user" ]
RUN [ "$(stat -c "%U %G" /dest01)" == "user01 user" ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -1032,8 +1021,7 @@ RUN [ "$(stat -c "%U %G" /dest01)" == "user01 user" ]
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- "build-arg:group": "user",
+ "build-arg:group": "user",
},
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
@@ -1045,29 +1033,26 @@ RUN [ "$(stat -c "%U %G" /dest01)" == "user01 user" ]
func testCopyThroughSymlinkContext(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM scratch
COPY link/foo .
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.Symlink("sub", "link"),
fstest.CreateDir("sub", 0700),
fstest.CreateFile("sub/foo", []byte(`contents`), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -1076,9 +1061,6 @@ COPY link/foo .
OutputDir: destDir,
},
},
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -1086,14 +1068,13 @@ COPY link/foo .
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "foo"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "foo"))
require.NoError(t, err)
require.Equal(t, "contents", string(dt))
}
func testCopyThroughSymlinkMultiStage(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM busybox AS build
@@ -1103,19 +1084,17 @@ COPY --from=build /sub/foo .
COPY --from=build /sub2/foo bar
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -1124,9 +1103,6 @@ COPY --from=build /sub2/foo bar
OutputDir: destDir,
},
},
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -1134,34 +1110,31 @@ COPY --from=build /sub2/foo bar
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "foo"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "foo"))
require.NoError(t, err)
require.Equal(t, "data", string(dt))
}
func testCopySocket(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM scratch
COPY . /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateSocket("socket.sock", 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -1170,9 +1143,6 @@ COPY . /
OutputDir: destDir,
},
},
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -1195,11 +1165,11 @@ ENTRYPOINT ["/nosuchcmd"]
RUN ["ls"]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -1228,19 +1198,17 @@ FROM scratch
COPY --from=build /out .
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
LocalDirs: map[string]string{
@@ -1256,7 +1224,7 @@ COPY --from=build /out .
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "out"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "out"))
require.NoError(t, err)
require.Equal(t, "bar-box-foo", string(dt))
}
@@ -1274,19 +1242,17 @@ FROM scratch
COPY --from=build /out .
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
LocalDirs: map[string]string{
@@ -1302,13 +1268,13 @@ COPY --from=build /out .
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "out"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "out"))
require.NoError(t, err)
require.Equal(t, "foo bar:box-foo:123 456", string(dt))
}
func testDefaultShellAndPath(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
+ integration.CheckFeatureCompat(t, sb, integration.FeatureOCIExporter)
f := getFrontend(t, sb)
dockerfile := []byte(`
@@ -1317,19 +1283,17 @@ ENTRYPOINT foo bar
COPY Dockerfile .
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
out := filepath.Join(destDir, "out.tar")
outW, err := os.Create(out)
@@ -1352,7 +1316,7 @@ COPY Dockerfile .
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "out.tar"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "out.tar"))
require.NoError(t, err)
m, err := testutil.ReadTarToMap(dt, false)
@@ -1398,7 +1362,7 @@ COPY Dockerfile .
}
func testExportMultiPlatform(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
+ integration.CheckFeatureCompat(t, sb, integration.FeatureOCIExporter, integration.FeatureMultiPlatform)
f := getFrontend(t, sb)
dockerfile := []byte(`
@@ -1409,7 +1373,8 @@ LABEL target=$TARGETPLATFORM
COPY arch-$TARGETARCH whoami
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("arch-arm", []byte(`i am arm`), 0600),
fstest.CreateFile("arch-amd64", []byte(`i am amd64`), 0600),
@@ -1417,15 +1382,12 @@ COPY arch-$TARGETARCH whoami
fstest.CreateFile("arch-ppc64le", []byte(`i am ppc64le`), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
LocalDirs: map[string]string{
@@ -1444,23 +1406,21 @@ COPY arch-$TARGETARCH whoami
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "windows_amd64/whoami"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "windows_amd64/whoami"))
require.NoError(t, err)
require.Equal(t, "i am amd64", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "linux_arm_v7/whoami"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "linux_arm_v7/whoami"))
require.NoError(t, err)
require.Equal(t, "i am arm", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "linux_s390x/whoami"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "linux_s390x/whoami"))
require.NoError(t, err)
require.Equal(t, "i am s390x", string(dt))
// repeat with oci exporter
- destDir, err = ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
out := filepath.Join(destDir, "out.tar")
outW, err := os.Create(out)
@@ -1483,7 +1443,7 @@ COPY arch-$TARGETARCH whoami
}, nil)
require.NoError(t, err)
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "out.tar"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "out.tar"))
require.NoError(t, err)
m, err := testutil.ReadTarToMap(dt, false)
@@ -1540,29 +1500,25 @@ COPY arch-$TARGETARCH whoami
// tonistiigi/fsutil#46
func testContextChangeDirToFile(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM scratch
COPY foo /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateDir("foo", 0700),
fstest.CreateFile("foo/bar", []byte(`contents`), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -1570,16 +1526,14 @@ COPY foo /
}, nil)
require.NoError(t, err)
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte(`contents2`), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -1588,9 +1542,6 @@ COPY foo /
OutputDir: destDir,
},
},
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -1598,35 +1549,31 @@ COPY foo /
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "foo"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "foo"))
require.NoError(t, err)
require.Equal(t, "contents2", string(dt))
}
func testNoSnapshotLeak(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM scratch
COPY foo /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte(`contents`), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -1638,9 +1585,6 @@ COPY foo /
require.NoError(t, err)
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -1657,7 +1601,6 @@ COPY foo /
// #1197
func testCopyFollowAllSymlinks(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM scratch
@@ -1665,27 +1608,20 @@ COPY foo /
COPY foo/sub bar
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("bar", []byte(`bar-contents`), 0600),
fstest.CreateDir("foo", 0700),
fstest.Symlink("../bar", "foo/sub"),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
-
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -1696,7 +1632,6 @@ COPY foo/sub bar
func testCopySymlinks(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM scratch
@@ -1704,7 +1639,8 @@ COPY foo /
COPY sub/l* alllinks/
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("bar", []byte(`bar-contents`), 0600),
fstest.Symlink("bar", "foo"),
@@ -1717,15 +1653,12 @@ COPY sub/l* alllinks/
fstest.CreateFile("sub/baz", []byte(`baz-contents`), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -1734,9 +1667,6 @@ COPY sub/l* alllinks/
OutputDir: destDir,
},
},
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -1744,19 +1674,19 @@ COPY sub/l* alllinks/
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "foo"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "foo"))
require.NoError(t, err)
require.Equal(t, "bar-contents", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "alllinks/l0"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "alllinks/l0"))
require.NoError(t, err)
require.Equal(t, "subfile-contents", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "alllinks/lfile"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "alllinks/lfile"))
require.NoError(t, err)
require.Equal(t, "lfile-contents", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "alllinks/l1"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "alllinks/l1"))
require.NoError(t, err)
require.Equal(t, "baz-contents", string(dt))
}
@@ -1771,11 +1701,9 @@ FROM scratch
COPY --from=0 /foo /foo
`)
- srcDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(srcDir)
+ srcDir := t.TempDir()
- err = ioutil.WriteFile(filepath.Join(srcDir, "Dockerfile"), dockerfile, 0600)
+ err := os.WriteFile(filepath.Join(srcDir, "Dockerfile"), dockerfile, 0600)
require.NoError(t, err)
resp := httpserver.Response{
@@ -1788,9 +1716,7 @@ COPY --from=0 /foo /foo
})
defer server.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -1810,7 +1736,7 @@ COPY --from=0 /foo /foo
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "foo"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "foo"))
require.NoError(t, err)
require.Equal(t, "foo-contents", string(dt))
}
@@ -1828,11 +1754,11 @@ FROM scratch
CMD ["test"]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -1861,11 +1787,11 @@ SHELL ["ls"]
ENTRYPOINT my entrypoint
`)
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
target = "docker.io/moby/cmdoverridetest2:latest"
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
@@ -1920,11 +1846,11 @@ FROM scratch
LABEL foo=bar
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -1953,12 +1879,12 @@ LABEL bar=baz
COPY foo .
`)
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte("foo-contents"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
target = "docker.io/moby/testpullscratch2:latest"
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
@@ -2012,9 +1938,7 @@ COPY foo .
def, err := echo.Marshal(sb.Context())
require.NoError(t, err)
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = c.Solve(sb.Context(), def, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -2030,7 +1954,7 @@ COPY foo .
}, nil)
require.NoError(t, err)
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "foo"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "foo"))
require.NoError(t, err)
require.Equal(t, "foo0", string(dt))
}
@@ -2043,11 +1967,11 @@ ARG tag=nosuchtag
FROM busybox:${tag}
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -2066,7 +1990,6 @@ FROM busybox:${tag}
}
func testDockerfileDirs(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
f := getFrontend(t, sb)
f.RequiresBuildctl(t)
@@ -2080,12 +2003,12 @@ func testDockerfileDirs(t *testing.T, sb integration.Sandbox) {
RUN cmp -s foo foo3
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte("bar"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
args, trace := f.DFCmdArgs(dir, dir)
defer os.RemoveAll(trace)
@@ -2108,17 +2031,17 @@ func testDockerfileDirs(t *testing.T, sb integration.Sandbox) {
require.NoError(t, err)
// different context and dockerfile directories
- dir1, err := tmpdir(
+ dir1, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir1)
- dir2, err := tmpdir(
+ dir2, err := integration.Tmpdir(
+ t,
fstest.CreateFile("foo", []byte("bar"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir2)
args, trace = f.DFCmdArgs(dir2, dir1)
defer os.RemoveAll(trace)
@@ -2135,7 +2058,6 @@ func testDockerfileDirs(t *testing.T, sb integration.Sandbox) {
}
func testDockerfileInvalidCommand(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
f := getFrontend(t, sb)
f.RequiresBuildctl(t)
dockerfile := []byte(`
@@ -2143,11 +2065,11 @@ func testDockerfileInvalidCommand(t *testing.T, sb integration.Sandbox) {
RUN invalidcmd
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
args, trace := f.DFCmdArgs(dir, dir)
defer os.RemoveAll(trace)
@@ -2162,7 +2084,6 @@ func testDockerfileInvalidCommand(t *testing.T, sb integration.Sandbox) {
}
func testDockerfileInvalidInstruction(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
f := getFrontend(t, sb)
f.RequiresBuildctl(t)
dockerfile := []byte(`
@@ -2170,11 +2091,11 @@ func testDockerfileInvalidInstruction(t *testing.T, sb integration.Sandbox) {
FNTRYPOINT ["/bin/sh", "-c", "echo invalidinstruction"]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -2193,7 +2114,6 @@ func testDockerfileInvalidInstruction(t *testing.T, sb integration.Sandbox) {
}
func testDockerfileADDFromURL(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
f := getFrontend(t, sb)
f.RequiresBuildctl(t)
@@ -2221,24 +2141,22 @@ FROM scratch
ADD %s /dest/
`, server.URL+"/foo"))
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
args, trace := f.DFCmdArgs(dir, dir)
defer os.RemoveAll(trace)
- destDir, err := tmpdir()
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
- cmd := sb.Cmd(args + fmt.Sprintf(" --exporter=local --exporter-opt output=%s", destDir))
+ cmd := sb.Cmd(args + fmt.Sprintf(" --output type=local,dest=%s", destDir))
err = cmd.Run()
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "dest/foo"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "dest/foo"))
require.NoError(t, err)
require.Equal(t, []byte("content1"), dt)
@@ -2248,25 +2166,23 @@ FROM scratch
ADD %s /dest/
`, server.URL+"/"))
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
args, trace = f.DFCmdArgs(dir, dir)
defer os.RemoveAll(trace)
- destDir, err = tmpdir()
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
- cmd = sb.Cmd(args + fmt.Sprintf(" --exporter=local --exporter-opt output=%s", destDir))
+ cmd = sb.Cmd(args + fmt.Sprintf(" --output type=local,dest=%s", destDir))
err = cmd.Run()
require.NoError(t, err)
destFile := filepath.Join(destDir, "dest/__unnamed__")
- dt, err = ioutil.ReadFile(destFile)
+ dt, err = os.ReadFile(destFile)
require.NoError(t, err)
require.Equal(t, []byte("content2"), dt)
@@ -2276,7 +2192,6 @@ ADD %s /dest/
}
func testDockerfileAddArchive(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
f := getFrontend(t, sb)
f.RequiresBuildctl(t)
@@ -2300,24 +2215,22 @@ FROM scratch
ADD t.tar /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("t.tar", buf.Bytes(), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
args, trace := f.DFCmdArgs(dir, dir)
defer os.RemoveAll(trace)
- destDir, err := tmpdir()
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
- cmd := sb.Cmd(args + fmt.Sprintf(" --exporter=local --exporter-opt output=%s", destDir))
+ cmd := sb.Cmd(args + fmt.Sprintf(" --output type=local,dest=%s", destDir))
require.NoError(t, cmd.Run())
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "foo"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "foo"))
require.NoError(t, err)
require.Equal(t, expectedContent, dt)
@@ -2334,24 +2247,22 @@ FROM scratch
ADD t.tar.gz /
`)
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("t.tar.gz", buf2.Bytes(), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
args, trace = f.DFCmdArgs(dir, dir)
defer os.RemoveAll(trace)
- destDir, err = tmpdir()
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
- cmd = sb.Cmd(args + fmt.Sprintf(" --exporter=local --exporter-opt output=%s", destDir))
+ cmd = sb.Cmd(args + fmt.Sprintf(" --output type=local,dest=%s", destDir))
require.NoError(t, cmd.Run())
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "foo"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "foo"))
require.NoError(t, err)
require.Equal(t, expectedContent, dt)
@@ -2361,24 +2272,22 @@ FROM scratch
COPY t.tar.gz /
`)
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("t.tar.gz", buf2.Bytes(), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
args, trace = f.DFCmdArgs(dir, dir)
defer os.RemoveAll(trace)
- destDir, err = tmpdir()
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
- cmd = sb.Cmd(args + fmt.Sprintf(" --exporter=local --exporter-opt output=%s", destDir))
+ cmd = sb.Cmd(args + fmt.Sprintf(" --output type=local,dest=%s", destDir))
require.NoError(t, cmd.Run())
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "t.tar.gz"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "t.tar.gz"))
require.NoError(t, err)
require.Equal(t, buf2.Bytes(), dt)
@@ -2398,23 +2307,21 @@ FROM scratch
ADD %s /
`, server.URL+"/t.tar.gz"))
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
args, trace = f.DFCmdArgs(dir, dir)
defer os.RemoveAll(trace)
- destDir, err = tmpdir()
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
- cmd = sb.Cmd(args + fmt.Sprintf(" --exporter=local --exporter-opt output=%s", destDir))
+ cmd = sb.Cmd(args + fmt.Sprintf(" --output type=local,dest=%s", destDir))
require.NoError(t, cmd.Run())
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "t.tar.gz"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "t.tar.gz"))
require.NoError(t, err)
require.Equal(t, buf2.Bytes(), dt)
@@ -2424,23 +2331,21 @@ FROM scratch
ADD %s /newname.tar.gz
`, server.URL+"/t.tar.gz"))
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
args, trace = f.DFCmdArgs(dir, dir)
defer os.RemoveAll(trace)
- destDir, err = tmpdir()
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
- cmd = sb.Cmd(args + fmt.Sprintf(" --exporter=local --exporter-opt output=%s", destDir))
+ cmd = sb.Cmd(args + fmt.Sprintf(" --output type=local,dest=%s", destDir))
require.NoError(t, cmd.Run())
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "newname.tar.gz"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "newname.tar.gz"))
require.NoError(t, err)
require.Equal(t, buf2.Bytes(), dt)
}
@@ -2483,17 +2388,15 @@ FROM scratch
ADD *.tar /dest
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("t.tar", buf.Bytes(), 0600),
fstest.CreateFile("b.tar", buf2.Bytes(), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -2513,18 +2416,17 @@ ADD *.tar /dest
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "dest/foo"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "dest/foo"))
require.NoError(t, err)
require.Equal(t, "content0", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "dest/bar"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "dest/bar"))
require.NoError(t, err)
require.Equal(t, "content1", string(dt))
}
func testDockerfileAddChownExpand(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM busybox
@@ -2534,12 +2436,12 @@ ADD --chown=${owner}:${group} foo /
RUN [ "$(stat -c "%u %G" /foo)" == "1000 nobody" ]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte(`foo-contents`), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -2547,8 +2449,7 @@ RUN [ "$(stat -c "%u %G" /foo)" == "1000 nobody" ]
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- "build-arg:group": "nobody",
+ "build-arg:group": "nobody",
},
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
@@ -2559,7 +2460,6 @@ RUN [ "$(stat -c "%u %G" /foo)" == "1000 nobody" ]
}
func testSymlinkDestination(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
f := getFrontend(t, sb)
f.RequiresBuildctl(t)
@@ -2582,25 +2482,23 @@ ADD t.tar /
COPY foo /symlink/
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", expectedContent, 0600),
fstest.CreateFile("t.tar", buf.Bytes(), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
args, trace := f.DFCmdArgs(dir, dir)
defer os.RemoveAll(trace)
- destDir, err := tmpdir()
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
- cmd := sb.Cmd(args + fmt.Sprintf(" --exporter=local --exporter-opt output=%s", destDir))
+ cmd := sb.Cmd(args + fmt.Sprintf(" --output type=local,dest=%s", destDir))
require.NoError(t, cmd.Run())
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "tmp/symlink-target/foo"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "tmp/symlink-target/foo"))
require.NoError(t, err)
require.Equal(t, expectedContent, dt)
}
@@ -2618,17 +2516,17 @@ FROM scratch
ENV foo=bar
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
args, trace := f.DFCmdArgs(dir, dir)
defer os.RemoveAll(trace)
target := "example.com/moby/dockerfilescratch:test"
- cmd := sb.Cmd(args + " --exporter=image --exporter-opt=name=" + target)
+ cmd := sb.Cmd(args + " --output type=image,name=" + target)
err = cmd.Run()
require.NoError(t, err)
@@ -2673,8 +2571,7 @@ ENV foo=bar
}
func testExposeExpansion(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
-
+ integration.CheckFeatureCompat(t, sb, integration.FeatureImageExporter)
f := getFrontend(t, sb)
dockerfile := []byte(`
@@ -2684,11 +2581,11 @@ EXPOSE $PORTS
EXPOSE 5000
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -2764,7 +2661,8 @@ Dockerfile
.dockerignore
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte(`foo-contents`), 0600),
fstest.CreateFile("bar", []byte(`bar-contents`), 0600),
@@ -2773,15 +2671,12 @@ Dockerfile
fstest.CreateFile(".dockerignore", dockerignore, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -2797,7 +2692,7 @@ Dockerfile
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "foo"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "foo"))
require.NoError(t, err)
require.Equal(t, "foo-contents", string(dt))
@@ -2817,7 +2712,7 @@ Dockerfile
require.Error(t, err)
require.True(t, errors.Is(err, os.ErrNotExist))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "bay"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "bay"))
require.NoError(t, err)
require.Equal(t, "bay-contents", string(dt))
}
@@ -2830,12 +2725,12 @@ FROM scratch
COPY . .
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile(".dockerignore", []byte("!\n"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
ctx, cancel := context.WithTimeout(sb.Context(), 15*time.Second)
defer cancel()
@@ -2866,11 +2761,11 @@ func testDockerfileLowercase(t *testing.T, sb integration.Sandbox) {
dockerfile := []byte(`FROM scratch
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
ctx := sb.Context()
@@ -2888,7 +2783,6 @@ func testDockerfileLowercase(t *testing.T, sb integration.Sandbox) {
}
func testExportedHistory(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
f := getFrontend(t, sb)
f.RequiresBuildctl(t)
@@ -2905,18 +2799,20 @@ RUN echo bar > foo4
RUN ["ls"]
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte("contents0"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
args, trace := f.DFCmdArgs(dir, dir)
defer os.RemoveAll(trace)
+ integration.CheckFeatureCompat(t, sb, integration.FeatureImageExporter)
+
target := "example.com/moby/dockerfilescratch:test"
- cmd := sb.Cmd(args + " --exporter=image --exporter-opt=name=" + target)
+ cmd := sb.Cmd(args + " --output type=image,name=" + target)
require.NoError(t, cmd.Run())
// TODO: expose this test to OCI worker
@@ -2966,17 +2862,8 @@ RUN ["ls"]
require.NotNil(t, ociimg.History[6].Created)
}
-func skipDockerd(t *testing.T, sb integration.Sandbox) {
- // TODO: remove me once dockerd supports the image and exporter.
- t.Helper()
- if os.Getenv("TEST_DOCKERD") == "1" {
- t.Skip("dockerd missing a required exporter, cache exporter, or entitlement")
- }
-}
-
func testUser(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
-
+ integration.CheckFeatureCompat(t, sb, integration.FeatureImageExporter)
f := getFrontend(t, sb)
dockerfile := []byte(`
@@ -3039,19 +2926,17 @@ COPY --from=base /out /
USER nobody
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -3067,11 +2952,11 @@ USER nobody
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "rootuser"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "rootuser"))
require.NoError(t, err)
require.Equal(t, "root\n", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "daemonuser"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "daemonuser"))
require.NoError(t, err)
require.Equal(t, "daemon\n", string(dt))
@@ -3120,9 +3005,45 @@ USER nobody
require.Equal(t, "nobody", ociimg.Config.User)
}
+// testUserAdditionalGids ensures that that the primary GID is also included in the additional GID list.
+// CVE-2023-25173: https://github.com/advisories/GHSA-hmfx-3pcx-653p
+func testUserAdditionalGids(t *testing.T, sb integration.Sandbox) {
+ f := getFrontend(t, sb)
+
+ dockerfile := []byte(`
+# Mimics the tests in https://github.com/containerd/containerd/commit/3eda46af12b1deedab3d0802adb2e81cb3521950
+FROM busybox
+SHELL ["/bin/sh", "-euxc"]
+RUN [ "$(id)" = "uid=0(root) gid=0(root) groups=0(root),10(wheel)" ]
+USER 1234
+RUN [ "$(id)" = "uid=1234 gid=0(root) groups=0(root)" ]
+USER 1234:1234
+RUN [ "$(id)" = "uid=1234 gid=1234 groups=1234" ]
+USER daemon
+RUN [ "$(id)" = "uid=1(daemon) gid=1(daemon) groups=1(daemon)" ]
+`)
+
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ c, err := client.New(sb.Context(), sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ }, nil)
+ require.NoError(t, err)
+}
+
func testCopyChown(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM busybox AS base
@@ -3139,22 +3060,20 @@ FROM scratch
COPY --from=base /out /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte(`foo-contents`), 0600),
fstest.CreateDir("bar", 0700),
fstest.CreateFile("bar/sub", nil, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -3164,8 +3083,7 @@ COPY --from=base /out /
},
},
FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- "build-arg:group": "nobody",
+ "build-arg:group": "nobody",
},
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
@@ -3174,22 +3092,21 @@ COPY --from=base /out /
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "fooowner"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "fooowner"))
require.NoError(t, err)
require.Equal(t, "daemon daemon\n", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "subowner"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "subowner"))
require.NoError(t, err)
require.Equal(t, "1000 nobody\n", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "foobisowner"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "foobisowner"))
require.NoError(t, err)
require.Equal(t, "1000 nobody\n", string(dt))
}
func testCopyChmod(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM busybox AS base
@@ -3206,22 +3123,19 @@ FROM scratch
COPY --from=base /out /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte(`foo-contents`), 0600),
fstest.CreateFile("bar", []byte(`bar-contents`), 0700),
)
-
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -3230,37 +3144,29 @@ COPY --from=base /out /
OutputDir: destDir,
},
},
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
},
}, nil)
- if !isFileOp {
- require.Contains(t, err.Error(), "chmod is not supported")
- return
- }
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "fooperm"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "fooperm"))
require.NoError(t, err)
require.Equal(t, "0644\n", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "barperm"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "barperm"))
require.NoError(t, err)
require.Equal(t, "0777\n", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "foobisperm"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "foobisperm"))
require.NoError(t, err)
require.Equal(t, "0000\n", string(dt))
}
func testCopyOverrideFiles(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM scratch AS base
@@ -3271,7 +3177,8 @@ COPY files/foo.go dest/foo.go
COPY files dest
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateDir("sub", 0700),
fstest.CreateDir("sub/dir1", 0700),
@@ -3281,15 +3188,12 @@ COPY files dest
fstest.CreateFile("files/foo.go", []byte(`foo.go-contents`), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -3298,9 +3202,6 @@ COPY files dest
OutputDir: destDir,
},
},
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -3308,18 +3209,17 @@ COPY files dest
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "sub/dir1/dir2/foo"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "sub/dir1/dir2/foo"))
require.NoError(t, err)
require.Equal(t, "foo-contents", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "dest/foo.go"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "dest/foo.go"))
require.NoError(t, err)
require.Equal(t, "foo.go-contents", string(dt))
}
func testCopyVarSubstitution(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM scratch AS base
@@ -3327,21 +3227,19 @@ ENV FOO bar
COPY $FOO baz
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("bar", []byte(`bar-contents`), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -3350,9 +3248,6 @@ COPY $FOO baz
OutputDir: destDir,
},
},
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -3360,14 +3255,13 @@ COPY $FOO baz
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "baz"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "baz"))
require.NoError(t, err)
require.Equal(t, "bar-contents", string(dt))
}
func testCopyWildcards(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM scratch AS base
@@ -3382,7 +3276,8 @@ COPY sub/dir1/. subdest5
COPY sub/dir1 subdest6
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo.go", []byte(`foo-contents`), 0600),
fstest.CreateFile("bar.go", []byte(`bar-contents`), 0600),
@@ -3392,15 +3287,12 @@ COPY sub/dir1 subdest6
fstest.CreateFile("sub/dir1/dir2/foo", []byte(`foo-contents`), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -3409,9 +3301,6 @@ COPY sub/dir1 subdest6
OutputDir: destDir,
},
},
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -3419,52 +3308,49 @@ COPY sub/dir1 subdest6
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "gofiles/foo.go"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "gofiles/foo.go"))
require.NoError(t, err)
require.Equal(t, "foo-contents", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "gofiles/bar.go"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "gofiles/bar.go"))
require.NoError(t, err)
require.Equal(t, "bar-contents", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "foo2.go"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "foo2.go"))
require.NoError(t, err)
require.Equal(t, "foo-contents", string(dt))
- if isFileOp { // non-fileop implementation is historically buggy
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "subdest/dir2/foo"))
- require.NoError(t, err)
- require.Equal(t, "foo-contents", string(dt))
- }
+ dt, err = os.ReadFile(filepath.Join(destDir, "subdest/dir2/foo"))
+ require.NoError(t, err)
+ require.Equal(t, "foo-contents", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "subdest2/foo"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "subdest2/foo"))
require.NoError(t, err)
require.Equal(t, "foo-contents", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "subdest3/bar"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "subdest3/bar"))
require.NoError(t, err)
require.Equal(t, "foo-contents", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "all/foo.go"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "all/foo.go"))
require.NoError(t, err)
require.Equal(t, "foo-contents", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "subdest4/dir2/foo"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "subdest4/dir2/foo"))
require.NoError(t, err)
require.Equal(t, "foo-contents", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "subdest5/dir2/foo"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "subdest5/dir2/foo"))
require.NoError(t, err)
require.Equal(t, "foo-contents", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "subdest6/dir2/foo"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "subdest6/dir2/foo"))
require.NoError(t, err)
require.Equal(t, "foo-contents", string(dt))
}
func testCopyRelative(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM busybox
@@ -3490,21 +3376,18 @@ COPY foo ../
RUN sh -c "[ $(cat /test5/foo) = 'hello' ]"
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte(`hello`), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
- FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- },
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
@@ -3514,7 +3397,6 @@ RUN sh -c "[ $(cat /test5/foo) = 'hello' ]"
}
func testAddURLChmod(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
f := getFrontend(t, sb)
f.RequiresBuildctl(t)
@@ -3540,19 +3422,17 @@ FROM scratch
COPY --from=build /dest /dest
`, server.URL+"/foo"))
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := tmpdir()
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -3568,7 +3448,7 @@ COPY --from=build /dest /dest
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "dest"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "dest"))
require.NoError(t, err)
require.Equal(t, []byte("0644\n0755\n0413\n"), dt)
}
@@ -3576,9 +3456,7 @@ COPY --from=build /dest /dest
func testDockerfileFromGit(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- gitDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(gitDir)
+ gitDir := t.TempDir()
dockerfile := `
FROM busybox AS build
@@ -3587,7 +3465,7 @@ FROM scratch
COPY --from=build foo bar
`
- err = ioutil.WriteFile(filepath.Join(gitDir, "Dockerfile"), []byte(dockerfile), 0600)
+ err := os.WriteFile(filepath.Join(gitDir, "Dockerfile"), []byte(dockerfile), 0600)
require.NoError(t, err)
err = runShell(gitDir,
@@ -3604,7 +3482,7 @@ COPY --from=build foo bar
COPY --from=build foo bar2
`
- err = ioutil.WriteFile(filepath.Join(gitDir, "Dockerfile"), []byte(dockerfile), 0600)
+ err = os.WriteFile(filepath.Join(gitDir, "Dockerfile"), []byte(dockerfile), 0600)
require.NoError(t, err)
err = runShell(gitDir,
@@ -3617,9 +3495,7 @@ COPY --from=build foo bar2
server := httptest.NewServer(http.FileServer(http.Dir(filepath.Join(gitDir))))
defer server.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -3638,7 +3514,7 @@ COPY --from=build foo bar2
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "bar"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "bar"))
require.NoError(t, err)
require.Equal(t, "fromgit", string(dt))
@@ -3647,9 +3523,7 @@ COPY --from=build foo bar2
require.True(t, errors.Is(err, os.ErrNotExist))
// second request from master branch contains both files
- destDir, err = ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
@@ -3664,11 +3538,11 @@ COPY --from=build foo bar2
}, nil)
require.NoError(t, err)
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "bar"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "bar"))
require.NoError(t, err)
require.Equal(t, "fromgit", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "bar2"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "bar2"))
require.NoError(t, err)
require.Equal(t, "fromgit", string(dt))
}
@@ -3709,9 +3583,7 @@ COPY foo bar
})
defer server.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -3731,7 +3603,7 @@ COPY foo bar
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "bar"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "bar"))
require.NoError(t, err)
require.Equal(t, "foo-contents", string(dt))
}
@@ -3744,19 +3616,17 @@ FROM scratch
COPY --from=busybox /etc/passwd test
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -3772,7 +3642,7 @@ COPY --from=busybox /etc/passwd test
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "test"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "test"))
require.NoError(t, err)
require.Contains(t, string(dt), "root")
@@ -3780,21 +3650,19 @@ COPY --from=busybox /etc/passwd test
dockerfile = []byte(`
FROM busybox AS golang
-RUN mkdir /usr/bin && echo -n foo > /usr/bin/go
+RUN mkdir -p /usr/bin && echo -n foo > /usr/bin/go
FROM scratch
COPY --from=golang /usr/bin/go go
`)
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
- destDir, err = ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -3810,7 +3678,7 @@ COPY --from=golang /usr/bin/go go
}, nil)
require.NoError(t, err)
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "go"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "go"))
require.NoError(t, err)
require.Contains(t, string(dt), "foo")
}
@@ -3826,20 +3694,18 @@ COPY --from=staGE0 bar baz
FROM scratch
COPY --from=stage1 baz bax
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte("foo-contents"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -3858,33 +3724,29 @@ COPY --from=stage1 baz bax
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "baz"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "baz"))
require.NoError(t, err)
require.Contains(t, string(dt), "foo-contents")
}
func testLabels(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
+ integration.CheckFeatureCompat(t, sb, integration.FeatureImageExporter)
f := getFrontend(t, sb)
dockerfile := []byte(`
FROM scratch
LABEL foo=bar
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
-
target := "example.com/moby/dockerfilelabels:test"
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
@@ -3940,7 +3802,6 @@ LABEL foo=bar
// #2008
func testWildcardRenameCache(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
f := getFrontend(t, sb)
dockerfile := []byte(`
@@ -3948,21 +3809,17 @@ FROM alpine
COPY file* /files/
RUN ls /files/file1
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("file1", []byte("foo"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
-
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
@@ -3985,6 +3842,7 @@ RUN ls /files/file1
}
func testOnBuildCleared(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureDirectPush)
f := getFrontend(t, sb)
registry, err := sb.NewRegistry()
@@ -3998,11 +3856,11 @@ FROM busybox
ONBUILD RUN mkdir -p /out && echo -n 11 >> /out/foo
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -4031,11 +3889,11 @@ ONBUILD RUN mkdir -p /out && echo -n 11 >> /out/foo
FROM %s
`, target))
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
target2 := registry + "/buildkit/testonbuild:child"
@@ -4062,15 +3920,13 @@ ONBUILD RUN mkdir -p /out && echo -n 11 >> /out/foo
COPY --from=base /out /
`, target2))
- dir, err = tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -4086,13 +3942,13 @@ ONBUILD RUN mkdir -p /out && echo -n 11 >> /out/foo
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "foo"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "foo"))
require.NoError(t, err)
require.Equal(t, "11", string(dt))
}
func testCacheMultiPlatformImportExport(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
+ integration.CheckFeatureCompat(t, sb, integration.FeatureDirectPush)
f := getFrontend(t, sb)
registry, err := sb.NewRegistry()
@@ -4110,11 +3966,11 @@ COPY --from=base unique /
COPY --from=base arch /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -4161,14 +4017,14 @@ COPY --from=base arch /
desc, provider, err := contentutil.ProviderFromRef(target + "-img")
require.NoError(t, err)
- imgMap, err := readIndex(sb.Context(), provider, desc)
+ imgs, err := testutil.ReadImages(sb.Context(), provider, desc)
require.NoError(t, err)
- require.Equal(t, 2, len(imgMap))
+ require.Equal(t, 2, len(imgs.Images))
- require.Equal(t, "amd64", string(imgMap["linux/amd64"].layers[1]["arch"].Data))
- dtamd := imgMap["linux/amd64"].layers[0]["unique"].Data
- dtarm := imgMap["linux/arm/v7"].layers[0]["unique"].Data
+ require.Equal(t, "amd64", string(imgs.Find("linux/amd64").Layers[1]["arch"].Data))
+ dtamd := imgs.Find("linux/amd64").Layers[0]["unique"].Data
+ dtarm := imgs.Find("linux/arm/v7").Layers[0]["unique"].Data
require.NotEqual(t, dtamd, dtarm)
for i := 0; i < 2; i++ {
@@ -4201,21 +4057,21 @@ COPY --from=base arch /
require.Equal(t, desc.Digest, desc2.Digest)
- imgMap, err = readIndex(sb.Context(), provider, desc2)
+ imgs, err = testutil.ReadImages(sb.Context(), provider, desc2)
require.NoError(t, err)
- require.Equal(t, 2, len(imgMap))
+ require.Equal(t, 2, len(imgs.Images))
- require.Equal(t, "arm", string(imgMap["linux/arm/v7"].layers[1]["arch"].Data))
- dtamd2 := imgMap["linux/amd64"].layers[0]["unique"].Data
- dtarm2 := imgMap["linux/arm/v7"].layers[0]["unique"].Data
+ require.Equal(t, "arm", string(imgs.Find("linux/arm/v7").Layers[1]["arch"].Data))
+ dtamd2 := imgs.Find("linux/amd64").Layers[0]["unique"].Data
+ dtarm2 := imgs.Find("linux/arm/v7").Layers[0]["unique"].Data
require.Equal(t, string(dtamd), string(dtamd2))
require.Equal(t, string(dtarm), string(dtarm2))
}
}
func testCacheImportExport(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
+ integration.CheckFeatureCompat(t, sb, integration.FeatureCacheExport)
f := getFrontend(t, sb)
registry, err := sb.NewRegistry()
@@ -4234,20 +4090,18 @@ COPY --from=base const /
COPY --from=base unique /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte("foobar"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
target := registry + "/buildkit/testexportdf:latest"
@@ -4271,18 +4125,16 @@ COPY --from=base unique /
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "const"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "const"))
require.NoError(t, err)
require.Equal(t, "foobar", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "unique"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "unique"))
require.NoError(t, err)
ensurePruneAll(t, c, sb)
- destDir, err = ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
@@ -4301,21 +4153,17 @@ COPY --from=base unique /
}, nil)
require.NoError(t, err)
- dt2, err := ioutil.ReadFile(filepath.Join(destDir, "const"))
+ dt2, err := os.ReadFile(filepath.Join(destDir, "const"))
require.NoError(t, err)
require.Equal(t, "foobar", string(dt2))
- dt2, err = ioutil.ReadFile(filepath.Join(destDir, "unique"))
+ dt2, err = os.ReadFile(filepath.Join(destDir, "unique"))
require.NoError(t, err)
require.Equal(t, string(dt), string(dt2))
-
- destDir, err = ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
}
func testReproducibleIDs(t *testing.T, sb integration.Sandbox) {
- skipDockerd(t, sb)
+ integration.CheckFeatureCompat(t, sb, integration.FeatureImageExporter)
f := getFrontend(t, sb)
dockerfile := []byte(`
@@ -4324,21 +4172,17 @@ ENV foo=bar
COPY foo /
RUN echo bar > bar
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte("foo-contents"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
-
target := "example.com/moby/dockerfileids:test"
opt := client.SolveOpt{
FrontendAttrs: map[string]string{},
@@ -4405,21 +4249,17 @@ COPY foo /
RUN echo bar > bar
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte("foobar"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
-
target := "example.com/moby/dockerfileexpids:test"
cacheTarget := registry + "/test/dockerfileexpids:cache"
opt := client.SolveOpt{
@@ -4487,19 +4327,17 @@ FROM scratch
COPY --from=s0 unique /
COPY --from=s1 unique2 /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
opt := client.SolveOpt{
FrontendAttrs: map[string]string{},
@@ -4518,9 +4356,7 @@ COPY --from=s1 unique2 /
_, err = f.Solve(sb.Context(), c, opt, nil)
require.NoError(t, err)
- destDir2, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir2 := t.TempDir()
opt.FrontendAttrs["no-cache"] = ""
opt.Exports[0].OutputDir = destDir2
@@ -4528,24 +4364,22 @@ COPY --from=s1 unique2 /
_, err = f.Solve(sb.Context(), c, opt, nil)
require.NoError(t, err)
- unique1Dir1, err := ioutil.ReadFile(filepath.Join(destDir, "unique"))
+ unique1Dir1, err := os.ReadFile(filepath.Join(destDir, "unique"))
require.NoError(t, err)
- unique1Dir2, err := ioutil.ReadFile(filepath.Join(destDir2, "unique"))
+ unique1Dir2, err := os.ReadFile(filepath.Join(destDir2, "unique"))
require.NoError(t, err)
- unique2Dir1, err := ioutil.ReadFile(filepath.Join(destDir, "unique2"))
+ unique2Dir1, err := os.ReadFile(filepath.Join(destDir, "unique2"))
require.NoError(t, err)
- unique2Dir2, err := ioutil.ReadFile(filepath.Join(destDir2, "unique2"))
+ unique2Dir2, err := os.ReadFile(filepath.Join(destDir2, "unique2"))
require.NoError(t, err)
require.NotEqual(t, string(unique1Dir1), string(unique1Dir2))
require.NotEqual(t, string(unique2Dir1), string(unique2Dir2))
- destDir3, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir3 := t.TempDir()
opt.FrontendAttrs["no-cache"] = "s1"
opt.Exports[0].OutputDir = destDir3
@@ -4553,10 +4387,10 @@ COPY --from=s1 unique2 /
_, err = f.Solve(sb.Context(), c, opt, nil)
require.NoError(t, err)
- unique1Dir3, err := ioutil.ReadFile(filepath.Join(destDir3, "unique"))
+ unique1Dir3, err := os.ReadFile(filepath.Join(destDir3, "unique"))
require.NoError(t, err)
- unique2Dir3, err := ioutil.ReadFile(filepath.Join(destDir3, "unique2"))
+ unique2Dir3, err := os.ReadFile(filepath.Join(destDir3, "unique2"))
require.NoError(t, err)
require.Equal(t, string(unique1Dir2), string(unique1Dir3))
@@ -4573,21 +4407,19 @@ FROM build-${TARGETOS}
COPY foo2 bar2
`, runtime.GOOS))
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte("d0"), 0600),
fstest.CreateFile("foo2", []byte("d1"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
opt := client.SolveOpt{
Exports: []client.ExportEntry{
@@ -4605,11 +4437,11 @@ COPY foo2 bar2
_, err = f.Solve(sb.Context(), c, opt, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "bar"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "bar"))
require.NoError(t, err)
require.Equal(t, "d0", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "bar2"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "bar2"))
require.NoError(t, err)
require.Equal(t, "d1", string(dt))
}
@@ -4626,19 +4458,17 @@ FROM scratch
COPY --from=build out .
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
opt := client.SolveOpt{
Exports: []client.ExportEntry{
@@ -4660,11 +4490,11 @@ COPY --from=build out .
_, err = f.Solve(sb.Context(), c, opt, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "platform"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "platform"))
require.NoError(t, err)
require.Equal(t, "darwin/ppc64le", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "os"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "os"))
require.NoError(t, err)
require.Equal(t, "freebsd", string(dt))
}
@@ -4682,19 +4512,17 @@ FROM scratch
COPY --from=build /out /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
opt := client.SolveOpt{
FrontendAttrs: map[string]string{
@@ -4717,14 +4545,12 @@ COPY --from=build /out /
_, err = f.Solve(sb.Context(), c, opt, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "out"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "out"))
require.NoError(t, err)
require.Equal(t, "hpvalue::npvalue::foocontents::::bazcontent", string(dt))
// repeat with changed default args should match the old cache
- destDir, err = ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
opt = client.SolveOpt{
FrontendAttrs: map[string]string{
@@ -4746,14 +4572,12 @@ COPY --from=build /out /
_, err = f.Solve(sb.Context(), c, opt, nil)
require.NoError(t, err)
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "out"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "out"))
require.NoError(t, err)
require.Equal(t, "hpvalue::npvalue::foocontents::::bazcontent", string(dt))
// changing actual value invalidates cache
- destDir, err = ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir = t.TempDir()
opt = client.SolveOpt{
FrontendAttrs: map[string]string{
@@ -4775,14 +4599,13 @@ COPY --from=build /out /
_, err = f.Solve(sb.Context(), c, opt, nil)
require.NoError(t, err)
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "out"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "out"))
require.NoError(t, err)
require.Equal(t, "hpvalue2::::foocontents2::::bazcontent", string(dt))
}
func testTarContext(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
dockerfile := []byte(`
FROM scratch
@@ -4823,8 +4646,7 @@ COPY foo /
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- "context": url,
+ "context": url,
},
Session: []session.Attachable{up},
}, nil)
@@ -4833,7 +4655,6 @@ COPY foo /
func testTarContextExternalDockerfile(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
- isFileOp := getFileOp(t, sb)
foo := []byte("contents")
@@ -4855,11 +4676,11 @@ func testTarContextExternalDockerfile(t *testing.T, sb integration.Sandbox) {
FROM scratch
COPY foo bar
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -4869,16 +4690,13 @@ COPY foo bar
url := up.Add(buf)
// repeat with changed default args should match the old cache
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
- "build-arg:BUILDKIT_DISABLE_FILEOP": strconv.FormatBool(!isFileOp),
- "context": url,
- "dockerfilekey": builder.DefaultLocalNameDockerfile,
- "contextsubdir": "sub/dir",
+ "context": url,
+ "dockerfilekey": builder.DefaultLocalNameDockerfile,
+ "contextsubdir": "sub/dir",
},
Session: []session.Attachable{up},
LocalDirs: map[string]string{
@@ -4893,7 +4711,7 @@ COPY foo bar
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "bar"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "bar"))
require.NoError(t, err)
require.Equal(t, string(dt), "contents")
}
@@ -4907,12 +4725,12 @@ func testFrontendUseForwardedSolveResults(t *testing.T, sb integration.Sandbox)
FROM scratch
COPY foo foo2
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
fstest.CreateFile("foo", []byte("data"), 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
frontend := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
res, err := c.Solve(ctx, gateway.SolveRequest{
@@ -4946,9 +4764,7 @@ COPY foo foo2
})
}
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = c.Build(sb.Context(), client.SolveOpt{
Exports: []client.ExportEntry{
@@ -4964,7 +4780,7 @@ COPY foo foo2
}, "", frontend, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "foo3"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "foo3"))
require.NoError(t, err)
require.Equal(t, dt, []byte("data"))
}
@@ -4976,9 +4792,7 @@ func testFrontendInputs(t *testing.T, sb integration.Sandbox) {
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
outMount := llb.Image("busybox").Run(
llb.Shlex(`sh -c "cat /dev/urandom | head -c 100 | sha256sum > /out/foo"`),
@@ -4997,7 +4811,7 @@ func testFrontendInputs(t *testing.T, sb integration.Sandbox) {
}, nil)
require.NoError(t, err)
- expected, err := ioutil.ReadFile(filepath.Join(destDir, "foo"))
+ expected, err := os.ReadFile(filepath.Join(destDir, "foo"))
require.NoError(t, err)
dockerfile := []byte(`
@@ -5005,11 +4819,11 @@ FROM scratch
COPY foo foo2
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
Exports: []client.ExportEntry{
@@ -5027,13 +4841,16 @@ COPY foo foo2
}, nil)
require.NoError(t, err)
- actual, err := ioutil.ReadFile(filepath.Join(destDir, "foo2"))
+ actual, err := os.ReadFile(filepath.Join(destDir, "foo2"))
require.NoError(t, err)
require.Equal(t, expected, actual)
}
func testFrontendSubrequests(t *testing.T, sb integration.Sandbox) {
f := getFrontend(t, sb)
+ if _, ok := f.(*clientFrontend); !ok {
+ t.Skip("only test with client frontend")
+ }
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -5044,15 +4861,11 @@ FROM scratch
COPY Dockerfile Dockerfile
`)
- if gf, ok := f.(*gatewayFrontend); ok {
- dockerfile = []byte(fmt.Sprintf("#syntax=%s\n\n%s", gf.gw, dockerfile))
- }
-
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
called := false
@@ -5070,6 +4883,7 @@ COPY Dockerfile Dockerfile
require.Equal(t, subrequests.RequestType("rpc"), req.Type)
require.NotEqual(t, req.Version, "")
require.True(t, len(req.Metadata) > 0)
+ require.Equal(t, "result.json", req.Metadata[0].Name)
}
}
require.True(t, hasDescribe)
@@ -5121,11 +4935,11 @@ RUN echo $HOSTNAME | grep foo
RUN echo $(hostname) | grep foo
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
@@ -5179,19 +4993,17 @@ FROM scratch
COPY --from=base /shmsize /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
@@ -5210,7 +5022,7 @@ COPY --from=base /shmsize /
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "shmsize"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "shmsize"))
require.NoError(t, err)
require.Contains(t, string(dt), `size=131072k`)
}
@@ -5224,19 +5036,17 @@ FROM scratch
COPY --from=base /ulimit /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
@@ -5255,7 +5065,7 @@ COPY --from=base /ulimit /
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "ulimit"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "ulimit"))
require.NoError(t, err)
require.Equal(t, `1062`, strings.TrimSpace(string(dt)))
}
@@ -5273,19 +5083,17 @@ FROM scratch
COPY --from=base /out /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
c, err := client.New(sb.Context(), sb.Address())
require.NoError(t, err)
defer c.Close()
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
@@ -5304,7 +5112,7 @@ COPY --from=base /out /
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "out"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "out"))
require.NoError(t, err)
require.Contains(t, strings.TrimSpace(string(dt)), `/foocgroup/buildkit/`)
}
@@ -5323,20 +5131,19 @@ FROM scratch
COPY --from=base /out /
`)
- dir, err := tmpdir(
+ dir, err := integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
f := getFrontend(t, sb)
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
+ destDir := t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
+ // Make sure image resolution works as expected, do not add a tag or locator.
"context:busybox": "docker-image://alpine",
},
LocalDirs: map[string]string{
@@ -5352,55 +5159,77 @@ COPY --from=base /out /
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "out"))
+ dt, err := os.ReadFile(filepath.Join(destDir, "out"))
require.NoError(t, err)
require.True(t, len(dt) > 0)
-}
-
-func testNamedLocalContext(t *testing.T, sb integration.Sandbox) {
- ctx := sb.Context()
- c, err := client.New(ctx, sb.Address())
- require.NoError(t, err)
- defer c.Close()
+ integration.CheckFeatureCompat(t, sb, integration.FeatureDirectPush)
- dockerfile := []byte(`
-FROM busybox AS base
-RUN cat /etc/alpine-release > /out
-FROM scratch
-COPY --from=base /o* /
+ // Now test with an image with custom envs
+ dockerfile = []byte(`
+FROM alpine:latest
+ENV PATH=/foobar:$PATH
+ENV FOOBAR=foobar
`)
- dir, err := tmpdir(
+ dir, err = integration.Tmpdir(
+ t,
fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
- outf := []byte(`dummy-result`)
+ registry, err := sb.NewRegistry()
+ if errors.Is(err, integration.ErrRequirements) {
+ t.Skip(err.Error())
+ }
+ require.NoError(t, err)
+ target := registry + "/buildkit/testnamedimagecontext:latest"
- dir2, err := tmpdir(
- fstest.CreateFile("out", outf, 0600),
- fstest.CreateFile("out2", outf, 0600),
- fstest.CreateFile(".dockerignore", []byte("out2\n"), 0600),
- )
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterImage,
+ Attrs: map[string]string{
+ "name": target,
+ "push": "true",
+ },
+ },
+ },
+ }, nil)
require.NoError(t, err)
- defer os.RemoveAll(dir2)
- f := getFrontend(t, sb)
+ dockerfile = []byte(`
+FROM busybox AS base
+RUN cat /etc/alpine-release > /out
+RUN env | grep PATH > /env_path
+RUN env | grep FOOBAR > /env_foobar
+FROM scratch
+COPY --from=base /out /
+COPY --from=base /env_path /
+COPY --from=base /env_foobar /
+ `)
- destDir, err := ioutil.TempDir("", "buildkit")
+ dir, err = integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
require.NoError(t, err)
- defer os.RemoveAll(destDir)
+
+ f = getFrontend(t, sb)
+
+ destDir = t.TempDir()
_, err = f.Solve(sb.Context(), c, client.SolveOpt{
FrontendAttrs: map[string]string{
- "context:base": "local:basedir",
+ "context:busybox": "docker-image://" + target,
},
LocalDirs: map[string]string{
builder.DefaultLocalNameDockerfile: dir,
builder.DefaultLocalNameContext: dir,
- "basedir": dir2,
},
Exports: []client.ExportEntry{
{
@@ -5411,284 +5240,1388 @@ COPY --from=base /o* /
}, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "out"))
+ dt, err = os.ReadFile(filepath.Join(destDir, "out"))
require.NoError(t, err)
require.True(t, len(dt) > 0)
- _, err = ioutil.ReadFile(filepath.Join(destDir, "out2"))
- require.Error(t, err)
- require.True(t, errors.Is(err, os.ErrNotExist))
+ dt, err = os.ReadFile(filepath.Join(destDir, "env_foobar"))
+ require.NoError(t, err)
+ require.Equal(t, "FOOBAR=foobar", strings.TrimSpace(string(dt)))
+
+ dt, err = os.ReadFile(filepath.Join(destDir, "env_path"))
+ require.NoError(t, err)
+ require.Contains(t, string(dt), "/foobar:")
}
-func testNamedInputContext(t *testing.T, sb integration.Sandbox) {
+func testNamedImageContextPlatform(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureDirectPush)
ctx := sb.Context()
c, err := client.New(ctx, sb.Address())
require.NoError(t, err)
defer c.Close()
- dockerfile := []byte(`
-FROM alpine
-ENV FOO=bar
-RUN echo first > /out
-`)
-
- dir, err := tmpdir(
- fstest.CreateFile("Dockerfile", dockerfile, 0600),
- )
+ registry, err := sb.NewRegistry()
+ if errors.Is(err, integration.ErrRequirements) {
+ t.Skip(err.Error())
+ }
require.NoError(t, err)
- defer os.RemoveAll(dir)
- dockerfile2 := []byte(`
-FROM base AS build
-RUN echo "foo is $FOO" > /foo
-FROM scratch
-COPY --from=build /foo /out /
-`)
+ // Build a base image and force buildkit to generate a manifest list.
+ dockerfile := []byte(`FROM --platform=$BUILDPLATFORM alpine:latest`)
+ target := registry + "/buildkit/testnamedimagecontextplatform:latest"
- dir2, err := tmpdir(
- fstest.CreateFile("Dockerfile", dockerfile2, 0600),
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
f := getFrontend(t, sb)
- b := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
- res, err := f.SolveGateway(ctx, c, gateway.SolveRequest{})
- if err != nil {
- return nil, err
- }
- ref, err := res.SingleRef()
- if err != nil {
- return nil, err
- }
- st, err := ref.ToState()
- if err != nil {
- return nil, err
- }
-
- def, err := st.Marshal(ctx)
- if err != nil {
- return nil, err
- }
-
- dt, ok := res.Metadata["containerimage.config"]
- if !ok {
- return nil, errors.Errorf("no containerimage.config in metadata")
- }
-
- dt, err = json.Marshal(map[string][]byte{
- "containerimage.config": dt,
- })
- if err != nil {
- return nil, err
- }
-
- res, err = f.SolveGateway(ctx, c, gateway.SolveRequest{
- FrontendOpt: map[string]string{
- "dockerfilekey": builder.DefaultLocalNameDockerfile + "2",
- "context:base": "input:base",
- "input-metadata:base": string(dt),
- },
- FrontendInputs: map[string]*pb.Definition{
- "base": def.ToPB(),
- },
- })
- if err != nil {
- return nil, err
- }
- return res, nil
- }
-
- product := "buildkit_test"
-
- destDir, err := ioutil.TempDir("", "buildkit")
- require.NoError(t, err)
- defer os.RemoveAll(destDir)
-
- _, err = c.Build(ctx, client.SolveOpt{
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ FrontendAttrs: map[string]string{
+ "build-arg:BUILDKIT_MULTI_PLATFORM": "true",
+ },
LocalDirs: map[string]string{
- builder.DefaultLocalNameDockerfile: dir,
- builder.DefaultLocalNameContext: dir,
- builder.DefaultLocalNameDockerfile + "2": dir2,
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
},
Exports: []client.ExportEntry{
{
- Type: client.ExporterLocal,
- OutputDir: destDir,
+ Type: client.ExporterImage,
+ Attrs: map[string]string{
+ "name": target,
+ "push": "true",
+ },
},
},
- }, product, b, nil)
+ }, nil)
require.NoError(t, err)
- dt, err := ioutil.ReadFile(filepath.Join(destDir, "out"))
+ dockerfile = []byte(`
+FROM --platform=$BUILDPLATFORM busybox AS target
+RUN echo hello
+`)
+
+ dir, err = integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
require.NoError(t, err)
- require.Equal(t, "first\n", string(dt))
- dt, err = ioutil.ReadFile(filepath.Join(destDir, "foo"))
+ f = getFrontend(t, sb)
+
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ FrontendAttrs: map[string]string{
+ "context:busybox": "docker-image://" + target,
+ // random platform that would never exist so it doesn't conflict with the build machine
+ // here we specifically want to make sure that the platform chosen for the image source is the one in the dockerfile not the target platform.
+ "platform": "darwin/ppc64le",
+ },
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ }, nil)
require.NoError(t, err)
- require.Equal(t, "foo is bar\n", string(dt))
}
-func testNamedMultiplatformInputContext(t *testing.T, sb integration.Sandbox) {
+func testNamedImageContextTimestamps(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureDirectPush)
ctx := sb.Context()
c, err := client.New(ctx, sb.Address())
require.NoError(t, err)
defer c.Close()
- dockerfile := []byte(`
-FROM --platform=$BUILDPLATFORM alpine
-ARG TARGETARCH
-ENV FOO=bar-$TARGETARCH
-RUN echo "foo $TARGETARCH" > /out
-`)
-
- dir, err := tmpdir(
- fstest.CreateFile("Dockerfile", dockerfile, 0600),
- )
+ registry, err := sb.NewRegistry()
+ if errors.Is(err, integration.ErrRequirements) {
+ t.Skip(err.Error())
+ }
require.NoError(t, err)
- defer os.RemoveAll(dir)
- dockerfile2 := []byte(`
-FROM base AS build
-RUN echo "foo is $FOO" > /foo
-FROM scratch
-COPY --from=build /foo /out /
-`)
+ f := getFrontend(t, sb)
- dir2, err := tmpdir(
- fstest.CreateFile("Dockerfile", dockerfile2, 0600),
+ dockerfile := []byte(`
+FROM alpine
+RUN echo foo >> /test
+`)
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
)
require.NoError(t, err)
- defer os.RemoveAll(dir)
-
- f := getFrontend(t, sb)
- b := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
- res, err := f.SolveGateway(ctx, c, gateway.SolveRequest{
- FrontendOpt: map[string]string{
- "platform": "linux/amd64,linux/arm64",
- },
- })
- if err != nil {
+ target := registry + "/buildkit/testnamedimagecontexttimestamps:latest"
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterImage,
+ Attrs: map[string]string{
+ "name": target,
+ "push": "true",
+ },
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+
+ desc, provider, err := contentutil.ProviderFromRef(target)
+ require.NoError(t, err)
+ img, err := testutil.ReadImage(sb.Context(), provider, desc)
+ require.NoError(t, err)
+
+ dirDerived, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ targetDerived := registry + "/buildkit/testnamedimagecontexttimestampsderived:latest"
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ FrontendAttrs: map[string]string{
+ "context:alpine": "docker-image://" + target,
+ },
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dirDerived,
+ builder.DefaultLocalNameContext: dirDerived,
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterImage,
+ Attrs: map[string]string{
+ "name": targetDerived,
+ "push": "true",
+ },
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+
+ desc, provider, err = contentutil.ProviderFromRef(targetDerived)
+ require.NoError(t, err)
+ imgDerived, err := testutil.ReadImage(sb.Context(), provider, desc)
+ require.NoError(t, err)
+
+ require.NotEqual(t, img.Img.Created, imgDerived.Img.Created)
+ diff := imgDerived.Img.Created.Sub(*img.Img.Created)
+ require.Greater(t, diff, time.Duration(0))
+ require.Less(t, diff, 10*time.Minute)
+}
+
+func testNamedImageContextScratch(t *testing.T, sb integration.Sandbox) {
+ ctx := sb.Context()
+
+ c, err := client.New(ctx, sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ dockerfile := []byte(`
+FROM busybox
+COPY < /out
+FROM scratch
+COPY --from=base /o* /
+`)
+
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ outf := []byte(`dummy-result`)
+
+ dir2, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("out", outf, 0600),
+ fstest.CreateFile("out2", outf, 0600),
+ fstest.CreateFile(".dockerignore", []byte("out2\n"), 0600),
+ )
+ require.NoError(t, err)
+
+ f := getFrontend(t, sb)
+
+ destDir := t.TempDir()
+
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ FrontendAttrs: map[string]string{
+ "context:base": "local:basedir",
+ },
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ "basedir": dir2,
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterLocal,
+ OutputDir: destDir,
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+
+ dt, err := os.ReadFile(filepath.Join(destDir, "out"))
+ require.NoError(t, err)
+ require.True(t, len(dt) > 0)
+
+ _, err = os.ReadFile(filepath.Join(destDir, "out2"))
+ require.Error(t, err)
+ require.True(t, errors.Is(err, os.ErrNotExist))
+}
+
+func testNamedOCILayoutContext(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureOCIExporter, integration.FeatureOCILayout)
+ // how this test works:
+ // 1- we use a regular builder with a dockerfile to create an image two files: "out" with content "first", "out2" with content "second"
+ // 2- we save the output to an OCI layout dir
+ // 3- we use another regular builder with a dockerfile to build using a referenced context "base", but override it to reference the output of the previous build
+ // 4- we check that the output of the second build matches our OCI layout, and not the referenced image
+ ctx := sb.Context()
+
+ c, err := client.New(ctx, sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ // create a tempdir where we will store the OCI layout
+ ocidir := t.TempDir()
+
+ ociDockerfile := []byte(`
+ FROM busybox:latest
+ WORKDIR /test
+ RUN sh -c "echo -n first > out"
+ RUN sh -c "echo -n second > out2"
+ ENV foo=bar
+ `)
+ inDir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", ociDockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ f := getFrontend(t, sb)
+
+ outW := bytes.NewBuffer(nil)
+
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: inDir,
+ builder.DefaultLocalNameContext: inDir,
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterOCI,
+ Output: fixedWriteCloser(nopWriteCloser{outW}),
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+
+ // extract the tar stream to the directory as OCI layout
+ m, err := testutil.ReadTarToMap(outW.Bytes(), false)
+ require.NoError(t, err)
+
+ for filename, content := range m {
+ fullFilename := path.Join(ocidir, filename)
+ err = os.MkdirAll(path.Dir(fullFilename), 0755)
+ require.NoError(t, err)
+ if content.Header.FileInfo().IsDir() {
+ err = os.MkdirAll(fullFilename, 0755)
+ require.NoError(t, err)
+ } else {
+ err = os.WriteFile(fullFilename, content.Data, 0644)
+ require.NoError(t, err)
+ }
+ }
+
+ var index ocispecs.Index
+ err = json.Unmarshal(m["index.json"].Data, &index)
+ require.NoError(t, err)
+ require.Equal(t, 1, len(index.Manifests))
+ digest := index.Manifests[0].Digest.Hex()
+
+ store, err := local.NewStore(ocidir)
+ ociID := "ocione"
+ require.NoError(t, err)
+
+ // we will use this simple dockerfile to test
+ // 1. busybox is used as is, but because we override the context for base,
+ // when we run `COPY --from=base`, it should take the /o* from the image in the store,
+ // rather than what we built on the first 2 lines here.
+ // 2. we override the context for `foo` to be our local OCI store, which has an `ENV foo=bar` override.
+ // As such, the `RUN echo $foo` step should have `$foo` set to `"bar"`, and so
+ // when we `COPY --from=imported`, it should have the content of `/outfoo` as `"bar"`
+ dockerfile := []byte(`
+FROM busybox AS base
+RUN cat /etc/alpine-release > out
+
+FROM foo AS imported
+RUN echo -n $foo > outfoo
+
+FROM scratch
+COPY --from=base /test/o* /
+COPY --from=imported /test/outfoo /
+`)
+
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ destDir := t.TempDir()
+
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ FrontendAttrs: map[string]string{
+ "context:base": fmt.Sprintf("oci-layout:%s@sha256:%s", ociID, digest),
+ "context:foo": fmt.Sprintf("oci-layout:%s@sha256:%s", ociID, digest),
+ },
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ OCIStores: map[string]content.Store{
+ ociID: store,
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterLocal,
+ OutputDir: destDir,
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+
+ dt, err := os.ReadFile(filepath.Join(destDir, "out"))
+ require.NoError(t, err)
+ require.True(t, len(dt) > 0)
+ require.Equal(t, []byte("first"), dt)
+
+ dt, err = os.ReadFile(filepath.Join(destDir, "out2"))
+ require.NoError(t, err)
+ require.True(t, len(dt) > 0)
+ require.Equal(t, []byte("second"), dt)
+
+ dt, err = os.ReadFile(filepath.Join(destDir, "outfoo"))
+ require.NoError(t, err)
+ require.True(t, len(dt) > 0)
+ require.Equal(t, []byte("bar"), dt)
+}
+
+func testNamedOCILayoutContextExport(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureOCIExporter, integration.FeatureOCILayout)
+ ctx := sb.Context()
+
+ c, err := client.New(ctx, sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ ocidir := t.TempDir()
+
+ dockerfile := []byte(`
+FROM scratch
+WORKDIR /test
+ENV foo=bar
+ `)
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ f := getFrontend(t, sb)
+
+ outW := bytes.NewBuffer(nil)
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ Exports: []client.ExportEntry{{
+ Type: client.ExporterOCI,
+ Output: fixedWriteCloser(nopWriteCloser{outW}),
+ }},
+ }, nil)
+ require.NoError(t, err)
+
+ m, err := testutil.ReadTarToMap(outW.Bytes(), false)
+ require.NoError(t, err)
+
+ for filename, content := range m {
+ fullFilename := path.Join(ocidir, filename)
+ err = os.MkdirAll(path.Dir(fullFilename), 0755)
+ require.NoError(t, err)
+ if content.Header.FileInfo().IsDir() {
+ err = os.MkdirAll(fullFilename, 0755)
+ require.NoError(t, err)
+ } else {
+ err = os.WriteFile(fullFilename, content.Data, 0644)
+ require.NoError(t, err)
+ }
+ }
+
+ var index ocispecs.Index
+ err = json.Unmarshal(m["index.json"].Data, &index)
+ require.NoError(t, err)
+ require.Equal(t, 1, len(index.Manifests))
+ digest := index.Manifests[0].Digest.Hex()
+
+ store, err := local.NewStore(ocidir)
+ ociID := "ocione"
+ require.NoError(t, err)
+
+ dockerfile = []byte(`
+FROM nonexistent AS base
+`)
+
+ dir, err = integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ outW = bytes.NewBuffer(nil)
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ FrontendAttrs: map[string]string{
+ "context:nonexistent": fmt.Sprintf("oci-layout:%s@sha256:%s", ociID, digest),
+ },
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ OCIStores: map[string]content.Store{
+ ociID: store,
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterOCI,
+ Output: fixedWriteCloser(nopWriteCloser{outW}),
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+
+ m, err = testutil.ReadTarToMap(outW.Bytes(), false)
+ require.NoError(t, err)
+
+ err = json.Unmarshal(m["index.json"].Data, &index)
+ require.NoError(t, err)
+ require.Equal(t, 1, len(index.Manifests))
+ digest = index.Manifests[0].Digest.Hex()
+
+ var mfst ocispecs.Manifest
+ require.NoError(t, json.Unmarshal(m["blobs/sha256/"+digest].Data, &mfst))
+ digest = mfst.Config.Digest.Hex()
+
+ var cfg ocispecs.Image
+ require.NoError(t, json.Unmarshal(m["blobs/sha256/"+digest].Data, &cfg))
+
+ require.Equal(t, "/test", cfg.Config.WorkingDir)
+ require.Contains(t, cfg.Config.Env, "foo=bar")
+}
+
+func testNamedInputContext(t *testing.T, sb integration.Sandbox) {
+ ctx := sb.Context()
+
+ c, err := client.New(ctx, sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ dockerfile := []byte(`
+FROM alpine
+ENV FOO=bar
+RUN echo first > /out
+`)
+
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ dockerfile2 := []byte(`
+FROM base AS build
+RUN echo "foo is $FOO" > /foo
+FROM scratch
+COPY --from=build /foo /out /
+`)
+
+ dir2, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile2, 0600),
+ )
+ require.NoError(t, err)
+
+ f := getFrontend(t, sb)
+
+ b := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
+ res, err := f.SolveGateway(ctx, c, gateway.SolveRequest{})
+ if err != nil {
+ return nil, err
+ }
+ ref, err := res.SingleRef()
+ if err != nil {
+ return nil, err
+ }
+ st, err := ref.ToState()
+ if err != nil {
+ return nil, err
+ }
+
+ def, err := st.Marshal(ctx)
+ if err != nil {
+ return nil, err
+ }
+
+ dt, ok := res.Metadata["containerimage.config"]
+ if !ok {
+ return nil, errors.Errorf("no containerimage.config in metadata")
+ }
+
+ dt, err = json.Marshal(map[string][]byte{
+ "containerimage.config": dt,
+ })
+ if err != nil {
+ return nil, err
+ }
+
+ res, err = f.SolveGateway(ctx, c, gateway.SolveRequest{
+ FrontendOpt: map[string]string{
+ "dockerfilekey": builder.DefaultLocalNameDockerfile + "2",
+ "context:base": "input:base",
+ "input-metadata:base": string(dt),
+ },
+ FrontendInputs: map[string]*pb.Definition{
+ "base": def.ToPB(),
+ },
+ })
+ if err != nil {
+ return nil, err
+ }
+ return res, nil
+ }
+
+ product := "buildkit_test"
+
+ destDir := t.TempDir()
+
+ _, err = c.Build(ctx, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ builder.DefaultLocalNameDockerfile + "2": dir2,
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterLocal,
+ OutputDir: destDir,
+ },
+ },
+ }, product, b, nil)
+ require.NoError(t, err)
+
+ dt, err := os.ReadFile(filepath.Join(destDir, "out"))
+ require.NoError(t, err)
+ require.Equal(t, "first\n", string(dt))
+
+ dt, err = os.ReadFile(filepath.Join(destDir, "foo"))
+ require.NoError(t, err)
+ require.Equal(t, "foo is bar\n", string(dt))
+}
+
+func testNamedMultiplatformInputContext(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureMultiPlatform)
+ ctx := sb.Context()
+
+ c, err := client.New(ctx, sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ dockerfile := []byte(`
+FROM --platform=$BUILDPLATFORM alpine
+ARG TARGETARCH
+ENV FOO=bar-$TARGETARCH
+RUN echo "foo $TARGETARCH" > /out
+`)
+
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ dockerfile2 := []byte(`
+FROM base AS build
+RUN echo "foo is $FOO" > /foo
+FROM scratch
+COPY --from=build /foo /out /
+`)
+
+ dir2, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile2, 0600),
+ )
+ require.NoError(t, err)
+
+ f := getFrontend(t, sb)
+
+ b := func(ctx context.Context, c gateway.Client) (*gateway.Result, error) {
+ res, err := f.SolveGateway(ctx, c, gateway.SolveRequest{
+ FrontendOpt: map[string]string{
+ "platform": "linux/amd64,linux/arm64",
+ },
+ })
+ if err != nil {
+ return nil, err
+ }
+
+ if len(res.Refs) != 2 {
+ return nil, errors.Errorf("expected 2 refs, got %d", len(res.Refs))
+ }
+
+ inputs := map[string]*pb.Definition{}
+ st, err := res.Refs["linux/amd64"].ToState()
+ if err != nil {
+ return nil, err
+ }
+ def, err := st.Marshal(ctx)
+ if err != nil {
+ return nil, err
+ }
+ inputs["base::linux/amd64"] = def.ToPB()
+
+ st, err = res.Refs["linux/arm64"].ToState()
+ if err != nil {
+ return nil, err
+ }
+ def, err = st.Marshal(ctx)
+ if err != nil {
+ return nil, err
+ }
+ inputs["base::linux/arm64"] = def.ToPB()
+
+ frontendOpt := map[string]string{
+ "dockerfilekey": builder.DefaultLocalNameDockerfile + "2",
+ "context:base::linux/amd64": "input:base::linux/amd64",
+ "context:base::linux/arm64": "input:base::linux/arm64",
+ "platform": "linux/amd64,linux/arm64",
+ }
+
+ dt, ok := res.Metadata["containerimage.config/linux/amd64"]
+ if !ok {
+ return nil, errors.Errorf("no containerimage.config in metadata")
+ }
+ dt, err = json.Marshal(map[string][]byte{
+ "containerimage.config": dt,
+ })
+ if err != nil {
+ return nil, err
+ }
+ frontendOpt["input-metadata:base::linux/amd64"] = string(dt)
+
+ dt, ok = res.Metadata["containerimage.config/linux/arm64"]
+ if !ok {
+ return nil, errors.Errorf("no containerimage.config in metadata")
+ }
+ dt, err = json.Marshal(map[string][]byte{
+ "containerimage.config": dt,
+ })
+ if err != nil {
+ return nil, err
+ }
+ frontendOpt["input-metadata:base::linux/arm64"] = string(dt)
+
+ res, err = f.SolveGateway(ctx, c, gateway.SolveRequest{
+ FrontendOpt: frontendOpt,
+ FrontendInputs: inputs,
+ })
+ if err != nil {
return nil, err
}
+ return res, nil
+ }
+
+ product := "buildkit_test"
+
+ destDir := t.TempDir()
+
+ _, err = c.Build(ctx, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ builder.DefaultLocalNameDockerfile + "2": dir2,
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterLocal,
+ OutputDir: destDir,
+ },
+ },
+ }, product, b, nil)
+ require.NoError(t, err)
+
+ dt, err := os.ReadFile(filepath.Join(destDir, "linux_amd64/out"))
+ require.NoError(t, err)
+ require.Equal(t, "foo amd64\n", string(dt))
+
+ dt, err = os.ReadFile(filepath.Join(destDir, "linux_amd64/foo"))
+ require.NoError(t, err)
+ require.Equal(t, "foo is bar-amd64\n", string(dt))
+
+ dt, err = os.ReadFile(filepath.Join(destDir, "linux_arm64/out"))
+ require.NoError(t, err)
+ require.Equal(t, "foo arm64\n", string(dt))
+
+ dt, err = os.ReadFile(filepath.Join(destDir, "linux_arm64/foo"))
+ require.NoError(t, err)
+ require.Equal(t, "foo is bar-arm64\n", string(dt))
+}
+
+func testSourceDateEpochWithoutExporter(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureOCIExporter, integration.FeatureSourceDateEpoch)
+ f := getFrontend(t, sb)
+
+ dockerfile := []byte(`
+FROM scratch
+ENTRYPOINT foo bar
+COPY Dockerfile .
+`)
+
+ dir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+ defer os.RemoveAll(dir)
+
+ c, err := client.New(sb.Context(), sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ destDir, err := os.MkdirTemp("", "buildkit")
+ require.NoError(t, err)
+ defer os.RemoveAll(destDir)
+
+ out := filepath.Join(destDir, "out.tar")
+ outW, err := os.Create(out)
+ require.NoError(t, err)
+
+ tm := time.Date(2015, time.October, 21, 7, 28, 0, 0, time.UTC)
+
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ FrontendAttrs: map[string]string{
+ "build-arg:SOURCE_DATE_EPOCH": fmt.Sprintf("%d", tm.Unix()),
+ },
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: dir,
+ builder.DefaultLocalNameContext: dir,
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterOCI,
+ // disable exporter epoch to make sure we test dockerfile
+ Attrs: map[string]string{"source-date-epoch": ""},
+ Output: fixedWriteCloser(outW),
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+
+ dt, err := os.ReadFile(filepath.Join(destDir, "out.tar"))
+ require.NoError(t, err)
+
+ m, err := testutil.ReadTarToMap(dt, false)
+ require.NoError(t, err)
+
+ var idx ocispecs.Index
+ err = json.Unmarshal(m["index.json"].Data, &idx)
+ require.NoError(t, err)
+
+ mlistHex := idx.Manifests[0].Digest.Hex()
+
+ var mfst ocispecs.Manifest
+ err = json.Unmarshal(m["blobs/sha256/"+mlistHex].Data, &mfst)
+ require.NoError(t, err)
+
+ var img ocispecs.Image
+ err = json.Unmarshal(m["blobs/sha256/"+mfst.Config.Digest.Hex()].Data, &img)
+ require.NoError(t, err)
+
+ require.Equal(t, tm.Unix(), img.Created.Unix())
+ for _, h := range img.History {
+ require.Equal(t, tm.Unix(), h.Created.Unix())
+ }
+}
+
+func testSBOMScannerImage(t *testing.T, sb integration.Sandbox) {
+ integration.CheckFeatureCompat(t, sb, integration.FeatureDirectPush, integration.FeatureSBOM)
+ ctx := sb.Context()
+
+ c, err := client.New(ctx, sb.Address())
+ require.NoError(t, err)
+ defer c.Close()
+
+ registry, err := sb.NewRegistry()
+ if errors.Is(err, integration.ErrRequirements) {
+ t.Skip(err.Error())
+ }
+ require.NoError(t, err)
+
+ f := getFrontend(t, sb)
+
+ dockerfile := []byte(`
+FROM busybox:latest
+COPY <<-"EOF" /scan.sh
+ set -e
+ cat < $BUILDKIT_SCAN_DESTINATION/spdx.json
+ {
+ "_type": "https://in-toto.io/Statement/v0.1",
+ "predicateType": "https://spdx.dev/Document",
+ "predicate": {"name": "sbom-scan"}
+ }
+ BUNDLE
+EOF
+CMD sh /scan.sh
+`)
+ scannerDir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ scannerTarget := registry + "/buildkit/testsbomscanner:latest"
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: scannerDir,
+ builder.DefaultLocalNameContext: scannerDir,
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterImage,
+ Attrs: map[string]string{
+ "name": scannerTarget,
+ "push": "true",
+ },
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+
+ dockerfile = []byte(`
+FROM scratch
+COPY < $BUILDKIT_SCAN_DESTINATION/spdx.json
+ {
+ "_type": "https://in-toto.io/Statement/v0.1",
+ "predicateType": "https://spdx.dev/Document",
+ "predicate": {"name": "core"}
+ }
+ BUNDLE
+ if [ "${BUILDKIT_SCAN_SOURCE_EXTRAS}" ]; then
+ for src in "${BUILDKIT_SCAN_SOURCE_EXTRAS}"/*; do
+ cat < $BUILDKIT_SCAN_DESTINATION/$(basename $src).spdx.json
+ {
+ "_type": "https://in-toto.io/Statement/v0.1",
+ "predicateType": "https://spdx.dev/Document",
+ "predicate": {"name": "extra"}
+ }
+ BUNDLE
+ done
+ fi
+EOF
+CMD sh /scan.sh
+`)
+
+ scannerDir, err := integration.Tmpdir(
+ t,
+ fstest.CreateFile("Dockerfile", dockerfile, 0600),
+ )
+ require.NoError(t, err)
+
+ scannerTarget := registry + "/buildkit/testsbomscannerargs:latest"
+ _, err = f.Solve(sb.Context(), c, client.SolveOpt{
+ LocalDirs: map[string]string{
+ builder.DefaultLocalNameDockerfile: scannerDir,
+ builder.DefaultLocalNameContext: scannerDir,
+ },
+ Exports: []client.ExportEntry{
+ {
+ Type: client.ExporterImage,
+ Attrs: map[string]string{
+ "name": scannerTarget,
+ "push": "true",
+ },
+ },
+ },
+ }, nil)
+ require.NoError(t, err)
+
+ // scan an image with no additional sboms
+ dockerfile = []byte(`
+FROM scratch as base
+COPY < **Note on whitespace**
+>
+> For backward compatibility, leading whitespace before comments (`#`) and
+> instructions (such as `RUN`) are ignored, but discouraged. Leading whitespace
+> is not preserved in these cases, and the following examples are therefore
+> equivalent:
+>
+> ```dockerfile
+> # this is a comment-line
+> RUN echo hello
+> RUN echo world
+> ```
+>
+> ```dockerfile
+> # this is a comment-line
+> RUN echo hello
+> RUN echo world
+> ```
+>
+> Note however, that whitespace in instruction _arguments_, such as the commands
+> following `RUN`, are preserved, so the following example prints ` hello world`
+> with leading whitespace as specified:
+>
+> ```dockerfile
+> RUN echo "\
+> hello\
+> world"
+> ```
+
+## Parser directives
+
+Parser directives are optional, and affect the way in which subsequent lines
+in a `Dockerfile` are handled. Parser directives do not add layers to the build,
+and will not be shown as a build step. Parser directives are written as a
+special type of comment in the form `# directive=value`. A single directive
+may only be used once.
+
+Once a comment, empty line or builder instruction has been processed, Docker
+no longer looks for parser directives. Instead it treats anything formatted
+as a parser directive as a comment and does not attempt to validate if it might
+be a parser directive. Therefore, all parser directives must be at the very
+top of a `Dockerfile`.
+
+Parser directives are not case-sensitive. However, convention is for them to
+be lowercase. Convention is also to include a blank line following any
+parser directives. Line continuation characters are not supported in parser
+directives.
+
+Due to these rules, the following examples are all invalid:
+
+Invalid due to line continuation:
+
+```dockerfile
+# direc \
+tive=value
+```
+
+Invalid due to appearing twice:
+
+```dockerfile
+# directive=value1
+# directive=value2
+
+FROM ImageName
+```
+
+Treated as a comment due to appearing after a builder instruction:
+
+```dockerfile
+FROM ImageName
+# directive=value
+```
+
+Treated as a comment due to appearing after a comment which is not a parser
+directive:
+
+```dockerfile
+# About my dockerfile
+# directive=value
+FROM ImageName
+```
+
+The unknown directive is treated as a comment due to not being recognized. In
+addition, the known directive is treated as a comment due to appearing after
+a comment which is not a parser directive.
+
+```dockerfile
+# unknowndirective=value
+# knowndirective=value
+```
+
+Non line-breaking whitespace is permitted in a parser directive. Hence, the
+following lines are all treated identically:
+
+```dockerfile
+#directive=value
+# directive =value
+# directive= value
+# directive = value
+# dIrEcTiVe=value
+```
+
+The following parser directives are supported:
+
+- `syntax`
+- `escape`
+
+### syntax
+
+
+
+This feature is only available when using the [BuildKit](https://docs.docker.com/build/buildkit/)
+backend, and is ignored when using the classic builder backend.
+
+See [Custom Dockerfile syntax](https://docs.docker.com/build/buildkit/dockerfile-frontend/)
+page for more information.
+
+### escape
+
+```dockerfile
+# escape=\ (backslash)
+```
+
+Or
+
+```dockerfile
+# escape=` (backtick)
+```
+
+The `escape` directive sets the character used to escape characters in a
+`Dockerfile`. If not specified, the default escape character is `\`.
+
+The escape character is used both to escape characters in a line, and to
+escape a newline. This allows a `Dockerfile` instruction to
+span multiple lines. Note that regardless of whether the `escape` parser
+directive is included in a `Dockerfile`, *escaping is not performed in
+a `RUN` command, except at the end of a line.*
+
+Setting the escape character to `` ` `` is especially useful on
+`Windows`, where `\` is the directory path separator. `` ` `` is consistent
+with [Windows PowerShell](https://technet.microsoft.com/en-us/library/hh847755.aspx).
+
+Consider the following example which would fail in a non-obvious way on
+`Windows`. The second `\` at the end of the second line would be interpreted as an
+escape for the newline, instead of a target of the escape from the first `\`.
+Similarly, the `\` at the end of the third line would, assuming it was actually
+handled as an instruction, cause it be treated as a line continuation. The result
+of this dockerfile is that second and third lines are considered a single
+instruction:
+
+```dockerfile
+FROM microsoft/nanoserver
+COPY testfile.txt c:\\
+RUN dir c:\
+```
+
+Results in:
+
+```console
+PS E:\myproject> docker build -t cmd .
+
+Sending build context to Docker daemon 3.072 kB
+Step 1/2 : FROM microsoft/nanoserver
+ ---> 22738ff49c6d
+Step 2/2 : COPY testfile.txt c:\RUN dir c:
+GetFileAttributesEx c:RUN: The system cannot find the file specified.
+PS E:\myproject>
+```
+
+One solution to the above would be to use `/` as the target of both the `COPY`
+instruction, and `dir`. However, this syntax is, at best, confusing as it is not
+natural for paths on `Windows`, and at worst, error prone as not all commands on
+`Windows` support `/` as the path separator.
+
+By adding the `escape` parser directive, the following `Dockerfile` succeeds as
+expected with the use of natural platform semantics for file paths on `Windows`:
+
+```dockerfile
+# escape=`
+
+FROM microsoft/nanoserver
+COPY testfile.txt c:\
+RUN dir c:\
+```
+
+Results in:
+
+```console
+PS E:\myproject> docker build -t succeeds --no-cache=true .
+
+Sending build context to Docker daemon 3.072 kB
+Step 1/3 : FROM microsoft/nanoserver
+ ---> 22738ff49c6d
+Step 2/3 : COPY testfile.txt c:\
+ ---> 96655de338de
+Removing intermediate container 4db9acbb1682
+Step 3/3 : RUN dir c:\
+ ---> Running in a2c157f842f5
+ Volume in drive C has no label.
+ Volume Serial Number is 7E6D-E0F7
+
+ Directory of c:\
+
+10/05/2016 05:04 PM 1,894 License.txt
+10/05/2016 02:22 PM Program Files
+10/05/2016 02:14 PM Program Files (x86)
+10/28/2016 11:18 AM 62 testfile.txt
+10/28/2016 11:20 AM Users
+10/28/2016 11:20 AM Windows
+ 2 File(s) 1,956 bytes
+ 4 Dir(s) 21,259,096,064 bytes free
+ ---> 01c7f3bef04f
+Removing intermediate container a2c157f842f5
+Successfully built 01c7f3bef04f
+PS E:\myproject>
+```
+
+## Environment replacement
+
+Environment variables (declared with [the `ENV` statement](#env)) can also be
+used in certain instructions as variables to be interpreted by the
+`Dockerfile`. Escapes are also handled for including variable-like syntax
+into a statement literally.
+
+Environment variables are notated in the `Dockerfile` either with
+`$variable_name` or `${variable_name}`. They are treated equivalently and the
+brace syntax is typically used to address issues with variable names with no
+whitespace, like `${foo}_bar`.
+
+The `${variable_name}` syntax also supports a few of the standard `bash`
+modifiers as specified below:
+
+- `${variable:-word}` indicates that if `variable` is set then the result
+ will be that value. If `variable` is not set then `word` will be the result.
+- `${variable:+word}` indicates that if `variable` is set then `word` will be
+ the result, otherwise the result is the empty string.
+
+In all cases, `word` can be any string, including additional environment
+variables.
+
+Escaping is possible by adding a `\` before the variable: `\$foo` or `\${foo}`,
+for example, will translate to `$foo` and `${foo}` literals respectively.
+
+Example (parsed representation is displayed after the `#`):
+
+```dockerfile
+FROM busybox
+ENV FOO=/bar
+WORKDIR ${FOO} # WORKDIR /bar
+ADD . $FOO # ADD . /bar
+COPY \$FOO /quux # COPY $FOO /quux
+```
+
+Environment variables are supported by the following list of instructions in
+the `Dockerfile`:
+
+- `ADD`
+- `COPY`
+- `ENV`
+- `EXPOSE`
+- `FROM`
+- `LABEL`
+- `STOPSIGNAL`
+- `USER`
+- `VOLUME`
+- `WORKDIR`
+- `ONBUILD` (when combined with one of the supported instructions above)
+
+Environment variable substitution will use the same value for each variable
+throughout the entire instruction. In other words, in this example:
+
+```dockerfile
+ENV abc=hello
+ENV abc=bye def=$abc
+ENV ghi=$abc
+```
+
+will result in `def` having a value of `hello`, not `bye`. However,
+`ghi` will have a value of `bye` because it is not part of the same instruction
+that set `abc` to `bye`.
+
+## .dockerignore file
+
+Before the docker CLI sends the context to the docker daemon, it looks
+for a file named `.dockerignore` in the root directory of the context.
+If this file exists, the CLI modifies the context to exclude files and
+directories that match patterns in it. This helps to avoid
+unnecessarily sending large or sensitive files and directories to the
+daemon and potentially adding them to images using `ADD` or `COPY`.
+
+The CLI interprets the `.dockerignore` file as a newline-separated
+list of patterns similar to the file globs of Unix shells. For the
+purposes of matching, the root of the context is considered to be both
+the working and the root directory. For example, the patterns
+`/foo/bar` and `foo/bar` both exclude a file or directory named `bar`
+in the `foo` subdirectory of `PATH` or in the root of the git
+repository located at `URL`. Neither excludes anything else.
+
+If a line in `.dockerignore` file starts with `#` in column 1, then this line is
+considered as a comment and is ignored before interpreted by the CLI.
+
+Here is an example `.dockerignore` file:
+
+```gitignore
+# comment
+*/temp*
+*/*/temp*
+temp?
+```
+
+This file causes the following build behavior:
+
+| Rule | Behavior |
+|:------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `# comment` | Ignored. |
+| `*/temp*` | Exclude files and directories whose names start with `temp` in any immediate subdirectory of the root. For example, the plain file `/somedir/temporary.txt` is excluded, as is the directory `/somedir/temp`. |
+| `*/*/temp*` | Exclude files and directories starting with `temp` from any subdirectory that is two levels below the root. For example, `/somedir/subdir/temporary.txt` is excluded. |
+| `temp?` | Exclude files and directories in the root directory whose names are a one-character extension of `temp`. For example, `/tempa` and `/tempb` are excluded. |
+
+
+Matching is done using Go's
+[filepath.Match](https://golang.org/pkg/path/filepath#Match) rules. A
+preprocessing step removes leading and trailing whitespace and
+eliminates `.` and `..` elements using Go's
+[filepath.Clean](https://golang.org/pkg/path/filepath/#Clean). Lines
+that are blank after preprocessing are ignored.
+
+Beyond Go's filepath.Match rules, Docker also supports a special
+wildcard string `**` that matches any number of directories (including
+zero). For example, `**/*.go` will exclude all files that end with `.go`
+that are found in all directories, including the root of the build context.
+
+Lines starting with `!` (exclamation mark) can be used to make exceptions
+to exclusions. The following is an example `.dockerignore` file that
+uses this mechanism:
+
+```gitignore
+*.md
+!README.md
+```
+
+All markdown files *except* `README.md` are excluded from the context.
+
+The placement of `!` exception rules influences the behavior: the last
+line of the `.dockerignore` that matches a particular file determines
+whether it is included or excluded. Consider the following example:
+
+```gitignore
+*.md
+!README*.md
+README-secret.md
+```
+
+No markdown files are included in the context except README files other than
+`README-secret.md`.
+
+Now consider this example:
+
+```gitignore
+*.md
+README-secret.md
+!README*.md
+```
+
+All of the README files are included. The middle line has no effect because
+`!README*.md` matches `README-secret.md` and comes last.
+
+You can even use the `.dockerignore` file to exclude the `Dockerfile`
+and `.dockerignore` files. These files are still sent to the daemon
+because it needs them to do its job. But the `ADD` and `COPY` instructions
+do not copy them to the image.
+
+Finally, you may want to specify which files to include in the
+context, rather than which to exclude. To achieve this, specify `*` as
+the first pattern, followed by one or more `!` exception patterns.
+
+> **Note**
+>
+> For historical reasons, the pattern `.` is ignored.
+
+## FROM
+
+```dockerfile
+FROM [--platform=] [AS ]
+```
+
+Or
+
+```dockerfile
+FROM [--platform=] [:] [AS ]
+```
+
+Or
+
+```dockerfile
+FROM [--platform=] [@] [AS ]
+```
+
+The `FROM` instruction initializes a new build stage and sets the
+[*Base Image*](https://docs.docker.com/glossary/#base-image) for subsequent instructions. As such, a
+valid `Dockerfile` must start with a `FROM` instruction. The image can be
+any valid image – it is especially easy to start by **pulling an image** from
+the [*Public Repositories*](https://docs.docker.com/docker-hub/repos/).
+
+- `ARG` is the only instruction that may precede `FROM` in the `Dockerfile`.
+ See [Understand how ARG and FROM interact](#understand-how-arg-and-from-interact).
+- `FROM` can appear multiple times within a single `Dockerfile` to
+ create multiple images or use one build stage as a dependency for another.
+ Simply make a note of the last image ID output by the commit before each new
+ `FROM` instruction. Each `FROM` instruction clears any state created by previous
+ instructions.
+- Optionally a name can be given to a new build stage by adding `AS name` to the
+ `FROM` instruction. The name can be used in subsequent `FROM` and
+ `COPY --from=` instructions to refer to the image built in this stage.
+- The `tag` or `digest` values are optional. If you omit either of them, the
+ builder assumes a `latest` tag by default. The builder returns an error if it
+ cannot find the `tag` value.
+
+The optional `--platform` flag can be used to specify the platform of the image
+in case `FROM` references a multi-platform image. For example, `linux/amd64`,
+`linux/arm64`, or `windows/amd64`. By default, the target platform of the build
+request is used. Global build arguments can be used in the value of this flag,
+for example [automatic platform ARGs](#automatic-platform-args-in-the-global-scope)
+allow you to force a stage to native build platform (`--platform=$BUILDPLATFORM`),
+and use it to cross-compile to the target platform inside the stage.
+
+### Understand how ARG and FROM interact
+
+`FROM` instructions support variables that are declared by any `ARG`
+instructions that occur before the first `FROM`.
+
+```dockerfile
+ARG CODE_VERSION=latest
+FROM base:${CODE_VERSION}
+CMD /code/run-app
+
+FROM extras:${CODE_VERSION}
+CMD /code/run-extras
+```
+
+An `ARG` declared before a `FROM` is outside of a build stage, so it
+can't be used in any instruction after a `FROM`. To use the default value of
+an `ARG` declared before the first `FROM` use an `ARG` instruction without
+a value inside of a build stage:
+
+```dockerfile
+ARG VERSION=latest
+FROM busybox:$VERSION
+ARG VERSION
+RUN echo $VERSION > image_version
+```
+
+## RUN
+
+RUN has 2 forms:
+
+- `RUN ` (*shell* form, the command is run in a shell, which by
+default is `/bin/sh -c` on Linux or `cmd /S /C` on Windows)
+- `RUN ["executable", "param1", "param2"]` (*exec* form)
+
+The `RUN` instruction will execute any commands in a new layer on top of the
+current image and commit the results. The resulting committed image will be
+used for the next step in the `Dockerfile`.
+
+Layering `RUN` instructions and generating commits conforms to the core
+concepts of Docker where commits are cheap and containers can be created from
+any point in an image's history, much like source control.
+
+The *exec* form makes it possible to avoid shell string munging, and to `RUN`
+commands using a base image that does not contain the specified shell executable.
+
+The default shell for the *shell* form can be changed using the `SHELL`
+command.
+
+In the *shell* form you can use a `\` (backslash) to continue a single
+RUN instruction onto the next line. For example, consider these two lines:
+
+```dockerfile
+RUN /bin/bash -c 'source $HOME/.bashrc; \
+echo $HOME'
+```
+
+Together they are equivalent to this single line:
+
+```dockerfile
+RUN /bin/bash -c 'source $HOME/.bashrc; echo $HOME'
+```
+
+To use a different shell, other than '/bin/sh', use the *exec* form passing in
+the desired shell. For example:
+
+```dockerfile
+RUN ["/bin/bash", "-c", "echo hello"]
+```
+
+> **Note**
+>
+> The *exec* form is parsed as a JSON array, which means that
+> you must use double-quotes (") around words not single-quotes (').
+
+Unlike the *shell* form, the *exec* form does not invoke a command shell.
+This means that normal shell processing does not happen. For example,
+`RUN [ "echo", "$HOME" ]` will not do variable substitution on `$HOME`.
+If you want shell processing then either use the *shell* form or execute
+a shell directly, for example: `RUN [ "sh", "-c", "echo $HOME" ]`.
+When using the exec form and executing a shell directly, as in the case for
+the shell form, it is the shell that is doing the environment variable
+expansion, not docker.
+
+> **Note**
+>
+> In the *JSON* form, it is necessary to escape backslashes. This is
+> particularly relevant on Windows where the backslash is the path separator.
+> The following line would otherwise be treated as *shell* form due to not
+> being valid JSON, and fail in an unexpected way:
+>
+> ```dockerfile
+> RUN ["c:\windows\system32\tasklist.exe"]
+> ```
+>
+> The correct syntax for this example is:
+>
+> ```dockerfile
+> RUN ["c:\\windows\\system32\\tasklist.exe"]
+> ```
+
+The cache for `RUN` instructions isn't invalidated automatically during
+the next build. The cache for an instruction like
+`RUN apt-get dist-upgrade -y` will be reused during the next build. The
+cache for `RUN` instructions can be invalidated by using the `--no-cache`
+flag, for example `docker build --no-cache`.
+
+See the [`Dockerfile` Best Practices
+guide](https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/) for more information.
+
+The cache for `RUN` instructions can be invalidated by [`ADD`](#add) and [`COPY`](#copy) instructions.
+
+### Known issues (RUN)
+
+- [Issue 783](https://github.com/docker/docker/issues/783) is about file
+ permissions problems that can occur when using the AUFS file system. You
+ might notice it during an attempt to `rm` a file, for example.
+
+ For systems that have recent aufs version (i.e., `dirperm1` mount option can
+ be set), docker will attempt to fix the issue automatically by mounting
+ the layers with `dirperm1` option. More details on `dirperm1` option can be
+ found at [`aufs` man page](https://github.com/sfjro/aufs3-linux/tree/aufs3.18/Documentation/filesystems/aufs)
+
+ If your system doesn't have support for `dirperm1`, the issue describes a workaround.
+
+## RUN --mount
+
+> **Note**
+>
+> Added in [`docker/dockerfile:1.2`](#syntax)
+
+`RUN --mount` allows you to create filesystem mounts that the build can access.
+This can be used to:
+
+- Create bind mount to the host filesystem or other build stages
+- Access build secrets or ssh-agent sockets
+- Use a persistent package management cache to speed up your build
+
+Syntax: `--mount=[type=][,option=[,option=]...]`
+
+### Mount types
+
+| Type | Description |
+|------------------------------------------|-----------------------------------------------------------------------------------------------------------|
+| [`bind`](#run---mounttypebind) (default) | Bind-mount context directories (read-only). |
+| [`cache`](#run---mounttypecache) | Mount a temporary directory to cache directories for compilers and package managers. |
+| [`secret`](#run---mounttypesecret) | Allow the build container to access secure files such as private keys without baking them into the image. |
+| [`ssh`](#run---mounttypessh) | Allow the build container to access SSH keys via SSH agents, with support for passphrases. |
+
+### RUN --mount=type=bind
+
+This mount type allows binding files or directories to the build container. A
+bind mount is read-only by default.
+
+| Option | Description |
+|----------------------|--------------------------------------------------------------------------------------|
+| `target`[^1] | Mount path. |
+| `source` | Source path in the `from`. Defaults to the root of the `from`. |
+| `from` | Build stage or image name for the root of the source. Defaults to the build context. |
+| `rw`,`readwrite` | Allow writes on the mount. Written data will be discarded. |
+
+### RUN --mount=type=cache
+
+This mount type allows the build container to cache directories for compilers
+and package managers.
+
+| Option | Description |
+|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `id` | Optional ID to identify separate/different caches. Defaults to value of `target`. |
+| `target`[^1] | Mount path. |
+| `ro`,`readonly` | Read-only if set. |
+| `sharing` | One of `shared`, `private`, or `locked`. Defaults to `shared`. A `shared` cache mount can be used concurrently by multiple writers. `private` creates a new mount if there are multiple writers. `locked` pauses the second writer until the first one releases the mount. |
+| `from` | Build stage to use as a base of the cache mount. Defaults to empty directory. |
+| `source` | Subpath in the `from` to mount. Defaults to the root of the `from`. |
+| `mode` | File mode for new cache directory in octal. Default `0755`. |
+| `uid` | User ID for new cache directory. Default `0`. |
+| `gid` | Group ID for new cache directory. Default `0`. |
+
+Contents of the cache directories persists between builder invocations without
+invalidating the instruction cache. Cache mounts should only be used for better
+performance. Your build should work with any contents of the cache directory as
+another build may overwrite the files or GC may clean it if more storage space
+is needed.
+
+#### Example: cache Go packages
+
+```dockerfile
+# syntax=docker/dockerfile:1
+FROM golang
+RUN --mount=type=cache,target=/root/.cache/go-build \
+ go build ...
+```
+
+#### Example: cache apt packages
+
+```dockerfile
+# syntax=docker/dockerfile:1
+FROM ubuntu
+RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
+RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
+ --mount=type=cache,target=/var/lib/apt,sharing=locked \
+ apt update && apt-get --no-install-recommends install -y gcc
+```
+
+Apt needs exclusive access to its data, so the caches use the option
+`sharing=locked`, which will make sure multiple parallel builds using
+the same cache mount will wait for each other and not access the same
+cache files at the same time. You could also use `sharing=private` if
+you prefer to have each build create another cache directory in this
+case.
+
+### RUN --mount=type=tmpfs
+
+This mount type allows mounting tmpfs in the build container.
+
+| Option | Description |
+|---------------------|-------------------------------------------------------|
+| `target`[^1] | Mount path. |
+| `size` | Specify an upper limit on the size of the filesystem. |
+
+### RUN --mount=type=secret
+
+This mount type allows the build container to access secure files such as
+private keys without baking them into the image.
+
+| Option | Description |
+|---------------------|---------------------------------------------------------------------------------------------------|
+| `id` | ID of the secret. Defaults to basename of the target path. |
+| `target` | Mount path. Defaults to `/run/secrets/` + `id`. |
+| `required` | If set to `true`, the instruction errors out when the secret is unavailable. Defaults to `false`. |
+| `mode` | File mode for secret file in octal. Default `0400`. |
+| `uid` | User ID for secret file. Default `0`. |
+| `gid` | Group ID for secret file. Default `0`. |
+
+#### Example: access to S3
+
+```dockerfile
+# syntax=docker/dockerfile:1
+FROM python:3
+RUN pip install awscli
+RUN --mount=type=secret,id=aws,target=/root/.aws/credentials \
+ aws s3 cp s3://... ...
+```
+
+```console
+$ docker buildx build --secret id=aws,src=$HOME/.aws/credentials .
+```
+
+### RUN --mount=type=ssh
+
+This mount type allows the build container to access SSH keys via SSH agents,
+with support for passphrases.
+
+| Option | Description |
+|---------------------|------------------------------------------------------------------------------------------------|
+| `id` | ID of SSH agent socket or key. Defaults to "default". |
+| `target` | SSH agent socket path. Defaults to `/run/buildkit/ssh_agent.${N}`. |
+| `required` | If set to `true`, the instruction errors out when the key is unavailable. Defaults to `false`. |
+| `mode` | File mode for socket in octal. Default `0600`. |
+| `uid` | User ID for socket. Default `0`. |
+| `gid` | Group ID for socket. Default `0`. |
+
+#### Example: access to Gitlab
+
+```dockerfile
+# syntax=docker/dockerfile:1
+FROM alpine
+RUN apk add --no-cache openssh-client
+RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
+RUN --mount=type=ssh \
+ ssh -q -T git@gitlab.com 2>&1 | tee /hello
+# "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here
+# with the type of build progress is defined as `plain`.
+```
+
+```console
+$ eval $(ssh-agent)
+$ ssh-add ~/.ssh/id_rsa
+(Input your passphrase here)
+$ docker buildx build --ssh default=$SSH_AUTH_SOCK .
+```
+
+You can also specify a path to `*.pem` file on the host directly instead of `$SSH_AUTH_SOCK`.
+However, pem files with passphrases are not supported.
+
+## RUN --network
+
+> **Note**
+>
+> Added in [`docker/dockerfile:1.1`](#syntax)
+
+`RUN --network` allows control over which networking environment the command
+is run in.
+
+Syntax: `--network=`
+
+### Network types
+
+| Type | Description |
+|----------------------------------------------|----------------------------------------|
+| [`default`](#run---networkdefault) (default) | Run in the default network. |
+| [`none`](#run---networknone) | Run with no network access. |
+| [`host`](#run---networkhost) | Run in the host's network environment. |
+
+### RUN --network=default
+
+Equivalent to not supplying a flag at all, the command is run in the default
+network for the build.
+
+### RUN --network=none
+
+The command is run with no network access (`lo` is still available, but is
+isolated to this process)
+
+#### Example: isolating external effects
+
+```dockerfile
+# syntax=docker/dockerfile:1
+FROM python:3.6
+ADD mypackage.tgz wheels/
+RUN --network=none pip install --find-links wheels mypackage
+```
+
+`pip` will only be able to install the packages provided in the tarfile, which
+can be controlled by an earlier build stage.
+
+### RUN --network=host
+
+The command is run in the host's network environment (similar to
+`docker build --network=host`, but on a per-instruction basis)
+
+> **Warning**
+>
+> The use of `--network=host` is protected by the `network.host` entitlement,
+> which needs to be enabled when starting the buildkitd daemon with
+> `--allow-insecure-entitlement network.host` flag or in [buildkitd config](https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md),
+> and for a build request with [`--allow network.host` flag](https://docs.docker.com/engine/reference/commandline/buildx_build/#allow).
+{:.warning}
+
+## RUN --security
+
+> **Note**
+>
+> Not yet available in stable syntax, use [`docker/dockerfile:1-labs`](#syntax) version.
+
+### RUN --security=insecure
+
+With `--security=insecure`, builder runs the command without sandbox in insecure
+mode, which allows to run flows requiring elevated privileges (e.g. containerd).
+This is equivalent to running `docker run --privileged`.
+
+> **Warning**
+>
+> In order to access this feature, entitlement `security.insecure` should be
+> enabled when starting the buildkitd daemon with
+> `--allow-insecure-entitlement security.insecure` flag or in [buildkitd config](https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md),
+> and for a build request with [`--allow security.insecure` flag](https://docs.docker.com/engine/reference/commandline/buildx_build/#allow).
+{:.warning}
+
+#### Example: check entitlements
+
+```dockerfile
+# syntax=docker/dockerfile:1-labs
+FROM ubuntu
+RUN --security=insecure cat /proc/self/status | grep CapEff
+```
+```text
+#84 0.093 CapEff: 0000003fffffffff
+```
+
+### RUN --security=sandbox
+
+Default sandbox mode can be activated via `--security=sandbox`, but that is no-op.
+
+## CMD
+
+The `CMD` instruction has three forms:
+
+- `CMD ["executable","param1","param2"]` (*exec* form, this is the preferred form)
+- `CMD ["param1","param2"]` (as *default parameters to ENTRYPOINT*)
+- `CMD command param1 param2` (*shell* form)
+
+There can only be one `CMD` instruction in a `Dockerfile`. If you list more than one `CMD`
+then only the last `CMD` will take effect.
+
+**The main purpose of a `CMD` is to provide defaults for an executing
+container.** These defaults can include an executable, or they can omit
+the executable, in which case you must specify an `ENTRYPOINT`
+instruction as well.
+
+If `CMD` is used to provide default arguments for the `ENTRYPOINT` instruction,
+both the `CMD` and `ENTRYPOINT` instructions should be specified with the JSON
+array format.
+
+> **Note**
+>
+> The *exec* form is parsed as a JSON array, which means that you must use
+> double-quotes (") around words not single-quotes (').
+
+Unlike the *shell* form, the *exec* form does not invoke a command shell.
+This means that normal shell processing does not happen. For example,
+`CMD [ "echo", "$HOME" ]` will not do variable substitution on `$HOME`.
+If you want shell processing then either use the *shell* form or execute
+a shell directly, for example: `CMD [ "sh", "-c", "echo $HOME" ]`.
+When using the exec form and executing a shell directly, as in the case for
+the shell form, it is the shell that is doing the environment variable
+expansion, not docker.
+
+When used in the shell or exec formats, the `CMD` instruction sets the command
+to be executed when running the image.
+
+If you use the *shell* form of the `CMD`, then the `` will execute in
+`/bin/sh -c`:
+
+```dockerfile
+FROM ubuntu
+CMD echo "This is a test." | wc -
+```
+
+If you want to **run your** `` **without a shell** then you must
+express the command as a JSON array and give the full path to the executable.
+**This array form is the preferred format of `CMD`.** Any additional parameters
+must be individually expressed as strings in the array:
+
+```dockerfile
+FROM ubuntu
+CMD ["/usr/bin/wc","--help"]
+```
+
+If you would like your container to run the same executable every time, then
+you should consider using `ENTRYPOINT` in combination with `CMD`. See
+[*ENTRYPOINT*](#entrypoint).
+
+If the user specifies arguments to `docker run` then they will override the
+default specified in `CMD`.
+
+> **Note**
+>
+> Do not confuse `RUN` with `CMD`. `RUN` actually runs a command and commits
+> the result; `CMD` does not execute anything at build time, but specifies
+> the intended command for the image.
+
+## LABEL
+
+```dockerfile
+LABEL