Support private registries #8550

csweichel · 2022-03-02T18:57:39Z

Description

This PR adds rudimentary support for images from private registries. It does that by virtue of project-level environment variables.

To use it, add an environment variable named GITPOD_IMAGE_AUTH to your project. The content of that environment variable looks like:
domain.com:<base64(username:password)>, anotherdomain.com:...

This format is akin to dockerconfig.json files, where the auth section contains the same base64 encoded string. For example, logging into to the GitHub registry (ghcr.io) using:

echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin

produces an entry akin to:

"auths": {
	"ghcr.io": {
		"auth": "WW91IGNoZWNrZWQgOikgTmVlZCBhIGpvYj8gRW1haWwgbWUgYXQgY2hyaXNAZ2l0cG9kLmlvCg=="
	}
},

That very base64 encoded string can be used. It would turn into:

Once this variable is present, private images can be used directly, e.g.

image: my-private-repo.com/super-secret-sauce:latest

Caveats

the Docker image is copied over to Gitpod's registry. There currently is no means to delete that image.
with this implementation, private images can only be used directly, not as part of a Dockerfile (i.e. FROM my-private-repo.com/secret-sauce:latest)
~~right now the project level environment variable must NOT be "hidden". I'm looking to change that.~~
the failure modes aren't great. There are a lot of things which might not work, e.g. using the wrong format. Users don't get a good error message, it's just that their image pull fails with a "cannot resolve base image: hostname required" error.
this PR removes the base image resolution as part of the workspace config creation. Consequently, if the image referred to in the .gitpod.yml refers to a "moving tag" (think docker.io/somthing:latest), and we need to rebuild the "gitpod workspace image", we'd re-resolve that tag, which might lead to changing workspace images across restarts for the very same workspace.
I've been a bit coarse with the project level environment variables - we can probably clean some of that up.

Related Issue(s)

Fixes #1699

How to test

See PR description. I tested with ghcr.io

Release Notes

Add support for private registries

Documentation

https://github.com/gitpod-io/website/issues/1704

codecov · 2022-03-02T19:04:40Z

Codecov Report

Merging #8550 (468e51f) into main (d030750) will increase coverage by 3.05%.
The diff coverage is 37.03%.

@@            Coverage Diff             @@
##             main    #8550      +/-   ##
==========================================
+ Coverage   12.31%   15.37%   +3.05%     
==========================================
  Files          20       35      +15     
  Lines        1161     3122    +1961     
==========================================
+ Hits          143      480     +337     
- Misses       1014     2608    +1594     
- Partials        4       34      +30

Flag	Coverage Δ
components-gitpod-cli-app	`11.17% <ø> (ø)`
components-image-builder-api-go-lib	`∅ <ø> (?)`
components-image-builder-bob-app	`∅ <ø> (?)`
components-image-builder-mk3-app	`35.58% <37.03%> (?)`
components-local-app-app-darwin-amd64	`?`
components-local-app-app-darwin-arm64	`?`
components-local-app-app-linux-amd64	`?`
components-local-app-app-linux-arm64	`?`
components-local-app-app-windows-386	`?`
components-local-app-app-windows-amd64	`?`
components-local-app-app-windows-arm64	`?`
install-installer-raw-app	`4.27% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...image-builder-mk3/pkg/orchestrator/orchestrator.go	`40.81% <37.03%> (ø)`
components/local-app/pkg/auth/auth.go
components/local-app/pkg/auth/pkce.go
...ents/image-builder-mk3/pkg/orchestrator/metrics.go	`48.14% <0.00%> (ø)`
...l/installer/pkg/components/ws-manager/tlssecret.go	`0.00% <0.00%> (ø)`
install/installer/pkg/common/display.go	`0.00% <0.00%> (ø)`
install/installer/pkg/common/ca.go	`0.00% <0.00%> (ø)`
...l/installer/pkg/components/ws-manager/configmap.go	`23.75% <0.00%> (ø)`
install/installer/pkg/common/render.go	`0.00% <0.00%> (ø)`
install/installer/pkg/common/networkpolicies.go	`0.00% <0.00%> (ø)`
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d030750...468e51f. Read the comment docs.

jankeromnes

Many thanks for implementing this super valuable feature! 🙏 💯

Briefly reviewed the environment variable code, and left a few suggestions in-line. TL;DR -- please only "hydrate" values just when they're actually needed, in order to not pass them around clear-text too much.

components/gitpod-db/src/project-db.ts

components/gitpod-db/src/typeorm/project-db-impl.ts

components/server/ee/src/prebuilds/prebuild-manager.ts

components/server/src/projects/projects-service.ts

components/server/src/workspace/gitpod-server-impl.ts

components/server/src/workspace/workspace-starter.ts

components/server/src/workspace/gitpod-server-impl.ts

components/server/src/workspace/workspace-starter.ts

csweichel · 2022-03-03T13:08:05Z

@jankeromnes Thank you for your feedback. I've reverted the early "hydration" of the project-level environment variables and pushed it back to the actual point of use. PTAL.

jankeromnes

Awesome, many thanks for the quick fix!

I've reviewed the environment variables handling code, and it looks good to me. 👍

I haven't looked too closely at the Go code or the image-source-provider.ts changes.

Please let me know if you'd like me to test this PR.

To use it, add an environment variable named GITPOD_IMAGE_AUTH to your project. The content of that environment variable looks like:
domain.com:<base64(username:password)> anotherdomain.com:...

Two minor thoughts:

The code seems to split on , instead of -- I assume it's a typo here
~~Why base64-encode the username:password part? (As a user, I'd find it more convenient to specify domain.com:username:password directly)~~ EDIT: Nevermind, I just noticed that the response is literally the next sentence 😅

sagor999

reviewed go related changes. only one minor comment.

components/image-builder-mk3/debug.sh

sagor999 · 2022-03-03T18:19:34Z

the Docker image is copied over to Gitpod's registry. There currently is no means to delete that image.

is that a local cluster level registry? or registry at eu.gcr.io?
and can someone else download that image from local registry somehow later on?

utam0k · 2022-03-03T23:40:27Z

components/image-builder-mk3/debug.sh

+# give the old pod time to disappear
+sleep 20
+gpctl debug logs image-builder-mk3


Suggested change

gpctl debug logs image-builder-mk3

gpctl debug logs image-builder-mk3

I'm not sure I understand the comment :)

This is the fix that the github warns about because there is no new line. There is no specific problem, so it is a minor nits.
https://thoughtbot.com/blog/no-newline-at-end-of-file

components/image-builder-bob/pkg/proxy/auth.go

csweichel · 2022-03-04T14:26:12Z

is that a local cluster level registry? or registry at eu.gcr.io?
and can someone else download that image from local registry somehow later on?

I just used ghcr.io and a GitHub access token.
You could also use a Gitpod workspace with a proxy which requires basic auth.

csweichel · 2022-03-07T07:33:12Z

/werft run

👍 started the job as gitpod-build-cw-fix-1699.7

csweichel · 2022-03-07T11:13:48Z

build is green again.

@geropl this needs your approval due to API changes.

ccll · 2022-04-09T01:52:21Z

Does this suppose to work with Gitlab Registry?
I got an error trying to use image from my self-hosted Gitlab Registry from a self-hosted Gitpod.

Might be related with #8938 and #8795.

Request createWorkspace failed with message: 13 INTERNAL: cannot resolve base image: hostname required

Unknown Error: { "code": -32603 }

Due to issue #8536 I could not setup variable per project, so I added the variable to global settings.

jhulance · 2022-04-25T12:50:24Z

I believe this doesn't work for self-hosted (yet?) because it makes use of Project vars, and the global Variables are ignored?

In order to get this working in the FROM line of a Dockerfile for a workspace, I believe the BOB_WSLAYER_AUTH env var needs to be set up by Image Builder MK3 when it starts a Image Build. This is the whole of the "auths" var from a standard Docker config.json with auth/creds for private registries. Not sure how that will get passed over to the Image Builder MK3 -- use ImagePullSecrets somewhere?

Keith-Hon · 2022-08-04T08:55:56Z

I am unable to use a private image from hub.docker.com even after I added GITPOD_IMAGE_AUTH accordingly. Any help?

csweichel · 2022-08-04T08:57:53Z

I am unable to use a private image from hub.docker.com even after I added GITPOD_IMAGE_AUTH accordingly. Any help?

What do you see when you open the workspace?
Do you use the private image directly, or as part of a Dockerfile?

Keith-Hon · 2022-08-04T09:07:22Z

These are the steps I took.

I logged in docker hub via cli and printed the token

Add a variable named GITPOD_IMAGE_AUTH in gitpod settings

Here is the result I get

jmls · 2022-08-04T09:09:03Z

you need to add the repo to a gitpod project , then add the variable to the project, not personal

Keith-Hon · 2022-08-04T09:18:23Z

I just tried create a project at https://gitpod.io/projects, and added the variable GITPOD_IMAGE_AUTH to the project's Settings

and then create a new workspace from that project

still no luck...

jmls · 2022-08-04T10:26:14Z

double and triple check the value of the token - I know this works because we're using this for every single one of our repos across dozens of developers every day

Getting the value of the token right was our biggest hurdle ;)

jmls · 2022-08-04T10:30:25Z

The value is domain:<base64 encoded value>

echo -n "<domain>:"; echo <username>:<password> | base64 -w0

are you sure the domain is index.docker.io ?

Keith-Hon · 2022-08-04T10:58:38Z

@jmls I am not sure if the domain is index.docker.io. But that's what I get from the docker login command. I found that the token is not the same with "echo : | base64 -w0" and the one from .json.

csweichel · 2022-08-04T12:58:27Z

@Keith-Hon

Do you use the private image directly, or as part of a Dockerfile?
How do you refer to the image: dockerhub-user/imagename:latest or index.docker.io/dockerhub-user/imagename:latest?

jmls · 2022-08-04T13:25:48Z

oh, I forgot that a custom image on the dockerfile is not supported atm

You have to specify a custom image in the gitpod.yml

Keith-Hon · 2022-08-04T13:39:00Z

@csweichel

dockerhub-user/imagename:latest

Keith-Hon · 2022-08-04T13:44:02Z

I have a .gitpod.yml file at the root of the repo, which declare the custom workspace image:

csweichel · 2022-08-04T13:52:30Z

Please try using docker.io as host, i.e.

echo -n "docker.io:"; echo <username>:<password> | base64 -w0

Reasoning: reference.ParseNormalizedName results in docker.io. Verified using oci-tool:

$ oci-tool resolve --familiar name csweichel/werft:latest
docker.io/csweichel/werft:latest@sha256:332fd69ac3784ac57580eb61f48176e95da0f1183b826c3d732b51508c1436d2

Keith-Hon · 2022-08-04T14:14:07Z

@csweichel, @jmls it's working now. Thanks a lot!

However, this line of command didn't work

echo -n "docker.io:"; echo <username>:<password> | base64 -w0

I succeeded by using the token obtained from

docker login
cat /home/gitpod/.docker/config.json

and added GITPOD_IMAGE_AUTH to the project's Variables by

docker.io:TOKEN_OBTAINED

Keith-Hon · 2022-08-04T14:19:40Z

Btw, it is a bit confusing that "Projects" tab resides inside user's Setting. I didn't notice there was a "project" specific scope setting and environment variables....

Keith-Hon · 2022-08-08T14:45:27Z

@csweichel, @jmls it's working now. Thanks a lot!

However, this line of command didn't work

echo -n "docker.io:"; echo <username>:<password> | base64 -w0

I succeeded by using the token obtained from

docker login cat /home/gitpod/.docker/config.json

and added GITPOD_IMAGE_AUTH to the project's Variables by

docker.io:TOKEN_OBTAINED

I tried again with this approach today. It only works if

the targeted docker image was public when the workspace was prebuilt, and then set to private in dockerhub
the tag in the .gitpod.yml must match that once-public-then-private image's tag

It doesn't work if the docker image was never public @csweichel

utam0k · 2022-08-08T23:49:18Z

@Keith-Hon @jmls Thanks for your report! Can I ask you to create a new issue to prioritize to address?

Keith-Hon · 2022-08-09T03:33:33Z

Ok created

@Keith-Hon @jmls Thanks for your report! Can I ask you to create a new issue to prioritize to address?

utam0k · 2022-08-09T04:12:44Z

@Keith-Hon Thanks!

mrzarquon · 2022-10-14T09:59:54Z

As a note here until docs are live - this snippet needs the second echo to have -n also, it's throwing a carriage return onto the string. Using the below example you don't need to extract the credential from config.json.

echo -n "docker.io:"; echo -n <username>:<password> | base64 -w0

roboquat added release-note do-not-merge/work-in-progress size/XXL labels Mar 2, 2022

csweichel force-pushed the cw/fix-1699 branch from cd98f73 to 429cf77 Compare March 3, 2022 09:01

csweichel marked this pull request as ready for review March 3, 2022 09:15

csweichel requested review from a team March 3, 2022 09:15

csweichel requested a review from geropl as a code owner March 3, 2022 09:15

roboquat removed the do-not-merge/work-in-progress label Mar 3, 2022

github-actions bot added team: webapp Issue belongs to the WebApp team team: workspace Issue belongs to the Workspace team labels Mar 3, 2022

jankeromnes reviewed Mar 3, 2022

View reviewed changes

csweichel force-pushed the cw/fix-1699 branch from 429cf77 to fa472c1 Compare March 3, 2022 13:06

jankeromnes reviewed Mar 3, 2022

View reviewed changes

sagor999 previously approved these changes Mar 3, 2022

View reviewed changes

components/image-builder-mk3/debug.sh Outdated Show resolved Hide resolved

utam0k reviewed Mar 3, 2022

View reviewed changes

utam0k reviewed Mar 4, 2022

View reviewed changes

components/image-builder-bob/pkg/proxy/auth.go Outdated Show resolved Hide resolved

csweichel dismissed sagor999’s stale review via 916a96d March 4, 2022 14:36

csweichel force-pushed the cw/fix-1699 branch from fa472c1 to 916a96d Compare March 4, 2022 14:36

csweichel force-pushed the cw/fix-1699 branch from 916a96d to 3ea8f6d Compare March 7, 2022 07:42

Support private registries

468e51f

csweichel force-pushed the cw/fix-1699 branch from 3ea8f6d to 468e51f Compare March 7, 2022 18:20

This was referenced May 18, 2022

[image-builder] Fix error when secret is a json #10031

Merged

Image builder does not support private container registry when gitpod uses same registry #10089

Closed

Enable referring a private registry in gitpod Dockerfile #10204

Closed

princerachit mentioned this pull request May 25, 2022

[image-builder-bob] Prioritize using additional auth #10235

Closed

mrsimonemms mentioned this pull request Jul 12, 2022

[KOTS]: add ability to use custom workspace image stored in different private registry #11308

Closed

Keith-Hon mentioned this pull request Aug 9, 2022

Unable to create a workspace from a private image from dockerhub #11982

Closed

kylos101 mentioned this pull request Jan 4, 2023

[image builder] count failures and differentiate between system and user #15573

Open

R0Wi mentioned this pull request Jul 20, 2023

Add support for private Docker registries in Dockerfile #18315

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support private registries #8550

Support private registries #8550

csweichel commented Mar 2, 2022 •

edited

Loading

codecov bot commented Mar 2, 2022 •

edited

Loading

jankeromnes left a comment

csweichel commented Mar 3, 2022

jankeromnes left a comment •

edited

Loading

sagor999 left a comment

sagor999 commented Mar 3, 2022

utam0k Mar 3, 2022

csweichel Mar 4, 2022

utam0k Mar 6, 2022

csweichel commented Mar 4, 2022 •

edited

Loading

csweichel commented Mar 7, 2022 •

edited by werft-gitpod-dev-com bot

Loading

csweichel commented Mar 7, 2022

ccll commented Apr 9, 2022

jhulance commented Apr 25, 2022

Keith-Hon commented Aug 4, 2022

csweichel commented Aug 4, 2022

Keith-Hon commented Aug 4, 2022

jmls commented Aug 4, 2022

Keith-Hon commented Aug 4, 2022 •

edited

Loading

jmls commented Aug 4, 2022

jmls commented Aug 4, 2022 •

edited

Loading

Keith-Hon commented Aug 4, 2022

csweichel commented Aug 4, 2022

jmls commented Aug 4, 2022

Keith-Hon commented Aug 4, 2022

Keith-Hon commented Aug 4, 2022 •

edited

Loading

csweichel commented Aug 4, 2022 •

edited

Loading

Keith-Hon commented Aug 4, 2022 •

edited

Loading

Keith-Hon commented Aug 4, 2022

Keith-Hon commented Aug 8, 2022

utam0k commented Aug 8, 2022

Keith-Hon commented Aug 9, 2022

utam0k commented Aug 9, 2022

mrzarquon commented Oct 14, 2022

	gpctl debug logs image-builder-mk3
	gpctl debug logs image-builder-mk3

Support private registries #8550

Support private registries #8550

Conversation

csweichel commented Mar 2, 2022 • edited Loading

Description

Caveats

Related Issue(s)

How to test

Release Notes

Documentation

codecov bot commented Mar 2, 2022 • edited Loading

Codecov Report

jankeromnes left a comment

Choose a reason for hiding this comment

csweichel commented Mar 3, 2022

jankeromnes left a comment • edited Loading

Choose a reason for hiding this comment

sagor999 left a comment

Choose a reason for hiding this comment

sagor999 commented Mar 3, 2022

utam0k Mar 3, 2022

Choose a reason for hiding this comment

csweichel Mar 4, 2022

Choose a reason for hiding this comment

utam0k Mar 6, 2022

Choose a reason for hiding this comment

csweichel commented Mar 4, 2022 • edited Loading

csweichel commented Mar 7, 2022 • edited by werft-gitpod-dev-com bot Loading

csweichel commented Mar 7, 2022

ccll commented Apr 9, 2022

jhulance commented Apr 25, 2022

Keith-Hon commented Aug 4, 2022

csweichel commented Aug 4, 2022

Keith-Hon commented Aug 4, 2022

jmls commented Aug 4, 2022

Keith-Hon commented Aug 4, 2022 • edited Loading

jmls commented Aug 4, 2022

jmls commented Aug 4, 2022 • edited Loading

Keith-Hon commented Aug 4, 2022

csweichel commented Aug 4, 2022

jmls commented Aug 4, 2022

Keith-Hon commented Aug 4, 2022

Keith-Hon commented Aug 4, 2022 • edited Loading

csweichel commented Aug 4, 2022 • edited Loading

Keith-Hon commented Aug 4, 2022 • edited Loading

Keith-Hon commented Aug 4, 2022

Keith-Hon commented Aug 8, 2022

utam0k commented Aug 8, 2022

Keith-Hon commented Aug 9, 2022

utam0k commented Aug 9, 2022

mrzarquon commented Oct 14, 2022

csweichel commented Mar 2, 2022 •

edited

Loading

codecov bot commented Mar 2, 2022 •

edited

Loading

jankeromnes left a comment •

edited

Loading

csweichel commented Mar 4, 2022 •

edited

Loading

csweichel commented Mar 7, 2022 •

edited by werft-gitpod-dev-com bot

Loading

Keith-Hon commented Aug 4, 2022 •

edited

Loading

jmls commented Aug 4, 2022 •

edited

Loading

Keith-Hon commented Aug 4, 2022 •

edited

Loading

csweichel commented Aug 4, 2022 •

edited

Loading

Keith-Hon commented Aug 4, 2022 •

edited

Loading