Resolve Issue When Using a Self-Hosted GitLab Instance for the Container Registry on a Self-Hosted Gitpod Instance #8795

jimmybrancaccio · 2022-03-14T13:53:07Z

Bug description

When attempting to use the container registry of a self-hosted GitLab instance the following error is displayed in the Gitpod UI when bringing up a new workspace:

Request createWorkspace failed with message: 13 INTERNAL: cannot resolve workspace image: hostname required
Unknown Error: { "code": -32603 }

This message seems a bit misleading if I am to also believe what is shown in the image-builder-mk3 logs:

Mon, Mar 14 2022 8:19:20 am     {"host":"registry.gitpodbread.com","level":"debug","message":"do request","request.header.accept":"application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*","request.header.user-agent":"containerd/1.5.5+unknown","request.method":"HEAD","severity":"DEBUG","time":"2022-03-14T13:19:20Z","url":"https://registry.gitpodbread.com/v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d"}

Mon, Mar 14 2022 8:19:20 am	{"host":"registry.gitpodbread.com","level":"debug","message":"fetch response received","response.header.content-length":"159","response.header.content-type":"application/json","response.header.date":"Mon, 14 Mar 2022 13:19:20 GMT","response.header.docker-distribution-api-version":"registry/2.0","response.header.server":"nginx","response.header.www-authenticate":"Bearer realm=\"https://gitlab.gitpodbread.com/jwt/auth\",service=\"container_registry\",scope=\"repository:workspace-images:pull\"","response.header.x-content-type-options":"nosniff","response.status":"401 Unauthorized","severity":"DEBUG","time":"2022-03-14T13:19:20Z","url":"https://registry.gitpodbread.com/v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d"}

Mon, Mar 14 2022 8:19:20 am	{"header":"Bearer realm=\"https://gitlab.gitpodbread.com/jwt/auth\",service=\"container_registry\",scope=\"repository:workspace-images:pull\"","host":"registry.gitpodbread.com","level":"debug","message":"Unauthorized","severity":"DEBUG","time":"2022-03-14T13:19:20Z"}

Mon, Mar 14 2022 8:19:20 am	{"host":"registry.gitpodbread.com","level":"debug","message":"do request","request.header.accept":"application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*","request.header.user-agent":"containerd/1.5.5+unknown","request.method":"HEAD","severity":"DEBUG","time":"2022-03-14T13:19:20Z","url":"https://registry.gitpodbread.com/v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d"}

Mon, Mar 14 2022 8:19:20 am	{"host":"registry.gitpodbread.com","level":"debug","message":"fetch response received","response.header.content-length":"159","response.header.content-type":"application/json","response.header.date":"Mon, 14 Mar 2022 13:19:20 GMT","response.header.docker-distribution-api-version":"registry/2.0","response.header.server":"nginx","response.header.www-authenticate":"Bearer realm=\"https://gitlab.gitpodbread.com/jwt/auth\",service=\"container_registry\",scope=\"repository:workspace-images:pull\",error=\"insufficient_scope\"","response.header.x-content-type-options":"nosniff","response.status":"401 Unauthorized","severity":"DEBUG","time":"2022-03-14T13:19:20Z","url":"https://registry.gitpodbread.com/v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d"}

Mon, Mar 14 2022 8:19:20 am	{"header":"Bearer realm=\"https://gitlab.gitpodbread.com/jwt/auth\",service=\"container_registry\",scope=\"repository:workspace-images:pull\",error=\"insufficient_scope\"","host":"registry.gitpodbread.com","level":"debug","message":"Unauthorized","severity":"DEBUG","time":"2022-03-14T13:19:20Z"}

Mon, Mar 14 2022 8:19:20 am	{"error":"authorization failed\ngithub.com/containerd/containerd/remotes/docker.init\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/resolver.go:46\nruntime.doInit\n\truntime/proc.go:6498\nruntime.doInit\n\truntime/proc.go:6475\nruntime.doInit\n\truntime/proc.go:6475\nruntime.doInit\n\truntime/proc.go:6475\nruntime.doInit\n\truntime/proc.go:6475\nruntime.main\n\truntime/proc.go:238\nruntime.goexit\n\truntime/asm_amd64.s:1581\nserver message: insufficient_scope\ngithub.com/containerd/containerd/remotes/docker.invalidAuthorization\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/authorizer.go:322\ngithub.com/containerd/containerd/remotes/docker.(*dockerAuthorizer).AddResponses\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/authorizer.go:136\ngithub.com/containerd/containerd/remotes/docker.(*request).retryRequest\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/resolver.go:603\ngithub.com/containerd/containerd/remotes/docker.(*request).doWithRetries\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/resolver.go:582\ngithub.com/containerd/containerd/remotes/docker.(*request).doWithRetries\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/resolver.go:589\ngithub.com/containerd/containerd/remotes/docker.(*dockerResolver).Resolve\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/resolver.go:280\ngithub.com/gitpod-io/gitpod/image-builder/pkg/resolve.(*StandaloneRefResolver).Resolve\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/resolve/resolve.go:98\ngithub.com/gitpod-io/gitpod/image-builder/pkg/resolve.(*PrecachingRefResolver).Resolve\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/resolve/resolve.go:245\ngithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator.(*Orchestrator).checkImageExists\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator/orchestrator.go:513\ngithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator.(*Orchestrator).ResolveWorkspaceImage\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator/orchestrator.go:185\ngithub.com/gitpod-io/gitpod/image-builder/api._ImageBuilder_ResolveWorkspaceImage_Handler.func1\n\tgithub.com/gitpod-io/gitpod/image-builder/api@v0.0.0-00010101000000-000000000000/imgbuilder_grpc.pb.go:218\ngithub.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1\n\tgithub.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/tracing/opentracing/server_interceptors.go:38\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\tgithub.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\tgithub.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/gitpod-io/gitpod/image-builder/api._ImageBuilder_ResolveWorkspaceImage_Handler\n\tgithub.com/gitpod-io/gitpod/image-builder/api@v0.0.0-00010101000000-000000000000/imgbuilder_grpc.pb.go:220\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\tgoogle.golang.org/grpc@v1.39.1/server.go:1293\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/grpc@v1.39.1/server.go:1618\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\tgoogle.golang.org/grpc@v1.39.1/server.go:941\nruntime.goexit\n\truntime/asm_amd64.s:1581\npull access denied, repository does not exist or may require authorization\ngithub.com/containerd/containerd/remotes/docker.(*dockerResolver).Resolve\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/resolver.go:283\ngithub.com/gitpod-io/gitpod/image-builder/pkg/resolve.(*StandaloneRefResolver).Resolve\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/resolve/resolve.go:98\ngithub.com/gitpod-io/gitpod/image-builder/pkg/resolve.(*PrecachingRefResolver).Resolve\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/resolve/resolve.go:245\ngithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator.(*Orchestrator).checkImageExists\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator/orchestrator.go:513\ngithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator.(*Orchestrator).ResolveWorkspaceImage\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator/orchestrator.go:185\ngithub.com/gitpod-io/gitpod/image-builder/api._ImageBuilder_ResolveWorkspaceImage_Handler.func1\n\tgithub.com/gitpod-io/gitpod/image-builder/api@v0.0.0-00010101000000-000000000000/imgbuilder_grpc.pb.go:218\ngithub.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1\n\tgithub.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/tracing/opentracing/server_interceptors.go:38\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\tgithub.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\tgithub.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/gitpod-io/gitpod/image-builder/api._ImageBuilder_ResolveWorkspaceImage_Handler\n\tgithub.com/gitpod-io/gitpod/image-builder/api@v0.0.0-00010101000000-000000000000/imgbuilder_grpc.pb.go:220\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\tgoogle.golang.org/grpc@v1.39.1/server.go:1293\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/grpc@v1.39.1/server.go:1618\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\tgoogle.golang.org/grpc@v1.39.1/server.go:941\nruntime.goexit\n\truntime/asm_amd64.s:1581","host":"registry.gitpodbread.com","level":"info","message":"trying next host","severity":"INFO","time":"2022-03-14T13:19:20Z"}

As you can see it looks like an authorization error. I have tried setting up the container-registry-token using my GitLab administrator account to create a personal access token as well as my "normal" user account but both cause the same error message as seen above. When I create the PAT I create it with the api, read_registry and write_registry scopes.

Here's some logs from GitLab (/var/log/gitlab/registry/current - 0.0.0.0 indicates my home public IP):

2022-03-14_13:46:49.89829 time="2022-03-14T08:46:49.898-05:00" level=debug msg="authorizing request" correlation_id=01FY4AWY4WN1XVK62G2ME885SD go_version=go1.17.6 root_repo=workspace-images vars_name=workspace-images vars_reference=68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d

2022-03-14_13:46:49.89923 time="2022-03-14T08:46:49.899-05:00" level=warning msg="error authorizing context: authorization token required" correlation_id=01FY4AWY4WN1XVK62G2ME885SD go_version=go1.17.6 root_repo=workspace-images vars_name=workspace-images vars_reference=68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d

2022-03-14_13:46:49.89940 {"content_type":"application/json","correlation_id":"01FY4AWY4WN1XVK62G2ME885SD","duration_ms":14,"host":"registry.gitpodbread.com","level":"info","method":"HEAD","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:44830","remote_ip":"0.0.0.0","status":401,"system":"http","time":"2022-03-14T08:46:49.899-05:00","ttfb_ms":14,"uri":"/v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d","user_agent":"containerd/1.5.5+unknown","written_bytes":159}

2022-03-14_13:46:50.08123 time="2022-03-14T08:46:50.081-05:00" level=debug msg="authorizing request" correlation_id=01FY4AWYAX5EBYH5YQ9GWQYHTX go_version=go1.17.6 root_repo=workspace-images vars_name=workspace-images vars_reference=68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d

2022-03-14_13:46:50.08229 time="2022-03-14T08:46:50.082-05:00" level=warning msg="error authorizing context: insufficient scope" correlation_id=01FY4AWYAX5EBYH5YQ9GWQYHTX go_version=go1.17.6 root_repo=workspace-images vars_name=workspace-images vars_reference=68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d

2022-03-14_13:46:50.08242 {"content_type":"application/json","correlation_id":"01FY4AWYAX5EBYH5YQ9GWQYHTX","duration_ms":4,"host":"registry.gitpodbread.com","level":"info","method":"HEAD","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:44832","remote_ip":"0.0.0.0","status":401,"system":"http","time":"2022-03-14T08:46:50.082-05:00","ttfb_ms":4,"uri":"/v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d","user_agent":"containerd/1.5.5+unknown","written_bytes":159}

And from /var/log/gitlab/nginx/gitlab_registry_access.log:

0.0.0.0 - - [14/Mar/2022:08:19:20 -0500] "HEAD /v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d HTTP/2.0" 401 0 "" "containerd/1.5.5+unknown" -
0.0.0.0 - - [14/Mar/2022:08:19:20 -0500] "HEAD /v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d HTTP/2.0" 401 0 "" "containerd/1.5.5+unknown" -
0.0.0.0 - - [14/Mar/2022:08:46:49 -0500] "HEAD /v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d HTTP/2.0" 401 0 "" "containerd/1.5.5+unknown" -
0.0.0.0 - - [14/Mar/2022:08:46:50 -0500] "HEAD /v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d HTTP/2.0" 401 0 "" "containerd/1.5.5+unknown" -

I also noticed in the messages from the image-builder-mk3 pod - pull access denied, repository does not exist or may require authorization so I am wondering if something else needs to be done when using GitLab as a container registry? Does a group, or project also need to be created before hand? For what it's worth, I do have a group called 'workspace-images' to which I have pushed the workspace-full:latest image to.

I am running my GitLab Container Registry instance under it's own domain (main GitLab is at gitlab.gitpodbread.com, registry is at registry.gitpodbread.com). Related gitlab.rb config:

registry_external_url 'https://registry.gitpodbread.com'
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "registry.gitpodbread.com"
registry['log_level'] = "debug"
registry_nginx['enable'] = true
registry_nginx['proxy_set_headers'] = {
    "Host" => "$http_host",
    "X-Real-IP" => "$remote_addr",
    "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
    "X-Forwarded-Proto" => "https",
    "X-Forwarded-Ssl" => "on"
}

Gitpod configuration - gitpod.config.yaml:

apiVersion: v1
authProviders:
  - kind: secret
    name: gitlab
blockNewUsers:
  enabled: false
  passlist: []
certificate:
  kind: secret
  name: https-certificates
containerRegistry:
  inCluster: false
  external:
    url: registry.gitpodbread.com
    certificate:
      kind: secret
      name: container-registry-token
database:
  inCluster: true
disableDefinitelyGp: false
domain: gitpodbread.com
kind: Full
metadata:
  region: local
objectStorage:
  inCluster: true
observability:
  logLevel: debug
openVSX:
  url: https://open-vsx.org
repository: eu.gcr.io/gitpod-core-dev/build
workspace:
  resources:
    requests:
      cpu: "1"
      memory: 4Gi
  runtime:
    containerdRuntimeDir: /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io
    containerdSocket: /run/k3s/containerd/containerd.sock
    fsShiftMethod: fuse

If any more of my setups settings or information can be provided to further troubleshoot this please let me know! cc. @mrsimonemms

Steps to reproduce

Spin up a self-hosted GitLab instance. Ideally it is publicly accessible, or at least accessible by Let's Encrypt so TLS certificates can be fetched and applied.
Create a normal user account and a PAT under that account with the api, read_registry and write_registry scopes.
Create a docker-registry secret in Kubernetes with that information.
Spin up a Gitpod instance and set it to use your GitLab Container Registry.
Attempt to create a new workspace, observe the error 'cannot resolve workspace image: hostname required'.

Workspace affected

No response

Expected behavior

I would expect that Gitpod be able to utilize the GitLab Container Registry for itself.

Example repository

No response

Anything else?

No response

Front conversations

The text was updated successfully, but these errors were encountered:

mrsimonemms · 2022-04-19T08:24:59Z

Possible duplicate of #7264

stale · 2022-08-11T23:18:33Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Fentonfi · 2022-12-23T10:18:23Z

Hi Team,

Is there any resolution for the above issue?

mrsimonemms added the team: delivery Issue belongs to the self-hosted team label Mar 14, 2022

ccll mentioned this issue Apr 9, 2022

Support private registries #8550

Merged

mrsimonemms added the team: workspace Issue belongs to the Workspace team label Apr 19, 2022

stale bot added the meta: stale This issue/PR is stale and will be closed soon label Aug 11, 2022

jimmybrancaccio added meta: never-stale This issue can never become stale and removed meta: stale This issue/PR is stale and will be closed soon labels Aug 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resolve Issue When Using a Self-Hosted GitLab Instance for the Container Registry on a Self-Hosted Gitpod Instance #8795

Resolve Issue When Using a Self-Hosted GitLab Instance for the Container Registry on a Self-Hosted Gitpod Instance #8795

jimmybrancaccio commented Mar 14, 2022 •

edited

Loading

mrsimonemms commented Apr 19, 2022

stale bot commented Aug 11, 2022

Fentonfi commented Dec 23, 2022

Resolve Issue When Using a Self-Hosted GitLab Instance for the Container Registry on a Self-Hosted Gitpod Instance #8795

Resolve Issue When Using a Self-Hosted GitLab Instance for the Container Registry on a Self-Hosted Gitpod Instance #8795

Comments

jimmybrancaccio commented Mar 14, 2022 • edited Loading

Bug description

Steps to reproduce

Workspace affected

Expected behavior

Example repository

Anything else?

mrsimonemms commented Apr 19, 2022

stale bot commented Aug 11, 2022

Fentonfi commented Dec 23, 2022

jimmybrancaccio commented Mar 14, 2022 •

edited

Loading