Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to run container jobs using services.github-runners #306373

Open
eliandoran opened this issue Apr 23, 2024 · 4 comments
Open

Unable to run container jobs using services.github-runners #306373

eliandoran opened this issue Apr 23, 2024 · 4 comments
Labels
0.kind: bug Something is broken

Comments

@eliandoran
Copy link
Contributor

Describe the bug

Jobs fail when using a container: declaration in GitHub workflows, using a non-NixOS image.

The container is well built and run, but the all the GitHub Actions run inside it will crash with:

/nix/store/jk6qarf9kzfl49v7mgpyhwmg1znhcax0-docker-24.0.5/bin/docker exec  6b346de7d0c9f8d99cdca256f1119c5d2af04625baa91e3ad1d0f4afbf978953 sh -c "cat /etc/*release | grep ^ID"
OCI runtime exec failed: exec failed: unable to start container process: exec: "/__e/node20/bin/node": stat /__e/node20/bin/node: no such file or directory: unknown

Adding a rudimentary log step reveals that the node executable is there:

ls -l /__e
total 0
lrwxrwxrwx 1 root root 57 Jan  1  1970 node20 -> /nix/store/6wq28hcbiclvmwgphry7jsy3w7mzypqc-nodejs-20.9.0

Steps To Reproduce

Steps to reproduce the behavior:

  1. Set up a GitHub runner using services.github-runners.
  2. Define a GitHub action like this one:
jobs:
  build:
    container:
      image: ubuntu:latest
    steps:
      - name: Checkout the repository
        uses: actions/checkout@v4

Expected behavior

GitHub Actions should execute OK even when run under containers.

Additional context

My suspicion is that we have a non-NixOS container (e.g. debian) that is trying to run to a NixOS executable (since the path to the symlink above does point to the Nix store).

Notify maintainers

@veehaitch , @thomasjm

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here

Will fill it later, since I don't have access to the runner at this moment.

Add a 👍 reaction to issues you find important.
@eliandoran eliandoran added the 0.kind: bug Something is broken label Apr 23, 2024
@thomasjm
Copy link
Contributor

Your suspicion sounds right. I'm not really sure how this works in a normal container: stanza outside of NixOS. I guess the GitHub Runner bind-mounts the necessary tools into the container?

In this case, it seems like it may be bind-mounting the Node.JS directory, but not the rest of its closure in the Nix store, so it fails to find shared libraries.

Maybe it'll work if you do this?

    container:
      image: ubuntu:latest
      volumes:
        - /nix:/nix

@eliandoran
Copy link
Contributor Author

@thomasjm , you're absolutely right. Mounting /nix in the container does seem to solve the issue.
Now I'm wondering how we could fix this without having to change the workflow itself (since it could cause compatibility issues when using the workflow on non-NixOS runners).

I think the issue stems from the dynamic library loader, having a hard-coded path to the loader in the Nix store.

@thomasjm
Copy link
Contributor

thomasjm commented Apr 25, 2024 •
edited
Loading

Probably not just the loader, but potentially all runtime dependencies of the chosen Node.js.

I think the right solution would be to configure GitHub runners on NixOS to mount the runtime closure of Node.js (not the full /nix dir, as that seems like too much) using the instructions here:

https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/customizing-the-containers-used-by-jobs

tl;dr from the docs: it looks like we need to set the ACTIONS_RUNNER_CONTAINER_HOOKS environment variable for the runner to point to an index.js where we define a prepare_job script, and the script should define jobContainer.systemMountVolumes to mount the paths we need.

@artslob
Copy link

artslob commented Dec 28, 2024 •
edited
Loading

Seems like its bug of github-runner rather then nix:

Since github-runner provides dynamically built NodeJs as volume from host machine into docker container, it causes a lot of possible errors when host machine and docker container are not compatible.

I tried to patch nodejs-20 to be built statically with overlay but failed.
Seems like simplest solution for me would to use action without nodejs dependency, for example actions/checkout@v1 or https://github.com/taiki-e/checkout-action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
0.kind: bug Something is broken
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@thomasjm @artslob @eliandoran