Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Composite Actions using Node 20 don't work with CentOS 7 #2906

Open
ajschmidt8 opened this issue Oct 4, 2023 · 14 comments
Open

Composite Actions using Node 20 don't work with CentOS 7 #2906

ajschmidt8 opened this issue Oct 4, 2023 · 14 comments
Labels
bug Something isn't working

Comments

@ajschmidt8
Copy link
Contributor

Problem

As mentioned in the GitHub blog post below, Actions is currently transitioning from Node 16 to Node 20.

However, Node 20 seems to be incompatible with CentOS 7, which is listed as a compatible OS for self-hosted runners below:

This incompatibility seems to stem from the fact that NodeJS version 20 supports glibc >=2.28 (as seen in the link below), but CentOS 7 only has glibc==2.17.

The consequence of this is that composite actions that have been updated to use Node 20 no longer work with CentOS 7 host machines or containers.

Here is an example of the error message that results from trying to run a composite action using Node 20 on a CentOS 7 container:

As a result of this incompatibility, we've had to intentionally downgrade our composite actions to a version that uses Node 16 so that our workflows continue to work.

To Reproduce

Run the following workflow to exhibit the error message

name: centos7

on:
  push:

jobs:
  run:
    runs-on: ubuntu-latest
    container: centos:7
    steps:
      - uses: actions/checkout@v4
      - run: echo "hello, world"

image

Expected behavior

Since the documentation below states that self-hosted runners support CentOS 7, I would expect the above workflow to run to completion without any errors

Runner Version and Platform

Ubuntu 22.04, runner version 2.309.0

@firewave
Copy link

firewave commented Jan 28, 2024

This also affects

ubuntu:16:04 (https://github.com/danmar/cppcheck/actions/runs/7677304162/job/20926809147?pr=5917)

/__e/node20/bin/node: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /__e/node20/bin/node)

ubuntu:18.04 (https://github.com/danmar/cppcheck/actions/runs/7677304162/job/20926809224?pr=5917)

/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)

@TomasNovo
Copy link

TomasNovo commented Feb 26, 2024

hey @ajschmidt8 ,
I'm currently facing this problem. I've tried several ways to use Node 16 but seems that the action always calls Node 20

/lib64/libm.so.6: version 'GLIBC_2.27' not found (required by /opt/actions-runner/externals/node20/bin/node)

how did you effectively managed to downgrade your composite actions to a version that uses Node 16 so that the workflows continue to work?

@troyxmccall
Copy link

@TomasNovo you'll need to audit all of your workflows and look up Node v16 compatible versions

ie:

FranzDiebold/github-env-vars-action@v2

would need to be rolled back to a specific version of

FranzDiebold/github-env-vars-action@v2.7.0

per https://github.com/FranzDiebold/github-env-vars-action/releases/tag/v2.7.0

@luciansmith
Copy link

I feel like this isn't getting the attention it needs: anyone using github actions to create manylinux2014 builds will no longer be able to build on that platform at all, if the proposed Node16->Node20 change becomes required. Is there any proposed mitigation of this? Am I incorrect in my assessment?

@alex
Copy link

alex commented May 11, 2024

Your assessment is correct. As a result, this change will have substantial negative impact on the Python community.

#3128 sketches out a direction Github could take to resolve this, but there has been no response thus far.

@dixyes
Copy link

dixyes commented May 14, 2024

For those who are struggling here,

I have a dirty way to make node 20 work with centos:7

  1. mount a dir with jobs.<name>.container.volumes, one ro on /__e/node20, another rw on /node20217
  2. download and extract unofficial node-v20.9.0-linux-x64-glibc-217.tar.xz into /node20217
  3. all workflow works

NOTE: since we cannot cache that tar before node is set-up, this is abusing nodejs distribution, so you should make your own docker image that contains that tar (rather than download it every build)

https://github.com/dixyes/ghactionsplay/blob/main/.github/workflows/glibc217node20.yml

https://github.com/dixyes/ghactionsplay/actions/runs/9075609934/job/24936557854

olliecheng added a commit to DavidsonGroup/flexiplex that referenced this issue Jul 11, 2024
acelyc111 added a commit to apache/incubator-pegasus that referenced this issue Jul 11, 2024
ruojieranyishen pushed a commit to ruojieranyishen/incubator-pegasus that referenced this issue Jul 17, 2024
amadio added a commit to amadio/xrootd that referenced this issue Jul 25, 2024
amadio added a commit to amadio/xrootd that referenced this issue Jul 25, 2024
amadio added a commit to amadio/xrootd that referenced this issue Jul 25, 2024
amadio added a commit to amadio/xrootd that referenced this issue Jul 25, 2024
@klepiz
Copy link

klepiz commented Aug 12, 2024

For those who are struggling here,

I have a dirty way to make node 20 work with centos:7

  1. mount a dir with jobs.<name>.container.volumes, one ro on /__e/node20, another rw on /node20217
  2. download and extract unofficial node-v20.9.0-linux-x64-glibc-217.tar.xz into /node20217
  3. all workflow works

NOTE: since we cannot cache that tar before node is set-up, this is abusing nodejs distribution, so you should make your own docker image that contains that tar (rather than download it every build)

https://github.com/dixyes/ghactionsplay/blob/main/.github/workflows/glibc217node20.yml

https://github.com/dixyes/ghactionsplay/actions/runs/9075609934/job/24936557854

Hello! this workaround worked for me for GH runner version 2.317 and 2.318 on a C7, but for 2.319 this is not working anymore and I am getting the issue

externals/node20/bin/node --version
externals/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by externals/node20/bin/node) 

Is there another way to make the latest GH runner version work on C7 + node20?

rudymatela added a commit to rudymatela/hello-haskell that referenced this issue Aug 26, 2024
... the Haskell image of it does not support running node20:

	/__e/node20/bin/node: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)
	/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)
	/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /__e/node20/bin/node)

I could try the following on GitHub actions:

    env:
        ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16
        ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true

as noted here: actions/runner#2906

However, it seems support for this was removed this spring anyway:

	https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

... or will be soon.
So I might as well stop running CI on older GHCs already...
@olsen232
Copy link

olsen232 commented Sep 5, 2024

+1 to fixing this issue
I'm using the Python manylinux2014 image but that still has the same issue. AFAICT there is no newer manylinux image than that, ie, Python best practice is still to build wheels on a linux image from 2014 that has an old GLIBC, but the builtin GH actions - due to using dynamically linked up-to-date nodejs - don't work with an old GLIBC. This is a pain and will surely only get worse as support is dropped for older node versions - this has already broken our build more than once as various actions we depend on switch to newer nodes, and each time I have to figure out how to pin whatever has gone wrong, which will only work as long as support is maintained for that old version.

I think using a statically linked node that just works for the built-in actions would be the way to go - as demonstrated here: #3128
Or - allow people to provide their own node that works for the container. I made sure that node was installed and working before using actions/checkout - but actions/checkout doesn't care, it installs a broken version of node anyway and uses that.

Lots of people have hit this already:
#337
#801
#2115
#2906
#3373

Shouren added a commit to Shouren/OpenMLDB that referenced this issue Sep 27, 2024
nmreadelf pushed a commit to 4paradigm/OpenMLDB that referenced this issue Oct 11, 2024
Shouren added a commit to Shouren/OpenMLDB that referenced this issue Oct 30, 2024
Shouren added a commit to Shouren/OpenMLDB that referenced this issue Oct 30, 2024
Shouren added a commit to Shouren/OpenMLDB that referenced this issue Oct 30, 2024
Shouren added a commit to 4paradigm/OpenMLDB that referenced this issue Oct 30, 2024
* fix(workflow): upgrade upload-artifact to v4

* fix(workflow): make action works by actions/runner#2906 (comment)
@troyxmccall
Copy link

troyxmccall commented Dec 8, 2024

this is my workaround for v2.321.0 on RHEL7 systems

grab a node20 compatible glibc dist (prebuilt) for linux x86_64 and save it to /node20217

curl -sL https://unofficial-builds.nodejs.org/download/release/v20.9.0/node-v20.9.0-linux-x64-glibc-217.tar.xz | tar -xJ --strip-components 1 -C /node20217

grab an anaconda runtime for an updated version of GCC

wget https://repo.anaconda.com/archive/Anaconda3-2021.05-Linux-x86_64.sh

install path to /anaconda3

sh Anaconda3-2021.05-Linux-x86_64.sh

su to your runner user (non-root user that runs the systemd spawns)

sudo su runner

overwrite the GHA node20 runtime with unofficial build glibc (this path will vary depending on where you installed the runner on your system)

cp /node20217/bin/node ~/actions-runner/externals/node20/bin/

set some globals in your runner users ~/.bashrc

export ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true
export LD_LIBRARY_PATH="/anaconda3/lib:/usr/lib64/:$LD_LIBRARY_PATH"
export CURL_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
export SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt

hijack some vars for our systemd service in runsvc.sh

~/actions-runner
❯ vim runsvc.sh

this allows us to use updated GLIBCXX_3.4.20 , etc

#!/bin/bash

# convert SIGTERM signal to SIGINT
# for more info on how to propagate SIGTERM to a child process see: http://veithen.github.io/2014/11/16/sigterm-propagation.html
trap 'kill -INT $PID' TERM INT

if [ -f ".path" ]; then
    # configure
    export PATH=`cat .path`
    echo ".path=${PATH}"
fi

nodever="node20"

# START NEW OVERRIDES
export LD_LIBRARY_PATH="/anaconda3/lib:/usr/lib64/:$LD_LIBRARY_PATH"
export CURL_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
export SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt
# END NEW OVERRIDES

# insert anything to setup env when running as a service
# run the host process which keep the listener alive
./externals/$nodever/bin/node ./bin/RunnerService.js &
PID=$!
wait $PID
trap - TERM INT
wait $PID

tell git to use your RHEL7 certs

git config --global http.sslCAinfo /etc/pki/tls/certs/ca-bundle.crt
go back to your root or sudo/wheel user and restart your service
sudo systemctl status actions.runner.org.runnername
check your service logs
✦ ❯ sudo systemctl status actions.runner.org.myrhel7box
● actions.runner.org.myrhel7box.service - GitHub Actions Runner (org.myrhel7box)
   Loaded: loaded (/etc/systemd/system/actions.runner.org.myrhel7box.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2024-12-08 14:33:37 EST; 28min ago
 Main PID: 19669 (runsvc.sh)
    Tasks: 22
   Memory: 10.0G
   CGroup: /system.slice/actions.runner.org.myrhel7box.service
           ├─19669 /bin/bash /home/runner/actions-runner/runsvc.sh
           ├─19675 ./externals/node20/bin/node ./bin/RunnerService.js
           └─19725 /home/runner/actions-runner/bin/Runner.Listener run --startuptype service

Dec 08 14:33:40 myrhel7box runsvc.sh[19669]: Current runner version: '2.321.0'
Dec 08 14:33:40 myrhel7box runsvc.sh[19669]: 2024-12-08 19:33:40Z: Listening for Jobs
Dec 08 14:42:52 myrhel7box runsvc.sh[19669]: 2024-12-08 19:42:52Z: Running job: checkout-and-build-site
Dec 08 14:45:18 myrhel7box runsvc.sh[19669]: 2024-12-08 19:45:18Z: Job checkout-and-build-site completed with result: Succeeded

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests