Update Nix and Node JS versions #3549

Rheeseyb · 2023-04-17T12:31:43Z

Problem:
The version of node.js we have been using is old and needed updating in order to build the latest version of VS Code.

Fix:
Update the node.js version to v16.18.1, and update nix to the latest stable version (22.11).
The reason for the nix update is because the previous version of node we were using was the latest v16 version available on the older nix release.

Updating nix revealed that a package we were using on the server for the AWS API (amazonka) had been effectively abandoned and marked as broken, so that has been replaced with a new package (aws). Whilst we were there we also updated the version of the haskell compiler so that we could continue to pull cached builds from hydra as part of the server build (without that we'd have to build every single package in the dependency chain, making the build very long and painful).

Updating Node introduced / flagged the following issues that are resolved in this PR:

An error Error: error:0308010C:digital envelope routines::unsupported is thrown by webpack builds due to its use of a crypto function now considered to be legacy. The workaround for this is to use the node flag --openssl-legacy-provider pretty much everywhere. Further details here.
The updated version of pnpm seems to have introduced some change that was causing ts-loader resolution issues in webpack builds, so we had to replace an exclude rule with an explicit include rule in the editor's webpack config file for that loader. I'm not entirely sure why this only affects the dependency on utopia-api and not utopia-vscode-common.
It seems that by using pnpx tsc in places we were using a different version of tsc to the one coming from the typescript package specified in the package.json, so those cases have been replaced with an explicit node script instead.
patch-package was causing issues with pnpm, and is no longer necessary as pnpm has native support for patching, so the patches and package.json have been updated to reflect that.
The custom install steps we had for utopia-api were not only unnecessary, but were actively causing issues during the build, so those have been removed now and replaced with an install and build step in the install-editor script used everywhere.

…in tmux.

github-actions · 2023-04-17T12:32:26Z

Try me

relativeci · 2023-04-17T15:55:16Z

Job #5649: Bundle Size — 49.78MiB (+0.41%).

5c12b96(current) vs c4aaee4 master#5645(baseline)

⚠️ Bundle removed 15 duplicate packages, 46 duplicate packages remaining
⚠️ Bundle introduced 2 new packages: utopia-api, utopia-vscode-common

Metrics (9 changes)

	Current Job #5649	Baseline Job #5645
Initial JS	`32.15MiB`(`-0.63%`)	`32.35MiB`
Initial CSS	`0B`	`0B`
Cache Invalidation	`73.22%`	`21.79%`
Chunks	`26`(`+8.33%`)	`24`
Assets	`30`(`+7.14%`)	`28`
Modules	`3618`(`-1.26%`)	`3664`
Duplicate Modules	`353`(`+0.28%`)	`352`
Duplicate Code	`17.28%`(`+4.85%`)	`16.48%`
Packages	`373`(`-3.37%`)	`386`
Duplicate Packages	`46`(`-24.59%`)	`61`

Total size by type (2 changes)

	Current Job #5649	Baseline Job #5645
CSS	`0B`	`0B`
Fonts	`0B`	`0B`
HTML	`11.57KiB` (`~-0.01%`)	`11.57KiB`
IMG	`0B`	`0B`
JS	`49.77MiB` (`+0.41%`)	`49.57MiB`
Media	`0B`	`0B`
Other	`0B`	`0B`

View job #5649 report View chore/update-nix-node branch activity

github-actions · 2023-04-17T16:07:41Z

Performance test results:

Rheeseyb · 2023-04-18T12:48:38Z

editor/patches/@emotion__react@11.1.2.patch

@@ -1,7 +1,7 @@
-diff --git a/node_modules/@emotion/react/dist/emotion-react.browser.cjs.js b/node_modules/@emotion/react/dist/emotion-react.browser.cjs.js
+diff --git a/dist/emotion-react.browser.cjs.js b/dist/emotion-react.browser.cjs.js


All of the patches are file renames (to match the style used by pnpm's native patching support to make updating the patches later easier), and the removal of /node_modules/<package_name> from the file paths used in the diffs (like has happened here), so you can ignore the patch diffs

Rheeseyb · 2023-04-18T12:49:23Z

editor/jest.config.js

@@ -116,7 +116,7 @@ module.exports = {
        '\\.[jt]sx?$': 'babel-jest',
      },
      roots: ['src', 'node_modules', '<rootDir>/node_modules'],
-      transformIgnorePatterns: ['/node_modules/(?!utopia-api)'], // this lets ts-jest work on `/node_modules/utopia-api` which is a simlink to `../utopia-api`.
+      transformIgnorePatterns: ['<rootDir>/node_modules/.pnpm/(?!file+..+)'],


I'm honestly not sure how this hadn't broken before, but it seems like we were on the edge of this breaking since we first switched to pnpm

Rheeseyb · 2023-04-18T12:50:47Z

editor/package.json

-    "lint": "pnpx eslint \"src/**/*.{js,ts,tsx}\" --quiet",
-    "lint-loud": "pnpx eslint \"src/**/*.{js,ts,tsx}\"",
+    "check-code": "tsc --version && tsc --noEmit && pnpm run lint && pnpm run check-dependency-cruiser && pnpm run prettier-check",
+    "lint": "eslint \"src/**/*.{js,ts,tsx}\" --quiet",


pnpx had been changed over a year ago to become an alias for pnpm dlx, which downloads and runs the latest version of a package, rather than using the locally installed version

wow, good to know!

Rheeseyb · 2023-04-18T12:52:41Z

editor/webpack.config.js

+          if (filePath.includes('node_modules')) {
+            // We need to use the ts-loader to load the utopia-api module,
+            // but nothing else from node_modules
+            return filePath.includes('/utopia-api/')


Again, it seems like we have been on the verge of this breaking since we made the switch to pnpm

I didn't know you can write a function here instead of a regex. this makes it much easier to maintain! bonus points for the comments!

Rheeseyb · 2023-04-18T12:55:20Z

shell.nix

+    (pkgs.writeScriptBin "install-utopia-api" ''
+      #!/usr/bin/env bash
+      set -e
+      cd $(${pkgs.git}/bin/git rev-parse --show-toplevel)/utopia-api
+      pnpm install
+      pnpm run build
+    '')


This was added to replace the script previously living in the editor's package.json, which was causing build issues

Rheeseyb · 2023-04-18T12:56:57Z

shell.nix

+  pnpm = node.pkgs.pnpm;
+  yarn = pkgs.yarn;

-  pnpmPkg = pkgs.nodePackages_latest.pnpm.override {
-    nativeBuildInputs = [ pkgs.makeWrapper ];
-
-    preRebuild = ''
-      sed 's/"link:/"file:/g' --in-place package.json
-    '';
-
-    postInstall = let
-      pnpmLibPath = pkgs.lib.makeBinPath [
-        node.passthru.python
-        node
-      ];
-    in ''
-      for prog in $out/lib/node_modules/pnpm/bin/*; do
-        wrapProgram "$prog" --prefix PATH : ${pnpmLibPath}
-      done
-    '';
-  };
-  pnpm = "${pnpmPkg}/lib/node_modules/pnpm/bin/pnpm.cjs";
-  pnpx = "${pnpmPkg}/lib/node_modules/pnpm/bin/pnpx.cjs";
-
-  yarn = "${pkgs.nodePackages_latest.yarn}/lib/node_modules/yarn/bin/yarn.js";
+  nodePackages = [
+    node
+    pnpm
+    (yarn.override { nodejs = node; })
+  ];


This is synching up the version of node used by pnpm and yarn

maltenuhn · 2023-04-19T17:15:29Z

FWIW

I've been able to build the project with this change, even on my cursed machine
I've had this branch open all day and making theme-related changes, including occasional full reloads, and everything worked as expected

balazsbajorics · 2023-04-20T08:48:39Z

editor/package.json

    "eslint-check": "eslint --quiet --ignore-path ./.eslintignore --config ./.eslintrc.js --ext .ts --ext .tsx src ../utopia-api/src ../utopia-vscode-extension/src ../utopia-vscode-common/src ../puppeteer-tests/src ../website-next/",
    "prettier-check": "prettier --ignore-path ../.prettierignore --config ../prettier.config.js --check src/** ../utopia-api/src/**/* ../utopia-vscode-common/src/** ../utopia-vscode-extension/src/** ../puppeteer-tests/src/** ../website-next/pages/** ../website-next/components/**",
    "parse-print-analysis": "npx -s sh ts-node --require ignore-styles --transpile-only ./src/scripts/parse-print-analysis.ts",
    "parse-debug": "npx -s sh ts-node --pretty --require ignore-styles --transpile-only ./src/scripts/parse-debug.ts",
    "fetch-tailwind-classes": "npx -s sh ts-node --pretty --require ignore-styles --transpile-only ./src/scripts/fetch-and-parse-tailwind-classes.ts && npx prettier --write ./src/core/third-party/tailwind-defaults.ts",
    "lint-editor": "lint-staged",
    "dev-fast": "APP_ENV=development REACT_APP_ENVIRONMENT_CONFIG=development REACT_APP_AUTH0_CLIENT_ID=$AUTH0_CLIENT_ID REACT_APP_AUTH0_ENDPOINT=$AUTH0_ENDPOINT REACT_APP_AUTH0_REDIRECT_URI=$AUTH0_REDIRECT_URI REACT_APP_COMMIT_HASH=`git rev-parse HEAD` vite --force",
-    "vite-build": "node --max_old_space_size=16384 ./node_modules/vite/bin/vite build -- -DEBUG"
+    "vite-build": "node --max_old_space_size=16384 ./node_modules/vite/bin/vite build -- -DEBUG",
+    "watch-tsc": "tsc --watch && NODE_OPTIONS='--max_old_space_size=4096 --openssl-legacy-provider'"


I think this will also make iTerm 2's cmd + click to open VSCode work properly!

balazsbajorics · 2023-04-20T08:50:43Z

editor/package.json

    "staging-print-json": "pnpm run staging --  --json=lib/staging-stats.json",
-    "install-utopia-api": "cd ../utopia-api && pnpm install && pnpm run build && pnpm run dts-bundle",
-    "preinstall": "npx only-allow pnpm && pnpm run install-utopia-api",
-    "postinstall": "patch-package",


Rheeseyb and others added 8 commits April 13, 2023 12:15

chore(all) Updating nix and nodejs

e82a61c

fix(editor) Fix TS errors introduced by node update

40fd143

chore(pr) Update server with more recent dependencies.

1cd1e7b

fix(webpack) Fix utopia-api loading with new version of pnpm

8c6b003

chore(pr) Fix the issues with backspace and terminal colour profiles …

1670a5c

…in tmux.

chore(all) Use --openssl-legacy-provider everywhere explicitly

c30bf85

chore(pr) Swapped out minio-hs for aws to support S3.

8153676

Merge branch 'master' into chore/update-nix-node

db09355

Rheeseyb requested review from gbalint, seanparsons, ruggi, balazsbajorics, enidemi and bkrmendy April 17, 2023 12:31

Rheeseyb added 4 commits April 17, 2023 14:24

fix(shell.nix) update x11 to xlibsWrapper

31cadff

fix(utopia-api,editor) Use updated version of patch-package

b588c26

fix(editor,utopia-api) Switch back to patch-package

1901714

fix(editor,utopia-api) Use pnpm native patching support

3100e45

Rheeseyb and others added 6 commits April 17, 2023 17:50

maybe(editor) Trying to get this damn thing working

72b9be3

fix(package.json) Syntax change when passing params to jest

e14b69e

chore(utopia-api) Remove dts and custom install steps

28f2604

chore(pr) Fixed up tests for server build.

d6a1ee4

fix(editor) Remove rogue pnpx

50ab3e6

fix(shell.nix) Fixed a missing yarn install

5c12b96

concrete-utopia deleted a comment from github-actions bot Apr 18, 2023

Rheeseyb commented Apr 18, 2023

View reviewed changes

bkrmendy approved these changes Apr 20, 2023

View reviewed changes

enidemi approved these changes Apr 20, 2023

View reviewed changes

gbalint approved these changes Apr 20, 2023

View reviewed changes

balazsbajorics reviewed Apr 20, 2023

View reviewed changes

balazsbajorics approved these changes Apr 20, 2023

View reviewed changes

Rheeseyb merged commit bcd5023 into master Apr 20, 2023

Rheeseyb deleted the chore/update-nix-node branch April 20, 2023 10:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Nix and Node JS versions #3549

Update Nix and Node JS versions #3549

Rheeseyb commented Apr 17, 2023 •

edited

Loading

github-actions bot commented Apr 17, 2023 •

edited

Loading

relativeci bot commented Apr 17, 2023 •

edited

Loading

github-actions bot commented Apr 17, 2023 •

edited

Loading

Rheeseyb Apr 18, 2023

Rheeseyb Apr 18, 2023

Rheeseyb Apr 18, 2023

balazsbajorics Apr 20, 2023

Rheeseyb Apr 18, 2023

balazsbajorics Apr 20, 2023

Rheeseyb Apr 18, 2023

Rheeseyb Apr 18, 2023

maltenuhn commented Apr 19, 2023

balazsbajorics Apr 20, 2023

balazsbajorics Apr 20, 2023

		@@ -1,7 +1,7 @@
		diff --git a/node_modules/@emotion/react/dist/emotion-react.browser.cjs.js b/node_modules/@emotion/react/dist/emotion-react.browser.cjs.js
		diff --git a/dist/emotion-react.browser.cjs.js b/dist/emotion-react.browser.cjs.js

Update Nix and Node JS versions #3549

Update Nix and Node JS versions #3549

Conversation

Rheeseyb commented Apr 17, 2023 • edited Loading

github-actions bot commented Apr 17, 2023 • edited Loading

relativeci bot commented Apr 17, 2023 • edited Loading

Job #5649: Bundle Size — 49.78MiB (+0.41%).

github-actions bot commented Apr 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maltenuhn commented Apr 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rheeseyb commented Apr 17, 2023 •

edited

Loading

github-actions bot commented Apr 17, 2023 •

edited

Loading

relativeci bot commented Apr 17, 2023 •

edited

Loading

github-actions bot commented Apr 17, 2023 •

edited

Loading