@typescript-bot perf test this

Heya @jakebailey, I've started to run the perf test suite on this PR at 1a42515. You can monitor the build here.

Update: The results are in!

@jakebailey
The results of the perf run you requested are in!

Not bad results 😉

Yeah, though this was already known when we merged modules and gave up var downleveling. Really unfortunate that this helps so much; I'm trying to figure out if there's some subset of variables that can be changed or if it's just a million papercuts.

What’s the concern here though? Just the build time?

Well, I wouldn't merge this as-is because (in no particular order):

• It introduces another third-party dep into the critical build chain, which right now is solely tsc and esbuild, and we have the ability to drop the latter if needed.
  • Though, arguably, the latter wouldn't be great for performance either, so what's the point in avoiding it?
• It slows down the build, and I don't really want to re-introduce the idea of a "release" build of TypeScript.
  • But, we can just keep let/const for tests and catch violations not already caught by TS that way.
• Loops get rewritten, which makes the files annoying to use while debugging; preferrably there would be a performance-focused variant of the transform which only did the "easy" ones.
  • Again, could be non-test only.
• babel deoptimizes the files because they're so big, which effectively makes them as hard to patch as minified versions of TypeScript, and we have an unreasonable number of people who do that. I half-mitigated this by rerunning the output through esbuild, though that was to try and avoid the perf diff showing a startup speed boots from the simple fact that the files are smaller.
  • But, it could be implemented in esbuild directly.
• Long term, I think it'd be bad to be ESM but have export var (which I believe is slower for consumers because they are live bindings and not const).
  • Then again, esbuild already does this, so maybe it is something that can be pushed down to esbuild, much to everyone's chagrin.

It slows down the build

On the whole though, the "big" build tasks don't actually get longer because esbuild/babel are running in parallel with dts emit / bundling, but the actual bundle tasks go from half a second to 8 seconds, which isn't great for the debugging loop.

I assume the performance hit for let/const vs. var is down to TDZ tracking. Which is unfortunate because var has enough footguns that I’d never use it directly in a codebase and I’m not running babel over my code as a build step just to change all the lets to vars.

Yep, it's TDZ. I'm trying to figure out if there's a limited subset that can be changed to make parsing faster but it's not really turning out well.

It’s weird because most potential TDZ violations can be detected syntactically so engines should have figured out how to optimize it away by now.

I would certainly think. But at this point I'm having a hard time justifying not doing some form of this change.

If people are stumbling on this, last time I tested this, it was 5-9%, and now it's more like 7-15%; the relative proportion has changed because we've increased the performance of the TS compiler since modules has merged meaning the fixed perf bump here is now a bigger overall proportion.

I hacked up TypeScript and ts-perf to run this PR against bun v0.5.5. It doesn't seem like the engine Bun uses (Apple's JavaScriptCore) is affected by the TDZ problem; I'd chalk up any perf differences below to noise.

Note that these benchmark results are not comparable to the ones earlier in this thread; I performed these tests on an entirely different perf-tuned machine in my office, not the corp-owned machine that we normally use. But, I probably will try and upstream my hacks to ts-perf as I've been wanting to test some more interesting hosts.

But, i'm going to rerun this just to be sure I didn't mess up somehow.

Nah, seems correct. A second go:

EDIT: Updated this table to say "bun"; I did that in the last one but forgot in this one. ts-perf doesn't currently support bun so this is a hack.

Note that Bun runs the transpiler on every file loaded either via import or require (not eval, not new Function, etc). It doesn’t convert let/const variables to var. I wouldn’t expect the results to be different l. it might not work, but you could try BUN_DISABLE_TRANSPILER=1 & it would rule out the possibility of Bun’s transpiler changing the output before being evaluated

I'm not sure what you mean; the comparison is main versus this PR, i.e. let/const versus var. So if they don't do this transformation (as you say), then the comparison shows that their engine doesn't have this TDZ performance hit, at least not to any great extent.

Unless you mean that I should try disabling the transpiler to make sure 100% they don't already do this transformation.

The environment variable which prevents the transpiler from running is here:

https://github.com/oven-sh/bun/blob/main/src/bundler.zig#L539

I think the results were already correct, just that since it’s two transpilers running it’s harder to be 100% sure. I sent the above message without re-reading the words I wrote very carefully. Bun doesn’t convert let or const to var, but does a bunch of other stuff. I do know that ESM imports bindings have virtually no overhead in JSC (there was a thread in Node’s issues about ESM imports being slower than var at some point)

Looping in @evanw because that may be an interesting optimization to do right in esbuild itself.

Looping in @evanw because that may be an interesting optimization to do right in esbuild itself.

See: evanw/esbuild#2889 (comment)

@typescript-bot new perf test bun

Heya @jakebailey, I've started to run the bun perf test suite on this PR at e67fbc0. You can monitor the build here.

Update: The results are in!

@jakebailey
The results of the perf run you requested are in!

@typescript-bot perf test public

Heya @jakebailey, I've started to run the public perf test suite on this PR at 0c249d9. You can monitor the build here.

Update: The results are in!

@jakebailey
The results of the perf run you requested are in!