-
-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Slow createProjectProgram/createWatchProgram #2398
Comments
Sorry, what is your issue about exactly? Could you please clarify? If all of your packages are in the |
@bradzacher Please compare the time in the command outputs, the former one has all subsequent createProjectProgram taking around 1s, and the later one seems much better. The overall performance is something like 40s vs 4s in my case. Unfortunately I am writing a boilerplate so that I would like a more general configuration if possible. I can use the later configuration suggested in the first comment, but just want to let you guys know this weird issue exists. Note these two configurations result in the same |
Which times are you talking about exactly? The thing I'd be more interested in is the total lint time for the workspace. |
Yes, it's 40s vs 4s in my case. The first configuration results in:
And the second configuration results in:
It is happening for every subsequent files, so the time accumulates. |
It's hard to provide any guidance here, as there isn't much information here. |
https://github.com/vilic/bug-repros/tree/eslint-2398 git clone --single-branch --branch eslint-2398 https://github.com/vilic/bug-repros.git
cd bug-repros
yarn
yarn lint Again, if it's not that obvious, please let me know so that I would try to minimize the repro when I get extra time. |
So this problem is because The typescript-eslint/packages/typescript-estree/src/parser.ts Lines 136 to 171 in a53f8c6
I've done some research and
The old code allowed regexes, so it's going to be too hard to change this right now. |
Fixes #2398 If the user has a particularly large node_modules folder and uses globs for `parserOption.project`, then the `glob` library can spend a decent chunk of time searching the `node_modules` folder. In some cases, this can be on the order of hundreds to thousands of milliseconds. This wouldn't be a problem, but for safety and correctness during a persistent parse, we have to do this check for every call to the parser (i.e. every file that's being linted). Over a whole project, this can easily add up to many, many seconds wasted. Previously, we: - applied the project globs, one by one - then manually excluded `tsconfig`s from the list. This meant that we are always slow. I remember I did this because I had issues getting `glob`'s `ignore` option to work at all. ## The solution `globby` is a better glob library: - it accepts an array of globs, which saves us doing manual looping - it supports exclusion globs (globs prefixed with `!`), which are evaluated as part of the glob process - it has caching built in by default This allows us to evaluate all of the `project` globs at once, as opposed to one at a time (so should be less duplicated work). This also allows us to evaluate the `projectFolderIgnoreList` at the same time as the `project` globs (so should be no useless work done). All of these together should cut the glob evaluation time down to ~50ms for the first parse, and ~2ms for each parse after that (due to caching). For comparison, previously, in bad cases we would spend ~3-500ms, per project, per parsed file. Example to illustrate how much faster this can potentially be: For a project that provides 2 globs and has 100 files. Before: 300ms * 2 * 100 = 60,000ms (60s) After: 50ms + 2 * 100 = 250ms This should also save a non-trival amount of time in other, more optimal setups. BREAKING CHANGE: - removes the ability to supply a `RegExp` to `projectFolderIgnoreList`, and changes the meaning of the string value from a regex to a glob.
Fixes #2398 If the user has a particularly large node_modules folder and uses globs for `parserOption.project`, then the `glob` library can spend a decent chunk of time searching the `node_modules` folder. In some cases, this can be on the order of hundreds to thousands of milliseconds. This wouldn't be a problem, but for safety and correctness during a persistent parse, we have to do this check for every call to the parser (i.e. every file that's being linted). Over a whole project, this can easily add up to many, many seconds wasted. Previously, we: - applied the project globs, one by one - then manually excluded `tsconfig`s from the list. This meant that we are always slow. I remember I did this because I had issues getting `glob`'s `ignore` option to work at all. ## The solution `globby` is a better glob library: - it accepts an array of globs, which saves us doing manual looping - it supports exclusion globs (globs prefixed with `!`), which are evaluated as part of the glob process - it has caching built in by default This allows us to evaluate all of the `project` globs at once, as opposed to one at a time (so should be less duplicated work). This also allows us to evaluate the `projectFolderIgnoreList` at the same time as the `project` globs (so should be no useless work done). All of these together should cut the glob evaluation time down to ~50ms for the first parse, and ~2ms for each parse after that (due to caching). For comparison, previously, in bad cases we would spend ~3-500ms, per project, per parsed file. Example to illustrate how much faster this can potentially be: For a project that provides 2 globs and has 100 files. Before: 300ms * 2 * 100 = 60,000ms (60s) After: 50ms + 2 * 100 = 250ms This should also save a non-trival amount of time in other, more optimal setups. BREAKING CHANGE: - removes the ability to supply a `RegExp` to `projectFolderIgnoreList`, and changes the meaning of the string value from a regex to a glob.
Fixes #2398 If the user has a particularly large node_modules folder and uses globs for `parserOption.project`, then the `glob` library can spend a decent chunk of time searching the `node_modules` folder. In some cases, this can be on the order of hundreds to thousands of milliseconds. This wouldn't be a problem, but for safety and correctness during a persistent parse, we have to do this check for every call to the parser (i.e. every file that's being linted). Over a whole project, this can easily add up to many, many seconds wasted. Previously, we: - applied the project globs, one by one - then manually excluded `tsconfig`s from the list. This meant that we are always slow. I remember I did this because I had issues getting `glob`'s `ignore` option to work at all. ## The solution `globby` is a better glob library: - it accepts an array of globs, which saves us doing manual looping - it supports exclusion globs (globs prefixed with `!`), which are evaluated as part of the glob process - it has caching built in by default This allows us to evaluate all of the `project` globs at once, as opposed to one at a time (so should be less duplicated work). This also allows us to evaluate the `projectFolderIgnoreList` at the same time as the `project` globs (so should be no useless work done). All of these together should cut the glob evaluation time down to ~50ms for the first parse, and ~2ms for each parse after that (due to caching). For comparison, previously, in bad cases we would spend ~3-500ms, per project, per parsed file. Example to illustrate how much faster this can potentially be: For a project that provides 2 globs and has 100 files. Before: 300ms * 2 * 100 = 60,000ms (60s) After: 50ms + 2 * 100 = 250ms This should also save a non-trival amount of time in other, more optimal setups. BREAKING CHANGE: - removes the ability to supply a `RegExp` to `projectFolderIgnoreList`, and changes the meaning of the string value from a regex to a glob.
Fixes #2398 If the user has a particularly large node_modules folder and uses globs for `parserOption.project`, then the `glob` library can spend a decent chunk of time searching the `node_modules` folder. In some cases, this can be on the order of hundreds to thousands of milliseconds. This wouldn't be a problem, but for safety and correctness during a persistent parse, we have to do this check for every call to the parser (i.e. every file that's being linted). Over a whole project, this can easily add up to many, many seconds wasted. Previously, we: - applied the project globs, one by one - then manually excluded `tsconfig`s from the list. This meant that we are always slow. I remember I did this because I had issues getting `glob`'s `ignore` option to work at all. ## The solution `globby` is a better glob library: - it accepts an array of globs, which saves us doing manual looping - it supports exclusion globs (globs prefixed with `!`), which are evaluated as part of the glob process - it has caching built in by default This allows us to evaluate all of the `project` globs at once, as opposed to one at a time (so should be less duplicated work). This also allows us to evaluate the `projectFolderIgnoreList` at the same time as the `project` globs (so should be no useless work done). All of these together should cut the glob evaluation time down to ~50ms for the first parse, and ~2ms for each parse after that (due to caching). For comparison, previously, in bad cases we would spend ~3-500ms, per project, per parsed file. Example to illustrate how much faster this can potentially be: For a project that provides 2 globs and has 100 files. Before: 300ms * 2 * 100 = 60,000ms (60s) After: 50ms + 2 * 100 = 250ms This should also save a non-trival amount of time in other, more optimal setups. BREAKING CHANGE: - removes the ability to supply a `RegExp` to `projectFolderIgnoreList`, and changes the meaning of the string value from a regex to a glob.
Fixes #2398 If the user has a particularly large node_modules folder and uses globs for `parserOption.project`, then the `glob` library can spend a decent chunk of time searching the `node_modules` folder. In some cases, this can be on the order of hundreds to thousands of milliseconds. This wouldn't be a problem, but for safety and correctness during a persistent parse, we have to do this check for every call to the parser (i.e. every file that's being linted). Over a whole project, this can easily add up to many, many seconds wasted. Previously, we: - applied the project globs, one by one - then manually excluded `tsconfig`s from the list. This meant that we are always slow. I remember I did this because I had issues getting `glob`'s `ignore` option to work at all. ## The solution `globby` is a better glob library: - it accepts an array of globs, which saves us doing manual looping - it supports exclusion globs (globs prefixed with `!`), which are evaluated as part of the glob process - it has caching built in by default This allows us to evaluate all of the `project` globs at once, as opposed to one at a time (so should be less duplicated work). This also allows us to evaluate the `projectFolderIgnoreList` at the same time as the `project` globs (so should be no useless work done). All of these together should cut the glob evaluation time down to ~50ms for the first parse, and ~2ms for each parse after that (due to caching). For comparison, previously, in bad cases we would spend ~3-500ms, per project, per parsed file. Example to illustrate how much faster this can potentially be: For a project that provides 2 globs and has 100 files. Before: 300ms * 2 * 100 = 60,000ms (60s) After: 50ms + 2 * 100 = 250ms This should also save a non-trival amount of time in other, more optimal setups. BREAKING CHANGE: - removes the ability to supply a `RegExp` to `projectFolderIgnoreList`, and changes the meaning of the string value from a regex to a glob.
Fixes #2398 If the user has a particularly large node_modules folder and uses globs for `parserOption.project`, then the `glob` library can spend a decent chunk of time searching the `node_modules` folder. In some cases, this can be on the order of hundreds to thousands of milliseconds. This wouldn't be a problem, but for safety and correctness during a persistent parse, we have to do this check for every call to the parser (i.e. every file that's being linted). Over a whole project, this can easily add up to many, many seconds wasted. Previously, we: - applied the project globs, one by one - then manually excluded `tsconfig`s from the list. This meant that we are always slow. I remember I did this because I had issues getting `glob`'s `ignore` option to work at all. ## The solution `globby` is a better glob library: - it accepts an array of globs, which saves us doing manual looping - it supports exclusion globs (globs prefixed with `!`), which are evaluated as part of the glob process - it has caching built in by default This allows us to evaluate all of the `project` globs at once, as opposed to one at a time (so should be less duplicated work). This also allows us to evaluate the `projectFolderIgnoreList` at the same time as the `project` globs (so should be no useless work done). All of these together should cut the glob evaluation time down to ~50ms for the first parse, and ~2ms for each parse after that (due to caching). For comparison, previously, in bad cases we would spend ~3-500ms, per project, per parsed file. Example to illustrate how much faster this can potentially be: For a project that provides 2 globs and has 100 files. Before: 300ms * 2 * 100 = 60,000ms (60s) After: 50ms + 2 * 100 = 250ms This should also save a non-trival amount of time in other, more optimal setups. BREAKING CHANGE: - removes the ability to supply a `RegExp` to `projectFolderIgnoreList`, and changes the meaning of the string value from a regex to a glob.
Repro
The
@mufan/code-*
packages below involves symlinks.Please let me know if this is not obvious enough so I can prepare an actual repro.
debug output
But if I change the project pattern and ignore the
node_modules
before ignore list applied, then it lints a lot faster:debug output
Versions
@typescript-eslint/typescript-estree
3.9.0
@typescript-eslint/parser
3.9.0
TypeScript
3.9.7
node
14.4.0
npm
n/a
The text was updated successfully, but these errors were encountered: