CLI: Cannot create a string longer than 0x1fffffe8 characters #492

taras-turchenko-moc · 2022-02-04T09:31:14Z

Issue description

We have front-end heavy application with ~40 entry points and i want to optimize build size

$ rm -rf ./tmp/webpack-stats.json

$ yarn build --progress --profile --json > ./tmp/webpack-stats.json && npx webpack-bundle-analyzer ./tmp/webpack-stats.json
Couldn't read webpack bundle stats from "/Users/tarasturchenko/Projects/work/ZipifyPages/tmp/webpack-stats.json":
Error: Cannot create a string longer than 0x1fffffe8 characters

$ ll ./tmp/webpack-stats.json
-rw-r--r--  1 tarasturchenko  staff   1.0G Feb  4 11:28 ./tmp/webpack-stats.json

Technical info

  System:
    OS: macOS 12.0.1
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Memory: 487.04 MB / 16.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.18.1 - ~/.nvm/versions/node/v14.18.1/bin/node
    Yarn: 1.22.17 - ~/.nvm/versions/node/v14.18.1/bin/yarn
    npm: 8.1.4 - ~/.nvm/versions/node/v14.18.1/bin/npm
  npmPackages:
    compression-webpack-plugin: ^9.0.0 => 9.0.0
    image-minimizer-webpack-plugin: ^3.1.2 => 3.1.2
    webpack: 5.64.1 => 5.64.1
    webpack-cli: 4.9.1 => 4.9.1
    webpack-manifest-plugin: ^4.0.2 => 4.0.2
    webpack-merge: ^5.8.0 => 5.8.0

Debug info

How do you use this module? As CLI utility or as plugin?

If CLI, what command was used? (e.g. webpack-bundle-analyzer -O path/to/stats.json)

npx webpack-bundle-analyzer ./tmp/webpack-stats.json

If plugin, what options were provided? (e.g. new BundleAnalyzerPlugin({ analyzerMode: 'disabled', generateStatsFile: true }))

What other Webpack plugins were used?

image-minimizer-webpack-plugin
mini-css-extract-plugin
compression-webpack-plugin
webpack-manifest-plugin
IgnorePlugin & ProvidePlugin
vue-loader

It would be nice to also attach webpack stats file.
It can be generated using these options:

new BundleAnalyzerPlugin({
  analyzerMode: 'disabled',
  generateStatsFile: true,
  // Excludes module sources from stats file so there won't be any sensitive data
  statsOptions: { source: false }
})

stats.json will be created in Webpack bundle output directory.

The text was updated successfully, but these errors were encountered:

valscion · 2022-02-04T11:31:19Z

Oh wow, I've never seen an error like that 😅 sounds like something is generating an awfully long string 😅

I don't know even where to start to fix this. Would you be open to do some googling and figure out if this error has showed up elsewhere and if there might be some common code patterns that might cause this error?

th0r · 2022-02-04T11:34:45Z

I think the reason is webpack-stats.json size is 1gb and we're using something like readFileSync to read it as a string.

th0r · 2022-02-04T12:30:43Z

I think here are the problematic code:

webpack-bundle-analyzer/src/analyzer.js

Lines 164 to 168 in e5b617e

    
           function readStatsFromFile(filename) { 
        
             return JSON.parse( 
        
               fs.readFileSync(filename, 'utf8') 
        
             ); 
        
           }

Maybe there is some package that parses JSON without reading it into a string first.
@sokra maybe you've already solved such problem?

taras-turchenko-moc · 2022-02-04T15:43:27Z

I'll try stream-json. Seems like this lib can do it but it takes too long. I started parsing 30 minutes ago and it's still running

Also i will try how works the webpack plugin with it

taras-turchenko-moc · 2022-02-04T16:07:06Z

It's finished and it takes ~30 minutes just to parse this json. Also i've tried the webpack plugin and it works fine

sushantmakin · 2022-02-09T11:06:18Z

Hi,
What is the way around this? the stats.json which is created is >512 MB (840 MB), hence it fails with the same error for me as well with the same error.

taras-turchenko-moc · 2022-02-10T08:28:14Z

@sushantmakin Webpack plugin works fine for me

https://github.com/webpack-contrib/webpack-bundle-analyzer#usage-as-a-plugin

tgudroe · 2022-05-05T19:03:26Z

Is there a solution to this? We also have a very large stats.json file that is failing when trying to analyze it.

taras-turchenko-moc · 2022-05-06T12:09:43Z

@tgudroe the webpack plugin works fine

tgudroe · 2022-05-06T12:19:56Z

Does that mean the CLI can't be used in my case and there is no hope of it working in the future?

taras-turchenko-moc · 2022-06-07T12:15:21Z

@tgudroe I'll take a look at this later. Maybe we can fix it by replacing fs.readfile to one of binary json readers

joshunger · 2022-06-10T15:07:07Z

When I upload the same stats.json to http://chrisbateman.github.io/webpack-visualizer/ I'm seeing this error. Is this related?

VM33:1 Uncaught SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at e.handleFileUpload (build.js:18:5124)
    at n.onloadend (build.js:23:9878)

tiberiuzuld · 2022-06-29T07:58:08Z

Hello,
I encountered this issue with a large angular app.
Using the changes in this PR worked for me #423

haven2world · 2022-09-26T06:40:24Z

Same issue. Hope this can be fixed.

valscion · 2022-09-26T07:06:48Z

The workaround is to use the plugin and not CLI

haven2world · 2022-09-27T03:18:14Z

The workaround is to use the plugin and not CLI

I knew that. But sometimes I need to compare the dependency graph between branches. It takes time to switch branch, install dependency and build the project. In this case, it's much more convenient to leverage the existing stat.json

valscion · 2023-02-13T08:40:10Z

This should now be fixed in v4.8.0 thanks to #423! 🚀

Feel free to take it for a spin and let us know if the issue wasn't solved with that PR as expected.

tiberiuzuld · 2023-07-04T14:21:13Z

Hello,
A bit late to the party, but only now I had the time to test this with our application.
I tested this fix and it worked for me to load a 548 MB file.

Thank you for the fix.

valscion added status: Needs triage type: Bug labels Feb 4, 2022

HumbertoL mentioned this issue May 8, 2022

chore: upgrade to react-scripts 5.0.1 callemall/create-react-app#8

Merged

valscion mentioned this issue Dec 7, 2022

Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory #543

Closed

valscion closed this as completed Feb 13, 2023

Yexiaoxing mentioned this issue Mar 20, 2023

Big stats file cannot be loaded via fs function relative-ci/bundle-stats#3195

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLI: Cannot create a string longer than 0x1fffffe8 characters #492

CLI: Cannot create a string longer than 0x1fffffe8 characters #492

taras-turchenko-moc commented Feb 4, 2022

valscion commented Feb 4, 2022

th0r commented Feb 4, 2022

th0r commented Feb 4, 2022 •

edited

Loading

taras-turchenko-moc commented Feb 4, 2022

taras-turchenko-moc commented Feb 4, 2022

sushantmakin commented Feb 9, 2022 •

edited

Loading

taras-turchenko-moc commented Feb 10, 2022

tgudroe commented May 5, 2022

taras-turchenko-moc commented May 6, 2022 •

edited

Loading

tgudroe commented May 6, 2022

taras-turchenko-moc commented Jun 7, 2022

joshunger commented Jun 10, 2022

tiberiuzuld commented Jun 29, 2022

haven2world commented Sep 26, 2022

valscion commented Sep 26, 2022

haven2world commented Sep 27, 2022

valscion commented Feb 13, 2023

tiberiuzuld commented Jul 4, 2023

CLI: Cannot create a string longer than 0x1fffffe8 characters #492

CLI: Cannot create a string longer than 0x1fffffe8 characters #492

Comments

taras-turchenko-moc commented Feb 4, 2022

Issue description

Technical info

Debug info

valscion commented Feb 4, 2022

th0r commented Feb 4, 2022

th0r commented Feb 4, 2022 • edited Loading

taras-turchenko-moc commented Feb 4, 2022

taras-turchenko-moc commented Feb 4, 2022

sushantmakin commented Feb 9, 2022 • edited Loading

taras-turchenko-moc commented Feb 10, 2022

tgudroe commented May 5, 2022

taras-turchenko-moc commented May 6, 2022 • edited Loading

tgudroe commented May 6, 2022

taras-turchenko-moc commented Jun 7, 2022

joshunger commented Jun 10, 2022

tiberiuzuld commented Jun 29, 2022

haven2world commented Sep 26, 2022

valscion commented Sep 26, 2022

haven2world commented Sep 27, 2022

valscion commented Feb 13, 2023

tiberiuzuld commented Jul 4, 2023

th0r commented Feb 4, 2022 •

edited

Loading

sushantmakin commented Feb 9, 2022 •

edited

Loading

taras-turchenko-moc commented May 6, 2022 •

edited

Loading