Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiler failing at optimization step with a big .ts file #2810

Closed
sebpiq opened this issue Jan 11, 2024 · 13 comments
Closed

Compiler failing at optimization step with a big .ts file #2810

sebpiq opened this issue Jan 11, 2024 · 13 comments
Labels

Comments

@sebpiq
Copy link

sebpiq commented Jan 11, 2024

Bug description

I am trying to compile a big file (~130k lines) that was machine generated, and is doing audio synthesis.
After a while the compiler returns the following error :

▌ Whoops, the AssemblyScript compiler has crashed during optimize :-(
▌ 
▌ Here is the stack trace hinting at the problem, perhaps it's useful?
▌ 
▌ RuntimeError: Aborted(). Build with -sASSERTIONS for more info.
▌     at x (file:///home/spiq/code/webpd/WebPd/node_modules/binaryen/index.js:6:3024)
▌     at p (file:///home/spiq/code/webpd/WebPd/node_modules/binaryen/index.js:9:3988)
▌     at wasm://wasm/016d0f36:wasm-function[92]:0x10aba
▌     at QB (file:///home/spiq/code/webpd/WebPd/node_modules/binaryen/index.js:9:108051)
▌     at wasm://wasm/016d0f36:wasm-function[1802]:0x1af2da
▌     at wasm://wasm/016d0f36:wasm-function[4982]:0x360e60
▌     at wasm://wasm/016d0f36:wasm-function[854]:0xee235
▌     at wasm://wasm/016d0f36:wasm-function[4984]:0x360ed2
▌     at DB (file:///home/spiq/code/webpd/WebPd/node_modules/binaryen/index.js:9:108308)
▌     at wasm://wasm/016d0f36:wasm-function[1388]:0x1396c6
▌ 
▌ If you see where the error is, feel free to send us a pull request. If not,
▌ please let us know: https://github.com/AssemblyScript/assemblyscript/issues
▌ 
▌ Thank you!

Some of the things I have tried :

  • doing only compile without optimization step : it works
  • reducing the file to a minimal test case : I couldn't. When I remove some of the code, it invariably starts working at some point.
  • dividing the code in two parts and trying to compile them separately : each of the two parts compiles + optimizes without any problem, which makes me think that it is probably due to the size of the file itself.

Steps to reproduce

Run npx asc -O tmp/bla.ts with the enclosed file (zipped because github doesn't support uploading .ts files) :
bla.zip

AssemblyScript version

v0.27.22

@sebpiq sebpiq added the bug label Jan 11, 2024
@sebpiq sebpiq changed the title Compiler failing at optimization step Compiler failing at optimization step with a big .ts file Jan 11, 2024
@CountBleck
Copy link
Member

Here is the complete error it seems:

Cannot enlarge memory, requested 2147504128 bytes, but the limit is 2147483648 bytes!
Cannot enlarge memory, requested 2147508224 bytes, but the limit is 2147483648 bytes!
Aborted(native code called abort())

▌ Whoops, the AssemblyScript compiler has crashed during optimize :-(
▌
▌ Here is the stack trace hinting at the problem, perhaps it's useful?
▌
▌ RuntimeError: Aborted(native code called abort())
▌     at abort (file:///Users/blumiere/Projects/assemblyscript/node_modules/binaryen/index.js:422:15)
▌     at _abort (file:///Users/blumiere/Projects/assemblyscript/node_modules/binaryen/index.js:3478:7)
▌     at binaryen_wasm.wasm.MixedArena::allocSpace(unsigned long, unsigned long) (wasm://wasm/binaryen_wasm.wasm-021889a6:wasm-function[813]:0x71d58)
▌     at invoke_iiii (file:///Users/blumiere/Projects/assemblyscript/node_modules/binaryen/index.js:5722:40)
▌     at binaryen_wasm.wasm.wasm::optimizeBlock(wasm::Block*, wasm::Module*, wasm::PassOptions&, wasm::BranchUtils::BranchSeekerCache&) (wasm://wasm/binaryen_wasm.wasm-021889a6:wasm-function[5986]:0x309bda)
▌     at binaryen_wasm.wasm.wasm::Walker<wasm::MergeBlocks, wasm::UnifiedExpressionVisitor<wasm::MergeBlocks, void>>::doVisitBlock(wasm::MergeBlocks*, wasm::Expression**) (wasm://wasm/binaryen_wasm.wasm-021889a6:wasm-function[5895]:0x30659f)
▌     at binaryen_wasm.wasm.wasm::Walker<wasm::MergeBlocks, wasm::UnifiedExpressionVisitor<wasm::MergeBlocks, void>>::walk(wasm::Expression*&) (wasm://wasm/binaryen_wasm.wasm-021889a6:wasm-function[5891]:0x305335)
▌     at binaryen_wasm.wasm.wasm::WalkerPass<wasm::PostWalker<wasm::MergeBlocks, wasm::UnifiedExpressionVisitor<wasm::MergeBlocks, void>>>::runOnFunction(wasm::Module*, wasm::Function*) (wasm://wasm/binaryen_wasm.wasm-021889a6:wasm-function[5893]:0x306521)
▌     at invoke_viii (file:///Users/blumiere/Projects/assemblyscript/node_modules/binaryen/index.js:5733:33)
▌     at binaryen_wasm.wasm.wasm::PassRunner::runPassOnFunction(wasm::Pass*, wasm::Function*) (wasm://wasm/binaryen_wasm.wasm-021889a6:wasm-function[2178]:0x1723f2)
▌
▌ If you see where the error is, feel free to send us a pull request. If not,
▌ please let us know: https://github.com/AssemblyScript/assemblyscript/issues
▌
▌ Thank you!

@CountBleck
Copy link
Member

CountBleck commented Jan 12, 2024

It seems to work if binaryen.js is built with a maximum of 4 GiB of memory (which is the hard maximum according to wasm-ld, likely because it's Wasm32).

Here's the gzipped bla.wasm by the way.

The time needed on my 15" mid-2014 MacBook Pro: node ../assemblyscript/bin/asc -O bla.ts -o bla.wasm 484.24s user 1.51s system 102% cpu 7:55.49 total

@sebpiq
Copy link
Author

sebpiq commented Jan 12, 2024

Thanks so much @CountBleck ! I have a couple of questions then :

  • when you say "built with a maximum of 4 GiB of memory" you mean the amount of memory that's available to binaryen for optimization ?
  • how would you go about detecting that error and reporting it in a more user friendly manner ?

@CountBleck
Copy link
Member

Yes, I mean the amount of memory available to Binaryen.

To detect that error, perhaps one could check the size of HEAP8 and compare it to the maximum.

Also, it might be good to bump the maximum memory in binaryen.js to 4 GiB regardless.

@sebpiq
Copy link
Author

sebpiq commented Jan 12, 2024

Also, it might be good to bump the maximum memory in binaryen.js to 4 GiB regardless.

How do you do that ? I see only a few options are forwarded to binaryen from asc

To detect that error, perhaps one could check the size of HEAP8 and compare it to the maximum.

Same ... any pointers on how I can access these ?

@MaxGraey
Copy link
Member

Binaryen should use add_link_flag("-sMAXIMUM_MEMORY=4gb") for that. Just need to add this flag here:
https://github.com/WebAssembly/binaryen/blob/main/CMakeLists.txt#L337

@CountBleck
Copy link
Member

Yeah, that's basically the flag I used.

How do you do that ? I see only a few options are forwarded to binaryen from asc

That's done when binaryen.js is built, using the flag Max said.

any pointers on how I can access these ?

That would likely be done within asc and not in your code.

@MaxGraey
Copy link
Member

Also, we could try to update binaryen.js to latest. Perhaps there are some fixes, improvements and less memory leakage or consumption.

@CountBleck
Copy link
Member

Also, we could try to update binaryen.js to latest. Perhaps there are some fixes, improvements and less memory leakage or consumption.

I have zero experience with updating AS's binaryen.js version, which is why I haven't just stuck the flag in the action yet.

Also, I tested AS with the latest version of binaryen.js, and it doesn't work with the default 2 GiB.

CountBleck added a commit to CountBleck/binaryen.js that referenced this issue Jan 13, 2024
This change is part of fixing AssemblyScript/assemblyscript#2810.
The stack size flag has been removed, as Binaryen has updated their
defaults accordingly.
CountBleck added a commit to AssemblyScript/binaryen.js that referenced this issue Jan 13, 2024
This change is part of fixing AssemblyScript/assemblyscript#2810.
The stack size flag has been removed, as Binaryen has updated their
defaults accordingly.
CountBleck added a commit to CountBleck/assemblyscript that referenced this issue Jan 14, 2024
This version of binaryen.js has the requisite 4 GiB max memory change.
Fixes AssemblyScript#2810.
CountBleck added a commit to CountBleck/assemblyscript that referenced this issue Jan 14, 2024
This version of binaryen.js has the requisite 4 GiB max memory change.
Fixes AssemblyScript#2810.
@CountBleck
Copy link
Member

@sebpiq Sadly, I'm not sure if there's a reliable way to detect OOM. I might look into it further, but the next version of AS (released in a day or two) should allow your code to be built.

CountBleck added a commit to CountBleck/assemblyscript that referenced this issue Jan 14, 2024
This version of binaryen.js has the requisite 4 GiB max memory change.
Fixes AssemblyScript#2810.
@sebpiq
Copy link
Author

sebpiq commented Jan 19, 2024

Thanks a lot !
Sorry I haven't responded before. I'll try with the newest version and see what I get.
Thanks also for all the great work with assemblyscript !

@MaxGraey
Copy link
Member

It may also be memory leak in binaryen: WebAssembly/binaryen#6239

@sebpiq
Copy link
Author

sebpiq commented Feb 10, 2024

Sorry for coming back so late to you ! I have upgraded to latest AssemblyScript version and indeed it works now ;)

Thanks so much for the help !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants