Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UndispatchedCoroutine in withContext creates unreasonable CPU pressure due to GC nepotism #3592

Closed
qwwdfsad opened this issue Jan 19, 2023 · 0 comments
Assignees
Labels

Comments

@qwwdfsad
Copy link
Contributor

This is a condensed description of the issue Space is suffering, so some details (e.g. the actual CPU impact, CPU snapshots, etc.) are omitted.

For coroutines, in order to keep track of ThreadContextElement, each undispatched (the one that doesn't change current coroutine dispatcher) withContext, a special thread-local variable is spawned in order to preserve and restore execution context in the corresponding thread.

It leads to multiple problems:

  • If the coroutine does not use thread-local elements, the corresponding system ThreadLocal is instantiated and queried, while being later reset to null
  • If system-wide ThreadContextElement is used in the application, then application spawns as much thread locals as there are calls to `withContext.

The rest is ThreadLocal implementation peculiarities:

  • ThreadLocal is actually stored in Thread's local map with open addressing by weak reference
  • ThreadLocal map entry is allocated on the first access (e.g. get()) even if it's null or wasn't set and has no initializer
  • If ThreadLocal is unreachable, it's collected, but its weak-reference slot is not
  • When thread-local map has reached its fill factor or encountered "stale" (GC-collected) entry, it attempts to clean it up with potential rehash
  • If coroutine lives long enough, it is promoted to old generation along with its ThreadLocal object
    • Because of that, entry can only be collected after an old generation GC which is rare enough event
    • Until then, any thread-local op that encounters rehash/stale entry will burn quite a lot of CPU processing 4k+ open-addressing hashmap
@qwwdfsad qwwdfsad added the bug label Jan 19, 2023
@qwwdfsad qwwdfsad self-assigned this Jan 19, 2023
qwwdfsad added a commit that referenced this issue Jan 19, 2023
…outine

* It addresses the problem with ThreadLocalMap.entries that may outlive the coroutine lifecycle and interfere with CPU consumption of other thread-locals on the same thread
* No test provided as this is a non-functioanl change. The only reasonable way to check it is to reflectively walk over Thread class which is prohibited by Java since 11+. The only way is to eyeball Thread.currentThread().threadLocals size in debugger in the properly crafted unit test

Fixes #3592
qwwdfsad added a commit that referenced this issue Jan 19, 2023
…outine

* It addresses the problem with ThreadLocalMap.entries that may outlive the coroutine lifecycle and interfere with CPU consumption of other thread-locals on the same thread
* No test provided as this is a non-functioanl change. The only reasonable way to check it is to reflectively walk over Thread class which is prohibited by Java since 11+. The only way is to eyeball Thread.currentThread().threadLocals size in debugger in the properly crafted unit test

Fixes #3592
qwwdfsad added a commit that referenced this issue Jan 23, 2023
…outine

* It addresses the problem with ThreadLocalMap.entries that may outlive the coroutine lifecycle and interfere with CPU consumption of other thread-locals on the same thread
* No test provided as this is a non-functioanl change. The only reasonable way to check it is to reflectively walk over Thread class which is prohibited by Java since 11+. The only way is to eyeball Thread.currentThread().threadLocals size in debugger in the properly crafted unit test

Fixes #3592
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant