Adding CPU core pinning and parking capability #416

HenrikHolst · 2023-05-03T20:39:22Z

First attempt at merging this patch set that adds cpu core pinning and parking capability. There is automatic logic that will try to detect if running on a CPU with non-uniform L3 cache (say 7900x3D or 7950x3D) or on a CPU with non-uniform frequencies (say Intel Alder Lake, Raptor Lake and so on with E-cores and P-cores).

By default the code will pin all the threads of the game on the cores with the better L3 or frequency but can also be configured to completely park the less desired cores (akin to how the xbox gamebar works in Windows on the 7900x3D and 7950x3D). Parking requires the aid of an new helper utility, cpucorectl, under polkit since that requires root privileges.

If logic fails or is inadequate then it is also possible to manually set which cores to pin to or which cores to park in the config file. There is also a built in safety check to make sure that the user/logic does not disable all cores by making sure that at least 4 cores have to be available for the game.

Added common files for the cpu core parking/pinning functionality

added build info for common-cpu to meson

added the cpu core parking/pinning settings to gamemode-config.c

added cpu core parking/pinning settings to gamemode-config.h

call the cpu core parking/pinning from gamemode-context.c

Added gamemode-cpu.c which contains the functions for cpu core parking and pinning

added the cpu core parking/pinning definitions to gamemode.h

Added build info for gamemade-cpu.c to meson

added the polkit policy for the cpucorectl utility so we can call it without being root

added polkit rules so we can run the cpucorectl utility without being root

added some info about the new cpu core parking and pinning settings in gamemode.ini

Added a utility to enable and disable cpu cores (aka core parking) since this requires root privileges

added build info for the new cpucorectl utility to meson

Added detection for big.LITTLE aka cpu:s where not all cores have the same frequency like on Intel Alder Lake and newer. The current logic allows a 5% difference in the max frequency due to some reports that those cpu:s doesn't always give back the exact same value (possible due to boosting capability).

use defines instead of values for the park_or_pin variable

use a define instead of values for the park_or_pin variable

if cpu core parking/pinning was disabled by the logic then there would be a double free at exit

cfebs · 2023-05-26T20:17:39Z

@HenrikHolst is there a good way to verify that the core pinning is working correctly while a game is running?
Want to give the branch a go!

HenrikHolst · 2023-05-26T21:05:38Z

@cfebs use ps or top to find the pid of the game once it is running and then you can use "taskset -ap pid-number" to see the cpu mask for the process and all it's threads, depending on the amount of cores your cpu have the default mask would be something like "ffff" if the process was allowed to run on all cores and something like "ff00" or "00ff" if it was masked out of 8 of the cores (each f here represents 4 cores)

cfebs · 2023-05-26T22:07:18Z

Thanks for the walk through! Here are my results with a 7950X3D and pin_cores=yes set in ~/.config/gamemode.ini

❯ pgrep supertuxkart
205593
❯ gamemoded --status=205593
gamemode is active and [205593] registered

Here's the result of taskset -ap 205593 https://gist.github.com/cfebs/d78f9fdafbaeafbc7eb750c3c5b789de so that would mean 48/61 threads are masked?

Was also just using watch -d on taskset and it was not changing over time FWIW.

If you would like any help testing other patches LMK!

HenrikHolst · 2023-05-26T22:57:15Z

Thanks for the walk through! Here are my results with a 7950X3D and pin_cores=yes set in ~/.config/gamemode.ini
❯ pgrep supertuxkart
205593
❯ gamemoded --status=205593
gamemode is active and [205593] registered
Here's the result of taskset -ap 205593 https://gist.github.com/cfebs/d78f9fdafbaeafbc7eb750c3c5b789de so that would mean 48/61 threads are masked?

Was also just using watch -d on taskset and it was not changing over time FWIW.

If you would like any help testing other patches LMK!

that looks correct yes, most likely the game launched new threads some time after launch so looks like we would have to store the pids that we watch and apply the mask on a regular basis to catch cases like this.

cfebs · 2023-05-26T23:46:24Z

Did another test with a steam game (if just to provide more info for other testers!). Also tried to debug which procs gamemode considers registered:

❯ ps haxo pid | sed 's/\s\+//g' | xargs -I{} gamemoded --status={} \; 2>&1 | grep '] registered'
gamemode is active and [244593] registered
gamemode is active and [244615] registered

The main proc that was taking most cpu time was different though: 244861 which is probably just a child of the main steam launcher procs.

Here's the taskset results:

pid 244593's current affinity mask: ff00ff
pid 244615's current affinity mask: ff00ff
244861 indeed looks like the actual game. Most are pinned but looks like same behavior of some launching later.

And here's my ps + cli args for those procs: https://gist.github.com/cfebs/50e8c6b4198cce20db713bbfad451734

looks like we would have to store the pids that we watch and apply the mask on a regular basis to catch cases like this

Gotcha, I'll keep using this build for as long as it's stable - thank you for the branch and the quick responses.

made core pinning optionally silent, used for when the reaper thread calls us repeatable so we don't create tons of unnecessary logs

Reapply the core pinning from the reaper thread to catch cases where the game launches threads after initial start

walk through /proc/pid/task to make sure that we set the thread affinity for every single thread in the process

HenrikHolst · 2023-05-27T02:44:14Z

Did another test with a steam game (if not to provide more info for other testers!). Also tried to debug which procs gamemode considers registered:
❯ ps haxo pid | sed 's/\s\+//g' | xargs -I{} gamemoded --status={} \; 2>&1 | grep '] registered'
gamemode is active and [244593] registered
gamemode is active and [244615] registered
The main proc that was taking most cpu time was different though: 244861 which is probably just a child of the main steam launcher procs.

Here's the taskset results:
* `pid 244593's current affinity mask: ff00ff`

* `pid 244615's current affinity mask: ff00ff`

* [244861](https://gist.github.com/cfebs/11510e9bc3459e6e5f690eb90656bfef) indeed looks like the actual game. Most are pinned but looks like same behavior of some launching later.
And here's my ps + cli args for those procs: https://gist.github.com/cfebs/50e8c6b4198cce20db713bbfad451734

looks like we would have to store the pids that we watch and apply the mask on a regular basis to catch cases like this

Gotcha, I'll keep using this build for as long as it's stable - thank you for the branch and the quick responses.

I have added some patches now that reapplies the core pinning in the gamemode reaper thread (by default it runs once every 5 seconds) and I also make sure that we really hit every single thread of the process.

cfebs · 2023-05-27T03:30:01Z

Awesome, here's the first test with supertuxkart:

pinning_tux_1.mp4

Looks great so far!

HenrikHolst · 2023-05-27T03:31:22Z

nice!, thanks for the confirmation!

cfebs · 2023-05-27T15:22:40Z

Here are some Steam tests! Each using gamemoderun %command% launch option and then observing with watch -n0.5 'pgrep $EXE | xargs taskset -ap'. Games launch around 7-10s mark.

So both look like there are a few threads that stay default masked. In DS3 there's more mask variety :)

In both cases using gamemoded --status was getting: gamemode is active but [...] not registered

HenrikHolst · 2023-05-27T15:58:58Z

could be that gamemode for some reason couldn't find the process id or something. One thing to try is to run "gamemodelist" to see if the process is listed as being registered.

Another is to edit /etc/xdg/systemd/user/gamemoded.service and change "ExecStart=/usr/bin/gamemoded" to "ExecStart=/usr/bin/gamemoded -l" and then do "systemctl --user daemon-reload && systemctl --user stop gamemoded.service" to let the change take effect (the -l makes gamemoded log more to the journal) and then run "journalctl --user /usr/bin/gamemoded -f" in a terminal and then run the game and see what gamemoded logs.

cfebs · 2023-05-28T20:24:01Z

Thanks for the logging tip.
I just re-installed pkg manager version of gamemode (v1.7) to test the behavior with Dark Souls 3 and:

❯ gamemodelist 
    PID    PPID USER      NI PSR COMMAND
 187873    6150 $USER     0  14 reaper
 187989  187965 $USER     0  12 python3
 187991  187989 $USER     0   4 steam.exe
 187993  187965 $USER     0  22 wineserver
 187997  187965 $USER     0   6 services.exe
 188000  187965 $USER     0  16 winedevice.exe
 188010  187965 $USER     0  17 winedevice.exe
 188043  187965 $USER     0   8 plugplay.exe
 188051  187965 $USER     0   6 svchost.exe
 188058  187965 $USER     0  21 explorer.exe
 188075  187965 $USER     0   8 rpcss.exe
 188085  187965 $USER     0  16 tabtip.exe
 188119  187965 $USER     0  23 DarkSoulsIII.ex

❯ gamemoded --status=188119
gamemode is active but [188119] not registered

❯ gamemoded --status=187965
gamemode is active but [187965] not registered

So I guess that's just how gamemode works: gamemodelist can list a pid but it's not returned as "registered" in --status. Good to know :)

Hope to see this merged! And LMK if need any more testing.

HenrikHolst · 2023-05-28T22:20:45Z

Thanks for the logging tip. I just re-installed pkg manager version of gamemode (v1.7) to test the behavior with Dark Souls 3 and:

❯ gamemodelist 
    PID    PPID USER      NI PSR COMMAND
 187873    6150 $USER     0  14 reaper
 187989  187965 $USER     0  12 python3
 187991  187989 $USER     0   4 steam.exe
 187993  187965 $USER     0  22 wineserver
 187997  187965 $USER     0   6 services.exe
 188000  187965 $USER     0  16 winedevice.exe
 188010  187965 $USER     0  17 winedevice.exe
 188043  187965 $USER     0   8 plugplay.exe
 188051  187965 $USER     0   6 svchost.exe
 188058  187965 $USER     0  21 explorer.exe
 188075  187965 $USER     0   8 rpcss.exe
 188085  187965 $USER     0  16 tabtip.exe
 188119  187965 $USER     0  23 DarkSoulsIII.ex

❯ gamemoded --status=188119
gamemode is active but [188119] not registered

❯ gamemoded --status=187965
gamemode is active but [187965] not registered

So I guess that's just how gamemode works: gamemodelist can list a pid but it's not returned as "registered" in --status. Good to know :)

Hope to see this merged! And LMK if need any more testing.

thanks for the tests! Hopefully this will increase the fps for the games that do get registered.

HenrikHolst · 2023-06-24T21:42:11Z

closes #412

greg2010 · 2023-09-05T12:53:34Z

Took it out for a spin with Starfield, can confirm that the affinity is correctly set for all child processes. Also, got a noticeable bump in FPS in crowded areas.

@afayaz-feral any chance this could be merged? This is a major boon for asymmetric CPU owners.

cfebs · 2023-12-04T15:14:04Z

🎉

The original PR #416 failed the format check, but this wasn't apparent until after merging.

HenrikHolst · 2023-12-09T17:36:49Z

7900x3d user here. noticed at least 3 to 12 more fps in Akila city of Starfield! (I was stuck at 60 there but now reaching above 70!) some time ago I disabled one entire CCD (one without 3d v-cache) through BIOS and tested the same scene in Akila city and had 0 fps gain yet gamemode can give more fps? how is this possible? thank you very much.

hard to say but could be that since the rest of the system, except just the game, can use the other CCD cores when the game is pinned instead of the entire CCD being disabled the game gets more cpu than it would otherwise.

HenrikHolst · 2023-12-09T19:11:17Z

BTW taskset -ap is always showing all fs for me and also mangohud is showing all cores are active when I use park_cores=yes in config file...

prob better to open a issue for that than to continue here on the MR but check the gamemode log, in order to park the cores an external utility have to be used which means that polkit is also used and if you are on say ubuntu < 23.04 or some version of debian then the included polkit rules are incomatible and have to be manually remade to work

HenrikHolst added 27 commits May 3, 2023 07:22

Added common-cpu.c and common-cpu.h

07ce663

Added common files for the cpu core parking/pinning functionality

updated meson.build to build common-cpu

c0053a2

added build info for common-cpu to meson

added the cpu core parking/pinning settings

4ecfd06

added the cpu core parking/pinning settings to gamemode-config.c

added cpu core parking/pinning settings

1d2ebdc

added cpu core parking/pinning settings to gamemode-config.h

call the cpu core parking/pinning from context

4a2816f

call the cpu core parking/pinning from gamemode-context.c

Added gamemode-cpu.c

5a4c76e

Added gamemode-cpu.c which contains the functions for cpu core parking and pinning

added the cpu core parking/pinning definitions

a03d179

added the cpu core parking/pinning definitions to gamemode.h

build gamemode-cpu

afd571e

Added build info for gamemade-cpu.c to meson

added polkit policy for cpucorectl

5baf489

added the polkit policy for the cpucorectl utility so we can call it without being root

added polkit rules for cpucorectl

944b720

added polkit rules so we can run the cpucorectl utility without being root

added cpu core parking/pinning settings to gamemode.ini

515d51f

added some info about the new cpu core parking and pinning settings in gamemode.ini

added cpucorectl.c

04d2ae7

Added a utility to enable and disable cpu cores (aka core parking) since this requires root privileges

added build info for cpucorectl

96cae1f

added build info for the new cpucorectl utility to meson

added info about suport for Intel E- and P-cores

2f8f81d

fixed to match clang-format

3cb94c5

fix to match clang-format

be3ac69

fix to match clang-format

58234d7

fix for clang-format

f54b230

fixed more coding style errors

443b1a4

fixed more coding style errors

20758fd

fixed more coding style errors

ec8a2e4

fixed more coding style errors

b53b170

use defines instead of value

9261c68

use defines instead of values for the park_or_pin variable

use define instead of value

16cbc83

use a define instead of values for the park_or_pin variable

fixed double free on exit from non use

c7ad09a

if cpu core parking/pinning was disabled by the logic then there would be a double free at exit

added info about the parking/pinning capability

c560232

HenrikHolst mentioned this pull request May 4, 2023

Automatic CPU Pinning for AMD 3D V-cache CPUs #412

Closed

HenrikHolst added 2 commits May 7, 2023 15:20

properly support config reloading

1d51bef

properly support config reloading

55a5aa5

properly support config reloading

6c1bd8a

HenrikHolst added 3 commits May 27, 2023 04:39

made pinning optionally silent

5937aba

made core pinning optionally silent, used for when the reaper thread calls us repeatable so we don't create tons of unnecessary logs

reapply the core pinning from the reaper thread

21d27aa

Reapply the core pinning from the reaper thread to catch cases where the game launches threads after initial start

pin every thread of the process

fa00e3f

walk through /proc/pid/task to make sure that we set the thread affinity for every single thread in the process

Merge branch 'FeralInteractive:master' into master

45b06a8

afayaz-feral mentioned this pull request Sep 11, 2023

New release #429

Closed

afayaz-feral merged commit 9614cec into FeralInteractive:master Dec 4, 2023

afayaz-feral added a commit that referenced this pull request Dec 4, 2023

Apply clang-format to CPU pinning and parking code

fad889d

The original PR #416 failed the format check, but this wasn't apparent until after merging.

afayaz-feral mentioned this pull request Dec 6, 2023

Alder lake / Big little optimisation #352

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding CPU core pinning and parking capability #416

Adding CPU core pinning and parking capability #416

HenrikHolst commented May 3, 2023

cfebs commented May 26, 2023

HenrikHolst commented May 26, 2023 •

edited

Loading

cfebs commented May 26, 2023

HenrikHolst commented May 26, 2023

cfebs commented May 26, 2023 •

edited

Loading

HenrikHolst commented May 27, 2023

cfebs commented May 27, 2023

HenrikHolst commented May 27, 2023

cfebs commented May 27, 2023

HenrikHolst commented May 27, 2023

cfebs commented May 28, 2023 •

edited

Loading

HenrikHolst commented May 28, 2023

HenrikHolst commented Jun 24, 2023

greg2010 commented Sep 5, 2023

cfebs commented Dec 4, 2023

HenrikHolst commented Dec 9, 2023

HenrikHolst commented Dec 9, 2023

Adding CPU core pinning and parking capability #416

Adding CPU core pinning and parking capability #416

Conversation

HenrikHolst commented May 3, 2023

cfebs commented May 26, 2023

HenrikHolst commented May 26, 2023 • edited Loading

cfebs commented May 26, 2023

HenrikHolst commented May 26, 2023

cfebs commented May 26, 2023 • edited Loading

HenrikHolst commented May 27, 2023

cfebs commented May 27, 2023

HenrikHolst commented May 27, 2023

cfebs commented May 27, 2023

HenrikHolst commented May 27, 2023

cfebs commented May 28, 2023 • edited Loading

HenrikHolst commented May 28, 2023

HenrikHolst commented Jun 24, 2023

greg2010 commented Sep 5, 2023

cfebs commented Dec 4, 2023

HenrikHolst commented Dec 9, 2023

HenrikHolst commented Dec 9, 2023

HenrikHolst commented May 26, 2023 •

edited

Loading

cfebs commented May 26, 2023 •

edited

Loading

cfebs commented May 28, 2023 •

edited

Loading