Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core: implement core_mutex_priority_inheritance #17040

Merged
merged 2 commits into from
Aug 5, 2022

Conversation

maribu
Copy link
Member

@maribu maribu commented Oct 22, 2021

Contribution description

This implements core_mutex_mitigate_priority_inversion, a module that employs the textbook solution of priority inheritance to mitigate priority inversion.

In addition, the corresponding test was cleaned up and automated:

  • drop timer dependency, just use mutex to stage the priority inversion scenario
  • reduce memory consumption so that Makefile.ci can be dropped by using fmt over stdio and reducing stack sizes
  • reduce the complexity of the scenario to the bare minimum required to produce the issue

Testing procedure

make flash test -C tests/thread_priority_inversion

should build, flash and run. It should fail without core_mutex_mitigate_priority_inversion (comment out line in the test's Makefile), but pass with if it is used.

Issues/PRs references

None

@github-actions github-actions bot added Area: doc Area: Documentation Area: tests Area: tests and testing framework labels Oct 22, 2021
@maribu maribu added Type: cleanup The issue proposes a clean-up / The PR cleans-up parts of the codebase / documentation and removed Area: doc Area: Documentation labels Oct 22, 2021
@fjmolinas
Copy link
Contributor

should build, flash and run. But the test should fail, since there currently is no mechanism to boost the low priority thread to the priority of the highest waiter for the resource it has exclusive access to.

Is #7445 a solution??

@fjmolinas
Copy link
Contributor

Just saw there are many PR regarding this open, are those test cases covered by this PR? #7455 #9306 #7444? Would be nice to have something that covers all.

@maribu
Copy link
Member Author

maribu commented Oct 23, 2021

Is #7445 a solution??

Yes, only needs a rebase and adaption to the changes to core/mutex that got in since 2018.

This is alternative regarding the test case used there, but IMO this PR is preferable due to its compatibility with all boards (no Makefile.ci). It does not test for priority inversion in case of core/msg - but IMO it is sensible to separate the tests, since they cover different synchronisation mechanisms.

@fjmolinas fjmolinas added this to the Release 2022.01 milestone Nov 18, 2021
@OlegHahm
Copy link
Member

OlegHahm commented Mar 9, 2022

Where are we at here?

@maribu
Copy link
Member Author

maribu commented Mar 9, 2022

The infrastructure to change the priority of a running thread is now in place. So half way there. (Or actually even more, extending the mutex to make use of it is relatively simple.) I think I can PR this today.

@maribu maribu force-pushed the tests/thread_priority_inversion branch from ed0f703 to 33b2845 Compare March 9, 2022 20:26
@github-actions github-actions bot added Area: core Area: RIOT kernel. Handle PRs marked with this with care! Area: doc Area: Documentation Area: Kconfig Area: Kconfig integration labels Mar 9, 2022
@maribu maribu changed the title tests/thread_priority_inversion: clean up and automate core: implement core_mutex_mitigate_priority_inversion Mar 9, 2022
@maribu
Copy link
Member Author

maribu commented Mar 9, 2022

I added the corresponding code in a new commit to this PR. It kind of makes sense to me have the fix in the corresponding test application (or rather it's cleanup) in the same PR.

@maribu maribu force-pushed the tests/thread_priority_inversion branch from 33b2845 to 6009a55 Compare May 17, 2022 11:54
@maribu maribu requested a review from MrKevinWeiss as a code owner May 17, 2022 11:54
@maribu
Copy link
Member Author

maribu commented May 17, 2022

Note that decoupling priority inheritance for mutex and msg makes sense, as there is a trade-off. In a use case that needs to mitigate priority inversion for shared bus accesses is not automatically affected by priority inversion via msg.

The issue of priority inversion can occur due to hardware limitations (e.g. a low prio and a high prio task needing to talk to hardware on the single I2C bus available) and for some use cases cannot be avoided. I have to admit that I didn't spent much time thinking about it, but since msg is not linked to shared hardware resource, I believe that priority inversion could be solved by carefully designing the software so that it cannot occur.

In any case, #7445 implements priority inheritance only for msg_send_receive(), not for msg_send() or msg_receive(). And even there the caller of msg_send_receive() would have to first to potentially boost the priority of the receiver to not block on sending for longer than expected, and afterwards boost the priority of the sender to not block longer than expected on receiving. So 25 % of the places where priority inheritance would be needed are actually implemented in #7445 - and given that it is stale for quite some time, I don't expect this to be fixed swiftly.

We can still add priority inheritance for msg later on, if that is indeed needed.

@benpicco
Copy link
Contributor

C++ does not like the mutex initialization macro 😕

@maribu
Copy link
Member Author

maribu commented May 18, 2022

Ah, yes. Named initializers ({ .name = 42 }) are a C++2x feature :( Let me see if I can find an initializer that both C and C++ like.

@github-actions github-actions bot added the Area: sys Area: System label May 18, 2022
@benpicco benpicco requested review from fabian18, OlegHahm and biboc June 14, 2022 07:44
@maribu
Copy link
Member Author

maribu commented Jun 14, 2022

The use of fwrite() to fix output when modules are intermixing fmt and stdio lead to stack overflows in the test here now :/

I wonder if we should just provide a tiny stdio implementation and make that a default module when newlib is in use :/

@maribu maribu force-pushed the tests/thread_priority_inversion branch from 286f025 to 1be5cd1 Compare June 14, 2022 11:30
@maribu
Copy link
Member Author

maribu commented Jun 14, 2022

I changed the mutex_t to store the pid rather than a pointer to the thread to safe a bit of RAM. Locally, the rmutex_t now fits in the ESP32's sturct __lock. Let's see if Murdock also likes it.

At least this forced me to re-run the test and notice that I had to increase the stack sizes due to #18162 being merged.

@benpicco
Copy link
Contributor

benpicco commented Aug 1, 2022

Please squash!

@maribu maribu force-pushed the tests/thread_priority_inversion branch from 1be5cd1 to 0aafdf8 Compare August 4, 2022 13:12
@maribu
Copy link
Member Author

maribu commented Aug 5, 2022

All green and test is still passing

@benpicco benpicco enabled auto-merge August 5, 2022 08:36
@maribu maribu force-pushed the tests/thread_priority_inversion branch from 0aafdf8 to 3446752 Compare August 5, 2022 11:09
@maribu
Copy link
Member Author

maribu commented Aug 5, 2022

Uncrustify should now also hopefully be happy

Copy link
Contributor

@kfessel kfessel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I trust the test run by @maribu and @benpicco
this is a feature long asked for (as indicated by a similar PR created in 2017) - lets check that box

@maribu
Copy link
Member Author

maribu commented Aug 5, 2022

🎉

@benpicco benpicco merged commit ba33ab5 into RIOT-OS:master Aug 5, 2022
@maribu maribu deleted the tests/thread_priority_inversion branch August 5, 2022 16:50
@maribu
Copy link
Member Author

maribu commented Aug 5, 2022

thx :)

@MrKevinWeiss
Copy link
Contributor

MrKevinWeiss commented Aug 30, 2022

Hmm just adding the core_mutex_priority_inheritance seems to break expose something the esp32-wroom-32 board. I am trying to investigate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: core Area: RIOT kernel. Handle PRs marked with this with care! Area: doc Area: Documentation Area: Kconfig Area: Kconfig integration Area: sys Area: System Area: tests Area: tests and testing framework CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Process: needs >1 ACK Integration Process: This PR requires more than one ACK Type: cleanup The issue proposes a clean-up / The PR cleans-up parts of the codebase / documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants