Hi.
I'm an acting Senior Lead Developer from Helsinki/Finland, with more than 30 years of experience in software development, doing sidekicks alongside my day job.
I have extensive knowledge and experience in embedding Linux on both custom and stock hardware. Last year I embedded Linux for a US gaming company for their arcade gaming cabinets with quite similar requirements, that was not Raspberry hardware though. On Raspberry I have developed embedded Linux and software for a company making medicinal dispensers for elderly in home care, and I actually use a Raspi 4 as my desktop =)
When I founded a company doing IoT systems we developed and manufactured own ARM based boards, and I was responsible for the bootloader and embedding Linux on it, so I'm very familiar also with building, configuring and also coding the kernel, as we had some unsopported hardware on the board, NAND and FRAM chips for example, that needed drivers to be written.
Buildroot is a good way to get minimal, but maybe after getting to know the whys behind the requirements and making some trials also Linux From Scratch or using debootstrap might be options that produce a bit more maintanable results. But if small is the main goal, even if it costs in development and maintenance, then buildroot for sure =)