Apple TV not waking from sleep / won't turn TV on after extended period of time.

I'm trying my hardest to only use the Apple TV on my TV, but it is just so buggy that I'm about to give up.

Latest Apple TV 4K

Samsung TV

CEC is enabled on the tv. Allow to control my tv is enabled on Apple TV.

If my Apple TV is on, then I can hold the power button and the Apple TV, as well as the Samsung TV will turn off. If I then try to turn it back on shortly after, then they both turn back on.

If I leave it for an extended period of time (maybe half hour +), then nothing happens when I hit the power button on my Apple TV remote. The TV does not turn on, and neither does the Apple TV.

It isn't really better when I use the Samsung TV Remote to control the Apple TV. The biggest problem is getting it to wake from sleep. Whenever I change it to the Apple TV source, the Apple TV does not wake up. Even if I hit the "power on" button that sometimes appears on the screen from Samsung. Half the time that button isn't there. I'm forced to get my phone and power it on from control centre remote control.

I'd like to just use my Apple TV as the one remote