We noticed that we were seeing large amounts of 504 responses when using the cloudflare_zone_requests_status_country_host metric particularly for heavily cached hosts. It turns out the Early Hints optimisation offered by Cloudflare is the ultimate cause for this inflated result. It also explains the incorrect 200 responses we were seeing.

Looking through Cloudflare's documentation they acknowledge this behaviour with respect to analytics:

You may see an influx of 504 responses with the RequestSource of earlyHintsCache in Cloudflare Logs when Early Hints is enabled, which is expected and benign. Requests from earlyHintsCache are internal subrequests for cached Early Hints, and they are neither end user requests, nor do they go to your origin. Their response status only indicates whether there are cached Early Hints for the request URI (200 on cache hit, 504 on cache miss). These requests are already filtered out in other views, such as Cache Analytics.

The suggested fix is to filter these requests out as included in this PR. With this in place we're seeing metric counts that match what we see in the Cloudflare dashboard.