/sig scalability
/assign

We can rule out fluentd-gcp upgrade, the version that we use (v.3.2.0) hasn't changed since 1.13

/assign @x13n

@x13n, could you take a look and help us debug?

Since the version didn't change, my guess is that some component(s) started generating more logs. Did CPU usage increase as well?

I believe it slightly increased, but it's not as visible as the memory increase, see http://perf-dash.k8s.io/#/?jobname=gce-100Nodes&metriccategoryname=E2E&metricname=DensityResources&PodName=fluentd-gcp%2Ffluentd-gcp&Resource=CPU

Interesting. Can we rule out backend latency increase somehow? If the mean latency increased, the buffers utilization would follow. Unfortunately, fluentd doesn't expose request latency metrics today, only success/failures: https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud/blob/master/lib/fluent/plugin/out_google_cloud.rb

/cc @qingling128 @bmoyles0117

Is it possible to perform some sort of bisecting to narrow the scope of search ?

Thanks

Not sure if it's related, but we observed some memory consumption difference between k8s versions < 1.10 and versions >=1.10: fluent/fluentd#2236 (comment). Originally we thought it's triggered by log rotation change in k8s. But it's only a hypothesis that has not been proved.

Bisection would be possible, but may be hard due to high variance of the memory consumption (see the first graph). Also we don't really have manpower to do it, we're currently facing few other regressions that have priority over this one.

Any other ideas how to debug this? Is fluentd-gcp exposing any prometheus metrics that might be useful to debug this? Can we compare some fluentd-gcp logs from a good run and bad run to see if there is anything that stands out?

I'm not sure eyeballing fluentd logs would expose memory problems.

fluentd_status_buffer_queue_length/fluentd_status_buffer_total_bytes metrics might be worth looking at, to confirm that the memory increase is indeed coming from the buffers.

I was wrong saying that the version of fluentd-gcp hasn't changed.
It was bumped up from 1.6.0 to 1.6.8 in #77224.
The timing correlates with the increase we see in the perf-dash, so it's the culprit most likely.

Sorry, this got deprioritized. The tests still flakes due to that, I think at this point the only reasonable action is to assume that the increase was caused by fluentd version upgrade and increase the constraints to add some headroom and deflake the test.

Will send a PR shortly.

Ref. #80212