So this is also seen in the apiserver:

It hopefully shouldn't be endpoint controller. Note that we don't start the second test before all namespaces from the previous one are deleted. That means endpoint controller wouldn't be able to update endpoints object becuase of non-existing namespace.
I hope that we would observe the higher number of errors somewhere if that's the case.

I looked into the apiserver logs, and can't find any calls that have 'load' mentioned in them after the first call which has 'density' mentioned in it. This could potentially mean that the memory usage is coming from watches (though not of those created by the e2e test, as they IIUC are closed after the load test finishes). So maybe watches from kube-proxies or kubelets?

I have one idea to check if it is sth around services. Let's disable them in our load test for the job and see.

It hopefully shouldn't be endpoint controller. Note that we don't start the second test before all namespaces from the previous one are deleted. That means endpoint controller wouldn't be able to update endpoints object becuase of non-existing namespace.

That's true. But IIUC it's still possible that endpoints-controller is using memory, for e.g to process watch events for endpoints updates (coming from load test deletion phase)?

I don't think it's watch related.
Yes - CM may accumulate some memory, because it allocated a lot in the past. And without memory-pressure not everything might have been cleared yet.

I don't think we should spent too much time on it now.

Makes sense.

I don't think we should spent too much time on it now.

Sure... But this is just taking very small time while I'm running bisection for the pod-startup regression in the background :)

@shyamjvs, @wojtek-t does this look more like a problem with the test, or does this look like a real user-affecting performance problem related to the other performance issues?

This needs more digging, but AFAIU it's not a regression and we've been seeing it for a while already I think.
Wrt users this may not be much of a problem, but I'd want to hear out @wojtek-t on this.

related to the other performance issues?

By "other performance issues" if you mean the recent ones that I've been hunting down (#60500, #60589), then I don't think so. Not 100% sure - but I believe they're different.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close