Now we have a different problem. Someone or something deleted all of the Jenkins GCE VMs in that project.

Sorry, I'm retarded. I did what i always do when there's a leak, free resources. Thought i was deleting disposable vms but i clearly messed that up.

well, I will take this opportunity to fully clean up the project.

The node e2e tests occasionally leak VMs. We need to improve our infrastructure to garbage-collect these.

Things should be running now, closing. More than 60% of the ips are unused.

So it looks like we go from 0-100, real quick. We're back at 90 something % usage of ips.
I think we're just naturally saturating and need a quota bump, and till that gets approved some runs might flake. Leaving this open in the interim so people don't keep opening issues for the same thing.

Cleaned up leaked node e2e VMs:

gcloud compute instances list --project=kubernetes-jenkins-pull --filter="NOT tags.items:do-not-delete AND creationTimestamp<'$(date +%Y-%m-%dT%H:%M:%S%z --date='1 hour ago')' AND NOT name~'.*jenkins.*'" --format='value(name)' | xargs gcloud compute instances delete --project=kubernetes-jenkins-pull --zone=us-central1-f

Hmm, looks like we're maxing out 150 ips now (#25171 (comment))

In-use IP addresses us-central1 
99%
148 of 150

Once bitten twice shy, @ixdy kick off your cleanup script? (or gc in a cron job till we fix the leak?)

Yeah, cleaning up now. I'll make it part of our daily cleanup, but we need to figure out why so many of the node e2e tests are getting stuck, which is the root cause. @pwittrock

/cc @kubernetes/sig-testing

Root cause (leaking VMs) fixed. Should be in the clear now.

@ixdy it occurred to me that an alert on quota > 75% indicates imminent flake. I'd subscribe to that if we had it.

I'm working on setting up monitoring and alerting of our GCP quotas. It's unnecessarily difficult to integrate, unfortunately.