Can one of the admins verify this patch?

this is ok to test

LGTM, fine to merge once performance tests are green.

One observation I'd like to note is that it looks like the shell script will get out of hand fairly quickly and we might need to switch to something a bit more manageable in the future (e.g. some python script to drive this)

It passes, but 900sec is 15min which could be a while for later tests. For instance, if we look at the pattern of build times over time, with the following script:

$ cat perf-tests.sh
#!/bin/bash
testtimes=" "
for ((t=500;t<=700;t++));
do
  testtime=$(curl --silent https://grpc-testing.appspot.com/job/gRPC_performance_pull_requests/$t/ | grep Took | cut -f2 -d"<" | cut -f2 -d">")
  if [ $(expr "$testtime" : '.*') -gt 1 ]; then
    echo "BUILD "$t" -> "$testtime
  fi
done
$
$ ./perf-tests.sh
BUILD 500 -> 2 min 36 sec
BUILD 501 -> 2 min 23 sec
BUILD 502 -> 2 min 36 sec
...

If I look at the average, it's 2 min 22 sec. If I plot the trend, it seems to expand quite dramatically lately by almost 4x than the average.

It might be good to keep this in mind, and maybe optimize around it so that it doesn't become a bottleneck later on.

~p

I guess that's what Jan was suggesting too :)