Lots of upgrade tests have been broken for 2 months #31148
Closed
Description
A couple dozen upgrade and skew e2e test suites have been broken for exactly the same amount of time (1 month, 23 days). I looked at 2, and they were both broken due to:
/workspace/sh2ju.sh: line 125: bc: command not found
This problem was introduced by #27825 in June, and apparently wasn't entirely fixed by #30556.
We lost 2 months of upgrade test coverage due to this. What can we do to notice such failures earlier?
Visible here, if you scroll all the way to the right:
https://k8s-testgrid.appspot.com/google-upgrade#gke-1.3-1.4-kubectl-skew&width=3
Also, we have repeated problems with the lack of portability of shell scripts. If we're not going to ban them, can we make them more hermetic?