Unit test flake Test_NewGoRoutineMap_Positive_WaitEmpty

```
=== RUN   Test_NewGoRoutineMap_Positive_WaitEmpty
--- FAIL: Test_NewGoRoutineMap_Positive_WaitEmpty (0.14s)
    goroutinemap_test.go:216: Error waiting for GoRoutineMap.Wait: timeout after 50ms
=== RUN   Test_NewGoRoutineMap_Positive_Wait
--- PASS: Test_NewGoRoutineMap_Positive_Wait (0.16s)
FAIL
FAIL    k8s.io/kubernetes/pkg/util/goroutinemap 0.676s
```
