pod.go / WaitUntilPodRunningBySelector() times out

[lubars]

I have seen this method time out in two ways:

1. The method takes a count parameter for expected number of pods; if this changes while the call is in progress (e.g. due to HPA), the number of pods will never match
2. If the StatefulSet has been deleted, the call to c.CoreV1().Pods().List() will never succeed

This prevents a StatefulSet with the same name in the same namespace from being redeployed until the kutil.ReadinessTimeout has expired.

I resolved this locally by a) Replacing the count with a function to retrieve the current count b) Verifying the StatefulSet still exists and exiting the call if it doesn't.