GKE deployment: implement scale down and timeout for fixing hanging integration test

[balopat]

Rarely, but GKE deployments are not happening when kubectl -f apply is executed. In this case the current gke_integration test just keeps waiting on the new version of the test app to be deployed, while the old version of the app is still running on the cluster.

1.) Before deployment we should scale down the old deployment to 0 instances, and only then run kubectl apply.
2.) A timeout policy should be implemented, in case the new deployment token has not appeared in 2 minutes, scale down and kubectl apply should be retried.

[cassand]

As GKE can run multiple applications at the same time, it might be preferable to just isolate the different applications (with a unique label), kubectl can fetch the state of an application by its label.

To ensure that the new application is created we can use kubectl run or create instead of apply.
An example is present in the Jetty Repository.

If we intend to run multiple applications we have to be careful about the cleanup phase (as the destruction of the cluster would lead the other tests to fail), this is currently the case in Jetty (the cluster tear-down after each test) .

[balopat]

There is no clashing of multiple apps yet: there is a label app: openjdk-spring-boot-app already.
You're right, we could use kubectl run and create, I guess we didn't use them because they recreate the Service as well, which takes more time - it's better slow then not working :) How much time do the Jetty deploys take?
But your pointing at another important aspect: we should leverage the work done in Jetty and get to a similar integration test model across our runtimes!