[Feature] Add ability to modify the image of a worker group in RayCluster with rolling upgrade or restart

[pkit]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

When updating image in RayCluster definition of a worker group: perform worker group restart or rolling update.

Use case

When updating docker image of a worker group currently the group will continue to run with the old image until cluster is restarted or re-created.
Right now I just re-create the cluster after each image version update.

Related issues

#2534

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[pkit]

We also experimented with scale to zero and back. But it may lead to version discrepancies between various worker groups. So it's safer to just kill everything in our case.

[andrewsykim]

@pkit what's your expectation for the head pod? Do you ever expect KubeRay to recreate it if you update the image even if it means losing the GCS state?

[pkit]

I don't care much about head pod. I manage all actors from the application anyway.
So if it restarts it's ok.
Obviously it would be nice if it would re-create the actors and be fault tolerant, but it's just "nice to have".

[Future-Outlier]

can you try:

1. raycluster CR's worker suspend -> true
2. update image
3. raycluster CR's worker suspend -> false

[Future-Outlier]

add a doc

[kevin85421]

@pkit: @win5923 is working on this #4185.

[pkit]

@kevin85421 looks good!