Skip to content

HIVE-3029: Update CI jobs to test presubmits on OCP 4.20 and periodics on OCP 4.21 #72576

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
huangmingxia wants to merge 1 commit into
base: master
Choose a base branch
from
Open

HIVE-3029: Update CI jobs to test presubmits on OCP 4.20 and periodics on OCP 4.21 #72576

huangmingxia wants to merge 1 commit into from

Conversation

@huangmingxia
Copy link
Contributor

@huangmingxia huangmingxia commented Dec 15, 2025 •
edited by openshift-ci bot
Loading

Updates Hive CI job configurations per HIVE-3029:

  • Presubmits now test against OCP 4.20
  • Periodics now test against OCP 4.21

@huangmingxia
HIVE-3029: Update test versions for presubmits and periodics
978d7c1 
- Presubmits now test against OCP 4.20
- Periodics now test against OCP 4.21
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Dec 15, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Dec 15, 2025 •
edited by openshift-ci bot
Loading

@huangmingxia: This pull request references HIVE-3029 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Updates Hive CI job configurations per HIVE-3029:

  • Presubmits now test against OCP 4.20
  • Periodics now test against OCP 4.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested review from dlom and suhanime December 15, 2025 02:55
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Dec 15, 2025

[REHEARSALNOTIFIER]
@huangmingxia: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-hive-master-periodic-images openshift/hive presubmit Ci-operator config changed
pull-ci-openshift-hive-master-coverage openshift/hive presubmit Ci-operator config changed
pull-ci-openshift-hive-master-e2e openshift/hive presubmit Ci-operator config changed
pull-ci-openshift-hive-master-e2e-azure openshift/hive presubmit Ci-operator config changed
pull-ci-openshift-hive-master-e2e-gcp openshift/hive presubmit Ci-operator config changed
pull-ci-openshift-hive-master-e2e-openstack openshift/hive presubmit Ci-operator config changed
pull-ci-openshift-hive-master-e2e-pool openshift/hive presubmit Ci-operator config changed
pull-ci-openshift-hive-master-e2e-vsphere openshift/hive presubmit Ci-operator config changed
pull-ci-openshift-hive-master-images openshift/hive presubmit Ci-operator config changed
pull-ci-openshift-hive-master-security openshift/hive presubmit Ci-operator config changed
pull-ci-openshift-hive-master-unit openshift/hive presubmit Ci-operator config changed
pull-ci-openshift-hive-master-verify openshift/hive presubmit Ci-operator config changed
periodic-ci-openshift-hive-master-periodic-e2e-gcp-weekly N/A periodic Ci-operator config changed
periodic-ci-openshift-hive-master-periodic-e2e-pool-weekly N/A periodic Ci-operator config changed
periodic-ci-openshift-hive-master-periodic-e2e-weekly N/A periodic Ci-operator config changed
periodic-ci-openshift-hive-master-periodic-e2e-azure-weekly N/A periodic Ci-operator config changed

Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 15, 2025

/hold

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 15, 2025
@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 15, 2025

/cc @2uasimojo

@openshift-ci openshift-ci bot requested a review from 2uasimojo December 15, 2025 02:58
@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 15, 2025

/pj-rehearse max

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Dec 15, 2025

@huangmingxia: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 15, 2025

/retest

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 15, 2025

/test e2e-vsphere

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 15, 2025

@huangmingxia: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test app-ci-config-dry

/test boskos-config

/test boskos-config-generation

/test build01-dry

/test build02-dry

/test build03-dry

/test build04-dry

/test build06-dry

/test build07-dry

/test build08-dry

/test build09-dry

/test build10-dry

/test build11-dry

/test check-gh-automation

/test check-gh-automation-tide

/test ci-operator-config

/test ci-operator-config-metadata

/test ci-operator-registry

/test ci-secret-bootstrap-config-validation

/test ci-testgrid-allow-list

/test clusterimageset-validate

/test config

/test core-valid

/test generated-config

/test generated-dashboards

/test hosted-mgmt-dry

/test image-mirroring-config-validation

/test jira-lifecycle-config

/test labels

/test openshift-image-mirror-mappings

/test ordered-prow-config

/test owners

/test pr-reminder-config

/test prow-config

/test prow-config-filenames

/test prow-config-semantics

/test pylint

/test release-config

/test release-controller-config

/test rover-groups-config-validation

/test secret-generator-config-valid

/test services-valid

/test stackrox-stackrox-stackrox-stackrox-check

/test step-registry-metadata

/test step-registry-shellcheck

/test sync-rover-groups

/test verified-config

/test vsphere02-dry

/test yamllint

The following commands are available to trigger optional jobs:

/test build05-dry

/test check-cluster-profiles-config

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-release-check-gh-automation

pull-ci-openshift-release-master-ci-operator-config

pull-ci-openshift-release-master-ci-operator-config-metadata

pull-ci-openshift-release-master-ci-operator-registry

pull-ci-openshift-release-master-config

pull-ci-openshift-release-master-core-valid

pull-ci-openshift-release-master-generated-config

pull-ci-openshift-release-master-ordered-prow-config

pull-ci-openshift-release-master-owners

pull-ci-openshift-release-master-prow-config-filenames

pull-ci-openshift-release-master-prow-config-semantics

pull-ci-openshift-release-master-release-controller-config

pull-ci-openshift-release-openshift-image-mirror-mappings

pull-ci-openshift-release-yamllint
Details

In response to this:

/test e2e-vsphere

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 15, 2025

/retest-required

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 15, 2025

/retest

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 15, 2025

/pj-rehearse pull-ci-openshift-hive-master-e2e-vsphere

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Dec 15, 2025

@huangmingxia: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 15, 2025

/pj-rehearse periodic-ci-openshift-hive-master-periodic-e2e-azure-weekly

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Dec 15, 2025

@huangmingxia: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 15, 2025

/pj-rehearse periodic-ci-openshift-hive-master-periodic-e2e-gcp-weekly

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Dec 15, 2025

@huangmingxia: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 15, 2025

/pj-rehearse periodic-ci-openshift-hive-master-periodic-e2e-azure-weekly

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Dec 15, 2025

@huangmingxia: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@2uasimojo
Copy link
Member

2uasimojo commented Dec 15, 2025

/lgtm

...pending rehearsals.

/pj-rehearse periodic-ci-openshift-hive-master-periodic-e2e-azure-weekly

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Dec 15, 2025

@2uasimojo: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 15, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 15, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 2uasimojo, huangmingxia

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 15, 2025
@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 16, 2025

/pj-rehearse periodic-ci-openshift-hive-master-periodic-e2e-azure-weekly

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Dec 16, 2025

@huangmingxia: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 16, 2025

Submit a PR openshift/hive#2816 to fix the test job failure in ci/rehearse/periodic-ci-openshift-hive-master-periodic-e2e-azure-weekly

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 16, 2025

/pj-rehearse periodic-ci-openshift-hive-master-periodic-e2e-azure-weekly

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Dec 16, 2025

@huangmingxia: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 17, 2025

/pj-rehearse periodic-ci-openshift-hive-master-periodic-e2e-azure-weekly

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Dec 17, 2025

@huangmingxia: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 17, 2025

For the periodic-ci-openshift-hive-master-periodic-e2e-azure-weekly CI failure:
The MachineSets filter issue has been addressed, but the infra MachinePool creation is currently failing. The issue is expected to be resolved by openshift/hive#2796.

I ran the tests locally and checked the status of MachineSets, Machines, and MachinePools. Error message is as follows:

time="2025-12-17T07:49:10-05:00" level=info msg="waiting for 3 nodes with machine annotation prefix 'hive-8852e5fb-d85b-4b-5tcj6-infra-'" pool=hive-8852e5fb-d85b-4ba8-9d3f-fee4ef6a7597-infra test=TestNewMachinePool
time="2025-12-17T07:49:10-05:00" level=info msg="Waiting for Nodes" client=node
    infra_test.go:194: 
                Error Trace:    /home/cloud-user/work-log/HIVE-3029/hive/test/e2e/postinstall/machinesets/infra_test.go:194
                Error:          Received unexpected error:
                                timed out waiting for nodes
                Test:           TestNewMachinePool
                Messages:       timed out waiting for nodes to be created
--- FAIL: TestNewMachinePool (924.06s)

e % oc get machines -A                                                                              
NAMESPACE               NAME                                                  PHASE     TYPE              REGION      ZONE   AGE
openshift-machine-api   hive-8852e5fb-d85b-4b-5tcj6-infra-centralus1-jkc2d    Failed                                         63m
openshift-machine-api   hive-8852e5fb-d85b-4b-5tcj6-infra-centralus2-hdllb    Failed                                         63m
openshift-machine-api   hive-8852e5fb-d85b-4b-5tcj6-infra-centralus3-gh8vl    Failed                                         63m
openshift-machine-api   hive-8852e5fb-d85b-4b-5tcj6-master-0                  Running   Standard_D8s_v3   centralus   1      109m
openshift-machine-api   hive-8852e5fb-d85b-4b-5tcj6-master-1                  Running   Standard_D8s_v3   centralus   2      109m
openshift-machine-api   hive-8852e5fb-d85b-4b-5tcj6-master-2                  Running   Standard_D8s_v3   centralus   3      109m
openshift-machine-api   hive-8852e5fb-d85b-4b-5tcj6-worker-centralus1-zw9rq   Running   Standard_D4s_v3   centralus   1      45m
openshift-machine-api   hive-8852e5fb-d85b-4b-5tcj6-worker-centralus2-zbpqd   Running   Standard_D4s_v3   centralus   2      48m
openshift-machine-api   hive-8852e5fb-d85b-4b-5tcj6-worker-centralus3-9768f   Running   Standard_D4s_v3   centralus   3      48m

 % oc describe machines -n openshift-machine-api hive-8852e5fb-d85b-4b-5tcj6-infra-centralus1-jkc2d
...
 Error Message:           failed to reconcile machine "hive-8852e5fb-d85b-4b-5tcj6-infra-centralus1-jkc2d": failed to create vm hive-8852e5fb-d85b-4b-5tcj6-infra-centralus1-jkc2d: failed to create VM: cannot 
create vm: PUT https://management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/hive-8852e5fb-d85b-4b-5tcj6-rg/providers/Microsoft.Compute/virtualMachines/hive-8852e5fb-d85b-4b-5t
cj6-infra-centralus1-jkc2d
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: GalleryImageNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "GalleryImageNotFound",
    "message": "\"The gallery image /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/hive-8852e5fb-d85b-4b-5tcj6-rg/providers/Microsoft.Compute/galleries/gallery_hive_8852e5fb_d85b_4b_5tcj6/i
mages/hive-8852e5fb-d85b-4b-5tcj6/versions/latest is not available in centralus region. Please contact image owner to replicate to this region, or change your requested region.\"",
    "target": "imageReference"
  }
}
--------------------------------------------------------------------------------

 % oc get machinepool hive-8852e5fb-d85b-4ba8-9d3f-fee4ef6a7597-infra -o yaml
...
machineSets:
  - errorMessage: |
      failed to reconcile machine "hive-8852e5fb-d85b-4b-5tcj6-infra-centralus2-hdllb": failed to create vm hive-8852e5fb-d85b-4b-5tcj6-infra-centralus2-hdllb: failed to create VM: cannot create vm: PUT https://management.azure.com/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/hive-8852e5fb-d85b-4b-5tcj6-rg/providers/Microsoft.Compute/virtualMachines/hive-8852e5fb-d85b-4b-5tcj6-infra-centralus2-hdllb
      --------------------------------------------------------------------------------
      RESPONSE 404: 404 Not Found
      ERROR CODE: GalleryImageNotFound
      --------------------------------------------------------------------------------
      {
        "error": {
          "code": "GalleryImageNotFound",
          "message": "\"The gallery image /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/hive-8852e5fb-d85b-4b-5tcj6-rg/providers/Microsoft.Compute/galleries/gallery_hive_8852e5fb_d85b_4b_5tcj6/images/hive-8852e5fb-d85b-4b-5tcj6/versions/latest is not available in centralus region. Please contact image owner to replicate to this region, or change your requested region.\"",
          "target": "imageReference"
        }
      }
      --------------------------------------------------------------------------------
    errorReason: InvalidConfiguration
    maxReplicas: 1
    minReplicas: 1
    name: hive-8852e5fb-d85b-4b-5tcj6-infra-centralus2
    replicas: 1

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 17, 2025

@huangmingxia: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-openshift-hive-master-periodic-e2e-azure-weekly 978d7c1 link unknown /pj-rehearse periodic-ci-openshift-hive-master-periodic-e2e-azure-weekly

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@2uasimojo
Copy link
Member

2uasimojo commented Dec 18, 2025

So did this fail e2e-azure because the ProviderSpec our MachinePool controller was generating via the old installer code is not compatible with the MAO in the new openshift? (Or the reverse, really.)

@huangmingxia
Copy link
Contributor Author

huangmingxia commented Dec 19, 2025

Thank you @2uasimojo Yes, I think so.
My understanding is that this issue matches what you described in HIVE-3027. According to the referenced thread, the installer was updated to allow Hive to pass an empty osImage, with mao falling back to the default when it’s unset.
I assume the installer code change would be picked up via a revendor PR.
However, I just took another look at the revendor PR, and it seems that the e2e-azure tests after patching the installer change are still not passing.
Could you please help me check if I have missed anything? thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dlom dlom Awaiting requested review from dlom

@suhanime suhanime Awaiting requested review from suhanime

@2uasimojo 2uasimojo Awaiting requested review from 2uasimojo

Assignees

@2uasimojo 2uasimojo

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@huangmingxia @openshift-ci-robot @2uasimojo