-
-
Notifications
You must be signed in to change notification settings - Fork 210
Open
Description
Currently, the remote CI is not reliable and will result in faliures and sporadic failures that are unrelated for the PR in question - that means the remote CI is not usable for practical purposes. Problems:
- it may still be that not all failures are temporarily decorated with
xfailandskipproperly - I think there is a even race condition in tests for the server, which results in sporadic failures, especially if multiple CI runs happen in parallel (which happens in every PR since there is a test matrix)
I am not sure how to address this best long-term; short-term I would suggest to simply skip all the flaky tests.
Mid-term, I think the server tests need to be decoupled from openml-python SDK tests, i.e., replaced by three things:
- mock backends for simple API tests
- a local server option that runs on the VM. This can follow the extension pattern introduced in the v1-v2 migration, see [ENH] V1 → V2 API Migration #1575
- tests for the server API decoupled from tests from
openml-python, possibly on a CRON and controlling for race conditions