Skip to content

[BUG] resolve failing and flaky tests in CI, and race conditions in tests using server #1586

@fkiraly

Description

@fkiraly

Currently, the remote CI is not reliable and will result in faliures and sporadic failures that are unrelated for the PR in question - that means the remote CI is not usable for practical purposes. Problems:

  • it may still be that not all failures are temporarily decorated with xfail and skip properly
  • I think there is a even race condition in tests for the server, which results in sporadic failures, especially if multiple CI runs happen in parallel (which happens in every PR since there is a test matrix)

I am not sure how to address this best long-term; short-term I would suggest to simply skip all the flaky tests.

Mid-term, I think the server tests need to be decoupled from openml-python SDK tests, i.e., replaced by three things:

  • mock backends for simple API tests
  • a local server option that runs on the VM. This can follow the extension pattern introduced in the v1-v2 migration, see [ENH] V1 → V2 API Migration #1575
  • tests for the server API decoupled from tests from openml-python, possibly on a CRON and controlling for race conditions

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions