pytest-subtests: subtest reports overwrite parent test result, causing incorrect status in UI and JSON output

[jasonwbarnett]

Description

Hi there! We've been using pytest-subtests in our integration test suite and noticed some unexpected behavior with how test results are reported to both the Buildkite Test Engine API (Tests tab in the UI) and the local JSON file used by bktec for retries.

It looks like the collector doesn't currently have any special handling for subtests, and I think the way subtest reports come through pytest_runtest_logreport may be causing issues. I wanted to flag this in case it's helpful.

Worth noting: as of pytest 9.0, subtests have been integrated directly into pytest core (previously they were only available via the separate pytest-subtests plugin). The feature is marked as experimental but the core functionality and usage are considered stable. So this is likely to become increasingly common in the wild, and it would be great if the collector handled it correctly.

What we observed

We have a test test_file_export_flow_xlsx that uses pytest-subtests with 4 sequential subtests (they're dependent steps in an e2e workflow). In a recent build:

• 3 subtests passed (SUBPASSED), 1 failed (SUBFAILED)
• The parent test was correctly reported as FAILED by pytest itself
• However, in the BK Test Engine UI (Tests tab), the test did not show up with a failed status
• Additionally, the JSON file written via --json appeared to contain incorrect results, which caused bktec to retry 13 tests when only 2 actually failed

What we think is happening

Looking at the code in buildkite_plugin.py, the update_test_result() method (around line 105) processes reports by looking up self.in_flight[report.nodeid]. Since subtests emit separate pytest_runtest_logreport calls for each subtest using the same report.nodeid as the parent test, each subtest's result overwrites the previous one.

So the sequence might look like:

1. Subtest 1 (SUBPASSED) -> test_data.passed()
2. Subtest 2 (SUBPASSED) -> test_data.passed()
3. Subtest 3 (SUBFAILED) -> test_data.failed()
4. Subtest 4 (SUBPASSED) -> test_data.passed() -- overwrites the failure
5. Parent test (FAILED) -> should restore failed, but this depends on the parent's call-phase report coming through correctly

If the parent's final FAILED report doesn't arrive as expected (or arrives with an unexpected when value), the last subtest's passed() would stick, and both the API upload and JSON output would show the test as passed.

Environment

• pytest 9.0.2
• pytest-subtests (latest)
• pytest-xdist 3.8.0
• buildkite-test-collector 1.3.1

Possible fix ideas

Not sure what the best approach would be, but a couple of thoughts:

• Ignore subtest reports and only use the parent test's final report for the result. This would be the simplest fix and would work well for our use case (sequential dependent subtests).
• Detect subtest reports (perhaps by checking for subtest-specific attributes on the report) and handle them separately, or skip them when updating the parent's result.

Happy to provide more details or help test a fix if that would be useful. Thanks!

[meghan-kradolfer]

Hi @jasonwbarnett! Thanks for raising this. The Test Engine team will have a look into it and keep you updated.

[meghan-kradolfer]

Hey @jasonwbarnett, we are probably leaning more towards option 1 (Ignore subtest reports). However, if you were wanting to go ahead with implementing a fix for this than that would be great, as this work might take a little while to be prioritised in our team.