Skip to content

Make test retries configurable and disable when debugging#2873

Open
m-akinc wants to merge 3 commits intomainfrom
users/makinc/no-reruns-for-test-debugging
Open

Make test retries configurable and disable when debugging#2873
m-akinc wants to merge 3 commits intomainfrom
users/makinc/no-reruns-for-test-debugging

Conversation

@m-akinc
Copy link
Contributor

@m-akinc m-akinc commented Feb 25, 2026

Pull Request

🤨 Rationale

When debugging a test failure locally, it isn't helpful to retry the test. We really should only be running retries for official validation runs, and arguably for non-debugging, local, test runs (as we likely want the behavior to mirror what happens on build agents).

👩‍💻 Implementation

  • jasmine-extensions/src/index.ts now checks a global TEST_RETRY_COUNT variable that is set by Vite (based on --retries=N argument is passed to karma start). If a value has been provided, it uses that. Otherwise, we fall back to the standard default of 4 retries (+ the initial run = 5 total runs)
  • added --retries=0 to the the debugging test commands (in package.json) for nimble/ok/spright-components. (The Angular test runner does not use Vite, so we can't do the same for those projects.)
  • minor tweaks to retry code (e.g. moved beforeEach calls to after the wait, in case the timing between setup and test run matters)
  • tweaked the output printed when retrying tests. Now looks like this:
> karma start karma.conf.cjs --browsers=ChromeDebugging --retries=2 --skip-tags SkipChrome

LOG: 'FAILED: Table keyboard navigation in scrollable viewport: when column header is scrolled out of view, tabbing to the table (focusing the table without any pointer events) focuses the header and scrolls it into view
        Expected 0 to be true.
        Expected undefined to be true.

Retrying... (1/2)'
LOG: 'FAILED: Table keyboard navigation in scrollable viewport: when column header is scrolled out of view, tabbing to the table (focusing the table without any pointer events) focuses the header and scrolls it into view
        Expected 0 to be true.
        Expected undefined to be true.

Retrying... (2/2)'
Chrome 139.0.0.0 (Windows 10) Table keyboard navigation in scrollable viewport: when column header is scrolled out of view, tabbing to the table (focusing the table without any pointer events) focuses the header and scrolls it into view FAILED
        Expected 0 to be true.
            at <Jasmine>
            at UserContext.<anonymous> (src/table/tests/table-keyboard-navigation.spec.ts:1213:17)
            at run (http://localhost:9876/@fs/C:/dev/nimble/packages/jasmine-extensions/dist/esm/retry-failed-tests.js:8:23)
            at spec.queueableFn.fn (http://localhost:9876/@fs/C:/dev/nimble/packages/jasmine-extensions/dist/esm/retry-failed-tests.js:53:41)
        Expected undefined to be true.
            at <Jasmine>
            at UserContext.<anonymous> (src/table/tests/table-keyboard-navigation.spec.ts:1214:22)
            at run (http://localhost:9876/@fs/C:/dev/nimble/packages/jasmine-extensions/dist/esm/retry-failed-tests.js:8:23)
            at spec.queueableFn.fn (http://localhost:9876/@fs/C:/dev/nimble/packages/jasmine-extensions/dist/esm/retry-failed-tests.js:53:41)
TOTAL: 1 FAILED, 0 SUCCESS

🧪 Testing

Manual testing

@m-akinc m-akinc marked this pull request as ready for review February 26, 2026 00:02
}
},
define: {
TEST_RETRY_COUNT: config.retries === undefined ? undefined : JSON.stringify(config.retries)
Copy link
Member

@rajsite rajsite Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want to rely on a pattern that's vite specific / coupled to build configuration. What I've had on my personal backlog is to turn this into a karma jasmine plugin and configure it that was like any other plug-in. Would have been helpful to chat about it before spending time but I'd like to push back on this implementation direction

Copy link
Member

@rajsite rajsite left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should discuss implementation direction

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants