Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(wpt): mark timed out tests as 'failed' #2644

Merged
merged 5 commits into from
Feb 6, 2024
Merged

test(wpt): mark timed out tests as 'failed' #2644

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
79a0fc6
test(wpt): refactor test statistics in WPTRunner class
rossilor95 Jan 26, 2024
ae2529a
test(wpt): mark timed out tests as 'failed'
rossilor95 Jan 26, 2024
ab08800
fixup
rossilor95 Jan 30, 2024
6058d42
fix: linting
rossilor95 Jan 30, 2024
3957a58
Merge branch 'main' into wpt-timeout-as-failure
rossilor95 Feb 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Merge branch 'main' into wpt-timeout-as-failure
  • Loading branch information
@rossilor95
rossilor95 authored Feb 6, 2024
commit 3957a588aff7511766b66f600dce2d88861592e8
13 changes: 10 additions & 3 deletions test/wpt/runner/runner.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,28 +274,35 @@ export class WPTRunner extends EventEmitter {
}

if (isFailure) {
let isExpectedFailure = false
this.#stats.failedTests += 1

const name = normalizeName(message.result.name)
const sanitizedMessage = sanitizeUnpairedSurrogates(message.result.message)

if (file.flaky?.includes(name)) {
isExpectedFailure = true
this.#stats.expectedFailures += 1
wptResult?.subtests.push({ ...testResult, message: sanitizedMessage, isExpectedFailure: true })
wptResult?.subtests.push({ ...testResult, message: sanitizedMessage, isExpectedFailure })
} else if (file.allowUnexpectedFailures || topLevel.allowUnexpectedFailures || file.fail?.includes(name)) {
if (!file.allowUnexpectedFailures && !topLevel.allowUnexpectedFailures) {
if (Array.isArray(file.fail)) {
this.#statusOutput[path] ??= []
this.#statusOutput[path].push(name)
}
}

isExpectedFailure = true
this.#stats.expectedFailures += 1
wptResult?.subtests.push({ ...testResult, message: sanitizedMessage, isExpectedFailure: true })
wptResult?.subtests.push({ ...testResult, message: sanitizedMessage, isExpectedFailure })
} else {
wptResult?.subtests.push({ ...testResult, message: sanitizedMessage, isExpectedFailure: false })
wptResult?.subtests.push({ ...testResult, message: sanitizedMessage, isExpectedFailure })
process.exitCode = 1
console.error(message.result)
}
if (!isExpectedFailure) {
process._rawDebug(`Failed test: ${path}`)
}
} else {
this.#stats.passedTests += 1
wptResult?.subtests.push(testResult)
Expand Down
Loading
You are viewing a condensed version of this merge commit. You can view the full changes here.