Skip to content

Unexpected non-OK test statuses will stop Interop scores updating #143

Open
@DanielRyanSmith

Description

If any non-OK test status occurs in any labeled test that is being scored for interoperability, and that test was not added to the list of non-OK statuses to ignore in the code, the entirety of the score update GitHub Action will fail, and interop scores will not be updated unless the test is either investigated and explicitly added to to the list of ignored tests statuses.

Because the results are checked for all major parallel browser runs over time, any test that has ever encountered a non-OK test status will need to be added to this list or the interop score will no longer update. This includes transient errors like the one for /css/css-scroll-snap/snap-at-user-scroll-end.html. This test incurred a non-OK test status (timeout) on this run, but has not seen another non-OK status afterward. However, the entirety of Interop scoring will be halted until this test is permanently added to the list of known non-OK test statuses indefinitely.

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions