A spec might allow for multiple responses depending on the gateway's capability.

For example, the specs for path gateway - range request header allows a gateway to return either:

• the minimal dag matched by a range request using multiple ranges,
• the minimal dag matched by the first range request,
• the whole dag.

We solve this by running a first check that requests an endpoint, stores the response's header through a side-effect, then uses an if / else flow to test between these capabilities (code).

@aschmahmann is working on adding more of these tests across the suite; before implementing many more, we might want to support this natively through the sugar API.

Also, since two gateways might pass the specs with very different feature sets, it would make sense to signal which features are supported in the conformance output.

Problem 1: Support many possible responses

At the moment, you may define a test with:

{
   Name: ...
   Request: Request().Path(...)....
   Response: Expect().Headers(...)....
},
{
   Name: ...
   Requests: Requests(Request()...., Request()..., ....) // multiple requests
   Response: Expect().Headers(...)....
   Responses: Responses().HaveTheSamePayload(),... // checker for multiple responses
}

We could add AnyOf operator:

{
   Name: ...
   Request: Request().Path(...)....
   Response: AnyOf(
       Response()...., // returned multiple ranges
       Response()...., // returned first range
       Response().... // returned whole file
   )
}

That would pass if the actual response matches any of the responses.
We could implement this as a /simple/ "Or" operator.

This opens up more complex operators like

{
   Name: ...
   Request: Request().Path(...)....
   Response: AllOf(
       Response()...., // shared response expectation,
       AnyOf(
           Response()...., // returned multiple ranges
           Response()...., // returned first range
           Response().... // returned whole file
       )
   )
}

Problem 2: Surface MUST / MAY / SHOULD features

Rough idea:

• If a spec is a MUST, the test should PASS or FAIL if the expected responses is matched
• If a spec is a MAY or SHOULD, the test should PASS or SKIP if the expected responses is matched

This will signal to users/implementers which features are supported or not.

It might be subtle to represent and evaluate these in go code: some of these would be "additive", this is how specs are written; we'd define three expectations, and the gateway MUST pass some, SHOULD pass some, and MAY pass others.

{
   Name: ...
   Request: Request().Path(...)....
   Response: AllOf(
           Response().MUST().... // eval => pass or fail
           Response().SHOULD().... // eval => pass or skip
           Response().MAY().... // eval => pass or skip
    )
}

But in the case of range requests, the expected responses are mutually exclusive; if the gateway returns the minimal dag that matches the range requests (SHOULD), then the response will be very different from the gateway that returns the whole dag (MUST). So the AnyOf operator should take these into account during evaluation:

{
   Name: ...
   Request: Request().Path(...)....
   Response: AnyOf(
           Response().MUST().... // eval last, if the other did not pass => pass or fail
           Response().SHOULD().... // eval second if the may did not pass => pass or skip
           Response().MAY().... // eval first => pass or skip
    )
}

Early exit

We'll have to be careful with AND/AllOf and early exit: if the first predicate fails, it's intuitive to exit the whole evaluation early, BUT in the case of testing, we should continue running all the tests in order to preserve the test counts.

Related

• Discussion in Slack
• PR
• Related: issue regarding dependency graph (why we used side-effects at first)