Use Schemathesis to test REST API #682

juhoinkinen · 2023-03-14T15:23:53Z

For testing the REST API there have been manually written tests since abandoning the unmaintained swagger-tester in #551. Already at that time Schemathesis was considered as a replacement of swagger-tester, but there was a dependency conflict that prevented using it.

Now Schemathesis can be installed to restore testing that the API actually conforms to the OpenAPI specification. Also the specification will now be validated.

With the default settings Schemathesis takes 12-15 s to run the tests in tests/test_swagger.py, but using @settings(max_examples=10) only 1-2 s.

~~Due to a bug the pyproject.yaml includes pinned starlette installation (starlette = ">=0.13,<0.21").~~ Edit: Pinning starlette is not necessary when pinning to recent version Schemathesis 3.18.*.

When fixing the errors noted by Schemathesis I marked some properties in the specification as nullable. is_trained and modification_time should be nullable, but backend_id probably should not be:

Annif/annif/openapi/annif.yaml

Line 173 in 6fc3157

nullable: true

I marked this only to make the test pass. One project for the testing config is deliberately missing the backend specification:

Annif/tests/projects.cfg

Lines 70 to 74 in 6fc3157

    
           [nobackend] 
        
           name=Dummy with no backend 
        
           language=en 
        
           vocab=dummy 
        
           analyzer=snowball(english)

I'm not really sure what to do for that.

codecov · 2023-03-14T15:27:36Z

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.01 ⚠️

Comparison is base (e9fd177) 99.57% compared to head (08bf4ce) 99.57%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #682      +/-   ##
==========================================
- Coverage   99.57%   99.57%   -0.01%     
==========================================
  Files          88       88              
  Lines        6170     6138      -32     
==========================================
- Hits         6144     6112      -32     
  Misses         26       26

Impacted Files	Coverage Δ
tests/conftest.py	`100.00% <ø> (ø)`
tests/test_openapi.py	`100.00% <100.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

pyproject.toml

osma · 2023-03-14T15:39:55Z

tests/test_swagger.py


-def test_swagger_cors(app_client):


Does schemathesis replace this test, and all other tests below? Or should we also keep these tests?

Also, I think we should rename test_swagger.py to something else, maybe test_openapi.py, because we're not actually using Swagger (2.0) anymore. I just forgot to rename this in the OpenAPI 3.0 PR #649

I added a custom check that is run in addition to the default Schemathesis tests to cover checking CORS support:

Annif/tests/test_openapi.py

Lines 8 to 10 in 08bf4ce

@schemathesis.check

def check_cors(response, case):

assert response.headers["access-control-allow-origin"] == "*"

Schemathesis validates the following on the responses:

not_a_server_error. The response has 5xx HTTP status;

status_code_conformance. The response status is not defined in the API schema;

content_type_conformance. The response content type is not defined in the API schema;

response_schema_conformance. The response content does not conform to the schema defined for this specific response;

response_headers_conformance. The response headers does not contain all defined headers.

If I make methods in rest.py to return nonsense strings, Schemathesis tests fail:

tests/test_openapi.py::test_api[GET /v1/] FAILED [ 20%] tests/test_openapi.py::test_api[GET /v1/projects] FAILED [ 40%] tests/test_openapi.py::test_api[GET /v1/projects/{project_id}] FAILED [ 60%] tests/test_openapi.py::test_api[POST /v1/projects/{project_id}/suggest] FAILED [ 80%] tests/test_openapi.py::test_api[POST /v1/projects/{project_id}/learn] FAILED [100%]

If I make error cases to return code 204 instead of 404/503/400 some tests fail:

tests/test_openapi.py::test_api[GET /v1/] PASSED [ 20%] tests/test_openapi.py::test_api[GET /v1/projects] PASSED [ 40%] tests/test_openapi.py::test_api[GET /v1/projects/{project_id}] FAILED [ 60%] tests/test_openapi.py::test_api[POST /v1/projects/{project_id}/suggest] FAILED [ 80%] tests/test_openapi.py::test_api[POST /v1/projects/{project_id}/learn] PASSED [100%]

(/learn method has 204 in the schema, so testing that is passed.)

Both cases are covered also by test_rest.py, so I would not keep the old tests here.

osma · 2023-03-14T15:57:51Z

is_trained and modification_time should be nullable, but backend_id probably should not be:

Agree, backend_id should not be nullable. We should ensure that the API never returns data like that.

…private Now API wont return this project and Schemathesis tests will not fail

sonarqubecloud · 2023-03-15T09:42:12Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

juhoinkinen · 2023-03-15T10:58:50Z

is_trained and modification_time should be nullable, but backend_id probably should not be:

Agree, backend_id should not be nullable. We should ensure that the API never returns data like that.

I marked nobackend project as private, which I think is enough for this PR.

osma

LVGTM

juhoinkinen · 2023-03-17T13:12:10Z

I started to feel that maybe some of the manually written tests should have not been deleted.

Schemathesis does use the examples defined in the API schema (indirectly stated in the CLI part of the documentation for the --hypothesis-phases=explicit option). This is true also when using it via pytest: I removed dummy-fi project from Annif test projects and 404 responses from the API schema, and this made those tests fail that use dummy-fi as an example project id in the schema. Reason for fails was shown to be

Received a response with a status code, which is not defined in the schema: 404

and in details "Project 'dummy-fi' not found".

But still this does not mean that e.g. the query parameters for /suggest-batch method are correctly passed and processed in rest.py. However it seems to need some amount of effort to make up an error that either tests in test_openapi.py or test_rest.py wont catch.

juhoinkinen added 3 commits March 14, 2023 16:43

Use Schemathesis for testing REST API

86af042

Fix errors noted by Schemathesis

2052d27

Speed up test by setting max_examples=10

6fc3157

juhoinkinen added the maintenance label Mar 14, 2023

juhoinkinen added this to the 0.61 milestone Mar 14, 2023

juhoinkinen marked this pull request as ready for review March 14, 2023 15:29

osma reviewed Mar 14, 2023

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

osma reviewed Mar 14, 2023

View reviewed changes

juhoinkinen added 4 commits March 14, 2023 18:45

Rename test_swagger.py to test_openapi.py

e0360a7

Add custom Schemathesis check for CORS support

9d1578e

Require installing more up-to-date version of Schemathesis

61bef14

Require again non-null backend-id in schema & make nobackend project …

08bf4ce

…private Now API wont return this project and Schemathesis tests will not fail

osma approved these changes Mar 15, 2023

View reviewed changes

juhoinkinen merged commit 3a5fc50 into main Mar 15, 2023

juhoinkinen deleted the schemathesis branch March 15, 2023 16:33

This was referenced Mar 20, 2023

Add REST API method batch-suggest #664

Merged

Project-targeted Schemathesis test and OpenAPI schema fix #685

Merged

juhoinkinen added a commit that referenced this pull request Mar 28, 2023

Restore swagger/openapi tests removed in #682

c52cf7b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Schemathesis to test REST API #682

Use Schemathesis to test REST API #682

juhoinkinen commented Mar 14, 2023 •

edited

Loading

codecov bot commented Mar 14, 2023 •

edited

Loading

osma Mar 14, 2023

juhoinkinen Mar 15, 2023

osma commented Mar 14, 2023

sonarqubecloud bot commented Mar 15, 2023

juhoinkinen commented Mar 15, 2023

osma left a comment

juhoinkinen commented Mar 17, 2023

	[nobackend]
	name=Dummy with no backend
	language=en
	vocab=dummy
	analyzer=snowball(english)

	@schemathesis.check
	def check_cors(response, case):
	assert response.headers["access-control-allow-origin"] == "*"

Use Schemathesis to test REST API #682

Use Schemathesis to test REST API #682

Conversation

juhoinkinen commented Mar 14, 2023 • edited Loading

codecov bot commented Mar 14, 2023 • edited Loading

Codecov Report

osma Mar 14, 2023

Choose a reason for hiding this comment

juhoinkinen Mar 15, 2023

Choose a reason for hiding this comment

osma commented Mar 14, 2023

sonarqubecloud bot commented Mar 15, 2023

juhoinkinen commented Mar 15, 2023

osma left a comment

Choose a reason for hiding this comment

juhoinkinen commented Mar 17, 2023

juhoinkinen commented Mar 14, 2023 •

edited

Loading

codecov bot commented Mar 14, 2023 •

edited

Loading