The recently added GH Actions workflow for rebuilding Docker images (#715) could also verify that the models trained on the previous image build work (results-wise) identically in the new image. It is quite undesirable that the models would work even slightly differently in different Docker image builds of the same Annif version.

These are the steps in the workflow that aim to verify the compatibility and results identicality of models:

1. Train models with all (trainable) algorithms using old image (the one in quay.io with the tag being rebuild)
2. Evaluate the models and store results in eval.prev.out file using old image
3. Evaluate the models and store results in eval.out file using new image
4. Compare eval.prev.out and eval.out with diff, and fail the workflow in the case of difference, unless the box for allowing this is checked when triggering the workflow

For both training and evaluation the tests/corpora/archaeology/fulltext/ corpus is used, which is fine for all algorithms I think. Although there could be some dedicated corpora for this.

Also, there could be a similar workflow for checking models compatibility when preparing an Annif release, instead of doing compatibility checks manually, so the compatibility-check steps could be moved to a separate action for reusability, like the prepare action of CI/CD workflow.

Note: I've been working on this in my own fork, to avoid accidental image pushes to quay.io.

TODO before merge:

• Switch the image used to compare current build to quay.io/natlibfi/annif from jinkinen/annif