Closed
Description
Using annif_app as in https://github.com/NatLibFi/Annif/blob/main/docker-compose.yml, the annif docker container was running a for while until pulling recently the image quay.io/natlibfi/annif:latest, since then it fails with the following error:
# docker logs --follow docker_annif
[2024-04-05 12:02:34 +0000] [1] [INFO] Starting gunicorn 21.2.0
[2024-04-05 12:02:34 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2024-04-05 12:02:34 +0000] [1] [INFO] Using worker: sync
[2024-04-05 12:02:34 +0000] [7] [INFO] Booting worker with pid: 7
/usr/local/lib/python3.10/site-packages/sklearn/base.py:376: InconsistentVersionWarning: Trying to unpickle estimator TfidfTransformer from version 1.3.2 when using version 1.4.1.post1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
warnings.warn(
/usr/local/lib/python3.10/site-packages/sklearn/base.py:376: InconsistentVersionWarning: Trying to unpickle estimator TfidfVectorizer from version 1.3.2 when using version 1 .4.1.post1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
warnings.warn(
/usr/local/lib/python3.10/site-packages/sklearn/base.py:376: InconsistentVersionWarning: Trying to unpickle estimator TfidfTransformer from version 1.3.2 when using version 1.4.1.post1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
warnings.warn(
/usr/local/lib/python3.10/site-packages/sklearn/base.py:376: InconsistentVersionWarning: Trying to unpickle estimator TfidfVectorizer from version 1.3.2 when using version 1 .4.1.post1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
warnings.warn(
WARNING:annif:Couldn't initialize backend 'fasttext': model data/projects/hogwarts/fasttext-model not found
WARNING:annif:Couldn't initialize backend 'tfidf': vectorizer file 'data/projects/tfidf-fi/vectorizer' not found
WARNING:annif:Couldn't initialize backend 'tfidf': vectorizer file 'data/projects/tfidf-sv/vectorizer' not found
WARNING:annif:Couldn't initialize backend 'tfidf': vectorizer file 'data/projects/tfidf-en/vectorizer' not found
WARNING:annif:Couldn't initialize backend 'fasttext': model data/projects/fasttext-fi/fasttext-model not found
WARNING:annif:Couldn't initialize backend 'fasttext': model data/projects/fasttext-sv/fasttext-model not found
WARNING:annif:Couldn't initialize backend 'fasttext': model data/projects/fasttext-en/fasttext-model not found
/usr/local/lib/python3.10/site-packages/sklearn/base.py:376: InconsistentVersionWarning: Trying to unpickle estimator CountVectorizer from version 1.3.2 when using version 1 .4.1.post1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
warnings.warn(
/usr/local/lib/python3.10/site-packages/sklearn/base.py:376: InconsistentVersionWarning: Trying to unpickle estimator DecisionTreeClassifier from version 1.3.2 when using ve rsion 1.4.1.post1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
warnings.warn(
/usr/local/lib/python3.10/site-packages/sklearn/base.py:376: InconsistentVersionWarning: Trying to unpickle estimator BaggingClassifier from version 1.3.2 when using version 1.4.1.post1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
warnings.warn(
2024-04-05 12:02:44.113014: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critica l operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-05 12:02:52.370582: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 28799200 exceeds 10% of free system memory.
2024-04-05 12:02:52.470843: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 28799200 exceeds 10% of free system memory.
2024-04-05 12:02:52.497902: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 28799200 exceeds 10% of free system memory.
2024-04-05 12:02:52.603658: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 14399600 exceeds 10% of free system memory.
2024-04-05 12:02:52.642776: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 28799200 exceeds 10% of free system memory.
[2024-04-05 12:02:54 +0000] [7] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 424, in import_app
app = app(*args, **kwargs)
File "/Annif/annif/__init__.py", line 59, in create_app
annif.registry.initialize_projects(cxapp.app)
File "/Annif/annif/registry.py", line 114, in initialize_projects
app.annif_registry = AnnifRegistry(projects_config_path, datadir, init_projects)
File "/Annif/annif/registry.py", line 41, in __init__
project.initialize()
File "/Annif/annif/project.py", line 132, in initialize
self._initialize_backend(parallel)
File "/Annif/annif/project.py", line 116, in _initialize_backend
self.backend.initialize(parallel)
File "/Annif/annif/backend/nn_ensemble.py", line 132, in initialize
self._model = load_model(
File "/usr/local/lib/python3.10/site-packages/keras/src/saving/saving_api.py", line 254, in load_model
return saving_lib.load_model(
File "/usr/local/lib/python3.10/site-packages/keras/src/saving/saving_lib.py", line 281, in load_model
raise e
File "/usr/local/lib/python3.10/site-packages/keras/src/saving/saving_lib.py", line 269, in load_model
_load_state(
File "/usr/local/lib/python3.10/site-packages/keras/src/saving/saving_lib.py", line 466, in _load_state
_load_container_state(
File "/usr/local/lib/python3.10/site-packages/keras/src/saving/saving_lib.py", line 534, in _load_container_state
_load_state(
File "/usr/local/lib/python3.10/site-packages/keras/src/saving/saving_lib.py", line 435, in _load_state
trackable.load_own_variables(weights_store.get(inner_path))
File "/usr/local/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 3531, in load_own_variables
raise ValueError(
ValueError: Layer 'dense' expected 2 variables, but received 0 variables during loading. Expected: ['dense/kernel:0', 'dense/bias:0']
[2024-04-05 12:02:54 +0000] [7] [INFO] Worker exiting (pid: 7)
[2024-04-05 12:02:57 +0000] [1] [ERROR] Worker (pid:7) exited with code 3
[2024-04-05 12:02:57 +0000] [1] [ERROR] Shutting down: Master
[2024-04-05 12:02:57 +0000] [1] [ERROR] Reason: Worker failed to boot.