Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update dependencies for v1.2 release #796

Merged
merged 22 commits into from
Sep 30, 2024
Merged

Update dependencies for v1.2 release #796

merged 22 commits into from
Sep 30, 2024

Conversation

juhoinkinen
Copy link
Member

@juhoinkinen juhoinkinen commented Jul 17, 2024 •
edited
Loading

Updates the outdated dependencies.

Depedencies, which are not upgraded to the newest releases:

  • numpy
    • Because tensorflow-cpu (2.17.0) depends on numpy (>=1.23.5,<2.0.0)
  • scipy
    • v1.14 requires Python >=3.10
  • spacy
    • v3.8 got released just before we are going to release Annif 1.2, and it doesn't seem to bring any improvements (maybe even performance regression for some cases?)

Also resolves two DeprecationWarnings by Schemathesis.

@juhoinkinen juhoinkinen added maintenance dependencies Pull requests that update a dependency file labels Jul 17, 2024
@juhoinkinen juhoinkinen added this to the 1.2 milestone Jul 17, 2024
@codecov Codecov
Copy link

codecov bot commented Aug 16, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.65%. Comparing base (c42a93f) to head (2d7509d).
Report is 27 commits behind head on main.

Additional details and impacted files 
@@           Coverage Diff            @@
##             main     #796    +/-   ##
========================================
  Coverage   99.65%   99.65%            
========================================
  Files          93       93            
  Lines        6889     7058   +169     
========================================
+ Hits         6865     7034   +169     
  Misses         24       24

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@juhoinkinen juhoinkinen force-pushed the update-dependencies-v1.2 branch from 70f7e8b to 66512f5 Compare August 22, 2024 07:24
@juhoinkinen
Copy link
Member Author

juhoinkinen commented Aug 22, 2024 •
edited
Loading

Rebased and force-pushed to tidy commit history and drop upgrade to scipy 1.13 for now (because not supporting Python 3.9, which it should support?).

Reminder to self: Scipy 1.13 needs these changes.

@juhoinkinen juhoinkinen mentioned this pull request Aug 22, 2024
Merged
@juhoinkinen juhoinkinen mentioned this pull request Sep 19, 2024
Merged
@juhoinkinen
Copy link
Member Author

I built Dockerimage from this branch and tried to deploy to OpenShift for the ai.dev.finto.fi instance. NN ensemble models fail to load:

 WARNING:annif:Operation failed backend 'nn_ensemble': loading Keras model from data/projects/yso-fi/nn-model.keras; model metadata: {'keras_version': '2.15.0', 'date_saved': '2024-04-26@22:40:55'}; you have Keras version 3.5.0. Original error message: "Could not deserialize class 'Functional' because its parent module keras.src.engine.functional cannot be imported. Full object config: {'module': 'keras.src.engine.functional', 'class_name': 'Functional', 'config': {'name': 'model', 'trainable': True, 'layers': [{'module': 'keras.layers', 'class_name': 'InputLayer', 'config': {'batch_input_shape': [None, 38815, 3], 'dtype': 'float32', 'sparse': False, 'ragged': False, 'name': 'input_1'}, 'registered_name': None, 'name': 'input_1', 'inbound_nodes': []}, {'module': 'keras.layers', 'class_name': 'Flatten', 'config': {'name': 'flatten', 'trainable': True, 'dtype': 'float32', 'data_format': 'channels_last'}, 'registered_name': None, 'build_config': {'input_shape': [None, 38815, 3]}, 'name': 'flatten', 'inbound_nodes': [[['input_1', 0, 0, {}]]]}, {'module': 'keras.layers', 'class_name': 'Dropout', 'config': {'name': 'dropout', 'trainable': True, 'dtype': 'float32', 'rate': 0.2, 'noise_shape': None, 'seed': None}, 'registered_name': None, 'build_config': {'input_shape': [None, 116445]}, 'name': 'dropout', 'inbound_nodes': [[['flatten', 0, 0, {}]]]}, {'module': 'keras.layers', 'class_name': 'Dense', 'config': {'name': 'dense', 'trainable': True, 'dtype': 'float32', 'units': 100, 'activation': 'relu', 'use_bias': True, 'kernel_initializer': {'module': 'keras.initializers', 'class_name': 'GlorotUniform', 'config': {'seed': None}, 'registered_name': None}, 'bias_initializer': {'module': 'keras.initializers', 'class_name': 'Zeros', 'config': {}, 'registered_name': None}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}, 'registered_name': None, 'build_config': {'input_shape': [None, 116445]}, 'name': 'dense', 'inbound_nodes': [[['dropout', 0, 0, {}]]]}, {'module': 'keras.layers', 'class_name': 'Dropout', 'config': {'name': 'dropout_1', 'trainable': True, 'dtype': 'float32', 'rate': 0.2, 'noise_shape': None, 'seed': None}, 'registered_name': None, 'build_config': {'input_shape': [None, 100]}, 'name': 'dropout_1', 'inbound_nodes': [[['dense', 0, 0, {}]]]}, {'module': 'annif.backend.nn_ensemble', 'class_name': 'MeanLayer', 'config': {'name': 'mean_layer', 'trainable': True, 'dtype': 'float32'}, 'registered_name': 'MeanLayer', 'build_config': {'input_shape': [None, 38815, 3]}, 'name': 'mean_layer', 'inbound_nodes': [[['input_1', 0, 0, {}]]]}, {'module': 'keras.layers', 'class_name': 'Dense', 'config': {'name': 'dense_1', 'trainable': True, 'dtype': 'float32', 'units': 38815, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'module': 'keras.initializers', 'class_name': 'Zeros', 'config': {}, 'registered_name': None}, 'bias_initializer': {'module': 'keras.initializers', 'class_name': 'Zeros', 'config': {}, 'registered_name': None}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}, 'registered_name': None, 'build_config': {'input_shape': [None, 100]}, 'name': 'dense_1', 'inbound_nodes': [[['dropout_1', 0, 0, {}]]]}, {'module': 'keras.layers', 'class_name': 'Add', 'config': {'name': 'add', 'trainable': True, 'dtype': 'float32'}, 'registered_name': None, 'build_config': {'input_shape': [[None, 38815], [None, 38815]]}, 'name': 'add', 'inbound_nodes': [[['mean_layer', 0, 0, {}], ['dense_1', 0, 0, {}]]]}], 'input_layers': [['input_1', 0, 0]], 'output_layers': [['add', 0, 0]]}, 'registered_name': 'Functional', 'build_config': {'input_shape': [None, 38815, 3]}, 'compile_config': {'optimizer': 'adam', 'loss': 'binary_crossentropy', 'metrics': ['top_k_categorical_accuracy'], 'loss_weights': None, 'weighted_metrics': None, 'run_eagerly': None, 'steps_per_execution': None, 'jit_compile': None}}"

@osma
Copy link
Member

osma commented Sep 26, 2024

@juhoinkinen

This seems like a problem with Keras modules that have been reorganized, so older imports (from inside the saved model) no longer work. Here were two suggestions from GPT-4 on how that might be fixed:

Use Custom Object Scopes

If you cannot change the TensorFlow version due to other dependencies, try using custom object scopes to map the missing or changed classes and functions:

from tensorflow.keras.models import load_model
from tensorflow.keras.layers import InputLayer, Flatten, Dropout, Dense
from tensorflow.keras import layers

# Assuming `MeanLayer` is a custom layer you have the code for
from your_custom_layers import MeanLayer  

custom_objects = {
    'Functional': layers.Functional,  # Adjust according to the actual required class
    'MeanLayer': MeanLayer
}

model = load_model('path_to_your_model.keras', custom_objects=custom_objects)

Adjust the Model Loading Code

Sometimes, simply adjusting the import statements or model loading code can resolve compatibility issues:

from tensorflow import keras
model = keras.models.load_model('path_to_your_model.keras', compile=False)

@juhoinkinen
Copy link
Member Author

juhoinkinen commented Sep 26, 2024 •
edited
Loading

The second suggestion by GPT-4 did not work, and for the first I don't know what layer/class here should be:

'Functional': layers.Functional,  # Adjust according to the actual required class

There is a Keras issue about the incompatibility of models by versions 2.15.0 and 3: keras-team/keras#20083 The convert_h5_format_v2_to_v3() function proposed there does not run (OSError: Unable to synchronously open file (file signature not found)).

@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

@juhoinkinen juhoinkinen mentioned this pull request Sep 30, 2024
Merged
osma
osma approved these changes Sep 30, 2024
View reviewed changes
Copy link
Member

@osma osma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@juhoinkinen juhoinkinen marked this pull request as ready for review September 30, 2024 08:28
@juhoinkinen juhoinkinen merged commit 8f2e905 into main Sep 30, 2024
17 checks passed
@juhoinkinen juhoinkinen deleted the update-dependencies-v1.2 branch September 30, 2024 08:30
@juhoinkinen juhoinkinen mentioned this pull request Sep 30, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@osma osma osma approved these changes

Assignees
No one assigned
Labels
dependencies Pull requests that update a dependency file maintenance
Projects
None yet
Milestone
1.2
Development

Successfully merging this pull request may close these issues.
2 participants
@juhoinkinen @osma