Batch suggest in ensemble backends #677
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Make the prediction on the batch in one call of the NN model
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## master #677 +/- ##
==========================================
+ Coverage 99.56% 99.57% +0.01%
==========================================
Files 87 87
Lines 6158 6157 -1
==========================================
Hits 6131 6131
+ Misses 27 26 -1
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
juhoinkinen
changed the title
osma
approved these changes
Mar 6, 2023
|
Kudos, SonarCloud Quality Gate passed!
|
|
I rerun the NN ensemble suggest and eval timings and updated the results table after the last change: it seemed to give a few seconds improvement more. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds the
_suggest_batchmethod to ensemble backends.In all ensemble backends (simple, PAV, NN) the suggestions from the source projects are fetched by using batched suggest calls (this is on project level, so whether a backend actually uses batched suggest depends on the backend).
In case of NN ensemble the prediction (now via
_merge_hit_sets_from_sources) is performed on the whole batch of the base suggestions in one call of the NN model. The prediction is made using the model's__call__()method instead ofpredict()as it is the recommended way for "small numbers of inputs that fit in one batch" and it should offer better performance; this should also fix #674.Also the EnsembleOptimizer is made to use batched suggest. Quickly testing the
hyperoptcommand on an ensemble project gives very similar weights and best NDCG scores before and after this PR.This PR improves somewhat the performance of NN ensemble suggest functionality while the results remain (only nearly?) identical (I think there were small differences in suggestion scores on my laptop). I think most of the increase comes from using
__call__of the model instead ofpredict.The below results are from runs at kj-kk using the current Finto AI YSO NN ensemble model (having MLLM, fastText and Omikuji base projects).
suggest
Targeting 6 times
tests/corpora/archaeology/fulltext/*.txt:eval
Targeting 200 documents from
kirjaesittelyt2021/yso/fin/test:With 1 job
With 4 jobs