Thanks for the 2nd commit, unfortunately seems like there's some issue causing the tests to fail: https://github.com/xhluca/bm25s/actions/runs/11994683583/job/33448614792#step:6:1

Perhaps running the tests locally could help diagnose the issue?

@xhluca Sorry, I didn't quite know how to test all the test files at once, so only checked the core/test_save_load.py. Now, I've run both core and comparison tests using python -m unittest tests/(core or comparison)/test_*.py cmd, it ran without error. Please review. Thanks

Can you elaborate on the changes to the test file: tests/core/test_tokenizer_misc.py ?

Can you elaborate on the changes to the test file: tests/core/test_tokenizer_misc.py ?

In my local, I saw that test_tokenizer_misc.py was failing as the ids were not matching the results. In my case, it was coming up to be [2,0,3]. Then I checked with the corpus and query:

corpus = [
            "a cat is a feline and likes to purr",
            "a dog is the human's best friend and loves to play",
            "a bird is a beautiful animal that can fly",
            "a fish is a creature that lives in water and swims",
        ]
query = "What is a fly?"

Thought that the id 2 "a bird is a beautiful animal that can fly" should be at least ranked first, which was also the result coming in my case. So, I edited the assertion.

It seems the tests are failing due to the changes to the test. Does github allow you to see the results?

btw can you add the test file from here: #79 (comment)

@xhluca Yes, github is allowing me to see the test results. I've changed back the assertion in test_tokenizer_misc.py. I also added the test as #79, which is passing in my local. In case, this doesn't pass in the github actions, let's please connect over a webcall.

Looks good!

Hi @xhluca, updated this hotfix/save-load-extended-character-corpora . Missed to add ensure-ascii=False while saving corpus. It is causing invalid escape sequence failure while retrieving along with corpus. Should I create another PR or can you re-open this one?

Please create a new one