When I used a customized tokenizer, the vocab is saved by retriever into vocab.index.json under the format of, for example:

{"0": 0, "1": 1, "2": 2, "3": 3, "4": 4, "5": 5, "6": 6, "7": 7, "8": 8, "9": 9, "10": 10, "11": 11}

when i retrieve by ids of query tokens like results, scores = reloaded_retriever.retrieve(query_tokens, corpus=titles, k=2) it led to an error when we query in a different corpus

File "env\Lib\site-packages\bm25s_init_.py", line 486, in get_scores_from_ids
raise ValueError(
ValueError: The maximum token ID in the query (12) is higher than the number of tokens in the index.This likely means that the query contains tokens that are not in the index.

In function retrieve of class BM25 in bm25s_init_.py, the retriever load the vocab this way:

        # if it's a list of list of tokens ids (int), we remove any integer not in the vocab_dict
        if is_list_of_list_of_type(query_tokens, type_=int):
            query_tokens_filtered = []
            for query in query_tokens:
                query_filtered = [
                    token_id for token_id in query if token_id in self.vocab_dict
                ]
                if len(query_filtered) == 0:
                    if "" not in self.vocab_dict:
                        self.vocab_dict[""] = max(self.vocab_dict.values()) + 1
                    query_filtered = [self.vocab_dict[""]]

                query_tokens_filtered.append(query_filtered)

            query_tokens = query_tokens_filtered

I think a quick fix is add .values in the query_filtered and it work:

    query_filtered = [
        token_id for token_id in query if token_id in self.vocab_dict.values()
    ]

Is this correct?