Crashes on out of range inputs depending on other inputs #1051
Open
Description
Using sentencepiece 0.1.99 in python 3.11.10, an out of range may cause crashes depending on which other valid inputs are part of the batch:
>>> tkn.load(str(Path("gemma2-9b") / "tokenizer.model"))
True
>>> tkn.decode_ids([255000])
'\ue0a2'
>>> tkn.decode_ids([256000])
...
IndexError: Out of range: piece id is out of range.
>>> tkn.decode_ids([[1, 256000]])
...
IndexError: Out of range: piece id is out of range.
>>> tkn.decode_ids([[2, 4], [1, 256000]])
terminate called after throwing an instance of 'sentencepiece::util::Status'
Aborted (core dumped)
This easy to resolve once you know what the issue is. But that's not 100% obvious from the crash message and even that message is obscured if this error is initially encountered in a notebook where it crashes the Jupyter kernel.
Activity