You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Commented out the 'if (maxTokenLength < 1 || vocabSize != modelVocabSize)' expression and it worked as it should be. So maybe sometimes the tokenizer and the model have not the same size of dictionary.
Maybe a warning would be enough in these cases.
The text was updated successfully, but these errors were encountered:
I used https://huggingface.co/sambanovasystems/SambaLingo-Hungarian-Chat-70B q40. Conversion was successful but it reported incompatible tokenizer.
Commented out the 'if (maxTokenLength < 1 || vocabSize != modelVocabSize)' expression and it worked as it should be. So maybe sometimes the tokenizer and the model have not the same size of dictionary.
Maybe a warning would be enough in these cases.
The text was updated successfully, but these errors were encountered: