I also encounted this issue and found the problem as incompatibility of different versions of transformers. You may choose to use older versions to get around the problem:

pip install crfm-helm==0.2.3
pip install transformers==4.28.1
pip install lm-eval==0.3.0

Update: I discussed the dependency problem with the authors and they recommend to use transformers==4.33, I've tried it on llama-2-7b-chat and got correct responses.

Thanks for the reply and reaching out to the authors. That worked for me using transformers==4.33, after applying the small changes in this PR #41.

Is there a plan to update to newer versions of transformers? If not, I'd be happy to update the installation instructions in the readme.

Thanks for the reply and reaching out to the authors. That worked for me using transformers==4.33, after applying the small changes in this PR #41.

Is there a plan to update to newer versions of transformers? If not, I'd be happy to update the installation instructions in the readme.

The authors also recommended to use a newly-built version based on llama-recipes (https://github.com/meta-llama/llama-recipes/tree/main/recipes/experimental/long_context/H2O), it works with transformers == 4.39.0. You may try it out!