Skip to content

[Question]: different eval results compared to the results in paper #99

Open
@unicorneeee

Description

Describe the issue

hello ,when I use the env as follows:
transformers:4.47.0
torch 2.3.0
triton 2.1.0
flash_attn 2.5.8
minference use the support_tp branch
however, I set the attn_type="hf" to evalute the infinitebench, there are different results compared to the results in paper:
results self-evalute (use llama3-8B-262K)
Llama-3-8B-Instruct-262k_hf,code_debug,24.62
Llama-3-8B-Instruct-262k_hf,math_find,18.00
Llama-3-8B-Instruct-262k_hf,longdialogue_qa_eng,0.50
Could U please provide the requirments of env when U test the infinitebench? thank you!

Metadata

Assignees

Labels

questionFurther information is requested

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions