Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evaluation code for reproducing results in the paper. #3

Open
ashim95 opened this issue Jul 19, 2022 · 0 comments
Open

Evaluation code for reproducing results in the paper. #3

ashim95 opened this issue Jul 19, 2022 · 0 comments

Comments

@ashim95
Copy link

ashim95 commented Jul 19, 2022

Hi @LilyNLP ,

Thank you for releasing your code. I had a couple of questions regarding reproducing the results shown in the paper.

  • Your codebase only has code for TextFooler attack. Is it possible to release codes for other attacks as well (PWWS etc.) so that I can reproduce your results?

  • The codes for SAFER and DISP do not support MNLI (text pair classification), which means you would have adapted those to your codebase. Is it possible to release those codes as well so I can exactly reproduce results in your paper?

  • The paper says that you use TextAttack for implementation, but I don't see it being used in the current implementation.

We use TextFooler† (Jin et al., 2019) as the major attack model for AWS. 
Moreover, we implement (Ren et al., 2019) and 
GENETIC (Alzantot et al., 2018) based on the TextAttack (Morris et al., 2020) 
code base to further verify the efficiency of our proposed method.

Thank you,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant