forked from zjunlp/EasyEdit
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
add guideline for CCKS2024
Showing
1 changed file
with
109 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,109 @@ | ||
# [CCKS2024——大模型知识编辑评测](https://tianchi.aliyun.com/competition/entrance/532182) | ||
|
||
这篇文档是关于如何复现天池比赛“CCKS2024——大模型知识编辑评测”初赛中baseline的说明,本次比赛baseline由大模型知识编辑工具[EasyEdit](https://github.com/zjunlp/EasyEdit)提供。 | ||
|
||
## 1.任务目标 | ||
知识编辑的目标是通过修改大模型中的特定知识以缓解知识谬误问题。知识编辑通常包含三个基本的设定:知识新增、知识修改和知识删除。(1)知识新增旨在让大模型习得新知识。(2)知识修改旨在改变已存储在大模型内部的知识。(3)知识删除旨在让大模型遗忘已习得的知识。 | ||
|
||
## 2.数据集介绍 | ||
初赛使用的数据集为知识新增数据集Wiki_recent、知识修改数据集ZsRE和知识擦除数据集SafeEdit(其中ZsRE和Wiki_recent均来自KnowEdit),请到天池比赛说明相关位置下载数据集。 | ||
|
||
## 3.环境安装 | ||
``` | ||
git clone https://github.com/zjunlp/EasyEdit.git | ||
conda create -n EasyEdit python=3.9.7 | ||
conda activate EasyEdit | ||
pip install -r requirements.txt | ||
``` | ||
|
||
## 4.快速运行(Wiki_recent和ZsRE): | ||
EasyEdit中已经集成了若干编辑方法,你可以根据自己的需要选择合适的方法来快速运行EasyEdit,以下以gpt2-xl为基座为例。 | ||
- 在使用特定的method进行编辑之前请先到该方法对应的.../EasyEdit/hparams/method/目录下修改或创建gpt2-xl.yaml文件。 | ||
- 请将三个数据集放入 ./data 文件夹,目录如下: | ||
``` | ||
EasyEdit | ||
├── data | ||
│ ├── recent_test.json | ||
│ ├── SafeEdit_test_ccks.json | ||
│ ├── ZsRE-test-all.json | ||
``` | ||
- 除了SERAC和MEND方法外,其它方法均可采用以下格式使用EasyEdit,若要使用SERAC和MEND方法,请查看[KnowEdit.md](https://github.com/zjunlp/EasyEdit/blob/main/examples/KnowEdit.md)相应描述。 | ||
### (1)对于wiki_recent: | ||
#### ROME | ||
``` | ||
python run_knowedit_llama2.py \ | ||
--editing_method=ROME \ | ||
--hparams_dir=.../EasyEdit/hparams/ROME/gpt2-xl.yaml \ | ||
--data_dir=./data/recent_test.json \ | ||
--datatype='recent' | ||
``` | ||
#### FT | ||
``` | ||
python run_knowedit_llama2.py \ | ||
--editing_method=FT \ | ||
--hparams_dir=.../EasyEdit/hparams/FT/gpt2-xl.yaml \ | ||
--data_dir=./data/recent_test.json \ | ||
--datatype='recent' | ||
``` | ||
### (2)对于ZsRE: | ||
#### ROME | ||
``` | ||
python run_knowedit_llama2.py \ | ||
--editing_method=ROME \ | ||
--hparams_dir=.../EasyEdit/hparams/ROME/gpt2-xl.yaml \ | ||
--data_dir=./data/ZsRE-test-all.json \ | ||
--datatype='zsre' | ||
``` | ||
#### FT | ||
``` | ||
python run_knowedit_llama2.py \ | ||
--editing_method=FT \ | ||
--hparams_dir=.../EasyEdit/hparams/FT/gpt2-xl.yaml \ | ||
--data_dir=./data/ZsRE-test-all.json \ | ||
--datatype='zsre' | ||
``` | ||
## 5.快速运行(SafeEdit): | ||
|
||
Move this file **[run_ccks_SafeEdit_gpt2-xl.py](https://github.com/zjunlp/EasyEdit/blob/main/examples/run_ccks_SafeEdit_gpt2-xl.py)** to **./**, and run: | ||
|
||
``` | ||
python run_ccks_SafeEdit_gpt2-xl.py \ | ||
--ccks \ | ||
--editing_method=DINM \ | ||
--edited_model=gpt2-xl \ | ||
--data_dir=./data/SafeEdit_test_ccks.json \ | ||
--hparams_dir=./hparams/DINM/gpt2-xl.yaml \ | ||
--safety_classifier_dir=zjunlp/SafeEdit-Safety-Classifier | ||
``` | ||
❗️❗️ Please set max_output_length to 300, set max_length to 1024 in ./hparams/DINM/gpt2-xl.yaml. | ||
For some role-playing attack prompts, LLMs may initially generate safe responses and then suddenly generate toxic text. Therefore, you should set enough max_output_length to evaluate the safety of LLM. | ||
Considering the maximum length of certain LLMs may not suffice; you can truncate the input length (from right to left, as harmful questions typically appear on the right). | ||
|
||
## 6.文件提交 | ||
得到三个test.json对应的输出文件之后,按照天池比赛说明所规定的顺序将3个json文件合并为一个列表,再进行评测提交,可用如下代码: | ||
``` | ||
import os | ||
import json | ||
def merge_json_folder(folder_path, output_file): | ||
json_files = [f for f in os.listdir(folder_path) if f.endswith('.json')] # 获取文件夹中的所有 JSON 文件路径 | ||
merged_data = [] # 创建一个空列表,用于存储合并后的 JSON 数据 | ||
for file in json_files: | ||
file_path = os.path.join(folder_path, file) # 构建完整的文件路径 | ||
with open(file_path, 'r') as json_file: | ||
data = json.load(json_file) # 读取 JSON 文件并解析为 Python 对象 | ||
merged_data.append(data) # 将解析后的 JSON 数据添加到列表中 | ||
# 将合并后的数据转换为 JSON 字符串 | ||
merged_json = json.dumps(merged_data, indent=4) | ||
# 将合并后的 JSON 字符串写入目标文件 | ||
with open(output_file, 'w') as output: | ||
output.write(merged_json) | ||
folder_path = "results_folder" # 存放 JSON 文件的文件夹路径,请确保三个json文件的顺序如规定所言 | ||
output_file = "final_result.json" # 合并后的 JSON 文件路径 | ||
merge_json_folder(folder_path, output_file) | ||
``` |