Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRAFT] Add support gradient cache for Reranker #148

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

[DRAFT] Add support gradient cache for Reranker #148

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
859c9e6
gc reranker
sigridjineth Aug 24, 2024
f29930c
reranker arguments
sigridjineth Aug 24, 2024
3d92da6
fix: training arguments
sigridjineth Aug 24, 2024
0127d00
fix: arguments
sigridjineth Aug 24, 2024
52616cd
fix: trainer
sigridjineth Aug 24, 2024
cd09b57
fix: trainer
sigridjineth Aug 24, 2024
6d11105
parser
sigridjineth Aug 24, 2024
e98bbf3
fix: paserr
sigridjineth Aug 24, 2024
851f01b
fix: parser
sigridjineth Aug 24, 2024
bb5d87c
fix: parser
sigridjineth Aug 24, 2024
70d65ab
fix: trainer
sigridjineth Aug 24, 2024
bfb8e5a
fix: trust_remote_code
sigridjineth Aug 24, 2024
ca4b04b
hotfix: gradient_checkpointing_enable
sigridjineth Aug 24, 2024
a79b647
fix: forward method
sigridjineth Aug 24, 2024
78ff580
fix: prediction step
sigridjineth Aug 24, 2024
43a642d
fix: trainer
sigridjineth Aug 24, 2024
0380e4e
fix: ddp
sigridjineth Aug 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
reranker arguments
  • Loading branch information
@sigridjineth
sigridjineth committed Aug 24, 2024
commit f29930c9cb0c2e41602d824f58116ba8b81a4fa1
1 change: 1 addition & 0 deletions examples/example_rankllama.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,5 @@ deepspeed --include localhost:4,5,6,7 --master_port 60000 --module tevatron.rera
--num_train_epochs 1 \
--logging_steps 10 \
--overwrite_output_dir
--gra
```
8 changes: 8 additions & 0 deletions src/tevatron/reranker/arguments.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,3 +116,11 @@ class DataArguments:
"enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta)."
},
)

@dataclass
class TevatronTrainingArguments(TrainingArguments):
warmup_ratio: float = field(default=0.1)

grad_cache: bool = field(default=False, metadata={"help": "Use gradient cache update"})
gc_q_chunk_size: int = field(default=4)
gc_p_chunk_size: int = field(default=32)