Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restore fp16 support on xla gpu device #22300

Merged
merged 1 commit into from
Mar 21, 2023
Merged

Restore fp16 support on xla gpu device #22300

merged 1 commit into from
Mar 21, 2023

Conversation

ymwangg
Copy link
Contributor

@ymwangg ymwangg commented Mar 21, 2023

#20684 accidentally disabled fp16 support on xla gpu device, which leads to significant performance regression. This PR restores this feature.

cc @jeffhataws @sgugger @Lokiiiiii

Tested with

GPU_NUM_DEVICES=1 python run_mlm.py \
    --model_name_or_path bert-base-uncased \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --overwrite_output_dir true \
    --output_dir /tmp/test-mlm \
    --per_gpu_train_batch_size 24 \
    --do_eval \
    --fp16 true \
    --do_train \
    --num_train_epochs 3 \
    --optim adamw_torch_xla

***** train metrics *****
  epoch                    =        3.0
  train_loss               =     1.7725
  train_runtime            = 0:04:58.00
  train_samples            =       4627
  train_samples_per_second =      46.58
  train_steps_per_second   =      1.943
INFO:__main__:*** Evaluate ***
[INFO|trainer.py:739] 2023-03-21 19:05:53,483 >> The following columns in the evaluation set don't have a corresponding argument in `BertForMaskedLM.forward` and have been ignored: special_tokens_mask. If special_tokens_mask are not expected by `BertForMaskedLM.forward`,  you can safely ignore this message.
[INFO|trainer.py:3072] 2023-03-21 19:05:53,487 >> ***** Running Evaluation *****
[INFO|trainer.py:3074] 2023-03-21 19:05:53,487 >>   Num examples = 479
[INFO|trainer.py:3077] 2023-03-21 19:05:53,487 >>   Batch size = 8
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [00:07<00:00,  8.38it/s]
***** eval metrics *****
  epoch                   =        3.0
  eval_loss               =     1.5811
  eval_runtime            = 0:00:29.83
  eval_samples            =        479
  eval_samples_per_second =     16.055
  eval_steps_per_second   =      2.011
  perplexity              =     4.8601

sgugger
sgugger approved these changes Mar 21, 2023
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should work as #21428 changed the logic so nothing needs to be done for bfloat16 anymore.

Thanks for the fix!

jeffhataws
jeffhataws reviewed Mar 21, 2023
View reviewed changes
src/transformers/trainer.py
@@ -598,7 +598,7 @@ def __init__(
logger.info(f"Using {args.half_precision_backend} half precision backend")

self.do_grad_scaling = False
if (args.fp16 or args.bf16) and not (args.deepspeed or is_sagemaker_mp_enabled() or is_torch_tpu_available()):
if (args.fp16 or args.bf16) and not (args.deepspeed or is_sagemaker_mp_enabled()):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will you separate bf16 out? We don't need to scale for bf16.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment and PR mentioned above.

Copy link
Contributor

@jeffhataws jeffhataws Mar 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this work for bf16 on XLA?

if args.fp16 and not (args.deepspeed or is_sagemaker_mp_enabled()):

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Let's merge this for XLA GPU then. I will check on my side and fix if needed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this check ensures bf16 won't run grad scaling.

self.do_grad_scaling = self.amp_dtype == torch.float16
if self.do_grad_scaling:

Please revise if you see anything wrong for trainium.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#22307 is the fix for Neuron.

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Mar 21, 2023 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

@sgugger
Copy link
Collaborator

sgugger commented Mar 21, 2023

Failure is unrelated and due to the branch being old, it's fixed on main, so merging.

@sgugger sgugger merged commit d35f729 into huggingface:main Mar 21, 2023
jeffhataws added a commit to jeffhataws/transformers that referenced this pull request Mar 22, 2023
@jeffhataws
Restore bf16 support for Neuron after PR huggingface#22300
3d6c1ba 
This PR fixes the "RuntimeError: No CUDA GPUs are available"
when running with --bf16 option on Neuron.

Related PRs:
huggingface#20684
huggingface#22300
@jeffhataws jeffhataws mentioned this pull request Mar 22, 2023
Merged
5 tasks
jeffhataws added a commit to jeffhataws/transformers that referenced this pull request Mar 23, 2023
@jeffhataws
Restore bf16 support for Neuron after PR huggingface#22300
9430d12 
This PR fixes the "RuntimeError: No CUDA GPUs are available"
when running with --bf16 option on Neuron.

Related PRs:
huggingface#20684
huggingface#22300
jeffhataws added a commit to jeffhataws/transformers that referenced this pull request Mar 23, 2023
@jeffhataws
Fix --bf16 option support for Neuron after PR huggingface#22300
7e907b3 
This PR fixes the "RuntimeError: No CUDA GPUs are available"
when running with --bf16 option on Neuron.

Related PRs:
huggingface#20684
huggingface#22300
jeffhataws added a commit to jeffhataws/transformers that referenced this pull request Mar 23, 2023
@jeffhataws
Fix --bf16 option support for Neuron after PR huggingface#22300
fd81746 
This PR fixes the "RuntimeError: No CUDA GPUs are available"
when running with --bf16 option on Neuron.

Related PRs:
huggingface#20684
huggingface#22300
sgugger pushed a commit that referenced this pull request Mar 23, 2023
@jeffhataws
Fix --bf16 option support for Neuron after PR #22300 (#22307)
ec9b18f 
This PR fixes the "RuntimeError: No CUDA GPUs are available"
when running with --bf16 option on Neuron.

Related PRs:
#20684
#22300
jeffhataws added a commit to jeffhataws/transformers that referenced this pull request Mar 29, 2023
@jeffhataws
Revert "Fix --bf16 option support for Neuron after PR huggingface#22300"
bc8f00e 
This reverts commit fd81746.
sgugger pushed a commit that referenced this pull request Mar 29, 2023
@jeffhataws
Revert "Fix --bf16 option support for Neuron after PR #22300" (#22451)
5e89a43 
This reverts commit fd81746.
raghavanone pushed a commit to raghavanone/transformers that referenced this pull request Apr 5, 2023
@ymwangg @raghavanone
Restore fp16 support on xla gpu device (huggingface#22300)
5d15c1b
raghavanone pushed a commit to raghavanone/transformers that referenced this pull request Apr 5, 2023
@jeffhataws @raghavanone
Fix --bf16 option support for Neuron after PR huggingface#22300 (hugg…
892ec86 
…ingface#22307)

This PR fixes the "RuntimeError: No CUDA GPUs are available"
when running with --bf16 option on Neuron.

Related PRs:
huggingface#20684
huggingface#22300
raghavanone pushed a commit to raghavanone/transformers that referenced this pull request Apr 5, 2023
@jeffhataws @raghavanone
Revert "Fix --bf16 option support for Neuron after PR huggingface#22300…
e832063 
…" (huggingface#22451)

This reverts commit fd81746.
novice03 pushed a commit to novice03/transformers that referenced this pull request Jun 23, 2023
@ymwangg @novice03
Restore fp16 support on xla gpu device (huggingface#22300)
f41ff75
novice03 pushed a commit to novice03/transformers that referenced this pull request Jun 23, 2023
@jeffhataws @novice03
Fix --bf16 option support for Neuron after PR huggingface#22300 (hugg…
1eb07b4 
…ingface#22307)

This PR fixes the "RuntimeError: No CUDA GPUs are available"
when running with --bf16 option on Neuron.

Related PRs:
huggingface#20684
huggingface#22300
novice03 pushed a commit to novice03/transformers that referenced this pull request Jun 23, 2023
@jeffhataws @novice03
Revert "Fix --bf16 option support for Neuron after PR huggingface#22300…
5cf5c92 
…" (huggingface#22451)

This reverts commit fd81746.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jeffhataws jeffhataws jeffhataws approved these changes

@sgugger sgugger sgugger approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ymwangg @HuggingFaceDocBuilderDev @sgugger @jeffhataws