Motivation

transformers provides Adafactor optimizer.
It is often used when training Stable Diffusion XL.

Use cases (Optional)

pip install transformers

optim_wrapper = dict(
    optimizer=dict(
        type='Adafactor',
        lr=1e-5,
        weight_decay=1e-2,
        scale_parameter=False,
        relative_step=False))

Checklist

1. Pre-commit or other linting tools are used to fix the potential lint issues.
2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
4. The documentation has been modified accordingly, like docstring or example tutorials.