This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

It would be great if this PR could be revived :)

Btw if you do, I think you can set is_trainable to True. Even though the PEFT support for quanto is still in PR state, it still works as is. The reason is that quanto QLinear modules are subclasses of nn.Linear, thus PEFT just applies a normal lora.Linear layer. Some features like merging won't work (hence the need for the PR), but training and inference work. Here is some code for testing, which passed with PEFT v0.12.0:

import os

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, DataCollatorForLanguageModeling, TrainingArguments, Trainer
from transformers.utils.quantization_config import QuantizationMethod
from peft import LoraConfig, get_peft_model
from optimum.quanto import QuantizedModelForCausalLM, qint2, qint4, qint8


os.environ["CUDA_VISISBLE_DEVICES"] = "0"

device = "cpu"
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
data = load_dataset("ybelkada/english_quotes_copy")
data = data.map(lambda samples: tokenizer(samples["quote"]), batched=True)


def main(weights):
    config = LoraConfig(
        r=16,
        lora_alpha=32,
        target_modules=["q_proj", "v_proj"],
        lora_dropout=0.05,
        bias="none",
        task_type="CAUSAL_LM",
    )

    model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m", device_map=device)
    QuantizedModelForCausalLM.quantize(model, weights=weights)

    class QuantizationConfig:
        quant_method = QuantizationMethod.QUANTO

    class HfQuantizer:
        is_trainable = True  #                                  <======= is False right now
        quantization_config = QuantizationConfig()

    model.hf_quantizer = HfQuantizer()
    model.is_quantized = True

    model = get_peft_model(model, config)
    assert "QLinear" in repr(model)
    assert "lora.Linear" in repr(model)

    trainer = Trainer(
        model=model,
        train_dataset=data["train"],
        args=TrainingArguments(
            per_device_train_batch_size=4,
            gradient_accumulation_steps=4,
            warmup_steps=2,
            max_steps=5,
            learning_rate=2e-4,
            logging_steps=1,
            output_dir="/tmp/peft/quanto",
        ),
        data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),
    )
    model.config.use_cache = False
    trainer.train()

if __name__ == "__main__":
    main(qint2)
    main(qint4)
    main(qint8)

(I do get a RuntimeError: CUDA error: an illegal memory access was encountered from quanto when running with CUDA, but that's a separate issue)

Hi there. I am considering pushing one last quanto version that:

• has optimum-quanto as single dependency,
• raises an exception whenever a symbol is imported from quanto (instead of optimum.quanto).

This is because there are more than 50k downloads of the obsolete quanto version every week, and I would like to redirect the users to the correct library.

I am however worried about the implications in transformers, so it would be great to either merge this pull-request or make sure it is harmless.

Hi there. I am considering pushing one last quanto version that:

has optimum-quanto as single dependency,
raises an exception whenever a symbol is imported from quanto (instead of optimum.quanto).
This is because there are more than 50k downloads of the obsolete quanto version every week, and I would like to redirect the users to the correct library.

I am however worried about the implications in transformers, so it would be great to either merge this pull-request or make sure it is harmless.

Thanks for the heads-up ! I will spend a bit of time today to finish this PR !

Hey! 🤗 Thanks for your contribution to the transformers library!

Before merging this pull request, slow tests CI should be triggered. To enable this:

• Add the run-slow label to the PR
• When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command [run-slow] followed by a comma separated list of all the models to be tested, i.e. [run_slow] model_to_test_1, model_to_test_2
  • If the pull request affects a lot of models, put at most 10 models in the commit message
• A transformers maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is here.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Generally LGTM, thanks Marc (and David for the recent release).

Just a question, should the is_trainable property also be updated (see #31732 (comment))?

Oh indeed ! I'll change it !