Model description

E.g., https://huggingface.co/CohereForAI/c4ai-command-r7b-12-2024

ValueError: Unsupported model type cohere2 rank=0

Supposedly a simpler addition, a key difference to earlier models seems to be that it interleaves layers with RoPE attention and layers with no positional encoding, which allows the model to attend to tokens at an arbitrary distance. This may be beneficial with system prompts I guess - I have no actual experience with this model, and cannot provide you with further details than the links here, please contact the authors.

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

Implementation in transformers: https://github.com/huggingface/transformers/blob/main/src/transformers/models/cohere2/modeling_cohere2.py

Pretrained model: https://huggingface.co/CohereForAI/c4ai-command-r7b-12-2024