Mamba_Chat_Demo.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jonbaer/googlecolab/blob/master/Mamba_Chat_Demo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Mamba-Chat Demo 🐍\n",
        "\n",
        "Mamba-Chat is a chat LLM based on a state space model architecture. You can find our Github repo [here](https://github.com/havenhq/mamba-chat)"
      ],
      "metadata": {
        "id": "v4OGyJu2rzEi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "1. First, make sure to select a GPU runtime. Then, install dependencies"
      ],
      "metadata": {
        "id": "rthDi2hFu3rH"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ml0er1AqpO0t"
      },
      "outputs": [],
      "source": [
        "!pip install causal-conv1d\n",
        "!pip install mamba-ssm"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "2. Fix python environment (see [here](https://github.com/pytorch/pytorch/issues/107960))"
      ],
      "metadata": {
        "id": "p6Oe7F2-u-L0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!export LC_ALL=\"en_US.UTF-8\"\n",
        "!export LD_LIBRARY_PATH=\"/usr/lib64-nvidia\"\n",
        "!export LIBRARY_PATH=\"/usr/local/cuda/lib64/stubs\"\n",
        "!ldconfig /usr/lib64-nvidia"
      ],
      "metadata": {
        "id": "PG57i14trNOF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "3. Initialize Model"
      ],
      "metadata": {
        "id": "Nj644eNPvStQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "from transformers import AutoTokenizer\n",
        "from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel\n",
        "\n",
        "device = \"cuda\"\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"havenhq/mamba-chat\")\n",
        "tokenizer.eos_token = \"<|endoftext|>\"\n",
        "tokenizer.pad_token = tokenizer.eos_token\n",
        "tokenizer.chat_template = AutoTokenizer.from_pretrained(\"HuggingFaceH4/zephyr-7b-beta\").chat_template\n",
        "\n",
        "model = MambaLMHeadModel.from_pretrained(\"havenhq/mamba-chat\", device=\"cuda\", dtype=torch.float16)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6tKE3s3lp_UJ",
        "outputId": "06ee6dbc-589b-48fd-df28-3d7f40d43ab9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "4. Talk to Mamba-Chat!"
      ],
      "metadata": {
        "id": "t0cvJLHCvVWN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "messages = []\n",
        "while True:\n",
        "    user_message = input(\"\\nYour message: \")\n",
        "    messages.append(dict(\n",
        "        role=\"user\",\n",
        "        content=user_message\n",
        "    ))\n",
        "\n",
        "    input_ids = tokenizer.apply_chat_template(messages, return_tensors=\"pt\", add_generation_prompt=True).to(\"cuda\")\n",
        "\n",
        "    out = model.generate(input_ids=input_ids, max_length=2000, temperature=0.9, top_p=0.7, eos_token_id=tokenizer.eos_token_id)\n",
        "\n",
        "    decoded = tokenizer.batch_decode(out)\n",
        "    messages.append(dict(\n",
        "        role=\"assistant\",\n",
        "        content=decoded[0].split(\"<|assistant|>\\n\")[-1])\n",
        "    )\n",
        "\n",
        "    print(\"Model:\", decoded[0].split(\"<|assistant|>\\n\")[-1])"
      ],
      "metadata": {
        "id": "4cMzClx-qN1x"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}