Copy_of_huggingface_pytorch_transformers.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of huggingface_pytorch-transformers.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jonbaer/googlecolab/blob/master/Copy_of_huggingface_pytorch_transformers.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jolBjW6ITADo",
        "colab_type": "text"
      },
      "source": [
        "### This notebook is optionally accelerated with a GPU runtime.\n",
        "### If you would like to use this acceleration, please select the menu option \"Runtime\" -> \"Change runtime type\", select \"Hardware Accelerator\" -> \"GPU\" and click \"SAVE\"\n",
        "\n",
        "----------------------------------------------------------------------\n",
        "\n",
        "# PyTorch-Transformers\n",
        "\n",
        "*Author: HuggingFace Team*\n",
        "\n",
        "**PyTorch implementations of popular NLP Transformers**\n",
        "\n",
        "\n",
        "# Model Description\n",
        "\n",
        "\n",
        "PyTorch-Transformers (formerly known as `pytorch-pretrained-bert`) is a library of state-of-the-art pre-trained models for Natural Language Processing (NLP).\n",
        "\n",
        "The library currently contains PyTorch implementations, pre-trained model weights, usage scripts and conversion utilities for the following models:\n",
        "\n",
        "1. **[BERT](https://github.com/google-research/bert)** (from Google) released with the paper [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.\n",
        "2. **[GPT](https://github.com/openai/finetune-transformer-lm)** (from OpenAI) released with the paper [Improving Language Understanding by Generative Pre-Training](https://blog.openai.com/language-unsupervised/) by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever.\n",
        "3. **[GPT-2](https://blog.openai.com/better-language-models/)** (from OpenAI) released with the paper [Language Models are Unsupervised Multitask Learners](https://blog.openai.com/better-language-models/) by Alec Radford*, Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**.\n",
        "4. **[Transformer-XL](https://github.com/kimiyoung/transformer-xl)** (from Google/CMU) released with the paper [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://arxiv.org/abs/1901.02860) by Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.\n",
        "5. **[XLNet](https://github.com/zihangdai/xlnet/)** (from Google/CMU) released with the paper [​XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) by Zhilin Yang*, Zihang Dai*, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le.\n",
        "6. **[XLM](https://github.com/facebookresearch/XLM/)** (from Facebook) released together with the paper [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample and Alexis Conneau.\n",
        "7. **[RoBERTa](https://github.com/pytorch/fairseq/tree/master/examples/roberta)** (from Facebook), released together with the paper a [Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.\n",
        "8. **[DistilBERT](https://github.com/huggingface/pytorch-transformers/tree/master/examples/distillation)** (from HuggingFace), released together with the blogpost [Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT](https://medium.com/huggingface/distilbert-8cf3380435b) by Victor Sanh, Lysandre Debut and Thomas Wolf.\n",
        "\n",
        "The components available here are based on the `AutoModel` and `AutoTokenizer` classes of the `pytorch-transformers` library.\n",
        "\n",
        "# Requirements\n",
        "\n",
        "Unlike most other PyTorch Hub models, BERT requires a few additional Python packages to be installed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dDFctcHDTADr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%bash\n",
        "pip install tqdm boto3 requests regex sentencepiece sacremoses"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nPASpZdtTADv",
        "colab_type": "text"
      },
      "source": [
        "# Usage\n",
        "\n",
        "The available methods are the following:\n",
        "- `config`: returns a configuration item corresponding to the specified model or pth.\n",
        "- `tokenizer`: returns a tokenizer corresponding to the specified model or path\n",
        "- `model`: returns a model corresponding to the specified model or path\n",
        "- `modelWithLMHead`: returns a model with a language modeling head corresponding to the specified model or path\n",
        "- `modelForSequenceClassification`: returns a model with a sequence classifier corresponding to the specified model or path\n",
        "- `modelForQuestionAnswering`: returns a model with  a question answering head corresponding to the specified model or path\n",
        "\n",
        "All these methods share the following argument: `pretrained_model_or_path`, which is a string identifying a pre-trained model or path from which an instance will be returned. There are several checkpoints available for each model, which are detailed below:\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "The available models are listed on the [pytorch-transformers documentation, pre-trained models section](https://huggingface.co/pytorch-transformers/pretrained_models.html).\n",
        "\n",
        "# Documentation\n",
        "\n",
        "Here are a few examples detailing the usage of each available method.\n",
        "\n",
        "\n",
        "## Tokenizer\n",
        "\n",
        "The tokenizer object allows the conversion from character strings to tokens understood by the different models. Each model has its own tokenizer, and some tokenizing methods are different across tokenizers. The complete documentation can be found [here](https://huggingface.co/pytorch-transformers/main_classes/tokenizer.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "attributes": {
          "classes": [
            "py"
          ],
          "id": ""
        },
        "id": "z7SDnXd4TADw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "tokenizer = torch.hub.load('huggingface/pytorch-transformers', 'tokenizer', 'bert-base-uncased')    # Download vocabulary from S3 and cache.\n",
        "tokenizer = torch.hub.load('huggingface/pytorch-transformers', 'tokenizer', './test/bert_saved_model/')  # E.g. tokenizer was saved using `save_pretrained('./test/saved_model/')`"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yJopUwpKTAD0",
        "colab_type": "text"
      },
      "source": [
        "## Models\n",
        "\n",
        "The model object is a model instance inheriting from a `nn.Module`. Each model is accompanied by their saving/loading methods, either from a local file or directory, or from a pre-trained configuration (see previously described `config`). Each model works differently, a complete overview of the different models can be found in the [documentation](https://huggingface.co/pytorch-transformers/pretrained_models.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "attributes": {
          "classes": [
            "py"
          ],
          "id": ""
        },
        "id": "uJCcMZN0TAD1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'model', 'bert-base-uncased')    # Download model and configuration from S3 and cache.\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'model', './test/bert_model/')  # E.g. model was saved using `save_pretrained('./test/saved_model/')`\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'model', 'bert-base-uncased', output_attentions=True)  # Update configuration during loading\n",
        "assert model.config.output_attentions == True\n",
        "# Loading from a TF checkpoint file instead of a PyTorch model (slower)\n",
        "config = AutoConfig.from_json_file('./tf_model/bert_tf_model_config.json')\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'model', './tf_model/bert_tf_checkpoint.ckpt.index', from_tf=True, config=config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DamRiorQTAD6",
        "colab_type": "text"
      },
      "source": [
        "## Models with a language modeling head\n",
        "\n",
        "Previously mentioned `model` instance with an additional language modeling head."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "attributes": {
          "classes": [
            "py"
          ],
          "id": ""
        },
        "id": "ypEp_GrCTAD7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'modelWithLMHead', 'bert-base-uncased')    # Download model and configuration from S3 and cache.\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'modelWithLMHead', './test/bert_model/')  # E.g. model was saved using `save_pretrained('./test/saved_model/')`\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'modelWithLMHead', 'bert-base-uncased', output_attentions=True)  # Update configuration during loading\n",
        "assert model.config.output_attentions == True\n",
        "# Loading from a TF checkpoint file instead of a PyTorch model (slower)\n",
        "config = AutoConfig.from_json_file('./tf_model/bert_tf_model_config.json')\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'modelWithLMHead', './tf_model/bert_tf_checkpoint.ckpt.index', from_tf=True, config=config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tf96azszTAD9",
        "colab_type": "text"
      },
      "source": [
        "## Models with a sequence classification head\n",
        "\n",
        "Previously mentioned `model` instance  with an additional sequence classification head."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "attributes": {
          "classes": [
            "py"
          ],
          "id": ""
        },
        "id": "l7Iiy6aQTAD-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'modelForSequenceClassification', 'bert-base-uncased')    # Download model and configuration from S3 and cache.\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'modelForSequenceClassification', './test/bert_model/')  # E.g. model was saved using `save_pretrained('./test/saved_model/')`\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'modelForSequenceClassification', 'bert-base-uncased', output_attention=True)  # Update configuration during loading\n",
        "assert model.config.output_attention == True\n",
        "# Loading from a TF checkpoint file instead of a PyTorch model (slower)\n",
        "config = AutoConfig.from_json_file('./tf_model/bert_tf_model_config.json')\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'modelForSequenceClassification', './tf_model/bert_tf_checkpoint.ckpt.index', from_tf=True, config=config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aSKRe_4MTAEB",
        "colab_type": "text"
      },
      "source": [
        "## Models with a question answering head\n",
        "\n",
        "Previously mentioned `model` instance  with an additional question answering head."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "attributes": {
          "classes": [
            "py"
          ],
          "id": ""
        },
        "id": "VneoT4HKTAEC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'modelForQuestionAnswering', 'bert-base-uncased')    # Download model and configuration from S3 and cache.\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'modelForQuestionAnswering', './test/bert_model/')  # E.g. model was saved using `save_pretrained('./test/saved_model/')`\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'modelForQuestionAnswering', 'bert-base-uncased', output_attention=True)  # Update configuration during loading\n",
        "assert model.config.output_attention == True\n",
        "# Loading from a TF checkpoint file instead of a PyTorch model (slower)\n",
        "config = AutoConfig.from_json_file('./tf_model/bert_tf_model_config.json')\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'modelForQuestionAnswering', './tf_model/bert_tf_checkpoint.ckpt.index', from_tf=True, config=config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R6yw8LuATAEG",
        "colab_type": "text"
      },
      "source": [
        "## Configuration\n",
        "\n",
        "The configuration is optional. The configuration object holds information concerning the model, such as the number of heads/layers, if the model should output attentions or hidden states, or if it should be adapted for TorchScript. Many parameters are available, some specific to each model. The complete documentation can be found [here](https://huggingface.co/pytorch-transformers/main_classes/configuration.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "attributes": {
          "classes": [
            "py"
          ],
          "id": ""
        },
        "id": "gxq2Ck_BTAEG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "config = torch.hub.load('huggingface/pytorch-transformers', 'config', 'bert-base-uncased')  # Download configuration from S3 and cache.\n",
        "config = torch.hub.load('huggingface/pytorch-transformers', 'config', './test/bert_saved_model/')  # E.g. config (or model) was saved using `save_pretrained('./test/saved_model/')`\n",
        "config = torch.hub.load('huggingface/pytorch-transformers', 'config', './test/bert_saved_model/my_configuration.json')\n",
        "config = torch.hub.load('huggingface/pytorch-transformers', 'config', 'bert-base-uncased', output_attention=True, foo=False)\n",
        "assert config.output_attention == True\n",
        "config, unused_kwargs = torch.hub.load('huggingface/pytorch-transformers', 'config', 'bert-base-uncased', output_attention=True, foo=False, return_unused_kwargs=True)\n",
        "assert config.output_attention == True\n",
        "assert unused_kwargs == {'foo': False}\n",
        "\n",
        "# Using the configuration with a model\n",
        "config = torch.hub.load('huggingface/pytorch-transformers', 'config', 'bert-base-uncased')\n",
        "config.output_attentions = True\n",
        "config.output_hidden_states = True\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'model', 'bert-base-uncased', config=config)\n",
        "# Model will now output attentions and hidden states as well\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WGSkc9_mTAEJ",
        "colab_type": "text"
      },
      "source": [
        "# Example Usage\n",
        "\n",
        "Here is an example on how to tokenize the input text to be fed as input to a BERT model, and then get the hidden states computed by such a model or predict masked tokens using language modeling BERT model.\n",
        "\n",
        "## First, tokenize the input"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M-wPP85FTAEK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "tokenizer = torch.hub.load('huggingface/pytorch-transformers', 'tokenizer', 'bert-base-cased')\n",
        "\n",
        "text_1 = \"Who was Jim Henson ?\"\n",
        "text_2 = \"Jim Henson was a puppeteer\"\n",
        "\n",
        "# Tokenized input with special tokens around it (for BERT: [CLS] at the beginning and [SEP] at the end)\n",
        "indexed_tokens = tokenizer.encode(text_1, text_2, add_special_tokens=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PCoxNpZ_TAEN",
        "colab_type": "text"
      },
      "source": [
        "## Using `BertModel` to encode the input sentence in a sequence of last layer hidden-states"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XeHCFgUFTAEO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Define sentence A and B indices associated to 1st and 2nd sentences (see paper)\n",
        "segments_ids = [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1]\n",
        "\n",
        "# Convert inputs to PyTorch tensors\n",
        "segments_tensors = torch.tensor([segments_ids])\n",
        "tokens_tensor = torch.tensor([indexed_tokens])\n",
        "\n",
        "model = torch.hub.load('huggingface/pytorch-transformers', 'model', 'bert-base-cased')\n",
        "\n",
        "with torch.no_grad():\n",
        "    encoded_layers, _ = model(tokens_tensor, token_type_ids=segments_tensors)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WuxN0oslTAES",
        "colab_type": "text"
      },
      "source": [
        "## Using `modelWithLMHead` to predict a masked token with BERT"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1nynMox9TAET",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Mask a token that we will try to predict back with `BertForMaskedLM`\n",
        "masked_index = 8\n",
        "indexed_tokens[masked_index] = tokenizer.mask_token_id\n",
        "tokens_tensor = torch.tensor([indexed_tokens])\n",
        "\n",
        "masked_lm__model = torch.hub.load('huggingface/pytorch-transformers', 'modelWithLMHead', 'bert-base-cased')\n",
        "\n",
        "with torch.no_grad():\n",
        "    predictions = masked_lm__model(tokens_tensor, token_type_ids=segments_tensors)\n",
        "\n",
        "# Get the predicted token\n",
        "predicted_index = torch.argmax(predictions[0][0], dim=1)[masked_index].item()\n",
        "predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]\n",
        "assert predicted_token == 'Jim'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p2Dm7-i-TAEW",
        "colab_type": "text"
      },
      "source": [
        "## Using `modelForQuestionAnswering` to do question answering with BERT"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4la4r66uTAEX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "question_answering_model = torch.hub.load('huggingface/pytorch-transformers', 'modelForQuestionAnswering', 'bert-large-uncased-whole-word-masking-finetuned-squad')\n",
        "question_answering_tokenizer = torch.hub.load('huggingface/pytorch-transformers', 'tokenizer', 'bert-large-uncased-whole-word-masking-finetuned-squad')\n",
        "\n",
        "# The format is paragraph first and then question\n",
        "text_1 = \"Jim Henson was a puppeteer\"\n",
        "text_2 = \"Who was Jim Henson ?\"\n",
        "indexed_tokens = question_answering_tokenizer.encode(text_1, text_2, add_special_tokens=True)\n",
        "segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]\n",
        "segments_tensors = torch.tensor([segments_ids])\n",
        "tokens_tensor = torch.tensor([indexed_tokens])\n",
        "\n",
        "# Predict the start and end positions logits\n",
        "with torch.no_grad():\n",
        "    start_logits, end_logits = question_answering_model(tokens_tensor, token_type_ids=segments_tensors)\n",
        "\n",
        "# get the highest prediction\n",
        "answer = question_answering_tokenizer.decode(indexed_tokens[torch.argmax(start_logits):torch.argmax(end_logits)+1])\n",
        "assert answer == \"puppeteer\"\n",
        "\n",
        "# Or get the total loss which is the sum of the CrossEntropy loss for the start and end token positions (set model to train mode before if used for training)\n",
        "start_positions, end_positions = torch.tensor([12]), torch.tensor([14])\n",
        "multiple_choice_loss = question_answering_model(tokens_tensor, token_type_ids=segments_tensors, start_positions=start_positions, end_positions=end_positions)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7gsGXDfXTAEZ",
        "colab_type": "text"
      },
      "source": [
        "## Using `modelForSequenceClassification` to do paraphrase classification with BERT"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5vcrDuc-TAEb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sequence_classification_model = torch.hub.load('huggingface/pytorch-transformers', 'modelForSequenceClassification', 'bert-base-cased-finetuned-mrpc')\n",
        "sequence_classification_tokenizer = torch.hub.load('huggingface/pytorch-transformers', 'tokenizer', 'bert-base-cased-finetuned-mrpc')\n",
        "\n",
        "text_1 = \"Jim Henson was a puppeteer\"\n",
        "text_2 = \"Who was Jim Henson ?\"\n",
        "indexed_tokens = sequence_classification_tokenizer.encode(text_1, text_2, add_special_tokens=True)\n",
        "segments_ids = [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1]\n",
        "segments_tensors = torch.tensor([segments_ids])\n",
        "tokens_tensor = torch.tensor([indexed_tokens])\n",
        "\n",
        "# Predict the sequence classification logits\n",
        "with torch.no_grad():\n",
        "    seq_classif_logits = sequence_classification_model(tokens_tensor, token_type_ids=segments_tensors)\n",
        "\n",
        "predicted_labels = torch.argmax(seq_classif_logits[0]).item()\n",
        "\n",
        "assert predicted_labels == 0  # In MRPC dataset this means the two sentences are not paraphrasing each other\n",
        "\n",
        "# Or get the sequence classification loss (set model to train mode before if used for training)\n",
        "labels = torch.tensor([1])\n",
        "seq_classif_loss = sequence_classification_model(tokens_tensor, token_type_ids=segments_tensors, labels=labels)"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}