Lab3_Deploy_MLPipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a SageMaker Pipeline to train and deploy a BERT-Based text classifier\n",
    "\n",
    "### Introduction\n",
    "\n",
    "In this lab, you will do the following:\n",
    "* Define and run a pipeline using a directed acyclic graph (DAG) with specific pipeline parameters and model hyper-parameters\n",
    "* Define a processing step that cleans, balances, transforms, and splits our dataset into train, validation, and test dataset\n",
    "* Define a training step that trains a model using the train and validation datasets\n",
    "* Define a processing step that evaluates the trained model's performance on the test dataset\n",
    "* Define a register model step that creates a model package from the trained model\n",
    "* Define a conditional step that checks the model's performance and conditionally registers the model for deployment\n",
    "\n",
    "### Table of Contents\n",
    "\n",
    "- [1. Configure the dataset and processing step](#c2w3-1.)\n",
    "  - [1.1. Configure S3 path for raw input data](#c2w3-1.1.)\n",
    "  - [1.2. Configure processing step](#c2w3-1.2.)\n",
    "    - [Exercise 1](#c2w3-ex-1)\n",
    "    - [Exercise 2](#c2w3-ex-2)\n",
    "- [2. Configure training step](#c2w3-2.)\n",
    "  - [2.1. Define parameters](#c2w3-2.1.)\n",
    "  - [2.2. Configure hyper-parameters](#c2w3-2.2.)\n",
    "  - [2.3. Configure model-evaluation metrics](#c2w3-2.3.)\n",
    "  - [2.4. Configure the `PyTorchEstimator`](#c2w3-2.4.)\n",
    "  - [2.5. Setup pipeline step caching](#c2w3-2.5.)\n",
    "  - [2.6. Configure the `TrainingStep`](#c2w3-2.6.)\n",
    "    - [Exercise 3](#c2w3-ex-3)\n",
    "- [3. Configure model-evaluation step](#c2w3-3.)\n",
    "- [4. Configure and register model step](#c2w3-4.)\n",
    "  - [4.1. Configure the model for deployment](#c2w3-4.1.)\n",
    "  - [4.2. Register the model for deployment](#c2w3-4.2.)\n",
    "    - [Exercise 4](#c2w3-ex-4)\n",
    "- [5. Create model for deployment step](#c2w3-5.)\n",
    "    - [Exercise 5](#c2w3-ex-5)\n",
    "    - [Exercise 6](#c2w3-ex-6)\n",
    "- [6. Check accuracy condition step](#c2w3-6.)\n",
    "- [7. Create pipeline](#c2w3-7.)\n",
    "  - [7.1. Define a pipeline of parameters, steps, and conditions](#c2w3-7.1.)\n",
    "  - [7.2. Start Pipeline](#c2w3-7.2.)\n",
    "  - [7.3. Wait for pipeline execution](#c2w3-7.3.)\n",
    "  - [7.4. Describe completed pipeline](#c2w3-7.4.)\n",
    "  - [7.5. Wait for the pipeline to complete](#c2w3-7.5.)\n",
    "- [8. Evaluate the model](#c2w3-8.)\n",
    "  - [8.1. Describe evaluation metrics](#c2w3-8.1.)\n",
    "    - [Exercise 7](#c2w3-ex-7)\n",
    "  - [8.2. Review the evaluation report](#c2w3-8.2.)\n",
    "  - [8.3. List pipeline artifacts](#c2w3-8.3.)\n",
    "    - [Exercise 8](#c2w3-ex-8)\n",
    "- [9. Deploy and test the model](#c2w3-9.)\n",
    "  - [9.1. Approve trained model](#c2w3-9.1.)\n",
    "  - [9.2. Deploy model](#c2w3-9.2.)\n",
    "  - [9.3. Create endpoint from registry](#c2w3-9.3.)\n",
    "  - [9.4. Test model](#c2w3-9.4.)\n",
    "  - [9.5. SageMaker Studio extensions](#c2w3-9.5.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Terminology**\n",
    "\n",
    "This notebook focuses on the following features of Amazon SageMaker Pipelines:\n",
    "\n",
    "* **Pipelines** - a directed acyclic graph (DAG) of steps and conditions to orchestrate SageMaker jobs and resource creation\n",
    "* **Processing job steps** - a simplified, managed experience on SageMaker to run data processing workloads, such as feature engineering, data validation, model evaluation, and model explainability\n",
    "* **Training job steps** - an iterative process that teaches a model to make predictions on new data by presenting examples from a training dataset\n",
    "* **Conditional step execution** - provides conditional execution of branches in a pipeline\n",
    "* **Registering models** - register a model in a model registry to create a deployable models in Amazon SageMaker\n",
    "* **Parameterized pipeline executions** - allows pipeline executions to vary by supplied parameters\n",
    "* **Model endpoint** - hosts the model as a REST endpoint to serve predictions from new data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**BERT Pipeline**\n",
    "\n",
    "The pipeline that you will create follows a typical machine learning application pattern of pre-processing, training, evaluation, and model registration.\n",
    "\n",
    "In the processing step, you will perform feature engineering to transform the `review_body` text into BERT embeddings using the pre-trained BERT model and split the dataset into train, validation and test files. The transformed dataset is stored in a feature store. To optimize for Tensorflow training, the transformed dataset files are saved using the TFRecord format in Amazon S3.\n",
    "\n",
    "In the training step, you will fine-tune the BERT model to the customer reviews dataset and add a new classification layer to predict the `sentiment` for a given `review_body`.\n",
    "\n",
    "In the evaluation step, you will take the trained model and a test dataset as input, and produce a JSON file containing classification evaluation metrics.\n",
    "\n",
    "In the condition step, you will register the trained model if the accuracy of the model, as determined by our evaluation step, exceeds a given threshold value. \n",
    "\n",
    "![](./images/bert_sagemaker_pipeline.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, install the required modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "sparkmagic 0.20.4 requires nest-asyncio==1.5.5, but you have nest-asyncio 1.5.7 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# please ignore warning messages during the installation\n",
    "!pip install --disable-pip-version-check -q sagemaker==2.35.0\n",
    "!pip install -q protobuf==3.20.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/boto3/compat.py:82: PythonDeprecationWarning: Boto3 will no longer support Python 3.7 starting December 13, 2023. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.8 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/\n",
      "  warnings.warn(warning, PythonDeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import sagemaker\n",
    "import logging\n",
    "import boto3\n",
    "import sagemaker\n",
    "import pandas as pd\n",
    "import json\n",
    "import botocore\n",
    "from botocore.exceptions import ClientError\n",
    "\n",
    "config = botocore.config.Config(user_agent_extra='dlai-pds/c2/w3')\n",
    "\n",
    "# low-level service client of the boto3 session\n",
    "sm = boto3.client(service_name='sagemaker', \n",
    "                  config=config)\n",
    "\n",
    "sm_runtime = boto3.client('sagemaker-runtime',\n",
    "                          config=config)\n",
    "\n",
    "sess = sagemaker.Session(sagemaker_client=sm,\n",
    "                         sagemaker_runtime_client=sm_runtime)\n",
    "\n",
    "bucket = sess.default_bucket()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = sess.boto_region_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setup the pipeline name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "timestamp = int(time.time())\n",
    "\n",
    "pipeline_name = 'BERT-pipeline-{}'.format(timestamp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-1.'></a>\n",
    "# 1. Configure the dataset and processing step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-1.1.'></a>\n",
    "### 1.1. Configure S3 path for raw input data\n",
    "\n",
    "The raw dataset is in the public S3 bucket. Let's start by specifying the S3 location of it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://dlai-practical-data-science/data/raw/\n"
     ]
    }
   ],
   "source": [
    "raw_input_data_s3_uri = 's3://dlai-practical-data-science/data/raw/'\n",
    "print(raw_input_data_s3_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List the files in the S3 bucket (in this case it will be just one file):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-30 02:21:06    8457214 womens_clothing_ecommerce_reviews.csv\n"
     ]
    }
   ],
   "source": [
    "!aws s3 ls $raw_input_data_s3_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-1.2.'></a>\n",
    "### 1.2. Configure processing step\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the pipeline workflow you will need to create workflow parameters of a specific type: integer, string, or float."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.parameters import (\n",
    "    ParameterInteger,\n",
    "    ParameterString,\n",
    "    ParameterFloat,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now set the parameters for the processing step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "processing_instance_type = ParameterString(\n",
    "    name=\"ProcessingInstanceType\",\n",
    "    default_value=\"ml.c5.2xlarge\"\n",
    ")\n",
    "\n",
    "processing_instance_count = ParameterInteger(\n",
    "    name=\"ProcessingInstanceCount\",\n",
    "    default_value=1\n",
    ")\n",
    "\n",
    "train_split_percentage = ParameterFloat(\n",
    "    name=\"TrainSplitPercentage\",\n",
    "    default_value=0.90,\n",
    ")\n",
    "\n",
    "validation_split_percentage = ParameterFloat(\n",
    "    name=\"ValidationSplitPercentage\",\n",
    "    default_value=0.05,\n",
    ")\n",
    "\n",
    "test_split_percentage = ParameterFloat(\n",
    "    name=\"TestSplitPercentage\",\n",
    "    default_value=0.05,\n",
    ")\n",
    "\n",
    "balance_dataset = ParameterString(\n",
    "    name=\"BalanceDataset\",\n",
    "    default_value=\"True\",\n",
    ")\n",
    "\n",
    "max_seq_length = ParameterInteger(\n",
    "    name=\"MaxSeqLength\",\n",
    "    default_value=128,\n",
    ")\n",
    "\n",
    "feature_store_offline_prefix = ParameterString(\n",
    "    name=\"FeatureStoreOfflinePrefix\",\n",
    "    default_value=\"reviews-feature-store-\" + str(timestamp),\n",
    ")\n",
    "\n",
    "feature_group_name = ParameterString(\n",
    "    name=\"FeatureGroupName\",\n",
    "    default_value=\"reviews-feature-group-\" + str(timestamp)\n",
    ")\n",
    "\n",
    "input_data = ParameterString(\n",
    "    name=\"InputData\",\n",
    "    default_value=raw_input_data_s3_uri,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting up scikit-learn-based processor, pass the SageMaker execution role, processing instance type and instance count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.sklearn.processing import SKLearnProcessor\n",
    "\n",
    "processor = SKLearnProcessor(\n",
    "    framework_version='0.23-1',\n",
    "    role=role,\n",
    "    instance_type=processing_instance_type,\n",
    "    instance_count=processing_instance_count,\n",
    "    env={'AWS_DEFAULT_REGION': region},                             \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you will use the processor instance to construct a `ProcessingStep`, along with the input and output channels and the code that will be executed when the pipeline invokes pipeline execution. This is very similar to a processor instance's `run` method, for those familiar with the existing Python SDK.\n",
    "\n",
    "Note the `\"sentiment-train\"`, `\"sentiment-validation\"` and `\"sentiment-test\"` named channels specified in the output configuration for the processing job. Such step `Properties` can be used in subsequent steps and will resolve to their runtime values at execution. In particular, you will call out this usage defining the training step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ProcessingStep(name='Processing', step_type=<StepTypeEnum.PROCESSING: 'Processing'>)\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "from sagemaker.workflow.steps import ProcessingStep\n",
    "\n",
    "processing_inputs=[\n",
    "    ProcessingInput(\n",
    "        input_name='raw-input-data',\n",
    "        source=input_data,\n",
    "        destination='/opt/ml/processing/input/data/',\n",
    "        s3_data_distribution_type='ShardedByS3Key'\n",
    "    )\n",
    "]\n",
    "\n",
    "processing_outputs=[\n",
    "    ProcessingOutput(output_name='sentiment-train',\n",
    "                     source='/opt/ml/processing/output/sentiment/train',\n",
    "                     s3_upload_mode='EndOfJob'),\n",
    "    ProcessingOutput(output_name='sentiment-validation',\n",
    "                     source='/opt/ml/processing/output/sentiment/validation',\n",
    "                     s3_upload_mode='EndOfJob'),\n",
    "    ProcessingOutput(output_name='sentiment-test',\n",
    "                     source='/opt/ml/processing/output/sentiment/test',\n",
    "                     s3_upload_mode='EndOfJob')\n",
    "]        \n",
    "\n",
    "processing_step = ProcessingStep(\n",
    "    name='Processing', \n",
    "    code='src/prepare_data.py',\n",
    "    processor=processor,\n",
    "    inputs=processing_inputs,\n",
    "    outputs=processing_outputs,\n",
    "    job_arguments=['--train-split-percentage', str(train_split_percentage.default_value),                   \n",
    "                   '--validation-split-percentage', str(validation_split_percentage.default_value),\n",
    "                   '--test-split-percentage', str(test_split_percentage.default_value),\n",
    "                   '--balance-dataset', str(balance_dataset.default_value),\n",
    "                   '--max-seq-length', str(max_seq_length.default_value),                   \n",
    "                   '--feature-store-offline-prefix', str(feature_store_offline_prefix.default_value),\n",
    "                   '--feature-group-name', str(feature_group_name.default_value)\n",
    "                  ]\n",
    ")        \n",
    "\n",
    "print(processing_step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can call out the properties of the processing job as an object using the command `processing_step.properties`. To print out and explore the attributes use `__dict__` method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"AppSpecification\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721fe1d0>\",\n",
      "    \"AutoMLJobArn\": \"<sagemaker.workflow.properties.Properties object at 0x7fab72203150>\",\n",
      "    \"CreationTime\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721feed0>\",\n",
      "    \"Environment\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721fe3d0>\",\n",
      "    \"ExitMessage\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721fecd0>\",\n",
      "    \"ExperimentConfig\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721fe0d0>\",\n",
      "    \"FailureReason\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721feb90>\",\n",
      "    \"LastModifiedTime\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721fefd0>\",\n",
      "    \"MonitoringScheduleArn\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721fe890>\",\n",
      "    \"NetworkConfig\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721fea50>\",\n",
      "    \"ProcessingEndTime\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721fee90>\",\n",
      "    \"ProcessingInputs\": \"<sagemaker.workflow.properties.PropertiesList object at 0x7fab7220e890>\",\n",
      "    \"ProcessingJobArn\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721fea90>\",\n",
      "    \"ProcessingJobName\": \"<sagemaker.workflow.properties.Properties object at 0x7fab7220ef50>\",\n",
      "    \"ProcessingJobStatus\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721fead0>\",\n",
      "    \"ProcessingOutputConfig\": \"<sagemaker.workflow.properties.Properties object at 0x7fab7220ef90>\",\n",
      "    \"ProcessingResources\": \"<sagemaker.workflow.properties.Properties object at 0x7fab7220ed50>\",\n",
      "    \"ProcessingStartTime\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721fedd0>\",\n",
      "    \"RoleArn\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721fe190>\",\n",
      "    \"StoppingCondition\": \"<sagemaker.workflow.properties.Properties object at 0x7fab721fe350>\",\n",
      "    \"TrainingJobArn\": \"<sagemaker.workflow.properties.Properties object at 0x7fab72203290>\",\n",
      "    \"_path\": \"Steps.Processing\",\n",
      "    \"_shape_name\": \"DescribeProcessingJobResponse\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# print out the list of the processing job properties\n",
    "print(json.dumps(\n",
    "    processing_step.properties.__dict__,\n",
    "    indent=4, sort_keys=True, default=str\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pull the channel `sentiment-train` from the output configuration of the processing job. Print out the attributes of the resulting object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"AppManaged\": \"<sagemaker.workflow.properties.Properties object at 0x7fab72227750>\",\n",
      "    \"FeatureStoreOutput\": \"<sagemaker.workflow.properties.Properties object at 0x7fab72227550>\",\n",
      "    \"OutputName\": \"<sagemaker.workflow.properties.Properties object at 0x7fab72227a50>\",\n",
      "    \"S3Output\": \"<sagemaker.workflow.properties.Properties object at 0x7fab72227450>\",\n",
      "    \"_path\": \"Steps.Processing.ProcessingOutputConfig.Outputs['sentiment-train']\",\n",
      "    \"_shape_name\": \"ProcessingOutput\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(\n",
    "    processing_step.properties.ProcessingOutputConfig.Outputs['sentiment-train'].__dict__, \n",
    "    indent=4, sort_keys=True, default=str\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can pull and print out attributes of the S3 output path related to the `sentiment-train` output channel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"__str__\": \"S3Uri\",\n",
      "    \"_path\": \"Steps.Processing.ProcessingOutputConfig.Outputs['sentiment-train'].S3Output.S3Uri\",\n",
      "    \"_shape_name\": \"S3Uri\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(\n",
    "    processing_step.properties.ProcessingOutputConfig.Outputs['sentiment-train'].S3Output.S3Uri.__dict__,\n",
    "    indent=4, sort_keys=True, default=str\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-ex-1'></a>\n",
    "### Exercise 1\n",
    "\n",
    "Pull and print out attributes of the S3 output path object related to the `sentiment-test` output channel.\n",
    "\n",
    "**Instructions**: Use the example in the cell above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"AppManaged\": \"<sagemaker.workflow.properties.Properties object at 0x7fab7222be50>\",\n",
      "    \"FeatureStoreOutput\": \"<sagemaker.workflow.properties.Properties object at 0x7fab7222b2d0>\",\n",
      "    \"OutputName\": \"<sagemaker.workflow.properties.Properties object at 0x7fab7222bd50>\",\n",
      "    \"S3Output\": \"<sagemaker.workflow.properties.Properties object at 0x7fab7222b550>\",\n",
      "    \"_path\": \"Steps.Processing.ProcessingOutputConfig.Outputs['sentiment-test']\",\n",
      "    \"_shape_name\": \"ProcessingOutput\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    processing_step.properties.ProcessingOutputConfig.Outputs['sentiment-test'].__dict__, # Replace all None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    indent=4, sort_keys=True, default=str\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These objects can be passed into the next steps of the workflow. Also, you can pull the arguments of the processing step with the corresponding function. The result is in the dictionary format. Review the keys of this dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['ProcessingResources', 'AppSpecification', 'RoleArn', 'ProcessingInputs', 'ProcessingOutputConfig', 'Environment'])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "processing_step.arguments.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pull and review processing inputs from the arguments of the processing step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'InputName': 'raw-input-data',\n",
       "  'AppManaged': False,\n",
       "  'S3Input': {'S3Uri': ParameterString(name='InputData', parameter_type=<ParameterTypeEnum.STRING: 'String'>, default_value='s3://dlai-practical-data-science/data/raw/'),\n",
       "   'LocalPath': '/opt/ml/processing/input/data/',\n",
       "   'S3DataType': 'S3Prefix',\n",
       "   'S3InputMode': 'File',\n",
       "   'S3DataDistributionType': 'ShardedByS3Key',\n",
       "   'S3CompressionType': 'None'}},\n",
       " {'InputName': 'code',\n",
       "  'AppManaged': False,\n",
       "  'S3Input': {'S3Uri': 's3://sagemaker-us-east-1-656717427230/sagemaker-scikit-learn-2023-09-17-18-36-55-860/input/code/prepare_data.py',\n",
       "   'LocalPath': '/opt/ml/processing/input/code',\n",
       "   'S3DataType': 'S3Prefix',\n",
       "   'S3InputMode': 'File',\n",
       "   'S3DataDistributionType': 'FullyReplicated',\n",
       "   'S3CompressionType': 'None'}}]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "processing_step.arguments['ProcessingInputs']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-ex-2'></a>\n",
    "### Exercise 2\n",
    "\n",
    "Pull and review configuration of the processing outputs from the arguments of the processing step.\n",
    "\n",
    "**Instructions**: Find the required key in the `arguments` dictionary and pull the corresponding value following the example above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Outputs': [{'OutputName': 'sentiment-train',\n",
       "   'AppManaged': False,\n",
       "   'S3Output': {'S3Uri': 's3://sagemaker-us-east-1-656717427230/sagemaker-scikit-learn-2023-09-17-18-36-50-637/output/sentiment-train',\n",
       "    'LocalPath': '/opt/ml/processing/output/sentiment/train',\n",
       "    'S3UploadMode': 'EndOfJob'}},\n",
       "  {'OutputName': 'sentiment-validation',\n",
       "   'AppManaged': False,\n",
       "   'S3Output': {'S3Uri': 's3://sagemaker-us-east-1-656717427230/sagemaker-scikit-learn-2023-09-17-18-36-50-637/output/sentiment-validation',\n",
       "    'LocalPath': '/opt/ml/processing/output/sentiment/validation',\n",
       "    'S3UploadMode': 'EndOfJob'}},\n",
       "  {'OutputName': 'sentiment-test',\n",
       "   'AppManaged': False,\n",
       "   'S3Output': {'S3Uri': 's3://sagemaker-us-east-1-656717427230/sagemaker-scikit-learn-2023-09-17-18-36-50-637/output/sentiment-test',\n",
       "    'LocalPath': '/opt/ml/processing/output/sentiment/test',\n",
       "    'S3UploadMode': 'EndOfJob'}}]}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "processing_step.arguments['ProcessingOutputConfig'] # Replace None\n",
    "### END SOLUTION - DO NOT delete this comment for grading purposes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-2.'></a>\n",
    "# 2. Configure training step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-2.1.'></a>\n",
    "### 2.1. Define parameters\n",
    "\n",
    "Setup the parameters for the workflow. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "freeze_bert_layer = ParameterString(\n",
    "    name=\"FreezeBertLayer\",\n",
    "    default_value=\"False\",\n",
    ")\n",
    "\n",
    "epochs = ParameterInteger(\n",
    "    name=\"Epochs\",\n",
    "    default_value=3\n",
    ")\n",
    "    \n",
    "learning_rate = ParameterFloat(\n",
    "    name=\"LearningRate\",\n",
    "    default_value=0.00001\n",
    ") \n",
    "    \n",
    "train_batch_size = ParameterInteger(\n",
    "    name=\"TrainBatchSize\",\n",
    "    default_value=64\n",
    ")\n",
    "\n",
    "train_steps_per_epoch = ParameterInteger(\n",
    "    name=\"TrainStepsPerEpoch\",\n",
    "    default_value=50\n",
    ")\n",
    "\n",
    "validation_batch_size = ParameterInteger(\n",
    "    name=\"ValidationBatchSize\",\n",
    "    default_value=64\n",
    ")\n",
    "\n",
    "validation_steps_per_epoch = ParameterInteger(\n",
    "    name=\"ValidationStepsPerEpoch\",\n",
    "    default_value=50\n",
    ")\n",
    "\n",
    "seed = ParameterInteger(\n",
    "    name=\"Seed\",\n",
    "    default_value=42\n",
    ")\n",
    "\n",
    "run_validation = ParameterString(\n",
    "    name=\"RunValidation\",\n",
    "    default_value=\"True\",\n",
    ")\n",
    "\n",
    "train_instance_count = ParameterInteger(\n",
    "    name=\"TrainInstanceCount\",\n",
    "    default_value=1\n",
    ")\n",
    "\n",
    "train_instance_type = ParameterString(\n",
    "    name=\"TrainInstanceType\",\n",
    "    default_value=\"ml.c5.9xlarge\"\n",
    ")\n",
    "\n",
    "train_volume_size = ParameterInteger(\n",
    "    name=\"TrainVolumeSize\",\n",
    "    default_value=256\n",
    ") \n",
    "\n",
    "input_mode = ParameterString(\n",
    "    name=\"InputMode\",\n",
    "    default_value=\"File\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-2.2.'></a>\n",
    "### 2.2. Configure hyper-parameters\n",
    "\n",
    "Setup the dictionary that will be passed into the hyperparameters argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hyperparameters={\n",
    "    'max_seq_length': max_seq_length,\n",
    "    'freeze_bert_layer': freeze_bert_layer,\n",
    "    'epochs': epochs,\n",
    "    'learning_rate': learning_rate,\n",
    "    'train_batch_size': train_batch_size,\n",
    "    'train_steps_per_epoch': train_steps_per_epoch,\n",
    "    'validation_batch_size': validation_batch_size,\n",
    "    'validation_steps_per_epoch': validation_steps_per_epoch,\n",
    "    'seed': seed,\n",
    "    'run_validation': run_validation\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-2.3.'></a>\n",
    "### 2.3. Configure model-evaluation metrics\n",
    "\n",
    "Choose loss and accuracy as the evaluation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "metric_definitions = [\n",
    "     {'Name': 'validation:loss', 'Regex': 'val_loss: ([0-9.]+)'},\n",
    "     {'Name': 'validation:accuracy', 'Regex': 'val_acc: ([0-9.]+)'},\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, these sample log lines...\n",
    "```\n",
    "[step: 100] val_loss: 0.55 - val_acc: 74.64%\n",
    "```\n",
    "...will produce the following metrics in CloudWatch:\n",
    "\n",
    "`validation:loss` =  0.55\n",
    "\n",
    "`validation:accuracy` = 74.64\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-2.4.'></a>\n",
    "### 2.4. Configure the `PyTorchEstimator`\n",
    "\n",
    "Configure an estimator and the input dataset. A typical training script loads data from the input channels, configures training with hyperparameters, trains a model, and saves a model to `model_dir` so that it can be hosted later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.pytorch import PyTorch as PyTorchEstimator\n",
    "\n",
    "estimator = PyTorchEstimator(\n",
    "    entry_point='train.py',\n",
    "    source_dir='src',\n",
    "    role=role,\n",
    "    instance_count=train_instance_count,\n",
    "    instance_type=train_instance_type,\n",
    "    volume_size=train_volume_size,\n",
    "    py_version='py3',\n",
    "    framework_version='1.6.0',\n",
    "    hyperparameters=hyperparameters,\n",
    "    metric_definitions=metric_definitions,\n",
    "    input_mode=input_mode\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-2.5.'></a>\n",
    "### 2.5. Setup pipeline step caching\n",
    "Step signature caching allows SageMaker Pipelines, before executing a step, to find a previous execution of a step that was called using the same arguments. Cache hit gets created if the previous execution is found. Then during execution instead of recomputing the step, pipelines propagates the values from the cache hit.\n",
    "\n",
    "Timeout period is defined using [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) format, it can contain a year, month, week, day, hour, and minute value. \n",
    "\n",
    "More details on SageMaker Pipeline step caching can be found [here](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-caching.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.steps import CacheConfig\n",
    "\n",
    "cache_config = CacheConfig(enable_caching=True, expire_after=\"PT1H\") # PT1H represents `one hour`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-2.6.'></a>\n",
    "### 2.6. Configure the `TrainingStep`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now configure the `TrainingStep` calling the outputs of the processing step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TrainingStep(name='Train', step_type=<StepTypeEnum.TRAINING: 'Training'>)\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.inputs import TrainingInput\n",
    "from sagemaker.workflow.steps import TrainingStep\n",
    "\n",
    "training_step = TrainingStep(\n",
    "    name='Train',\n",
    "    estimator=estimator,\n",
    "    inputs={\n",
    "        'train': TrainingInput(\n",
    "            s3_data=processing_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                'sentiment-train'\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type='text/csv'\n",
    "        ),\n",
    "        'validation': TrainingInput(\n",
    "            s3_data=processing_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                'sentiment-validation'\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type='text/csv'\n",
    "        )\n",
    "    },\n",
    "    cache_config=cache_config\n",
    ")\n",
    "\n",
    "print(training_step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-ex-3'></a>\n",
    "### Exercise 3\n",
    "\n",
    "Use `__dict__` method to print out attributes of the training step properties. Briefly review the result. The attributes match the object model of the [DescribeTrainingJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeTrainingJob.html) response object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'_path': 'Steps.Train',\n",
       " '_shape_name': 'DescribeTrainingJobResponse',\n",
       " 'TrainingJobName': <sagemaker.workflow.properties.Properties at 0x7fab71df53d0>,\n",
       " 'TrainingJobArn': <sagemaker.workflow.properties.Properties at 0x7fab71df5410>,\n",
       " 'TuningJobArn': <sagemaker.workflow.properties.Properties at 0x7fab71df5450>,\n",
       " 'LabelingJobArn': <sagemaker.workflow.properties.Properties at 0x7fab71df5490>,\n",
       " 'AutoMLJobArn': <sagemaker.workflow.properties.Properties at 0x7fab71df54d0>,\n",
       " 'ModelArtifacts': <sagemaker.workflow.properties.Properties at 0x7fab71df5510>,\n",
       " 'TrainingJobStatus': <sagemaker.workflow.properties.Properties at 0x7fab71df5590>,\n",
       " 'SecondaryStatus': <sagemaker.workflow.properties.Properties at 0x7fab71df55d0>,\n",
       " 'FailureReason': <sagemaker.workflow.properties.Properties at 0x7fab71df5610>,\n",
       " 'HyperParameters': <sagemaker.workflow.properties.Properties at 0x7fab71df5650>,\n",
       " 'AlgorithmSpecification': <sagemaker.workflow.properties.Properties at 0x7fab71df5690>,\n",
       " 'RoleArn': <sagemaker.workflow.properties.Properties at 0x7fab71df5990>,\n",
       " 'InputDataConfig': <sagemaker.workflow.properties.PropertiesList at 0x7fab71df59d0>,\n",
       " 'OutputDataConfig': <sagemaker.workflow.properties.Properties at 0x7fab71df5a10>,\n",
       " 'ResourceConfig': <sagemaker.workflow.properties.Properties at 0x7fab71df5b10>,\n",
       " 'VpcConfig': <sagemaker.workflow.properties.Properties at 0x7fab71df5cd0>,\n",
       " 'StoppingCondition': <sagemaker.workflow.properties.Properties at 0x7fab71df5d90>,\n",
       " 'CreationTime': <sagemaker.workflow.properties.Properties at 0x7fab71df5e50>,\n",
       " 'TrainingStartTime': <sagemaker.workflow.properties.Properties at 0x7fab71df5e90>,\n",
       " 'TrainingEndTime': <sagemaker.workflow.properties.Properties at 0x7fab71df5ed0>,\n",
       " 'LastModifiedTime': <sagemaker.workflow.properties.Properties at 0x7fab71df5f10>,\n",
       " 'SecondaryStatusTransitions': <sagemaker.workflow.properties.PropertiesList at 0x7fab71df5f50>,\n",
       " 'FinalMetricDataList': <sagemaker.workflow.properties.PropertiesList at 0x7fab71df5f90>,\n",
       " 'EnableNetworkIsolation': <sagemaker.workflow.properties.Properties at 0x7fab71df5fd0>,\n",
       " 'EnableInterContainerTrafficEncryption': <sagemaker.workflow.properties.Properties at 0x7fab7223ff50>,\n",
       " 'EnableManagedSpotTraining': <sagemaker.workflow.properties.Properties at 0x7fab7223ff10>,\n",
       " 'CheckpointConfig': <sagemaker.workflow.properties.Properties at 0x7fab7223fe50>,\n",
       " 'TrainingTimeInSeconds': <sagemaker.workflow.properties.Properties at 0x7fab7223fd90>,\n",
       " 'BillableTimeInSeconds': <sagemaker.workflow.properties.Properties at 0x7fab7223fed0>,\n",
       " 'DebugHookConfig': <sagemaker.workflow.properties.Properties at 0x7fab7223fe90>,\n",
       " 'ExperimentConfig': <sagemaker.workflow.properties.Properties at 0x7fab7223fd50>,\n",
       " 'DebugRuleConfigurations': <sagemaker.workflow.properties.PropertiesList at 0x7fab7223fbd0>,\n",
       " 'TensorBoardOutputConfig': <sagemaker.workflow.properties.Properties at 0x7fab7223fb90>,\n",
       " 'DebugRuleEvaluationStatuses': <sagemaker.workflow.properties.PropertiesList at 0x7fab7223fc50>,\n",
       " 'ProfilerConfig': <sagemaker.workflow.properties.Properties at 0x7fab7223f910>,\n",
       " 'ProfilerRuleConfigurations': <sagemaker.workflow.properties.PropertiesList at 0x7fab7223f950>,\n",
       " 'ProfilerRuleEvaluationStatuses': <sagemaker.workflow.properties.PropertiesList at 0x7fab7223f990>,\n",
       " 'ProfilingStatus': <sagemaker.workflow.properties.Properties at 0x7fab7223f6d0>,\n",
       " 'RetryStrategy': <sagemaker.workflow.properties.Properties at 0x7fab7223f890>,\n",
       " 'Environment': <sagemaker.workflow.properties.Properties at 0x7fab7223f610>,\n",
       " 'WarmPoolStatus': <sagemaker.workflow.properties.Properties at 0x7fab7223f7d0>}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "training_step.properties.__dict__ # Replace all None\n",
    "### END SOLUTION - DO NOT delete this comment for grading purposes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-3.'></a>\n",
    "# 3. Configure model-evaluation step\n",
    "\n",
    "First, develop an evaluation script that will be specified in the model evaluation processing step. The evaluation script users the trained model and the test dataset to produce a JSON file with classification evaluation metrics such as accuracy.\n",
    "\n",
    "After pipeline execution, you will examine the resulting `evaluation.json` for analysis.\n",
    "\n",
    "The evaluation script performs the following steps:\n",
    "* loads in the model\n",
    "* reads in the test data\n",
    "* issues a bunch of predictions against the test data\n",
    "* builds a classification report, including accuracy\n",
    "* saves the evaluation report to the evaluation directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an instance of the `SKLearnProcessor` to run our evaluation script as a scikit-learn-based SageMaker processing job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.sklearn.processing import SKLearnProcessor\n",
    "\n",
    "evaluation_processor = SKLearnProcessor(\n",
    "    framework_version='0.23-1',\n",
    "    role=role,\n",
    "    instance_type=processing_instance_type,\n",
    "    instance_count=processing_instance_count,\n",
    "    env={'AWS_DEFAULT_REGION': region},\n",
    "    max_runtime_in_seconds=7200\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setup the output `PropertyFile`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.properties import PropertyFile\n",
    "\n",
    "evaluation_report = PropertyFile(\n",
    "    name='EvaluationReport',\n",
    "    output_name='metrics',\n",
    "    path='evaluation.json'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the processor instance to construct a `ProcessingStep`, along with the input and output channels and the code that will be executed when the pipeline invokes pipeline execution. This is very similar to a processor instance's `run` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "\n",
    "evaluation_step = ProcessingStep(\n",
    "    name='EvaluateModel',\n",
    "    processor=evaluation_processor,\n",
    "    code='src/evaluate_model_metrics.py',\n",
    "    inputs=[\n",
    "        ProcessingInput(\n",
    "            source=training_step.properties.ModelArtifacts.S3ModelArtifacts,\n",
    "            destination='/opt/ml/processing/input/model'\n",
    "        ),\n",
    "        ProcessingInput(\n",
    "            source=processing_step.properties.ProcessingOutputConfig.Outputs['sentiment-test'].S3Output.S3Uri,\n",
    "            destination='/opt/ml/processing/input/data'\n",
    "        )\n",
    "    ],\n",
    "    outputs=[\n",
    "        ProcessingOutput(output_name='metrics', \n",
    "                         s3_upload_mode='EndOfJob',\n",
    "                         source='/opt/ml/processing/output/metrics/'),\n",
    "    ],\n",
    "    job_arguments=[\n",
    "        '--max-seq-length', str(max_seq_length.default_value),\n",
    "    ],\n",
    "    property_files=[evaluation_report],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-4.'></a>\n",
    "# 4. Configure and register model step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-4.1.'></a>\n",
    "### 4.1. Configure the model for deployment\n",
    "\n",
    "Use the estimator instance that was used for the training step to construct an instance of `RegisterModel`. The result of executing `RegisterModel` in a pipeline is a model package. A model package is a reusable model artifacts abstraction that packages all ingredients necessary for inference. Primarily, it consists of an inference specification that defines the inference image to use along with an optional model weights location.\n",
    "\n",
    "A model package group is a collection of model packages. You can create a model package group for a specific ML business problem, and you can keep adding versions/model packages into it. Typically, customers are expected to create a ModelPackageGroup for a SageMaker workflow pipeline so that they can keep adding versions/model packages to the group for every workflow pipeline run.\n",
    "\n",
    "The construction of `RegisterModel` is very similar to an estimator instance's `register` method, for those familiar with the existing Python SDK.\n",
    "\n",
    "In particular, you will pass in the `S3ModelArtifacts` from the `training_step` properties.\n",
    "\n",
    "Of note, here you will be provided a specific model package group name which will be used in the Model Registry and Continuous Integration/Continuous Deployment (CI/CD) work later on. Let's setup the variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_approval_status = ParameterString(\n",
    "    name=\"ModelApprovalStatus\",\n",
    "    default_value=\"PendingManualApproval\"\n",
    ")\n",
    "\n",
    "deploy_instance_type = ParameterString(\n",
    "    name=\"DeployInstanceType\",\n",
    "    default_value=\"ml.m5.large\"\n",
    ")\n",
    "\n",
    "deploy_instance_count = ParameterInteger(\n",
    "    name=\"DeployInstanceCount\",\n",
    "    default_value=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BERT-Reviews-1694975588\n"
     ]
    }
   ],
   "source": [
    "model_package_group_name = f\"BERT-Reviews-{timestamp}\"\n",
    "\n",
    "print(model_package_group_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Configure the `ModelMetrics` to be stored as metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<sagemaker.model_metrics.ModelMetrics object at 0x7fab7005e7d0>\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.model_metrics import MetricsSource, ModelMetrics \n",
    "\n",
    "model_metrics = ModelMetrics(\n",
    "    model_statistics=MetricsSource(\n",
    "        s3_uri=\"{}/evaluation.json\".format(\n",
    "            evaluation_step.arguments[\"ProcessingOutputConfig\"][\"Outputs\"][0][\"S3Output\"][\"S3Uri\"]\n",
    "        ),\n",
    "        content_type=\"application/json\"\n",
    "    )\n",
    ")\n",
    "\n",
    "print(model_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define deployment image for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:1.6.0-cpu-py36\n"
     ]
    }
   ],
   "source": [
    "inference_image_uri = sagemaker.image_uris.retrieve(\n",
    "    framework=\"pytorch\",\n",
    "    region=region,\n",
    "    version=\"1.6.0\",\n",
    "    py_version=\"py36\",\n",
    "    instance_type=deploy_instance_type,\n",
    "    image_scope=\"inference\"\n",
    ")\n",
    "print(inference_image_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-4.2.'></a>\n",
    "### 4.2. Register the model for deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-ex-4'></a>\n",
    "### Exercise 4\n",
    "\n",
    "Configure the register model step.\n",
    "\n",
    "**Instructions**: Pass the inference image defined above into the `image_uri` argument of the function `RegisterModel`. Review the rest of the arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.step_collections import RegisterModel\n",
    "\n",
    "register_step = RegisterModel(\n",
    "    name=\"RegisterModel\",\n",
    "    estimator=estimator,\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    image_uri=inference_image_uri, # Replace None \n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    model_data=training_step.properties.ModelArtifacts.S3ModelArtifacts,\n",
    "    content_types=[\"application/jsonlines\"],\n",
    "    response_types=[\"application/jsonlines\"],\n",
    "    inference_instances=[deploy_instance_type],\n",
    "    transform_instances=[deploy_instance_type], # batch transform is not used in this lab\n",
    "    model_package_group_name=model_package_group_name,\n",
    "    approval_status=model_approval_status,\n",
    "    model_metrics=model_metrics\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-5.'></a>\n",
    "# 5. Create model for deployment step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-ex-5'></a>\n",
    "### Exercise 5\n",
    "\n",
    "Configure model for deployment.\n",
    "\n",
    "**Instructions**: Pass the same inference image into the `image_uri` argument of the function `Model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.model import Model\n",
    "\n",
    "model_name = 'bert-model-{}'.format(timestamp)\n",
    "\n",
    "model = Model(\n",
    "    name=model_name,\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    image_uri=inference_image_uri, # Replace None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    model_data=training_step.properties.ModelArtifacts.S3ModelArtifacts,\n",
    "    sagemaker_session=sess,\n",
    "    role=role,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now configure create model input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.inputs import CreateModelInput\n",
    "\n",
    "create_inputs = CreateModelInput(\n",
    "    instance_type=deploy_instance_type, \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-ex-6'></a>\n",
    "### Exercise 6\n",
    "\n",
    "Configure create model step for the workflow.\n",
    "\n",
    "**Instructions**: Pass defined above model (the model object, not its name) and model inputs configuration into the related arguments of the function `CreateModelStep`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.steps import CreateModelStep\n",
    "\n",
    "create_step = CreateModelStep(\n",
    "    name=\"CreateModel\",\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    model=model, # Replace None\n",
    "    inputs=create_inputs, # Replace None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-6.'></a>\n",
    "# 6. Check accuracy condition step\n",
    "\n",
    "Finally, you would like to only register this model if the accuracy of the model, as determined by our evaluation step `evaluation_step`, exceeded some value. A `ConditionStep` allows for pipelines to support conditional execution in the pipeline DAG based on conditions of step properties. \n",
    "\n",
    "Below, you will:\n",
    "\n",
    "* define a minimum accuracy value as a parameter\n",
    "* define a `ConditionGreaterThan` on the accuracy value found in the output of the evaluation step, `evaluation_step`.\n",
    "* use the condition in the list of conditions in a `ConditionStep`\n",
    "* pass the `RegisterModel` step collection into the `if_steps` of the `ConditionStep`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "min_accuracy_value = ParameterFloat(\n",
    "    name=\"MinAccuracyValue\",\n",
    "    default_value=0.33 # random choice from three classes\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo\n",
    "from sagemaker.workflow.condition_step import (\n",
    "    ConditionStep,\n",
    "    JsonGet,\n",
    ")\n",
    "\n",
    "minimum_accuracy_condition = ConditionGreaterThanOrEqualTo(\n",
    "    left=JsonGet(\n",
    "        step=evaluation_step,\n",
    "        property_file=evaluation_report,\n",
    "        json_path=\"metrics.accuracy.value\",\n",
    "    ),\n",
    "    right=min_accuracy_value # minimum accuracy threshold\n",
    ")\n",
    "\n",
    "minimum_accuracy_condition_step = ConditionStep(\n",
    "    name=\"AccuracyCondition\",\n",
    "    conditions=[minimum_accuracy_condition],\n",
    "    if_steps=[register_step, create_step], # successfully exceeded or equaled the minimum accuracy, continue with model registration\n",
    "    else_steps=[], # did not exceed the minimum accuracy, the model will not be registered\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-7.'></a>\n",
    "# 7. Create pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-7.1.'></a>\n",
    "### 7.1. Define a pipeline of parameters, steps, and conditions\n",
    "\n",
    "Let's tie it all up into a workflow pipeline so you can execute it, and even schedule it.\n",
    "\n",
    "A pipeline requires a `name`, `parameters`, and `steps`. Names must be unique within an `(account, region)` pair so you can append the timestamp to the name to reduce the chance of name conflict.\n",
    "\n",
    "Note:\n",
    "\n",
    "* All the parameters used in the definitions must be present.\n",
    "* Steps passed into the pipeline need not be in the order of execution. The SageMaker workflow service will resolve the _data dependency_ DAG as steps the execution complete.\n",
    "* Steps must be unique to either pipeline step list or a single condition step if/else list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.pipeline import Pipeline\n",
    "\n",
    "pipeline = Pipeline(\n",
    "    name=pipeline_name,\n",
    "    parameters=[\n",
    "        input_data,\n",
    "        processing_instance_count,\n",
    "        processing_instance_type,\n",
    "        max_seq_length,\n",
    "        balance_dataset,\n",
    "        train_split_percentage,\n",
    "        validation_split_percentage,\n",
    "        test_split_percentage,\n",
    "        feature_store_offline_prefix,\n",
    "        feature_group_name,\n",
    "        epochs,\n",
    "        learning_rate,\n",
    "        train_batch_size,\n",
    "        train_steps_per_epoch,\n",
    "        validation_batch_size,\n",
    "        validation_steps_per_epoch,\n",
    "        freeze_bert_layer,\n",
    "        seed,\n",
    "        train_instance_count,\n",
    "        train_instance_type,\n",
    "        train_volume_size,        \n",
    "        input_mode,\n",
    "        run_validation,\n",
    "        min_accuracy_value,\n",
    "        model_approval_status,\n",
    "        deploy_instance_type,\n",
    "        deploy_instance_count\n",
    "    ],\n",
    "    steps=[processing_step, training_step, evaluation_step, minimum_accuracy_condition_step],\n",
    "    sagemaker_session=sess,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine the JSON of the pipeline definition that meets the SageMaker Workflow Pipeline DSL specification.\n",
    "\n",
    "By examining the definition, you are also confirming that the pipeline was well-defined, and that the parameters and step properties resolve correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No finished training job found associated with this estimator. Please make sure this estimator is only used for building workflow config\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Metadata': {},\n",
      " 'Parameters': [{'DefaultValue': 's3://dlai-practical-data-science/data/raw/',\n",
      "                 'Name': 'InputData',\n",
      "                 'Type': 'String'},\n",
      "                {'DefaultValue': 1,\n",
      "                 'Name': 'ProcessingInstanceCount',\n",
      "                 'Type': 'Integer'},\n",
      "                {'DefaultValue': 'ml.c5.2xlarge',\n",
      "                 'Name': 'ProcessingInstanceType',\n",
      "                 'Type': 'String'},\n",
      "                {'DefaultValue': 128,\n",
      "                 'Name': 'MaxSeqLength',\n",
      "                 'Type': 'Integer'},\n",
      "                {'DefaultValue': 'True',\n",
      "                 'Name': 'BalanceDataset',\n",
      "                 'Type': 'String'},\n",
      "                {'DefaultValue': 0.9,\n",
      "                 'Name': 'TrainSplitPercentage',\n",
      "                 'Type': 'Float'},\n",
      "                {'DefaultValue': 0.05,\n",
      "                 'Name': 'ValidationSplitPercentage',\n",
      "                 'Type': 'Float'},\n",
      "                {'DefaultValue': 0.05,\n",
      "                 'Name': 'TestSplitPercentage',\n",
      "                 'Type': 'Float'},\n",
      "                {'DefaultValue': 'reviews-feature-store-1694975588',\n",
      "                 'Name': 'FeatureStoreOfflinePrefix',\n",
      "                 'Type': 'String'},\n",
      "                {'DefaultValue': 'reviews-feature-group-1694975588',\n",
      "                 'Name': 'FeatureGroupName',\n",
      "                 'Type': 'String'},\n",
      "                {'DefaultValue': 3, 'Name': 'Epochs', 'Type': 'Integer'},\n",
      "                {'DefaultValue': 1e-05,\n",
      "                 'Name': 'LearningRate',\n",
      "                 'Type': 'Float'},\n",
      "                {'DefaultValue': 64,\n",
      "                 'Name': 'TrainBatchSize',\n",
      "                 'Type': 'Integer'},\n",
      "                {'DefaultValue': 50,\n",
      "                 'Name': 'TrainStepsPerEpoch',\n",
      "                 'Type': 'Integer'},\n",
      "                {'DefaultValue': 64,\n",
      "                 'Name': 'ValidationBatchSize',\n",
      "                 'Type': 'Integer'},\n",
      "                {'DefaultValue': 50,\n",
      "                 'Name': 'ValidationStepsPerEpoch',\n",
      "                 'Type': 'Integer'},\n",
      "                {'DefaultValue': 'False',\n",
      "                 'Name': 'FreezeBertLayer',\n",
      "                 'Type': 'String'},\n",
      "                {'DefaultValue': 42, 'Name': 'Seed', 'Type': 'Integer'},\n",
      "                {'DefaultValue': 1,\n",
      "                 'Name': 'TrainInstanceCount',\n",
      "                 'Type': 'Integer'},\n",
      "                {'DefaultValue': 'ml.c5.9xlarge',\n",
      "                 'Name': 'TrainInstanceType',\n",
      "                 'Type': 'String'},\n",
      "                {'DefaultValue': 256,\n",
      "                 'Name': 'TrainVolumeSize',\n",
      "                 'Type': 'Integer'},\n",
      "                {'DefaultValue': 'File', 'Name': 'InputMode', 'Type': 'String'},\n",
      "                {'DefaultValue': 'True',\n",
      "                 'Name': 'RunValidation',\n",
      "                 'Type': 'String'},\n",
      "                {'DefaultValue': 0.33,\n",
      "                 'Name': 'MinAccuracyValue',\n",
      "                 'Type': 'Float'},\n",
      "                {'DefaultValue': 'PendingManualApproval',\n",
      "                 'Name': 'ModelApprovalStatus',\n",
      "                 'Type': 'String'},\n",
      "                {'DefaultValue': 'ml.m5.large',\n",
      "                 'Name': 'DeployInstanceType',\n",
      "                 'Type': 'String'},\n",
      "                {'DefaultValue': 1,\n",
      "                 'Name': 'DeployInstanceCount',\n",
      "                 'Type': 'Integer'}],\n",
      " 'Steps': [{'Arguments': {'AppSpecification': {'ContainerArguments': ['--train-split-percentage',\n",
      "                                                                      '0.9',\n",
      "                                                                      '--validation-split-percentage',\n",
      "                                                                      '0.05',\n",
      "                                                                      '--test-split-percentage',\n",
      "                                                                      '0.05',\n",
      "                                                                      '--balance-dataset',\n",
      "                                                                      'True',\n",
      "                                                                      '--max-seq-length',\n",
      "                                                                      '128',\n",
      "                                                                      '--feature-store-offline-prefix',\n",
      "                                                                      'reviews-feature-store-1694975588',\n",
      "                                                                      '--feature-group-name',\n",
      "                                                                      'reviews-feature-group-1694975588'],\n",
      "                                               'ContainerEntrypoint': ['python3',\n",
      "                                                                       '/opt/ml/processing/input/code/prepare_data.py'],\n",
      "                                               'ImageUri': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-scikit-learn:0.23-1-cpu-py3'},\n",
      "                          'Environment': {'AWS_DEFAULT_REGION': 'us-east-1'},\n",
      "                          'ProcessingInputs': [{'AppManaged': False,\n",
      "                                                'InputName': 'raw-input-data',\n",
      "                                                'S3Input': {'LocalPath': '/opt/ml/processing/input/data/',\n",
      "                                                            'S3CompressionType': 'None',\n",
      "                                                            'S3DataDistributionType': 'ShardedByS3Key',\n",
      "                                                            'S3DataType': 'S3Prefix',\n",
      "                                                            'S3InputMode': 'File',\n",
      "                                                            'S3Uri': {'Get': 'Parameters.InputData'}}},\n",
      "                                               {'AppManaged': False,\n",
      "                                                'InputName': 'code',\n",
      "                                                'S3Input': {'LocalPath': '/opt/ml/processing/input/code',\n",
      "                                                            'S3CompressionType': 'None',\n",
      "                                                            'S3DataDistributionType': 'FullyReplicated',\n",
      "                                                            'S3DataType': 'S3Prefix',\n",
      "                                                            'S3InputMode': 'File',\n",
      "                                                            'S3Uri': 's3://sagemaker-us-east-1-656717427230/sagemaker-scikit-learn-2023-09-17-18-45-09-224/input/code/prepare_data.py'}}],\n",
      "                          'ProcessingOutputConfig': {'Outputs': [{'AppManaged': False,\n",
      "                                                                  'OutputName': 'sentiment-train',\n",
      "                                                                  'S3Output': {'LocalPath': '/opt/ml/processing/output/sentiment/train',\n",
      "                                                                               'S3UploadMode': 'EndOfJob',\n",
      "                                                                               'S3Uri': 's3://sagemaker-us-east-1-656717427230/sagemaker-scikit-learn-2023-09-17-18-36-50-637/output/sentiment-train'}},\n",
      "                                                                 {'AppManaged': False,\n",
      "                                                                  'OutputName': 'sentiment-validation',\n",
      "                                                                  'S3Output': {'LocalPath': '/opt/ml/processing/output/sentiment/validation',\n",
      "                                                                               'S3UploadMode': 'EndOfJob',\n",
      "                                                                               'S3Uri': 's3://sagemaker-us-east-1-656717427230/sagemaker-scikit-learn-2023-09-17-18-36-50-637/output/sentiment-validation'}},\n",
      "                                                                 {'AppManaged': False,\n",
      "                                                                  'OutputName': 'sentiment-test',\n",
      "                                                                  'S3Output': {'LocalPath': '/opt/ml/processing/output/sentiment/test',\n",
      "                                                                               'S3UploadMode': 'EndOfJob',\n",
      "                                                                               'S3Uri': 's3://sagemaker-us-east-1-656717427230/sagemaker-scikit-learn-2023-09-17-18-36-50-637/output/sentiment-test'}}]},\n",
      "                          'ProcessingResources': {'ClusterConfig': {'InstanceCount': {'Get': 'Parameters.ProcessingInstanceCount'},\n",
      "                                                                    'InstanceType': {'Get': 'Parameters.ProcessingInstanceType'},\n",
      "                                                                    'VolumeSizeInGB': 30}},\n",
      "                          'RoleArn': 'arn:aws:iam::656717427230:role/sagemaker-studio-vpc-firewall-us-east-1-sagemaker-execution-role'},\n",
      "            'Name': 'Processing',\n",
      "            'Type': 'Processing'},\n",
      "           {'Arguments': {'AlgorithmSpecification': {'EnableSageMakerMetricsTimeSeries': True,\n",
      "                                                     'MetricDefinitions': [{'Name': 'validation:loss',\n",
      "                                                                            'Regex': 'val_loss: '\n",
      "                                                                                     '([0-9.]+)'},\n",
      "                                                                           {'Name': 'validation:accuracy',\n",
      "                                                                            'Regex': 'val_acc: '\n",
      "                                                                                     '([0-9.]+)'}],\n",
      "                                                     'TrainingImage': '763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.6.0-cpu-py3',\n",
      "                                                     'TrainingInputMode': {'Get': 'Parameters.InputMode'}},\n",
      "                          'DebugHookConfig': {'CollectionConfigurations': [],\n",
      "                                              'S3OutputPath': 's3://sagemaker-us-east-1-656717427230/'},\n",
      "                          'HyperParameters': {'epochs': '3',\n",
      "                                              'freeze_bert_layer': '\"False\"',\n",
      "                                              'learning_rate': '1e-05',\n",
      "                                              'max_seq_length': '128',\n",
      "                                              'run_validation': '\"True\"',\n",
      "                                              'sagemaker_container_log_level': '20',\n",
      "                                              'sagemaker_job_name': '\"pytorch-training-2023-09-17-18-45-09-388\"',\n",
      "                                              'sagemaker_program': '\"train.py\"',\n",
      "                                              'sagemaker_region': '\"us-east-1\"',\n",
      "                                              'sagemaker_submit_directory': '\"s3://sagemaker-us-east-1-656717427230/pytorch-training-2023-09-17-18-45-09-388/source/sourcedir.tar.gz\"',\n",
      "                                              'seed': '42',\n",
      "                                              'train_batch_size': '64',\n",
      "                                              'train_steps_per_epoch': '50',\n",
      "                                              'validation_batch_size': '64',\n",
      "                                              'validation_steps_per_epoch': '50'},\n",
      "                          'InputDataConfig': [{'ChannelName': 'train',\n",
      "                                               'ContentType': 'text/csv',\n",
      "                                               'DataSource': {'S3DataSource': {'S3DataDistributionType': 'FullyReplicated',\n",
      "                                                                               'S3DataType': 'S3Prefix',\n",
      "                                                                               'S3Uri': {'Get': \"Steps.Processing.ProcessingOutputConfig.Outputs['sentiment-train'].S3Output.S3Uri\"}}}},\n",
      "                                              {'ChannelName': 'validation',\n",
      "                                               'ContentType': 'text/csv',\n",
      "                                               'DataSource': {'S3DataSource': {'S3DataDistributionType': 'FullyReplicated',\n",
      "                                                                               'S3DataType': 'S3Prefix',\n",
      "                                                                               'S3Uri': {'Get': \"Steps.Processing.ProcessingOutputConfig.Outputs['sentiment-validation'].S3Output.S3Uri\"}}}}],\n",
      "                          'OutputDataConfig': {'S3OutputPath': 's3://sagemaker-us-east-1-656717427230/'},\n",
      "                          'ProfilerConfig': {'S3OutputPath': 's3://sagemaker-us-east-1-656717427230/'},\n",
      "                          'ProfilerRuleConfigurations': [{'RuleConfigurationName': 'ProfilerReport-1694976309',\n",
      "                                                          'RuleEvaluatorImage': '503895931360.dkr.ecr.us-east-1.amazonaws.com/sagemaker-debugger-rules:latest',\n",
      "                                                          'RuleParameters': {'rule_to_invoke': 'ProfilerReport'}}],\n",
      "                          'ResourceConfig': {'InstanceCount': {'Get': 'Parameters.TrainInstanceCount'},\n",
      "                                             'InstanceType': {'Get': 'Parameters.TrainInstanceType'},\n",
      "                                             'VolumeSizeInGB': {'Get': 'Parameters.TrainVolumeSize'}},\n",
      "                          'RoleArn': 'arn:aws:iam::656717427230:role/sagemaker-studio-vpc-firewall-us-east-1-sagemaker-execution-role',\n",
      "                          'StoppingCondition': {'MaxRuntimeInSeconds': 86400}},\n",
      "            'CacheConfig': {'Enabled': True, 'ExpireAfter': 'PT1H'},\n",
      "            'Name': 'Train',\n",
      "            'Type': 'Training'},\n",
      "           {'Arguments': {'AppSpecification': {'ContainerArguments': ['--max-seq-length',\n",
      "                                                                      '128'],\n",
      "                                               'ContainerEntrypoint': ['python3',\n",
      "                                                                       '/opt/ml/processing/input/code/evaluate_model_metrics.py'],\n",
      "                                               'ImageUri': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-scikit-learn:0.23-1-cpu-py3'},\n",
      "                          'Environment': {'AWS_DEFAULT_REGION': 'us-east-1'},\n",
      "                          'ProcessingInputs': [{'AppManaged': False,\n",
      "                                                'InputName': 'input-1',\n",
      "                                                'S3Input': {'LocalPath': '/opt/ml/processing/input/model',\n",
      "                                                            'S3CompressionType': 'None',\n",
      "                                                            'S3DataDistributionType': 'FullyReplicated',\n",
      "                                                            'S3DataType': 'S3Prefix',\n",
      "                                                            'S3InputMode': 'File',\n",
      "                                                            'S3Uri': {'Get': 'Steps.Train.ModelArtifacts.S3ModelArtifacts'}}},\n",
      "                                               {'AppManaged': False,\n",
      "                                                'InputName': 'input-2',\n",
      "                                                'S3Input': {'LocalPath': '/opt/ml/processing/input/data',\n",
      "                                                            'S3CompressionType': 'None',\n",
      "                                                            'S3DataDistributionType': 'FullyReplicated',\n",
      "                                                            'S3DataType': 'S3Prefix',\n",
      "                                                            'S3InputMode': 'File',\n",
      "                                                            'S3Uri': {'Get': \"Steps.Processing.ProcessingOutputConfig.Outputs['sentiment-test'].S3Output.S3Uri\"}}},\n",
      "                                               {'AppManaged': False,\n",
      "                                                'InputName': 'code',\n",
      "                                                'S3Input': {'LocalPath': '/opt/ml/processing/input/code',\n",
      "                                                            'S3CompressionType': 'None',\n",
      "                                                            'S3DataDistributionType': 'FullyReplicated',\n",
      "                                                            'S3DataType': 'S3Prefix',\n",
      "                                                            'S3InputMode': 'File',\n",
      "                                                            'S3Uri': 's3://sagemaker-us-east-1-656717427230/sagemaker-scikit-learn-2023-09-17-18-45-09-545/input/code/evaluate_model_metrics.py'}}],\n",
      "                          'ProcessingOutputConfig': {'Outputs': [{'AppManaged': False,\n",
      "                                                                  'OutputName': 'metrics',\n",
      "                                                                  'S3Output': {'LocalPath': '/opt/ml/processing/output/metrics/',\n",
      "                                                                               'S3UploadMode': 'EndOfJob',\n",
      "                                                                               'S3Uri': 's3://sagemaker-us-east-1-656717427230/sagemaker-scikit-learn-2023-09-17-18-42-30-564/output/metrics'}}]},\n",
      "                          'ProcessingResources': {'ClusterConfig': {'InstanceCount': {'Get': 'Parameters.ProcessingInstanceCount'},\n",
      "                                                                    'InstanceType': {'Get': 'Parameters.ProcessingInstanceType'},\n",
      "                                                                    'VolumeSizeInGB': 30}},\n",
      "                          'RoleArn': 'arn:aws:iam::656717427230:role/sagemaker-studio-vpc-firewall-us-east-1-sagemaker-execution-role',\n",
      "                          'StoppingCondition': {'MaxRuntimeInSeconds': 7200}},\n",
      "            'Name': 'EvaluateModel',\n",
      "            'PropertyFiles': [{'FilePath': 'evaluation.json',\n",
      "                               'OutputName': 'metrics',\n",
      "                               'PropertyFileName': 'EvaluationReport'}],\n",
      "            'Type': 'Processing'},\n",
      "           {'Arguments': {'Conditions': [{'LeftValue': {'Std:JsonGet': {'Path': 'metrics.accuracy.value',\n",
      "                                                                        'PropertyFile': {'Get': 'Steps.EvaluateModel.PropertyFiles.EvaluationReport'}}},\n",
      "                                          'RightValue': {'Get': 'Parameters.MinAccuracyValue'},\n",
      "                                          'Type': 'GreaterThanOrEqualTo'}],\n",
      "                          'ElseSteps': [],\n",
      "                          'IfSteps': [{'Arguments': {'InferenceSpecification': {'Containers': [{'Image': '763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:1.6.0-cpu-py36',\n",
      "                                                                                                'ModelDataUrl': {'Get': 'Steps.Train.ModelArtifacts.S3ModelArtifacts'}}],\n",
      "                                                                                'SupportedContentTypes': ['application/jsonlines'],\n",
      "                                                                                'SupportedRealtimeInferenceInstanceTypes': [{'Get': 'Parameters.DeployInstanceType'}],\n",
      "                                                                                'SupportedResponseMIMETypes': ['application/jsonlines'],\n",
      "                                                                                'SupportedTransformInstanceTypes': [{'Get': 'Parameters.DeployInstanceType'}]},\n",
      "                                                     'ModelApprovalStatus': {'Get': 'Parameters.ModelApprovalStatus'},\n",
      "                                                     'ModelMetrics': {'ModelQuality': {'Statistics': {'ContentType': 'application/json',\n",
      "                                                                                                      'S3Uri': 's3://sagemaker-us-east-1-656717427230/sagemaker-scikit-learn-2023-09-17-18-42-30-564/output/metrics/evaluation.json'}}},\n",
      "                                                     'ModelPackageGroupName': 'BERT-Reviews-1694975588'},\n",
      "                                       'Name': 'RegisterModel',\n",
      "                                       'Type': 'RegisterModel'},\n",
      "                                      {'Arguments': {'ExecutionRoleArn': 'arn:aws:iam::656717427230:role/sagemaker-studio-vpc-firewall-us-east-1-sagemaker-execution-role',\n",
      "                                                     'PrimaryContainer': {'Environment': {},\n",
      "                                                                          'Image': '763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:1.6.0-cpu-py36',\n",
      "                                                                          'ModelDataUrl': {'Get': 'Steps.Train.ModelArtifacts.S3ModelArtifacts'}}},\n",
      "                                       'Name': 'CreateModel',\n",
      "                                       'Type': 'Model'}]},\n",
      "            'Name': 'AccuracyCondition',\n",
      "            'Type': 'Condition'}],\n",
      " 'Version': '2020-12-01'}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "from pprint import pprint\n",
    "\n",
    "definition = json.loads(pipeline.definition())\n",
    "\n",
    "pprint(definition)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Ignore the `WARNING` below_\n",
    "\n",
    "Create pipeline using the `create` method and then print the Amazon Resource Name (ARN) of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No finished training job found associated with this estimator. Please make sure this estimator is only used for building workflow config\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arn:aws:sagemaker:us-east-1:656717427230:pipeline/BERT-pipeline-1694975588\n"
     ]
    }
   ],
   "source": [
    "response = pipeline.create(role_arn=role)\n",
    "\n",
    "pipeline_arn = response[\"PipelineArn\"]\n",
    "print(pipeline_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Ignore the `WARNING` ^^ above ^^_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-7.2.'></a>\n",
    "### 7.2. Start Pipeline\n",
    "Let's submit our pipeline definition to the Amazon SageMaker Pipeline service. The role passed in will be used by the service to create all the jobs defined in the steps. You will start the pipeline using the parameters passed into the `start()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arn:aws:sagemaker:us-east-1:656717427230:pipeline/BERT-pipeline-1694975588/execution/t7zfx2f3fk5v\n"
     ]
    }
   ],
   "source": [
    "execution = pipeline.start(\n",
    "    parameters=dict(\n",
    "        InputData=raw_input_data_s3_uri,\n",
    "        ProcessingInstanceCount=1,\n",
    "        ProcessingInstanceType='ml.c5.2xlarge',\n",
    "        MaxSeqLength=128,\n",
    "        BalanceDataset='True',\n",
    "        TrainSplitPercentage=0.9,\n",
    "        ValidationSplitPercentage=0.05,\n",
    "        TestSplitPercentage=0.05,\n",
    "        FeatureStoreOfflinePrefix='reviews-feature-store-'+str(timestamp),\n",
    "        FeatureGroupName='reviews-feature-group-'+str(timestamp),\n",
    "        Epochs=3,\n",
    "        LearningRate=0.000012,\n",
    "        TrainBatchSize=64,\n",
    "        TrainStepsPerEpoch=50,\n",
    "        ValidationBatchSize=64,\n",
    "        ValidationStepsPerEpoch=64,\n",
    "        FreezeBertLayer='False',\n",
    "        Seed=42,         \n",
    "        TrainInstanceCount=1,\n",
    "        TrainInstanceType='ml.c5.9xlarge',\n",
    "        TrainVolumeSize=256,\n",
    "        InputMode='File',\n",
    "        RunValidation='True',\n",
    "        MinAccuracyValue=0.01,\n",
    "        ModelApprovalStatus='PendingManualApproval', \n",
    "        DeployInstanceType='ml.m5.large',\n",
    "        DeployInstanceCount=1 \n",
    "    )\n",
    ")\n",
    "\n",
    "print(execution.arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-7.3.'></a>\n",
    "### 7.3. Wait for pipeline execution\n",
    "\n",
    "Now you can describe execution instance and list the steps in the execution to find out more about the execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'CreatedBy': {'DomainId': 'd-91qnc0mzf0uu',\n",
      "               'UserProfileArn': 'arn:aws:sagemaker:us-east-1:656717427230:user-profile/d-91qnc0mzf0uu/sagemaker-user-profile-us-east-1',\n",
      "               'UserProfileName': 'sagemaker-user-profile-us-east-1'},\n",
      " 'CreationTime': datetime.datetime(2023, 9, 17, 18, 45, 44, 857000, tzinfo=tzlocal()),\n",
      " 'LastModifiedBy': {'DomainId': 'd-91qnc0mzf0uu',\n",
      "                    'UserProfileArn': 'arn:aws:sagemaker:us-east-1:656717427230:user-profile/d-91qnc0mzf0uu/sagemaker-user-profile-us-east-1',\n",
      "                    'UserProfileName': 'sagemaker-user-profile-us-east-1'},\n",
      " 'LastModifiedTime': datetime.datetime(2023, 9, 17, 18, 45, 44, 857000, tzinfo=tzlocal()),\n",
      " 'PipelineArn': 'arn:aws:sagemaker:us-east-1:656717427230:pipeline/BERT-pipeline-1694975588',\n",
      " 'PipelineExecutionArn': 'arn:aws:sagemaker:us-east-1:656717427230:pipeline/BERT-pipeline-1694975588/execution/t7zfx2f3fk5v',\n",
      " 'PipelineExecutionDisplayName': 'execution-1694976344946',\n",
      " 'PipelineExecutionStatus': 'Executing',\n",
      " 'ResponseMetadata': {'HTTPHeaders': {'content-length': '815',\n",
      "                                      'content-type': 'application/x-amz-json-1.1',\n",
      "                                      'date': 'Sun, 17 Sep 2023 18:45:55 GMT',\n",
      "                                      'x-amzn-requestid': '191dab88-948f-4266-b0da-847564ee80a4'},\n",
      "                      'HTTPStatusCode': 200,\n",
      "                      'RequestId': '191dab88-948f-4266-b0da-847564ee80a4',\n",
      "                      'RetryAttempts': 0}}\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "execution_run = execution.describe()\n",
    "pprint(execution_run)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the execution display name and its ARN:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "execution-1694976344946\n"
     ]
    }
   ],
   "source": [
    "execution_run_name = execution_run['PipelineExecutionDisplayName']\n",
    "print(execution_run_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arn:aws:sagemaker:us-east-1:656717427230:pipeline/BERT-pipeline-1694975588/execution/t7zfx2f3fk5v\n"
     ]
    }
   ],
   "source": [
    "pipeline_execution_arn = execution_run['PipelineExecutionArn']\n",
    "print(pipeline_execution_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-7.4.'></a>\n",
    "### 7.4. Describe completed pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for the first step to start running and print the information about it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'StepName': 'Processing',\n",
       "  'StartTime': datetime.datetime(2023, 9, 17, 18, 45, 45, 756000, tzinfo=tzlocal()),\n",
       "  'StepStatus': 'Executing',\n",
       "  'AttemptCount': 0,\n",
       "  'Metadata': {'ProcessingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:656717427230:processing-job/pipelines-t7zfx2f3fk5v-Processing-pob4BfBffL'}}}]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "time.sleep(30)\n",
    "\n",
    "execution.list_steps()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-7.5.'></a>\n",
    "### 7.5. Wait for the pipeline to complete\n",
    "\n",
    "To get the information about the pipeline execution you can use low-level service client of the boto3 session. It is also useful for other operations that you will see below.\n",
    "\n",
    "In the code below you will be observing the pipeline execution summary and waiting for the execution status to change from `Executing` to `Succeeded`.\n",
    "\n",
    "### _This cell will take approximately 30-45 minutes to run._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing\n",
      "Please wait...\n",
      "Please wait...\n",
      "Please wait...\n",
      "Please wait...\n",
      "Please wait...\n",
      "Please wait...\n",
      "Please wait...\n",
      "Please wait...\n",
      "Please wait...\n",
      "Please wait...\n",
      "Please wait...\n",
      "Please wait...\n",
      "Please wait...\n",
      "Please wait...\n",
      "Please wait...\n",
      "[{'PipelineExecutionArn': 'arn:aws:sagemaker:us-east-1:656717427230:pipeline/BERT-pipeline-1694975588/execution/t7zfx2f3fk5v',\n",
      "  'PipelineExecutionDisplayName': 'execution-1694976344946',\n",
      "  'PipelineExecutionStatus': 'Succeeded',\n",
      "  'StartTime': datetime.datetime(2023, 9, 17, 18, 45, 44, 857000, tzinfo=tzlocal())}]\n",
      "CPU times: user 16.7 s, sys: 673 ms, total: 17.4 s\n",
      "Wall time: 31min 4s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "import time\n",
    "from pprint import pprint\n",
    "\n",
    "sm = boto3.Session().client(service_name='sagemaker', region_name=region)\n",
    "\n",
    "executions_response = sm.list_pipeline_executions(PipelineName=pipeline_name)['PipelineExecutionSummaries']\n",
    "pipeline_execution_status = executions_response[0]['PipelineExecutionStatus']\n",
    "print(pipeline_execution_status)\n",
    "\n",
    "while pipeline_execution_status=='Executing':\n",
    "    try:\n",
    "        executions_response = sm.list_pipeline_executions(PipelineName=pipeline_name)['PipelineExecutionSummaries']\n",
    "        pipeline_execution_status = executions_response[0]['PipelineExecutionStatus']\n",
    "    except Exception as e:\n",
    "        print('Please wait...')\n",
    "        time.sleep(30)    \n",
    "    \n",
    "pprint(executions_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Wait for the pipeline ^^ above ^^ to complete._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can list the execution steps to check out the status and artifacts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Succeeded\n"
     ]
    }
   ],
   "source": [
    "pipeline_execution_status = executions_response[0]['PipelineExecutionStatus']\n",
    "print(pipeline_execution_status)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arn:aws:sagemaker:us-east-1:656717427230:pipeline/BERT-pipeline-1694975588/execution/t7zfx2f3fk5v\n"
     ]
    }
   ],
   "source": [
    "pipeline_execution_arn = executions_response[0]['PipelineExecutionArn']\n",
    "print(pipeline_execution_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-8.'></a>\n",
    "# 8. Evaluate the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-8.1.'></a>\n",
    "### 8.1. Describe evaluation metrics\n",
    "\n",
    "Examine the resulting model evaluation after the pipeline completes. Download the resulting evaluation.json file from S3 and print the report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Transform output s3://sagemaker-us-east-1-656717427230/sagemaker-scikit-learn-2023-09-17-18-36-50-637/output/sentiment-train\n"
     ]
    }
   ],
   "source": [
    "processing_job_name = None\n",
    "\n",
    "# pull the processing step name\n",
    "for execution_step in reversed(execution.list_steps()):\n",
    "    if execution_step['StepName'] == 'Processing':\n",
    "        processing_job_name=execution_step['Metadata']['ProcessingJob']['Arn'].split('/')[-1]\n",
    "\n",
    "# get the description of the processing job\n",
    "describe_transform_processing_job_response = sm.describe_processing_job(ProcessingJobName=processing_job_name)\n",
    "\n",
    "# get the output S3 path\n",
    "transform_output_s3_uri = describe_transform_processing_job_response['ProcessingOutputConfig']['Outputs'][0]['S3Output']['S3Uri']\n",
    "print('Transform output {}'.format(transform_output_s3_uri))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-09-17 18:57:28    4894822 sagemaker-scikit-learn-2023-09-17-18-36-50-637/output/sentiment-train/part-algo-1-womens_clothing_ecommerce_reviews.tsv\n"
     ]
    }
   ],
   "source": [
    "# list the files in the resulting output S3 path\n",
    "!aws s3 ls --recursive $transform_output_s3_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-ex-7'></a>\n",
    "### Exercise 7\n",
    "\n",
    "Pull the name of the model-evaluation step and then get the S3 path of the evaluation metrics, which will contain the evaluation report.\n",
    "\n",
    "**Instructions**: Find the execution step with the step name `EvaluateModel` following the example above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation output s3://sagemaker-us-east-1-656717427230/sagemaker-scikit-learn-2023-09-17-18-36-50-637/output/sentiment-train\n"
     ]
    }
   ],
   "source": [
    "processing_job_name = None\n",
    "\n",
    "for execution_step in reversed(execution.list_steps()):\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    if execution_step['StepName'] == 'Processing': # Replace all None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "        processing_job_name=execution_step['Metadata']['ProcessingJob']['Arn'].split('/')[-1]\n",
    "\n",
    "describe_evaluation_processing_job_response = sm.describe_processing_job(ProcessingJobName=processing_job_name)\n",
    "\n",
    "evaluation_metrics_s3_uri = describe_evaluation_processing_job_response['ProcessingOutputConfig']['Outputs'][0]['S3Output']['S3Uri']\n",
    "print('Evaluation output {}'.format(evaluation_metrics_s3_uri))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-8.2.'></a>\n",
    "### 8.2. Review the evaluation report\n",
    "\n",
    "Download the evaluation report and print the accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "ename": "NoSuchKey",
     "evalue": "An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNoSuchKey\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-62-ee6d3d97a401>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m evaluation_json = sagemaker.s3.S3Downloader.read_file(\"{}/evaluation.json\".format(\n\u001b[0;32m----> 4\u001b[0;31m     \u001b[0mevaluation_metrics_s3_uri\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m ))\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/sagemaker/s3.py\u001b[0m in \u001b[0;36mread_file\u001b[0;34m(s3_uri, sagemaker_session)\u001b[0m\n\u001b[1;32m    163\u001b[0m         \u001b[0mbucket\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey_prefix\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparse_s3_url\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0ms3_uri\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    164\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 165\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msagemaker_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_s3_file\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbucket\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbucket\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey_prefix\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkey_prefix\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    166\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    167\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mstaticmethod\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36mread_s3_file\u001b[0;34m(self, bucket, key_prefix)\u001b[0m\n\u001b[1;32m    331\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    332\u001b[0m         \u001b[0;31m# Explicitly passing a None kms_key to boto3 throws a validation error.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 333\u001b[0;31m         \u001b[0ms3_object\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0ms3\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_object\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mBucket\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbucket\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mKey\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkey_prefix\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    334\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    335\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0ms3_object\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"Body\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"utf-8\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_api_call\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    533\u001b[0m                 )\n\u001b[1;32m    534\u001b[0m             \u001b[0;31m# The \"self\" in this scope is referring to the BaseClient.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 535\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_api_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moperation_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    536\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    537\u001b[0m         \u001b[0m_api_call\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpy_operation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_make_api_call\u001b[0;34m(self, operation_name, api_params)\u001b[0m\n\u001b[1;32m    978\u001b[0m             \u001b[0merror_code\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Error\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Code\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    979\u001b[0m             \u001b[0merror_class\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_code\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merror_code\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 980\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0merror_class\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparsed_response\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moperation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    981\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    982\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNoSuchKey\u001b[0m: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist."
     ]
    }
   ],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "evaluation_json = sagemaker.s3.S3Downloader.read_file(\"{}/evaluation.json\".format(\n",
    "    evaluation_metrics_s3_uri\n",
    "))\n",
    "\n",
    "pprint(json.loads(evaluation_json))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-8.3.'></a>\n",
    "### 8.3. List pipeline artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-ex-8'></a>\n",
    "### Exercise 8\n",
    "\n",
    "Find and print the ARN and job name of the training job. \n",
    "\n",
    "**Instructions**: Find the execution step with the step name `Train` following the example above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'AttemptCount': 0,\n",
      " 'EndTime': datetime.datetime(2023, 9, 17, 19, 12, 6, 60000, tzinfo=tzlocal()),\n",
      " 'Metadata': {'TrainingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:656717427230:training-job/pipelines-t7zfx2f3fk5v-Train-nY59PgBWVB'}},\n",
      " 'StartTime': datetime.datetime(2023, 9, 17, 18, 57, 35, 671000, tzinfo=tzlocal()),\n",
      " 'StepName': 'Train',\n",
      " 'StepStatus': 'Succeeded'}\n",
      "Training job ARN: arn:aws:sagemaker:us-east-1:656717427230:training-job/pipelines-t7zfx2f3fk5v-Train-nY59PgBWVB\n",
      "Training job Name: pipelines-t7zfx2f3fk5v-Train-nY59PgBWVB\n"
     ]
    }
   ],
   "source": [
    "training_job_arn=None\n",
    "\n",
    "for execution_step in execution.list_steps():\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    if execution_step['StepName'] == 'Train': # Replace all None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "        training_job_arn = execution_step['Metadata']['TrainingJob']['Arn']        \n",
    "        pprint(execution_step)\n",
    "        break\n",
    "print('Training job ARN: {}'.format(training_job_arn))\n",
    "        \n",
    "training_job_name = training_job_arn.split('/')[-1]\n",
    "print('Training job Name: {}'.format(training_job_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using similar approach you can find and print the pipeline artifacts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "processing_job_name=None\n",
    "training_job_name=None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'AttemptCount': 0,\n",
      " 'EndTime': datetime.datetime(2023, 9, 17, 18, 57, 35, 209000, tzinfo=tzlocal()),\n",
      " 'Metadata': {'ProcessingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:656717427230:processing-job/pipelines-t7zfx2f3fk5v-Processing-pob4BfBffL'}},\n",
      " 'StartTime': datetime.datetime(2023, 9, 17, 18, 45, 45, 756000, tzinfo=tzlocal()),\n",
      " 'StepName': 'Processing',\n",
      " 'StepStatus': 'Succeeded'}\n",
      "Processing job name: pipelines-t7zfx2f3fk5v-Processing-pob4BfBffL\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name/Source</th>\n",
       "      <th>Direction</th>\n",
       "      <th>Type</th>\n",
       "      <th>Association Type</th>\n",
       "      <th>Lineage Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>s3://...-18-45-32-318/input/code/prepare_data.py</td>\n",
       "      <td>Input</td>\n",
       "      <td>DataSet</td>\n",
       "      <td>ContributedTo</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>s3://dlai-practical-data-science/data/raw/</td>\n",
       "      <td>Input</td>\n",
       "      <td>DataSet</td>\n",
       "      <td>ContributedTo</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>68331...om/sagemaker-scikit-learn:0.23-1-cpu-py3</td>\n",
       "      <td>Input</td>\n",
       "      <td>Image</td>\n",
       "      <td>ContributedTo</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>s3://...09-17-18-36-50-637/output/sentiment-test</td>\n",
       "      <td>Output</td>\n",
       "      <td>DataSet</td>\n",
       "      <td>Produced</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>s3://...18-36-50-637/output/sentiment-validation</td>\n",
       "      <td>Output</td>\n",
       "      <td>DataSet</td>\n",
       "      <td>Produced</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>s3://...9-17-18-36-50-637/output/sentiment-train</td>\n",
       "      <td>Output</td>\n",
       "      <td>DataSet</td>\n",
       "      <td>Produced</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Name/Source Direction     Type  \\\n",
       "0  s3://...-18-45-32-318/input/code/prepare_data.py     Input  DataSet   \n",
       "1        s3://dlai-practical-data-science/data/raw/     Input  DataSet   \n",
       "2  68331...om/sagemaker-scikit-learn:0.23-1-cpu-py3     Input    Image   \n",
       "3  s3://...09-17-18-36-50-637/output/sentiment-test    Output  DataSet   \n",
       "4  s3://...18-36-50-637/output/sentiment-validation    Output  DataSet   \n",
       "5  s3://...9-17-18-36-50-637/output/sentiment-train    Output  DataSet   \n",
       "\n",
       "  Association Type Lineage Type  \n",
       "0    ContributedTo     artifact  \n",
       "1    ContributedTo     artifact  \n",
       "2    ContributedTo     artifact  \n",
       "3         Produced     artifact  \n",
       "4         Produced     artifact  \n",
       "5         Produced     artifact  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'AttemptCount': 0,\n",
      " 'EndTime': datetime.datetime(2023, 9, 17, 19, 12, 6, 60000, tzinfo=tzlocal()),\n",
      " 'Metadata': {'TrainingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:656717427230:training-job/pipelines-t7zfx2f3fk5v-Train-nY59PgBWVB'}},\n",
      " 'StartTime': datetime.datetime(2023, 9, 17, 18, 57, 35, 671000, tzinfo=tzlocal()),\n",
      " 'StepName': 'Train',\n",
      " 'StepStatus': 'Succeeded'}\n",
      "Training job name: pipelines-t7zfx2f3fk5v-Train-nY59PgBWVB\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name/Source</th>\n",
       "      <th>Direction</th>\n",
       "      <th>Type</th>\n",
       "      <th>Association Type</th>\n",
       "      <th>Lineage Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>s3://...18-36-50-637/output/sentiment-validation</td>\n",
       "      <td>Input</td>\n",
       "      <td>DataSet</td>\n",
       "      <td>ContributedTo</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>s3://...9-17-18-36-50-637/output/sentiment-train</td>\n",
       "      <td>Input</td>\n",
       "      <td>DataSet</td>\n",
       "      <td>ContributedTo</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>76310...onaws.com/pytorch-training:1.6.0-cpu-py3</td>\n",
       "      <td>Input</td>\n",
       "      <td>Image</td>\n",
       "      <td>ContributedTo</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>s3://...k5v-Train-nY59PgBWVB/output/model.tar.gz</td>\n",
       "      <td>Output</td>\n",
       "      <td>Model</td>\n",
       "      <td>Produced</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Name/Source Direction     Type  \\\n",
       "0  s3://...18-36-50-637/output/sentiment-validation     Input  DataSet   \n",
       "1  s3://...9-17-18-36-50-637/output/sentiment-train     Input  DataSet   \n",
       "2  76310...onaws.com/pytorch-training:1.6.0-cpu-py3     Input    Image   \n",
       "3  s3://...k5v-Train-nY59PgBWVB/output/model.tar.gz    Output    Model   \n",
       "\n",
       "  Association Type Lineage Type  \n",
       "0    ContributedTo     artifact  \n",
       "1    ContributedTo     artifact  \n",
       "2    ContributedTo     artifact  \n",
       "3         Produced     artifact  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'AttemptCount': 0,\n",
      " 'EndTime': datetime.datetime(2023, 9, 17, 19, 17, 54, 490000, tzinfo=tzlocal()),\n",
      " 'Metadata': {'ProcessingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:656717427230:processing-job/pipelines-t7zfx2f3fk5v-EvaluateModel-ReeHTxB4BH'}},\n",
      " 'StartTime': datetime.datetime(2023, 9, 17, 19, 12, 7, 22000, tzinfo=tzlocal()),\n",
      " 'StepName': 'EvaluateModel',\n",
      " 'StepStatus': 'Succeeded'}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name/Source</th>\n",
       "      <th>Direction</th>\n",
       "      <th>Type</th>\n",
       "      <th>Association Type</th>\n",
       "      <th>Lineage Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>s3://...902/input/code/evaluate_model_metrics.py</td>\n",
       "      <td>Input</td>\n",
       "      <td>DataSet</td>\n",
       "      <td>ContributedTo</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>s3://...09-17-18-36-50-637/output/sentiment-test</td>\n",
       "      <td>Input</td>\n",
       "      <td>DataSet</td>\n",
       "      <td>ContributedTo</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>s3://...k5v-Train-nY59PgBWVB/output/model.tar.gz</td>\n",
       "      <td>Input</td>\n",
       "      <td>Model</td>\n",
       "      <td>ContributedTo</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>68331...om/sagemaker-scikit-learn:0.23-1-cpu-py3</td>\n",
       "      <td>Input</td>\n",
       "      <td>Image</td>\n",
       "      <td>ContributedTo</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>s3://...n-2023-09-17-18-42-30-564/output/metrics</td>\n",
       "      <td>Output</td>\n",
       "      <td>DataSet</td>\n",
       "      <td>Produced</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Name/Source Direction     Type  \\\n",
       "0  s3://...902/input/code/evaluate_model_metrics.py     Input  DataSet   \n",
       "1  s3://...09-17-18-36-50-637/output/sentiment-test     Input  DataSet   \n",
       "2  s3://...k5v-Train-nY59PgBWVB/output/model.tar.gz     Input    Model   \n",
       "3  68331...om/sagemaker-scikit-learn:0.23-1-cpu-py3     Input    Image   \n",
       "4  s3://...n-2023-09-17-18-42-30-564/output/metrics    Output  DataSet   \n",
       "\n",
       "  Association Type Lineage Type  \n",
       "0    ContributedTo     artifact  \n",
       "1    ContributedTo     artifact  \n",
       "2    ContributedTo     artifact  \n",
       "3    ContributedTo     artifact  \n",
       "4         Produced     artifact  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'AttemptCount': 0,\n",
      " 'EndTime': datetime.datetime(2023, 9, 17, 19, 17, 55, 894000, tzinfo=tzlocal()),\n",
      " 'Metadata': {'Condition': {'Outcome': 'True'}},\n",
      " 'StartTime': datetime.datetime(2023, 9, 17, 19, 17, 55, 265000, tzinfo=tzlocal()),\n",
      " 'StepName': 'AccuracyCondition',\n",
      " 'StepStatus': 'Succeeded'}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "None"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'AttemptCount': 0,\n",
      " 'EndTime': datetime.datetime(2023, 9, 17, 19, 17, 57, 652000, tzinfo=tzlocal()),\n",
      " 'Metadata': {'Model': {'Arn': 'arn:aws:sagemaker:us-east-1:656717427230:model/pipelines-t7zfx2f3fk5v-createmodel-jbkfcvfndr'}},\n",
      " 'StartTime': datetime.datetime(2023, 9, 17, 19, 17, 56, 262000, tzinfo=tzlocal()),\n",
      " 'StepName': 'CreateModel',\n",
      " 'StepStatus': 'Succeeded'}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "None"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'AttemptCount': 0,\n",
      " 'EndTime': datetime.datetime(2023, 9, 17, 19, 17, 57, 329000, tzinfo=tzlocal()),\n",
      " 'Metadata': {'RegisterModel': {'Arn': 'arn:aws:sagemaker:us-east-1:656717427230:model-package/BERT-Reviews-1694975588/1'}},\n",
      " 'StartTime': datetime.datetime(2023, 9, 17, 19, 17, 56, 262000, tzinfo=tzlocal()),\n",
      " 'StepName': 'RegisterModel',\n",
      " 'StepStatus': 'Succeeded'}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name/Source</th>\n",
       "      <th>Direction</th>\n",
       "      <th>Type</th>\n",
       "      <th>Association Type</th>\n",
       "      <th>Lineage Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>s3://...k5v-Train-nY59PgBWVB/output/model.tar.gz</td>\n",
       "      <td>Input</td>\n",
       "      <td>Model</td>\n",
       "      <td>ContributedTo</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>76310...aws.com/pytorch-inference:1.6.0-cpu-py36</td>\n",
       "      <td>Input</td>\n",
       "      <td>Image</td>\n",
       "      <td>ContributedTo</td>\n",
       "      <td>artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BERT-Reviews-1694975588-1-PendingManualApprova...</td>\n",
       "      <td>Input</td>\n",
       "      <td>Approval</td>\n",
       "      <td>ContributedTo</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BERT-Reviews-1694975588-1694978276-aws-model-p...</td>\n",
       "      <td>Output</td>\n",
       "      <td>ModelGroup</td>\n",
       "      <td>AssociatedWith</td>\n",
       "      <td>context</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         Name/Source Direction        Type  \\\n",
       "0   s3://...k5v-Train-nY59PgBWVB/output/model.tar.gz     Input       Model   \n",
       "1   76310...aws.com/pytorch-inference:1.6.0-cpu-py36     Input       Image   \n",
       "2  BERT-Reviews-1694975588-1-PendingManualApprova...     Input    Approval   \n",
       "3  BERT-Reviews-1694975588-1694978276-aws-model-p...    Output  ModelGroup   \n",
       "\n",
       "  Association Type Lineage Type  \n",
       "0    ContributedTo     artifact  \n",
       "1    ContributedTo     artifact  \n",
       "2    ContributedTo       action  \n",
       "3   AssociatedWith      context  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import time\n",
    "from sagemaker.lineage.visualizer import LineageTableVisualizer\n",
    "\n",
    "viz = LineageTableVisualizer(sagemaker.session.Session())\n",
    "\n",
    "for execution_step in reversed(execution.list_steps()):\n",
    "    pprint(execution_step)\n",
    "    if execution_step['StepName'] == 'Processing':\n",
    "        processing_job_name=execution_step['Metadata']['ProcessingJob']['Arn'].split('/')[-1]\n",
    "        print('Processing job name: {}'.format(processing_job_name))\n",
    "        display(viz.show(processing_job_name=processing_job_name))\n",
    "    elif execution_step['StepName'] == 'Train':\n",
    "        training_job_name=execution_step['Metadata']['TrainingJob']['Arn'].split('/')[-1]\n",
    "        print('Training job name: {}'.format(training_job_name))\n",
    "        display(viz.show(training_job_name=training_job_name))\n",
    "    else:\n",
    "        display(viz.show(pipeline_execution_step=execution_step))\n",
    "        time.sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-9.'></a>\n",
    "# 9. Deploy and test the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-9.1.'></a>\n",
    "### 9.1. Approve trained model\n",
    "\n",
    "The pipeline created a model package version within the specified model package group and an approval status of `PendingManualApproval`.  This requires a separate step to manually approve the model before deploying to production.\n",
    "\n",
    "You can approve the model using the SageMaker Studio UI or programmatically as shown below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the model package ARN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arn:aws:sagemaker:us-east-1:656717427230:model-package/BERT-Reviews-1694975588/1\n"
     ]
    }
   ],
   "source": [
    "for execution_step in execution.list_steps():\n",
    "    if execution_step['StepName'] == 'RegisterModel':\n",
    "        model_package_arn = execution_step['Metadata']['RegisterModel']['Arn']\n",
    "        break\n",
    "print(model_package_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update the model package with the `Approved` status to prepare for deployment.\n",
    "\n",
    "The model must be `Approved` before it can be deployed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'ModelPackageArn': 'arn:aws:sagemaker:us-east-1:656717427230:model-package/BERT-Reviews-1694975588/1',\n",
      " 'ResponseMetadata': {'HTTPHeaders': {'content-length': '102',\n",
      "                                      'content-type': 'application/x-amz-json-1.1',\n",
      "                                      'date': 'Sun, 17 Sep 2023 19:21:16 GMT',\n",
      "                                      'x-amzn-requestid': '5e970bac-e082-420b-9ffe-1c4e7af7d3c2'},\n",
      "                      'HTTPStatusCode': 200,\n",
      "                      'RequestId': '5e970bac-e082-420b-9ffe-1c4e7af7d3c2',\n",
      "                      'RetryAttempts': 0}}\n"
     ]
    }
   ],
   "source": [
    "model_package_update_response = sm.update_model_package(\n",
    "    ModelPackageArn=model_package_arn,\n",
    "    ModelApprovalStatus=\"Approved\",\n",
    ")\n",
    "\n",
    "pprint(model_package_update_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-9.2.'></a>\n",
    "### 9.2. Deploy model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the model ARN and the model name from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RegisterModel\n",
      "CreateModel\n",
      "arn:aws:sagemaker:us-east-1:656717427230:model/pipelines-t7zfx2f3fk5v-createmodel-jbkfcvfndr\n",
      "pipelines-t7zfx2f3fk5v-createmodel-jbkfcvfndr\n"
     ]
    }
   ],
   "source": [
    "for execution_step in execution.list_steps():\n",
    "    print(execution_step['StepName'])\n",
    "    if execution_step['StepName'] == 'CreateModel':\n",
    "        model_arn = execution_step['Metadata']['Model']['Arn']\n",
    "        break\n",
    "print(model_arn)\n",
    "\n",
    "model_name = model_arn.split('/')[-1]\n",
    "print(model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-9.3.'></a>\n",
    "### 9.3. Create endpoint from registry"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Configure the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bert-model-epc-1694975588\n"
     ]
    }
   ],
   "source": [
    "endpoint_config_name = 'bert-model-epc-{}'.format(timestamp)\n",
    "print(endpoint_config_name)\n",
    "\n",
    "create_endpoint_config_response = sm.create_endpoint_config(\n",
    "    EndpointConfigName = endpoint_config_name,\n",
    "    ProductionVariants=[{\n",
    "        'InstanceType':'ml.m5.xlarge',\n",
    "        'InitialVariantWeight':1,\n",
    "        'InitialInstanceCount':1,\n",
    "        'ModelName': model_name,\n",
    "        'VariantName':'AllTraffic'}])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EndpointName=bert-model-ep-1694975588\n",
      "arn:aws:sagemaker:us-east-1:656717427230:endpoint/bert-model-ep-1694975588\n"
     ]
    }
   ],
   "source": [
    "pipeline_endpoint_name = 'bert-model-ep-{}'.format(timestamp)\n",
    "print(\"EndpointName={}\".format(pipeline_endpoint_name))\n",
    "\n",
    "create_endpoint_response = sm.create_endpoint(\n",
    "    EndpointName=pipeline_endpoint_name,\n",
    "    EndpointConfigName=endpoint_config_name)\n",
    "print(create_endpoint_response['EndpointArn'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/endpoints/bert-model-ep-1694975588\">SageMaker REST Endpoint</a></b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/endpoints/{}\">SageMaker REST Endpoint</a></b>'.format(region, pipeline_endpoint_name)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait until the endpoint is deployed.\n",
    "\n",
    "### _This cell will take approximately 5-10 minutes to run._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for endpoint to be in `InService`...\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "while True:\n",
    "    try: \n",
    "        waiter = sm.get_waiter('endpoint_in_service')\n",
    "        print('Waiting for endpoint to be in `InService`...')\n",
    "        waiter.wait(EndpointName=pipeline_endpoint_name)\n",
    "        break;\n",
    "    except:\n",
    "        print('Waiting for endpoint...')\n",
    "        endpoint_status = sm.describe_endpoint(EndpointName=pipeline_endpoint_name)['EndpointStatus']\n",
    "        print('Endpoint status: {}'.format(endpoint_status))\n",
    "        if endpoint_status == 'Failed':\n",
    "            break\n",
    "        time.sleep(30)\n",
    "        \n",
    "print('Endpoint deployed.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Wait until the endpoint ^^ above ^^ is deployed._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-9.4.'></a>\n",
    "### 9.4. Test model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict the `sentiment` with `review_body` samples and review the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.predictor import Predictor\n",
    "from sagemaker.serializers import JSONLinesSerializer\n",
    "from sagemaker.deserializers import JSONLinesDeserializer\n",
    "\n",
    "inputs = [\n",
    "    {\"features\": [\"I love this product!\"]},\n",
    "    {\"features\": [\"OK, but not great.\"]},\n",
    "    {\"features\": [\"This is not the right product.\"]},\n",
    "]\n",
    "\n",
    "predictor = Predictor(\n",
    "    endpoint_name=pipeline_endpoint_name,\n",
    "    serializer=JSONLinesSerializer(),\n",
    "    deserializer=JSONLinesDeserializer(),\n",
    "    sagemaker_session=sess\n",
    ")\n",
    "\n",
    "predicted_classes = predictor.predict(inputs)\n",
    "\n",
    "for predicted_class in predicted_classes:\n",
    "    print(\"Predicted class {} with probability {}\".format(predicted_class['predicted_label'], predicted_class['probability']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w3-9.5.'></a>\n",
    "### 9.5. SageMaker Studio extensions\n",
    "\n",
    "SageMaker Studio provides a rich set of features to visually inspect SageMaker resources including pipelines, training jobs, and endpoints. Please take time to explore it opening the facet shown in the following image.\n",
    "\n",
    "![](images/sm_studio_extensions_pipelines.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have just deployed an end-to-end pipeline with BERT and SageMaker Pipelines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upload the notebook into S3 bucket for grading purposes.\n",
    "\n",
    "**Note**: you may need to click on \"Save\" button before the upload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!aws s3 cp ./C2_W3_Assignment.ipynb s3://$bucket/C2_W3_Assignment_Learner.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please go to the main lab window and click on `Submit` button (see the `Finish the lab` section of the instructions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}