cookbook/integration_langserve.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DAQz4b8eMk3b"
      },
      "source": [
        "---\n",
        "description: Examples on how to natively integrate with LangChain's Langserve\n",
        "category: Integrations\n",
        "---\n",
        "\n",
        "# Cookbook: Langserve Integration"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6f6yUZG2kuff"
      },
      "source": [
        "[Langserve](https://python.langchain.com/docs/langserve/) (Python)\n",
        "> LangServe helps developers deploy LangChain runnables and chains as a REST API.\n",
        ">\n",
        "> This library is integrated with FastAPI and uses pydantic for data validation.\n",
        ">\n",
        "> In addition, it provides a client that can be used to call into runnables deployed on a server. A JavaScript client is available in LangChain.js.\n",
        "\n",
        "This cookbook demonstrates how to trace applications deployed via Langserve with Langfuse (using the [LangChain integration](https://langfuse.com/docs/integrations/langchain)). We'll run both the server and the client in this notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RhFKjAOYk6d-"
      },
      "source": [
        "## Setup\n",
        "Install dependencies and configure environment"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KK5uk9JBNFYC",
        "outputId": "e7703ceb-a126-4bdd-f030-147abd9a50db"
      },
      "outputs": [],
      "source": [
        "!pip install fastapi sse_starlette httpx langserve langfuse langchain-openai langchain"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lnNfBrDy2a8B"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# Get keys for your project from the project settings page\n",
        "# https://cloud.langfuse.com\n",
        "os.environ[\"LANGFUSE_PUBLIC_KEY\"] = \"\"\n",
        "os.environ[\"LANGFUSE_SECRET_KEY\"] = \"\"\n",
        "os.environ[\"LANGFUSE_HOST\"] = \"https://cloud.langfuse.com\" # 🇪🇺 EU region\n",
        "# os.environ[\"LANGFUSE_HOST\"] = \"https://us.cloud.langfuse.com\" # 🇺🇸 US region\n",
        "\n",
        "# Your openai key\n",
        "os.environ[\"OPENAI_API_KEY\"] = \"\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V9Xap-l62ptn"
      },
      "source": [
        "## Simple LLM Call Example\n",
        "\n",
        "Initialize the Langfuse client and configure the LLM with Langfuse as callback handler. Add to Fastapi via Langserve's `add_routes()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QCUxy1f820dn"
      },
      "outputs": [],
      "source": [
        "from langchain_openai import ChatOpenAI\n",
        "from langchain_core.runnables.config import RunnableConfig\n",
        "from langfuse import Langfuse\n",
        "from langfuse.callback import CallbackHandler\n",
        "from fastapi import FastAPI\n",
        "from langserve import add_routes\n",
        "\n",
        "langfuse_handler = CallbackHandler()\n",
        "\n",
        "# Tests the SDK connection with the server\n",
        "langfuse_handler.auth_check()\n",
        "\n",
        "llm = ChatOpenAI()\n",
        "\n",
        "config = RunnableConfig(callbacks=[langfuse_handler])\n",
        "\n",
        "llm_with_langfuse = llm.with_config(config)\n",
        "\n",
        "# Setup server\n",
        "app = FastAPI()\n",
        "\n",
        "# Add Langserve route\n",
        "add_routes(\n",
        "    app,\n",
        "    llm_with_langfuse,\n",
        "    path=\"/test-simple-llm-call\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oHlJtl2AdNpH"
      },
      "source": [
        "\n",
        "\n",
        "*Note: We use TestClient in this example to be able to run the server in a notebook*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kekipGdp4_tk",
        "outputId": "1fb7ec39-0559-4bbb-f417-0cdcabec9fe4"
      },
      "outputs": [],
      "source": [
        "from fastapi.testclient import TestClient\n",
        "\n",
        "# Initialize TestClient\n",
        "client = TestClient(app)\n",
        "\n",
        "# Test simple route\n",
        "response = client.post(\"/test-simple-llm-call/invoke\", json={\"input\": \"Tell me a joke?\"})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UgHaaxS_cJT1"
      },
      "source": [
        "Example trace: https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/5f32e2e7-9508-4280-b47b-e0356bc3c81e\n",
        "\n",
        "![Trace of Langserve Simple LLM Call](https://langfuse.com/images/cookbook/integration_langserve_simple.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eIQsCJQF9rhI"
      },
      "source": [
        "## LCEL example"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2D2t0h1j-X6c",
        "outputId": "58591cfb-2927-4179-c54f-9aa671129c69"
      },
      "outputs": [],
      "source": [
        "from langchain.prompts import ChatPromptTemplate\n",
        "from langchain.schema import StrOutputParser\n",
        "from langserve import add_routes\n",
        "\n",
        "# Create Chain\n",
        "prompt = ChatPromptTemplate.from_template(\"Tell me a joke about {topic}\")\n",
        "\n",
        "chain = prompt | llm | StrOutputParser()\n",
        "\n",
        "# Add new route\n",
        "add_routes(\n",
        "    app,\n",
        "    chain.with_config(config),\n",
        "    path=\"/test-chain\",\n",
        ")\n",
        "\n",
        "# Test chain route\n",
        "response = client.post(\"/test-chain/invoke\", json={\"input\": {\"topic\": \"Berlin\"}})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sd1xjUVHcPAX"
      },
      "source": [
        "Example trace: https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/261d1006-74ff-4b67-8baf-afdfc827aee2\n",
        "\n",
        "![Trace of Langserve LCEL Example](https://langfuse.com/images/cookbook/integration_langserve_chain.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "39MTCpb9OsY_"
      },
      "source": [
        "## Agent Example"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OLqZMRrR4gjO"
      },
      "outputs": [],
      "source": [
        "from langchain_core.tools import tool\n",
        "from langchain_core.utils.function_calling import convert_to_openai_tool\n",
        "from langchain.agents.format_scratchpad.openai_tools import (\n",
        "    format_to_openai_tool_messages,\n",
        ")\n",
        "from langchain.agents import AgentExecutor\n",
        "from langchain.agents.output_parsers.openai_tools import OpenAIToolsAgentOutputParser\n",
        "from langserve.pydantic_v1 import BaseModel\n",
        "from langchain_core.prompts import MessagesPlaceholder\n",
        "\n",
        "class Input(BaseModel):\n",
        "    input: str\n",
        "\n",
        "prompt = ChatPromptTemplate.from_messages(\n",
        "    [\n",
        "        (\"system\", \"You are a helpful assistant.\"),\n",
        "        (\"user\", \"{input}\"),\n",
        "        MessagesPlaceholder(variable_name=\"agent_scratchpad\"),\n",
        "    ]\n",
        ")\n",
        "\n",
        "@tool\n",
        "def word_length(word: str) -> int:\n",
        "    \"\"\"Returns a counter word\"\"\"\n",
        "    return len(word)\n",
        "\n",
        "tools = [word_length]\n",
        "\n",
        "llm_with_tools = llm.bind(tools=[convert_to_openai_tool(tool) for tool in tools])\n",
        "\n",
        "agent = (\n",
        "    {\n",
        "        \"input\": lambda x: x[\"input\"],\n",
        "        \"agent_scratchpad\": lambda x: format_to_openai_tool_messages(\n",
        "            x[\"intermediate_steps\"]\n",
        "        ),\n",
        "    }\n",
        "    | prompt\n",
        "    | llm_with_tools\n",
        "    | OpenAIToolsAgentOutputParser()\n",
        ")\n",
        "agent_executor = AgentExecutor(agent=agent, tools=tools)\n",
        "\n",
        "agent_config = RunnableConfig({\"run_name\": \"agent\"}, callbacks=[langfuse_handler])\n",
        "\n",
        "add_routes(\n",
        "    app,\n",
        "    agent_executor.with_types(input_type=Input).with_config(\n",
        "        agent_config\n",
        "    ),\n",
        "    path=\"/test-agent\",\n",
        ")\n",
        "\n",
        "response = client.post(\"/test-agent/invoke\", json={\"input\": {\"input\": \"How long is Leonardo DiCaprios last name?\"}})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1caM-ElUcTdE"
      },
      "source": [
        "Example trace: https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/ed1d57f9-2f35-4e72-8150-b061f21840a7\n",
        "\n",
        "![Trace of Langserve Agent example](https://langfuse.com/images/cookbook/integration_langserve_agent.png)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}