DocETL is a tool for creating and executing data processing pipelines, especially suited for complex document processing tasks. It offers:

1. An interactive UI playground for iterative prompt engineering and pipeline development
2. A Python package for running production pipelines from the command line or Python code

• Conversation Generator
• Text-to-speech
• YouTube Transcript Topic Extraction

• UI/UX Thoughts
• Using Gleaning to Improve Output Quality
• Deep Dive on Resolve Operator

There are two main ways to use DocETL:

The UI Playground helps you iteratively develop your pipeline:

• Experiment with different prompts and see results in real-time
• Build your pipeline step by step
• Export your finalized pipeline configuration for production use

To run the playground locally, you can either:

• Use Docker (recommended for quick start): make docker
• Set up the development environment manually

See the Playground Setup Guide for detailed instructions.

If you want to use DocETL as a Python package:

• Python 3.10 or later
• OpenAI API key

pip install docetl

Create a .env file in your project directory:

OPENAI_API_KEY=your_api_key_here  # Required for LLM operations (or the key for the LLM of your choice)

To see examples of how to use DocETL, check out the tutorial.

To run the UI playground locally, you have two options:

The easiest way to get the playground running:

1. Create the required environment files:

Create .env in the root directory:

OPENAI_API_KEY=your_api_key_here
BACKEND_ALLOW_ORIGINS=
BACKEND_HOST=0.0.0.0
BACKEND_PORT=8000
BACKEND_RELOAD=True
FRONTEND_HOST=0.0.0.0
FRONTEND_PORT=3000

Create .env.local in the website directory:

OPENAI_API_KEY=sk-xxx
OPENAI_API_BASE=https://api.openai.com/v1
MODEL_NAME=gpt-4o-mini

NEXT_PUBLIC_BACKEND_HOST=localhost
NEXT_PUBLIC_BACKEND_PORT=8000

2. Run Docker:

make docker

This will:

• Create a Docker volume for persistent data
• Build the DocETL image
• Run the container with the UI accessible at http://localhost:3000

To clean up Docker resources (note that this will delete the Docker volume):

make docker-clean

For development or if you prefer not to use Docker:

1. Clone the repository:

git clone https://github.com/ucbepic/docetl.git
cd docetl

2. Set up environment variables in .env in the root/top-level directory:

OPENAI_API_KEY=your_api_key_here
BACKEND_ALLOW_ORIGINS=
BACKEND_HOST=localhost
BACKEND_PORT=8000
BACKEND_RELOAD=True
FRONTEND_HOST=0.0.0.0
FRONTEND_PORT=3000

And create an .env.local file in the website directory with the following:

OPENAI_API_KEY=sk-xxx
OPENAI_API_BASE=https://api.openai.com/v1
MODEL_NAME=gpt-4o-mini

NEXT_PUBLIC_BACKEND_HOST=localhost
NEXT_PUBLIC_BACKEND_PORT=8000

3. Install dependencies:

make install      # Install Python package
make install-ui   # Install UI dependencies

Note that the OpenAI API key, base, and model name are for the UI assistant only; not the DocETL pipeline execution engine.

4. Start the development server:

make run-ui-dev

5. Visit http://localhost:3000/playground to access the interactive UI.

If you're planning to contribute or modify DocETL, you can verify your setup by running the test suite:

make tests-basic  # Runs basic test suite (costs < $0.01 with OpenAI)

For detailed documentation and tutorials, visit our documentation.