Technical Report | Model Weights | Evaluation Code | Examples

flow-judge is a lightweight library for evaluating LLM applications with Flow-Judge-v0.1.

Flow-Judge-v0.1 is an open, small yet powerful language model evaluator trained on a synthetic dataset containing LLM system evaluation data by Flow AI.

You can learn more about the unique features of our model in the technical report.

• Support for multiple model types: Hugging Face Transformers and vLLM
• Extensible architecture for custom metrics
• Pre-defined evaluation metrics
• Ease of custom metric and rubric creation
• Batched evaluation for efficient processing
• Integrations with most popular frameworks like Llama Index

Install flow-judge using pip:

pip install -e ".[vllm,hf]"
pip install 'flash_attn>=2.6.3' --no-build-isolation

Extras available:

• dev to install development dependencies
• hf to install Hugging Face Transformers dependencies
• vllm to install vLLM dependencies
• llamafile to install Llamafile dependencies

Here's a simple example to get you started:

from flow_judge import Vllm, Llamafile, Hf, EvalInput, FlowJudge
from flow_judge.metrics import RESPONSE_FAITHFULNESS_5POINT
from IPython.display import Markdown, display

# If you are running on an Ampere GPU or newer, create a model using VLLM
model = Vllm()

# If you have other applications open taking up VRAM, you can use less VRAM by setting gpu_memory_utilization to a lower value.
# model = Vllm(gpu_memory_utilization=0.70)

# Or if not running on Ampere GPU or newer, create a model using no flash attn and Hugging Face Transformers
# model = Hf(flash_attn=False)

# Or create a model using Llamafile if not running an Nvidia GPU & running a Silicon MacOS for example
# model = Llamafile()

# Initialize the judge
faithfulness_judge = FlowJudge(
    metric=RESPONSE_FAITHFULNESS_5POINT,
    model=model
)

# Sample to evaluate
query = """..."""
context = """...""""
response = """..."""

# Create an EvalInput
# We want to evaluate the response to the customer issue based on the context and the user instructions
eval_input = EvalInput(
    inputs=[
        {"query": query},
        {"context": context},
    ],
    output={"response": response},
)

# Run the evaluation
result = faithfulness_judge.evaluate(eval_input, save_results=False)

# Display the result
display(Markdown(f"__Feedback:__\n{result.feedback}\n\n__Score:__\n{result.score}"))

The library supports multiple inference backends to accommodate different hardware configurations and performance needs:

1. vLLM:

   • Best for NVIDIA GPUs with Ampere architecture or newer (e.g., RTX 3000 series, A100, H100)
   • Offers the highest performance and throughput
   • Requires CUDA-compatible GPU

   from flow_judge import Vllm
   
   model = Vllm()

2. Hugging Face Transformers:

   • Compatible with a wide range of hardware, including older NVIDIA GPUs
   • Supports CPU inference (slower but universally compatible)
   • It is slower than vLLM but generally compatible with more hardware.

   If you are running on an Ampere GPU or newer:

   from flow_judge import Hf
   
   model = Hf()

   If you are not running on an Ampere GPU or newer, disable flash attention:

   from flow_judge import Hf
   
   model = Hf(flash_attn=False)

3. Llamafile:

   • Ideal for non-NVIDIA hardware, including Apple Silicon
   • Provides good performance on CPUs
   • Self-contained, easy to deploy option

   from flow_judge import Llamafile
   
   model = Llamafile()

Choose the inference backend that best matches your hardware and performance requirements. The library provides a unified interface for all these options, making it easy to switch between them as needed.

Flow-Judge-v0.1 was trained to handle any custom metric that can be expressed as a combination of evaluation criteria and rubric, and required inputs and outputs.

For convenience, flow-judge library comes with pre-defined metrics such as RESPONSE_CORRECTNESS or RESPONSE_FAITHFULNESS. You can check the full list by running:

from flow_judge.metrics import list_all_metrics

list_all_metrics()

For efficient processing of multiple inputs, you can use the batch_evaluate method:

# Read the sample data
import json
from flow_judge import Vllm, EvalInput, FlowJudge
from flow_judge.metrics import RESPONSE_FAITHFULNESS_5POINT
from IPython.display import Markdown, display

# Initialize the model
model = Vllm()

# Initialize the judge
faithfulness_judge = FlowJudge(
    metric=RESPONSE_FAITHFULNESS_5POINT,
    model=model
)

# Load some sampledata
with open("sample_data/csr_assistant.json", "r") as f:
    data = json.load(f)

# Create a list of inputs and outputs
inputs_batch = [
    [
        {"query": sample["query"]},
        {"context": sample["context"]},
    ]
    for sample in data
]
outputs_batch = [{"response": sample["response"]} for sample in data]

# Create a list of EvalInput
eval_inputs_batch = [EvalInput(inputs=inputs, output=output) for inputs, output in zip(inputs_batch, outputs_batch)]

# Run the batch evaluation
results = faithfulness_judge.batch_evaluate(eval_inputs_batch, save_results=False)

# Visualizing the results
for i, result in enumerate(results):
    display(Markdown(f"__Sample {i+1}:__"))
    display(Markdown(f"__Feedback:__\n{result.feedback}\n\n__Score:__\n{result.score}"))
    display(Markdown("---"))

Create your own evaluation metrics:

from flow_judge.metrics import CustomMetric, RubricItem

custom_metric = CustomMetric(
    name="My Custom Metric",
    criteria="Evaluate based on X, Y, and Z.",
    rubric=[
        RubricItem(score=0, description="Poor performance"),
        RubricItem(score=1, description="Good performance"),
    ],
    required_inputs=["query"],
    required_output="response"
)

judge = FlowJudge(metric=custom_metric, config="Flow-Judge-v0.1-AWQ")

We support an integration with Llama Index evaluation module and Haystack:

• Llama Index tutorial
• Haystack tutorial

Note that we are currently working on adding more integrations with other frameworks in the near future.

1. Clone the repository:

   git clone https://github.com/flowaicom/flow-judge.git
   cd flow-judge

2. Create a virtual environment:

   virtualenv ./.venv

   or

   python -m venv ./.venv

3. Activate the virtual environment:

   • On Windows:

     venv\Scripts\activate

   • On macOS and Linux:

     source venv/bin/activate

4. Install the package in editable mode with development dependencies:

   pip install -e ".[dev]"

   or

   pip install -e ".[dev,vllm]"

   for vLLM support.

5. Set up pre-commit hooks:

   pre-commit install

6. Run pre-commit on all files:

   pre-commit run --all-files

7. You're now ready to start developing! You can run the main script with:

   python -m flow_judge

Remember to always activate your virtual environment when working on the project. To deactivate the virtual environment when you're done, simply run:

deactivate

To run the tests for Flow-Judge, follow these steps:

1. Navigate to the root directory of the project in your terminal.

2. Run the tests using pytest:

   pytest tests/

   This will discover and run all the tests in the tests/ directory.

3. If you want to run a specific test file, you can do so by specifying the file path:

   pytest tests/test_flow_judge.py

4. For more verbose output, you can use the -v flag:

   pytest -v tests/

Contributions to flow-judge are welcome! Please follow these steps:

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Please ensure that your code adheres to the project's coding standards and passes all tests.

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Flow-Judge is developed and maintained by the Flow AI team. We appreciate the contributions and feedback from the AI community in making this tool more robust and versatile.