Decomposition of paper Q&A using humans and language models

• Design principles
• Running ICE locally
  • Requirements
  • Setup
  • Running ICE on the command line
    • Human data collection
    • GPT
  • Streamlit
  • Evaluation
  • Evaluate in-app QA results
• Development
  • Adding new Python dependencies
  • Contributions

• Recipes are decompositions of a task into subtasks.

  The meaning of a recipe is: If a human executed these steps and did a good job at each workspace in isolation, the overall answer would be good. This decomposition may be informed by what we think ML can do at this point, but the recipe itself (as an abstraction) doesn’t know about specific agents.

• Agents perform atomic subtasks of predefined shapes, like completion, scoring, or classification.

  Agents don't know which recipe is calling them. Agents don’t maintain state between subtasks. Agents generally try to complete all subtasks they're asked to complete (however badly), but some will not have implementations for certain task types.

• The mode in which a recipe runs is a global setting that can affect every agent call. For instance, whether to use humans or agents. Recipes can also run with certain RecipeSettings, which can map a task type to a specific agent_name, which can modify which agent is used for that specfic type of task.

1. Install Docker Desktop

1. Add required secrets to .env. See .env.example for a model.

2. Start ICE in its own terminal and leave it running:

   scripts/run-local.sh

3. Go through the tutorial.

scripts/run-recipe.sh --mode machine

You can run on the iteration gold standards of a specific recipe like this:

scripts/run-recipe.sh --mode machine -r placebotree -q placebo -g iterate

To run over multiple gold standard splits, just provide them separated by spaces:

scripts/run-recipe.sh --mode machine -r placebotree -q placebo -g iterate validation

These require the streamlit variant of the Docker image:

STREAMLIT=1 scripts/run-local.sh

Run the streamlit apps like this:

scripts/run-streamlit.sh

This opens a multi-page app that lets you select specific scripts.

To add a page, simply create a script in the streamlits/pages folder.

When you run a recipe, ICE will evaluate the results based on the gold standards in gold_standards/. You'll see the results on-screen, and they'll be saved as CSVs in data/evaluation_csvs/. You can then upload the CSVs to the "Performance dashboard" and "Individual paper eval" tables in the ICE Airtable.

1. Set up both ice and elicit-next so that they can run on your computer
2. Switch to the eval branch of elicit-next, or a branch from the eval branch. This branch should contain the QA code and gold standards that you want to evaluate.
3. If the ice QA gold standards (gold_standards/gold_standards.csv) may not be up-to-date, download this Airtable view (all rows, all fields) as a CSV and save it as gold_standards/gold_standards.csv
4. Duplicate the All rows, all fields view in Airtable, then in your duplicated view, filter to exactly the gold standards you'd like to evaluate and download it as a CSV. Save that CSV as api/eval/gold_standards/gold_standards.csv in elicit-next
5. Make sure api/eval/papers in elicit-next contains all of the gold standard papers you want to evaluate
6. In ice, run scripts/eval-in-app-qa.sh <path to elicit-next>. If you have elicit-next cloned as a sibling of ice, this would be scripts/eval-in-app-qa.sh $(pwd)/../elicit-next/.

This will generate the same sort of eval as for ICE recipes.

TORCH=1 scripts/run-local.sh

Cheap integration tests:

scripts/run-recipe.sh --mode test

Unit tests:

scripts/run-tests.sh

1. Manually add the dependency to pyproject.toml
2. Update the lock file and install the changes:

docker compose exec ice poetry lock --no-update
docker compose exec ice poetry install # if you're running a variant image, pass --extras streamlit or --extras torch

The lockfile update step will take about 15 minutes.

You do not need to stop, rebuild, and restart the docker containers.

To upgrade poetry to a new version:

1. In the Dockerfile, temporarily change pip install -r poetry-requirements.txt to pip install poetry==DESIRED_VERSION
2. Generate a new poetry-requirements.txt:

   docker compose -f docker-compose.yml -f docker-compose.build.yml up -d
   docker compose exec ice bash -c 'pip freeze > poetry-requirements.txt'

3. Revert the Dockerfile changes

Before making a PR, check linting, types, tests, etc:

scripts/checks.sh

Reminder: Traces contain source code, so be sure you want to share all the code called by your recipe.

1. Publish the trace to https://github.com/oughtinc/static and wait for the github-pages workflow to finish.
2. Add the trace information to ui/helpers/recipes.ts.