Skip to content

How to configure Vivaria to fetch agents and tasks from a Git host

Context

Vivaria can run in two modes:

  1. Users must upload agent code, task code, and secrets to Vivaria using the appropriate flags on viv run and viv task start
  2. Users push changes to agents, tasks, and secrets to repositories on a Git hosting service (e.g. GitHub). Vivaria reads

This how-to covers how to set up Vivaria to support the second mode.

Required setup

  1. A repository on the Git host that holds all your METR Task Standard task families. Each task family should have its own top-level directory in the repository. E.g. github.com/my-org/my-metr-tasks
  2. A collection of repositories on the Git host, one per Vivaria agent, all under the same top-level path on the Git host. E.g. github.com/my-org-agents. (This can be the same organization as owns the task repo, or a different one.)

Instructions

Set ALLOW_GIT_OPERATIONS: true in the environment section for the server and background-process-runner in docker-compose.override.yml (if running under Docker Compose, see docker-compose.dev.yml for an example) or server/.env (if not).

Then, add the following to your .env.server or server/.env:

# Make sure you fill in the placeholders (e.g. ${USERNAME})

# Don't forget to change github.com if you're using a different Git hosting service.
TASK_REPO_URL=https://${USERNAME}:${GITHUB_ACCESS_TOKEN}@github.com/my-org/my-metr-tasks

# Although this environment variable references GitHub specifically,
# Vivaria should be able to support non-GitHub hosting services.
GITHUB_AGENT_ORG= # e.g. my-org-agents

# Although this environment variable references GitHub specifically,
# Vivaria should be able to support non-GitHub hosting services.
# Don't forget to change github.com if you're using a different Git hosting service.
GITHUB_AGENT_HOST=https://${USERNAME}:${GITHUB_ACCESS_TOKEN}@github.com

Git LFS support for large assets

If your task needs to use a large asset such as a training dataset, you can use Git LFS to manage it.

  1. Add the large asset to the repository, e.g. under ${TASK_FAMILY_NAME}/assets
  2. Use git lfs track from the ${TASK_FAMILY_NAME} directory to start tracking the asset.
  3. git add ${TASK_FAMILY_NAME}/.gitattributes ${TASK_FAMILY_NAME}/assets

It's important that the .gitattributes file is created in the task family directory, not in the assets subdirectory or in the root of the repository.