Skip to content

How to create a new agent

In Vivaria, agents are arbitrary Python code that interacts with Vivaria using a library called pyhooks.

If you want to create a totally new agent, all you need to do is create a folder named after the new agent and add a main.py and a requirements.txt.

In practice, it's easiest to duplicate an existing agent and alter it to your liking. Check out this agent to begin adapting it to your use case.

Here's a non-exhaustive list of things that an agent could do:

  • Be written in a programming language other than Python (main.py still needs to exist but can invoke e.g. a binary that was compiled from C or Go)
  • Install arbitrary dependencies (e.g. a headless browser)

pyhooks

pyhooks is a Python library that Vivaria agents use to communicate with Vivaria. Agents communicate with Vivaria to get completions from LLM APIs and to record the trajectory of a run, e.g. LLM API requests made, actions taken, results of actions, errors encountered.

pyhooks also contains:

  1. Tools shared between METR agents (e.g. code for running shell commands, code for interacting with a Python interpreter session that persists between Python tool calls)
  2. Code for tracking an agent's process ID, output, and exit code, then sending them to Vivaria

OpenAI clone API

Vivaria provides a limited clone of the OpenAI API. Right now, the clone API supports the v1 API's /models, /chat/completions, and /embeddings endpoints.

The clone API is useful for running third-party agents that expect to have access to an OpenAI-shaped API, e.g. AutoGPT.

Calling the OpenAI clone API has several advantages over calling the OpenAI API directly:

  1. Vivaria makes the OpenAI clone API accessible in all task environments, even no-internet ones
  2. Requests to the OpenAI clone API count towards token usage limits
  3. Vivaria automatically logs OpenAI clone API requests as generation trace entries in the agent branch's trace

To use this API, agents can use an OpenAI API client or call the API endpoints using requests or another library. Agents should make sure to use the values of the OPENAI_API_KEY and OPENAI_API_BASE_URL environment variables that Vivaria provides to them.

Keeping old agents around

Once you've run a particular commit of a particular agent, we suggest not deleting its repository or removing. That's so that you and other users can find the agent's code in the future.