Configuring experiments

​

Configuration

​

Field reference

​

Top level fields

• apiversion (string, default: v1alpha1)
  Configuration schema version.
• experiment_name (string, required)
  Name of the experiment. Must be a valid DNS label ([a-z0-9-], max 63 characters, no leading -). If it ends with -, a random suffix is appended for uniqueness.

​

repo

• source (string)
  Where the experiment source code comes from. Can be:
  • A remote git URL — https://, ssh://, or git@.
  • A local directory path (absolute or relative; ~ and environment variables are expanded). The directory is uploaded directly, so you can run experiments on uncommitted local changes.
  If omitted (null), no source is uploaded — the base image must already contain the code at the sandbox workdir.
• files (list of strings, default: [])
  Files and directories to include from source. An empty list includes everything. Patterns are applied in order; prefix an entry with ! to exclude. Globs (*, ?, […]) are supported. Hidden files and symlinks are skipped.
  repo:
    files:
      - src              # include the src directory
      - "*.py"           # include top-level Python files
      - "!src/secrets"   # exclude a subdirectory
  
• branch (string, default: main)
  Branch to use when cloning a remote source.
• evaluation_script (string, default: evaluation.py)
  Script that evaluates experiment results (path relative to repo root).
• evaluation_arguments (list, default: [])
  Splits the evaluation into a multi-evaluator, with one entry per evaluator. Each entry is passed to evaluation_script as its command-line arguments. Write a single value for one argument, or a list for several. Requires aggregation_script.
  repo:
    evaluation_arguments:
      - '0'                 # runs: evaluation_script 0
      - ['--slice', '1']    # runs: evaluation_script --slice 1
  
• aggregation_script (string, default: null)
  Script that combines the per-evaluator results into the experiment’s final result. Only valid together with evaluation_arguments. See multi-evaluator.
• target_code (list of strings, default: [])
  Code for agents to evolve. Must be a YAML list — see file list syntax. An empty list allows the Hive to evolve any file in the specified codebase (except for the evaluation script). Prefix an entry ! to exclude it from evolution, e.g.,
  repo:
    target_code:
      - "!fixed.py"
  
  allows the Hive to evolve any file except for fixed.py and the evaluation script.
• additional_context (list of strings, default: [])
  Additional files to include as context. Must be a YAML list — see file list syntax.

​

File list syntax

target_code:
  - main.py               # entire file
  - main.py:1-50          # lines 1 to 50
  - main.py:1-10&21-30    # multiple ranges
  - evolve.py             # another file

​

Multi-evaluator

1. For each entry in evaluation_arguments, the Hive runs evaluation_script with those arguments. Each sub-evaluator emits the usual final-line JSON: {"status": "success", "result": {"fitness": ...}} on success, or {"status": "failed", "error": ...} on failure.
2. The aggregation_script receives a single argument: the path to a JSON file holding the list of per-evaluator results, in evaluation_arguments order. Failed evaluators appear in that list too, so the aggregator must check each status field.
3. The aggregation script’s own output follows the same contract as evaluation_script, and its fitness becomes the experiment’s fitness.

import json
import sys

if __name__ == "__main__":
    with open(sys.argv[1]) as f:
        results = json.load(f)          # one entry per evaluator, in order

    # Failed evaluators are included, so handle them explicitly.
    if any(r["status"] != "success" for r in results):
        print(json.dumps({"status": "failed", "error": "an evaluation failed"}))
        sys.exit(0)

    # Combine the per-evaluator fitnesses however your objective requires.
    fitness = sum(r["result"]["fitness"] for r in results)
    print(json.dumps({"status": "success", "result": {"fitness": fitness}}))

repo:
  evaluation_script: evaluator.py
  evaluation_arguments:
    - '0'
    - ['--slice', '1']
  aggregation_script: aggregator.py
  target_code:
    - main.py

runtime:
  # One sandbox per sub-evaluation (2) times the number of parallel attempts (1).
  num_sandboxes: 2

​

runtime

• num_sandboxes (integer, default: 1)
  Number of parallel sandboxes to run. (Formerly num_agents, now deprecated but still accepted.)
• max_runtime_seconds (integer, default: -1)
  Maximum execution time in seconds. -1 = unlimited.
• max_iterations (integer, default: -1)
  Maximum iterations per agent. -1 = unlimited.
• stochastic_evaluator (boolean, default: false)
  Whether the evaluator is stochastic. When true, the Hive accounts for evaluation noise by re-evaluating high-variance candidates so fitnesses can be compared reliably.

​

sandbox

• base_image (string, required)
  Docker base image (e.g. python:3.14-slim).
• workdir (string, default: /app)
  Working directory inside the container.
• setup_script (string, default: null)
  Shell script to run before the experiment starts (e.g. install dependencies). Omit or set to null if no setup is needed.
• evaluation_timeout (integer, default: 60)
  Maximum time (seconds) for the evaluation script to run.
• envs (list)
  Environment variables to set in the container. Each entry has a name and value:
  envs:
    - name: DATABASE_URL
      value: postgres://localhost/mydb
  
• services (list)
  Sidecar services to run alongside the sandbox. See sandbox.services below.

​

sandbox.resources

• cpu (string, default: "1")
  CPU limit (e.g. "2", "500m").
• memory (string, default: "2Gi")
  Memory limit (e.g. "4Gi", "512Mi").
• accelerators (string)
  GPU resources (e.g. a100-80gb:8).
• shmsize (string)
  Size of /dev/shm (e.g. "1Gi"). This is only supported if accelerators is unspecified, or if an AWS accelerator is specified.

​

sandbox.services[]

• name (string, required)
  Service name.
• image (string, required)
  Docker image for the service.
• ports (list)
  Ports to expose. Each entry has a port (integer) and optional protocol (TCP or UDP, defaults to TCP).
• envs (list)
  Environment variables (same format as sandbox.envs).
• command (list of strings)
  Container entrypoint override.
• args (list of strings)
  Arguments to the entrypoint.
• resources.cpu (string, default: "1")
  CPU limit for this service.
• resources.memory (string, default: "2Gi")
  Memory limit for this service.

​

provider.regions

• regions (string or list of strings, default: us)
  Where the sandboxes run, and so where the experiment’s code and data are processed: us for only the US region, or all for every region. Setting all may help with GPU availability, since sandboxes can land on capacity in any region. Only set all if your code and data may leave the US.
  provider:
    regions: all
  

​

prompt

• context (string)
  Experiment-specific context to provide to agents.
• ideas (list of strings)
  Ideas that will be randomly sampled and injected into the Hive.
  ideas:
    - "Try batching database writes"
    - "Consider async I/O for network calls"
  
• attachments (list of strings, default: null) PDF reference material for agents. Accepts local PDF files, local directories, and PDF URLs.

​

PDF attachments

prompt:
  attachments:
    - papers/reference.pdf
    - background/
    - https://arxiv.org/pdf/2605.10327

​

agents

• preset (string)
  A preset tier of agents, by name. One of low-effort, medium-effort, or high-effort.
  agents:
    preset: high-effort
  
• models (list)
  A custom ensemble of models. Each entry takes:
  • model (string, required) — The model to run. Run hive list models to see the names.
  • weight (float, default: 1) — How likely the model is to be used, relative to the rest of the ensemble. Must be greater than 0.
  agents:
    models:
      - model: openai.gpt-5.6-terra
        weight: 2.0                    # Used twice as often as the model below
      - model: anthropic.claude-opus-5
        weight: 1.0
  

​

Full example

apiversion: v1alpha1
experiment_name: my-experiment-

agents:
  preset: high-effort

repo:
  source: https://github.com/your-org/your-repo.git
  branch: main
  evaluation_script: evaluation.py
  target_code:
    - main.py:1-50
  additional_context:
    - utils.py:10-30

runtime:
  num_sandboxes: 10
  max_runtime_seconds: 3600
  max_iterations: 100
  stochastic_evaluator: false

sandbox:
  base_image: python:3.14-slim
  workdir: /app
  evaluation_timeout: 60
  setup_script: |
    pip install -r requirements.txt
    python setup.py
  resources:
    cpu: "2"
    memory: "4Gi"
    accelerators: a100-80gb:8
  envs:
    - name: DATABASE_URL
      value: postgres://localhost/mydb
    - name: DEBUG
      value: "true"
  services:
    - name: redis
      image: redis:7-alpine
      ports:
        - port: 6379
          protocol: TCP
      resources:
        cpu: "500m"
        memory: "512Mi"
    - name: worker
      image: my-worker:latest
      command: ["python"]
      args: ["-m", "worker", "--queue", "default"]
      envs:
        - name: WORKER_CONCURRENCY
          value: "4"
      resources:
        cpu: "1"
        memory: "1Gi"

prompt:
  context: "Focus on optimizing the data pipeline for throughput"
  ideas:
    - "Try batching database writes"
    - "Consider async I/O for network calls"
    - "Explore caching intermediate results"
  attachments:
    - https://arxiv.org/pdf/2605.10327

​

Next steps