Prepare an experiment

1. A target — which files (or lines) the Hive is allowed to modify
2. An evaluator — a script that scores each candidate solution
3. A sandbox — the environment where evaluations run

• Is the code you want to improve clearly separable from the surrounding harness?
• Do you have a quantifiable metric? If a higher score doesn’t mean a better algorithm, you’re measuring the wrong thing.
• Would domain knowledge or directional hints help narrow the search?

​

Define the target

repo:
  target_code:
    - evolve.py

​

Write the evaluator

repo:
  evaluation_script: path/to/evaluation.py

import json
import sys

if __name__ == "__main__":
    result = run_candidate()              # exercise the evolved code
    if not is_correct(result):            # verify output before scoring
        # A failed evaluation is a normal outcome — report it via JSON and
        # exit 0. Reserve a non-zero exit for genuine crashes.
        print(json.dumps({
            "status": "failed",
            "error": "incorrect output",
        }))
        sys.exit(0)

    fitness = score(result)               # quantify quality (higher is better)
    print(json.dumps({
        "status": "success",
        "result": {"fitness": fitness},
    }))

​

Feedback Format

​

Single Objective

{
    "status": "success",
    "result": {
        "fitness": 0.85,
        "feedback_message": "Throughput improved by 23% over baseline"
    }
}

​

Multi Objective

{
    "status": "success",
    "result": {
        "fitness": {
            "speedup": 2.48,
            "accuracy": 0.94
        },
        "feedback_message": "forward() accounts for 73% of total runtime; matmul on line 42 is the primary bottleneck"
    }
}

​

Failed Evaluation

{"status": "failed", "error": "Output did not match the expected result"}

​

Tips and tricks

• Enforce correctness explicitly. Never assume the output of the evaluator is correct. You should always add tests or checks to guarantee the code produces correct results before scoring performance. Your evaluator should return a failure if the code produces incorrect results, regardless of performance.
• Guard against reward hacking. The Hive will exploit any shortcut that inflates the score — caching results, short-circuiting logic, or producing hardcoded outputs. Build validation checks into your evaluator and treat unexpectedly large score jumps with skepticism.
• Make evaluations deterministic. If your evaluator has stochasticity, run it multiple times and report the mean or median. Noisy scores make it harder for the Hive to distinguish genuine improvements.
• Stochastic evaluations. If your evaluator can’t be made deterministic, the Hive supports an adaptive approach to re-sample high-variance results, see runtime.stochastic_evaluator.
• Keep evaluations fast. The Hive iterates faster with quick feedback. Balance evaluation thoroughness with speed — consider if a lighter test suite provides sufficient signal compared to an exhaustive one.
• Use relative scoring for timed benchmarks. Hardware performance can vary between sandboxes. If optimizing for runtime, instead of reporting raw times, run a baseline alongside each candidate and report the speedup factor.
• Maximize by convention. The Hive always assumes objectives should be maximized. If you have a quantity you want to minimize, return the negative of that value as the fitness.

​

Configure the sandbox

sandbox:
  base_image: python:3.14-slim
  setup_script: |
    pip install -r requirements.txt
  resources:
    cpu: "2"
    memory: "4Gi"
    shmsize: "1Gi"
    accelerators: a100-80gb:8
  evaluation_timeout: 60

• base_image: The Docker image used as the base environment. Because evaluation scripts are run with python3, the base image must include python3. The Hive will attempt to install the following system-dependencies if missing: rsync, git, jq, iptables, and util-linux — using one of the following supported package managers: apt-get (Debian/Ubuntu), yum (RHEL/CentOS), or apk (Alpine). Good choices are the official python images (e.g. python:3.14-slim) or, for GPU workloads, an ML image that ships Python such as pytorch/pytorch. You can also build and use your own custom base image, as explained here.
• setup_script: Shell commands that run once when the sandbox is first created. Use this to install dependencies or download data.
• resources: CPU, RAM, and shared memory allocated to the sandbox. GPUs can be added with the format <accelerator-name>:<num-gpus>.

  Accelerator	GPU
  a100-80gb	NVIDIA A100 80GB
  a100-40gb	NVIDIA A100 40GB
  h100	NVIDIA H100
  h200	NVIDIA H200
  b200	NVIDIA B200
  a10	NVIDIA A10
  t4	NVIDIA T4
  l4	NVIDIA L4
  l40s	NVIDIA L40S

  If you don’t have a strict constraint on target hardware, allocate resources with some headroom. Resource exhaustion is reported as an evaluation failure, and high-performing solutions can consume more resources than expected.
• evaluation_timeout: Maximum time in seconds before an evaluation is terminated.
  Set this with some headroom — high-performing candidates can take longer than expected, and an overly tight timeout may discard good solutions.

​

Provide context

prompt:
  context: |
    This is some sample context about the problem.
    It can be formatted like this to span multiple lines.

prompt:
  ideas:
    - This is one idea to explore.
    - This is a different idea to explore.

repo:
  additional_context:
    - context/file.py
    - another/context/file.py