> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hiverge.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Prepare an experiment

> How to prepare a codebase to run a custom Hive experiment

Setting up a Hive experiment requires three things:

1. **A target** — which files (or lines) the Hive is allowed to modify
2. **An evaluator** — a script that scores each candidate solution
3. **A sandbox** — the environment where evaluations run

Optionally, you can also provide natural language context to guide the search.

The Hive can evolve arbitrarily large codebases, but like a human researcher, it performs best when its task is well-defined. Before starting, consider:

* Is the code you want to improve clearly separable from the surrounding harness?
* Do you have a quantifiable metric? If a higher score doesn't mean a better algorithm, you're measuring the wrong thing.
* Would domain knowledge or directional hints help narrow the search?

## Define the target

The Hive needs to know which files in your codebase it is allowed to modify. This is specified in the configuration YAML using the `repo.target_code` field:

```yaml theme={null}
repo:
  target_code:
    - evolve.py
```

It is possible to specify multiple files to evolve, and specific line ranges too, see [here](/gettingstarted/cli/configuration#file-list-syntax).

Everything outside these paths is treated as fixed infrastructure. This lets you isolate the algorithm or heuristic you want to improve while keeping the surrounding harness stable.

## Write the evaluator

One of the most important parts to setting up a Hive experiment is defining an evaluator. Concretely, this is a Python script specified in the configuration YAML file as a local path relative to the codebase root.

```yaml theme={null}
repo:
  evaluation_script: path/to/evaluation.py
```

This script is run as `python3 path/to/evaluation.py`, and must print a JSON object to the final line of stdout. A typical evaluation script looks something like:

```python theme={null}
import json
import sys

if __name__ == "__main__":
    result = run_candidate()              # exercise the evolved code
    if not is_correct(result):            # verify output before scoring
        # A failed evaluation is a normal outcome — report it via JSON and
        # exit 0. Reserve a non-zero exit for genuine crashes.
        print(json.dumps({
            "status": "failed",
            "error": "incorrect output",
        }))
        sys.exit(0)

    fitness = score(result)               # quantify quality (higher is better)
    print(json.dumps({
        "status": "success",
        "result": {"fitness": fitness},
    }))
```

### Feedback Format

The evaluation script communicates its feedback to the hive-coordinator via stdout. This is done by writing a JSON on the last line of stdout.

<Warning>
  The JSON must be printed as a single line on the last line of the evaluator's output.
</Warning>

#### Single Objective

On successful evaluation, the JSON should have the following form:

```json theme={null}
{
    "status": "success",
    "result": {
        "fitness": 0.85,
        "feedback_summary": "Throughput improved by 23% over baseline"
    }
}
```

The Hive optimizes for an algorithm which maximizes the quantity reported under `result.fitness`.

#### Multi Objective

You can also define an evaluator with multiple objectives to be optimized simultaneously by reporting `result.fitness` as a mapping of objective names to scores, e.g.,

```json theme={null}
{
    "status": "success",
    "result": {
        "fitness": {
            "speedup": 2.48,
            "accuracy": 0.94
        },
        "feedback_summary": "forward() accounts for 73% of total runtime; matmul on line 42 is the primary bottleneck"
    }
}
```

#### Failed Evaluation

When an evaluation fails — for example the candidate code fails correctness checks or errors out — report the failure with an `error` message instead:

```json theme={null}
{"status": "failed", "error": "Output did not match the expected result"}
```

| Field                     | Type                                     | Description                                                                                                                                        |
| ------------------------- | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `status`                  | string                                   | `"success"` or `"failed"`                                                                                                                          |
| `result.fitness`          | number, or mapping of strings to numbers | A single score, or — for multi-objective — a dictionary mapping each objective's name to its numeric score (required when `status` is `"success"`) |
| `result.feedback_summary` | string (optional)                        | Summary passed back to the agent                                                                                                                   |
| `error`                   | string                                   | Description of what went wrong (required when `status` is `"failed"`)                                                                              |

### Tips and tricks

The Hive optimizes for the score your evaluator returns — make sure that score faithfully represents what you actually want to improve.

* **Enforce correctness explicitly.** Never assume the output of the evaluator is correct. You should always add tests or checks to guarantee the code produces correct results before scoring performance. Your evaluator should return a failure if the code produces incorrect results, regardless of performance.
* **Guard against reward hacking.** The Hive will exploit any shortcut that inflates the score — caching results, short-circuiting logic, or producing hardcoded outputs. Build validation checks into your evaluator and treat unexpectedly large score jumps with skepticism.
* **Make evaluations deterministic.** If your evaluator has stochasticity, run it multiple times and report the mean or median. Noisy scores make it harder for the Hive to distinguish genuine improvements.
* **Stochastic evaluations.** If your evaluator can't be made deterministic, the Hive supports an adaptive approach to re-sample high-variance results, see [`runtime.stochastic_evaluator`](/gettingstarted/cli/configuration#runtime).
* **Keep evaluations fast.** The Hive iterates faster with quick feedback. Balance evaluation thoroughness with speed — consider if a lighter test suite provides sufficient signal compared to an exhaustive one.
* **Use relative scoring for timed benchmarks.** Hardware performance can vary between sandboxes. If optimizing for runtime, instead of reporting raw times, run a baseline alongside each candidate and report the speedup factor.
* **Maximize by convention.** The Hive always assumes objectives should be maximized. If you have a quantity you want to minimize, return the negative of that value as the fitness.

## Configure the sandbox

Once the evaluator is implemented, it is also important to consider what environment to run the code in. Each candidate solution is evaluated inside a sandboxed environment which can be customized in the [configuration YAML](/gettingstarted/cli/configuration#sandbox) as follows:

```yaml theme={null}
sandbox:
  base_image: python:3.14-slim
  setup_script: |
    pip install -r requirements.txt
  resources:
    cpu: "2"
    memory: "4Gi"
    shmsize: "1Gi"
    accelerators: a100-80gb:8
  evaluation_timeout: 60
```

Below, we briefly explain the most important fields and how to use them. See [this page](/gettingstarted/cli/configuration#sandbox) for a full list of available options.

* `base_image`: The [Docker](https://www.docker.com/) image used as the base environment. Because evaluation scripts are run with `python3`, the base image must include `python3`. The Hive will attempt to install the following system-dependencies if missing: `rsync`, `git`, `jq`, `iptables`, and `util-linux` — using one of the following supported package managers: `apt-get` (Debian/Ubuntu), `yum` (RHEL/CentOS), or `apk` (Alpine). Good choices are the official `python` images (e.g. `python:3.14-slim`) or, for GPU workloads, an ML image that ships Python such as `pytorch/pytorch`.
* `setup_script`: Shell commands that run once when the sandbox is first created. Use this to install dependencies or download data.
* `resources`: CPU, RAM, and shared memory allocated to the sandbox. GPUs can be added with the format `<accelerator-name>:<num-gpus>`.
  | Accelerator | GPU              |
  | ----------- | ---------------- |
  | `a100-80gb` | NVIDIA A100 80GB |
  | `a100-40gb` | NVIDIA A100 40GB |
  | `h100`      | NVIDIA H100      |
  | `h200`      | NVIDIA H200      |
  | `b200`      | NVIDIA B200      |
  | `a10`       | NVIDIA A10       |
  | `t4`        | NVIDIA T4        |
  | `l4`        | NVIDIA L4        |
  | `l40s`      | NVIDIA L40S      |
  <Tip>
    If you don't have a strict constraint on target hardware, allocate resources with some headroom. Resource exhaustion is reported as an evaluation failure, and high-performing solutions can consume more resources than expected.
  </Tip>
* `evaluation_timeout`: Maximum time in seconds before an evaluation is terminated.
  <Tip>
    Set this with some headroom — high-performing candidates can take longer than expected, and an overly tight timeout may discard good solutions.
  </Tip>

## Provide context

You can provide natural language context to guide the Hive's search. This is specified in the `prompt.context` field:

```yaml theme={null}
prompt:
  context: |
    This is some sample context about the problem.
    It can be formatted like this to span multiple lines.
```

Use this to describe the problem domain, suggest directions to explore, or set soft constraints on approaches the Hive should avoid.

If you have multiple distinct ideas you want the Hive to explore, list them in the `prompt.ideas` field:

```yaml theme={null}
prompt:
  ideas:
    - This is one idea to explore.
    - This is a different idea to explore.
```

On each iteration, the Hive will randomly sample one of these ideas to guide its approach.

Finally, if there are any files in the codebase that provide important context to writing the desired algorithm, such as internal dependencies, these can be specified in the `repo.additional_context` field.

```yaml theme={null}
repo:
  additional_context:
    - context/file.py
    - another/context/file.py
```