How to compare AI models, effort settings and instructions fairly

1. Fix the task, the workload and the success criterion

Before comparing anything, write down three things and keep them the same for both setups:

• Task: the job, for example summarising support tickets.
• Workload: the exact set of inputs both setups will run on, such as the same 40 archived tickets.
• Success criterion: how a reviewer decides an output is good enough, for example “names the customer’s issue and the next step”.

If the workload or criterion changes between runs, the comparison stops being fair.

2. Run both setups and count attempts, including retries

Run the baseline (what you use today) and the candidate (what you want to try) on the same workload. Count every attempt, including retries after a failure or timeout, and count how many attempts met the criterion. Use the same number of attempts for both, so the pass counts are comparable.

3. Total the tokens, cost and latency, retries included

Add up total input tokens, total output tokens, total API cost and total latency across all attempts, retries included. If outputs also go through an automated check or a second review pass, include that overhead in the totals too. Retries and review are part of the real cost of a setup, so leaving them out flatters whichever setup failed more often.

Keep this to usage-based API cost. Do not fold subscription or seat charges into the comparison; they are billed differently and belong in your spend records.

4. Keep measured values and estimates apart

Mark each setup as an observation you measured, or as an estimate you worked out from list prices or a smaller sample. Both are useful, but they should never be mixed in one conclusion. An estimate is a reason to run a proper test, not a result.

A worked example (fictional)

A fictional company wants to know whether its support-summary workflow can use a lower effort setting with a revised instruction. Both setups ran on the same 40 archived tickets, with 4 retries each, so 44 attempts each. All figures below are made up for illustration.

The candidate met the criterion slightly more often, used more input tokens because its instructions are longer, and finished faster at a lower recorded cost for this workload. A one-off result like this describes this workload on this date. It does not prove equal quality on other work or a lasting saving, so the team keeps both setups recorded and retests when anything changes.

Recording a comparison in Vaze

Run the tests outside Vaze, with tools and data you are authorised to use. In Vaze, enter labels and aggregate counts only. Never paste prompts, outputs, ticket contents or private test sets into it.

1. Open More, then Environment, and select the asset the comparison belongs to, such as the support-summary instruction set. If it is not recorded yet, choose Add asset.
2. Under Optimisation comparisons, choose Record a comparison.
3. Fill in Task label, Workload / test-set label, Success criterion and Observation date (UTC).
4. For both Baseline and Candidate, enter the Model and dated version, Effort setting, Instruction / prompt revision label and Evidence (Observed · entered by you or Estimated scenario), then Attempts (including retries) and Attempts meeting the criterion.
5. Optionally add Total input tokens, Total output tokens, Total API cost and Total latency (ms), then choose Save comparison.

Recording a comparison needs write access to your company’s dashboard, and the asset must not be retired. Entering costs also needs Spend access: cost fields and the currency are only shown to people with it, and others see Withheld · Spend access required. Vaze asks for the same number of attempts on both sides. Saved observations cannot be edited, so to correct or repeat a test you record a new comparison.

When something changes, retest

Each comparison remembers the recorded setup it was saved against: the asset’s record in Environment and the records of the assets linked directly to it. If any of those records is edited or retired, the comparison shows Setup changed · retest instead of Recorded setup unchanged. That is the prompt to run the workload again and record a fresh comparison.

Vaze compares its own records only. It does not see changes inside the underlying files, prompts or systems, so update the record when you change one of those, and retest.