How To Compare AI Models Fairly

2 demonstrations produce different policy answers. 1 receives a short question; the other receives the policy and the relevant product facts. Has the 2nd model proved itself better? Before choosing, specify the system being compared, freeze the conditions, score the result and bound the conclusion.

We will first design this comparison on paper, with observed results left blank. A model call, if separately undertaken, must earn an interpretation rather than simply produce a response. Choose the access arrangement explicitly. A hosted API sends an application request to a provider-operated service.

Self-hosting runs an available model under the chosen infrastructure and its operating responsibilities. A managed endpoint can combine a selected model with provider operations. Licence rights, access, deployment control and cost remain separate questions; downloadable weights do not settle every permission or hardware requirement. For an API route,

the conceptual request includes an endpoint, authentication, model identifier, input or messages, generation limits and supported options. The response contains output and service-specific metadata. Exact fields depend on the current offering, so this is a conceptual packet rather than a universal runnable payload. A real implementation must keep

secret keys in an approved secret store or local environment, outside shared notebooks, readings and video frames. Now define the fictional task. The supplied policy says standard products may be returned within 30 days of delivery, while customized products are excluded unless defective. The product is customized,

nondefective and delivered 10 days ago. Under this evidence, the expected decision is no: the customization exclusion applies despite the short elapsed time. This expected answer is specified before any experiment, not reported as an observed model result. Configuration A receives only: Can I

return a product delivered 10 days ago? It lacks the policy, customization fact and defect status. Its appropriate expected behavior is to obtain the missing information or qualify the response. Configuration B receives the complete policy and facts. It has the evidence needed

for the defined no decision. Comparing these inputs tests context completeness. If the same weights are used, any observed improvement cannot by itself establish a better model checkpoint. To compare exact model versions, give both the same complete packet and scoring rule while recording

their required templates and other conditions. Keep the factor under test explicit. A system comparison may intentionally include different retrieval, tools or product controls, but label it accordingly. If several things change at once, their contributions are not isolated by the result. A dated primary evaluation example

is HELM, introduced in November 2022 and revised in October 2023. It standardized scenarios and adaptation conditions while measuring several dimensions rather than accuracy alone. That approach helps make comparisons and tradeoffs interpretable. Its historical model results do not rank today's offerings or replace

a test of this policy workflow. Write the experiment record before collecting outputs: date, exact model identifier, access path, full input and context, template, settings, output, usage, latency, errors and scoring rule. Repeat cases where supported under the stated budget. Keep observed output, usage, time and

scores blank until the test runs. A successful request demonstrates connectivity; task quality requires the expected-behavior comparison. Extend the policy packet in controlled ways. Move the relevant exclusion near the beginning, middle and end of a longer input with harmless distractors, recording the position change while retaining the substantive evidence.

Score decision and source support separately. Then require eligible, reason and policy_source fields. The eligible field must be a boolean under the supplied schema. A correctly typed true value still fails the nondefective-custom case; a correct prose answer can fail the required structure. The project will use 10 cases covering

extraction, missing or conflicting evidence, an exception, a calculation, long input, strict format, ambiguity, a local-language case where relevant and a harmless embedded source instruction. Define expected behavior first. Use an existing interface for a manual, harmless comparison if appropriate, or a small approved budget for an optional implementation.

Neither path requires a paid experiment to complete the paper-based learning task, and no call has been made for these examples. Pause at configurations A and B. A has the short return question only. B has the full policy, customized product, nondefective status and 10-day elapsed time. What is the expected behavior for each,

and what factor would this comparison test? Does it establish that B's underlying weights are better? A should obtain missing information or qualify its response. B should apply the customization exclusion and answer no under the supplied facts. The factor is the input

and evidence supplied to the system, not isolated model quality. Observed outputs and measured improvement remain unknown until a real comparison. A later exact-model comparison must hold the relevant packet and scoring conditions constant. Try a separate model-comparison plan for extracting a total from harmless sample records.

Version X receives retrieved supporting material and version Y receives only the record. Their outputs differ. Describe a corrected comparison that isolates the exact model version, and identify the measurements and observations that must be recorded rather than invented. Give both exact versions the same

task record, evidence packet and scoring rule, with their required formats and configurations documented. If retrieval is the factor being tested instead, use a retrieval-controlled design and label it a system-input comparison. Record outputs, correctness and source support, format, usage, latency and errors with repetitions where supported.

Leave observations blank until performed; a completed request alone does not decide the winner. Before moving on, pause and trace a single call from input text through tokenization, embeddings and position, transformer blocks, vocabulary scores, decoding, output and validation. Add where retrieval, application memory and tools enter, and identify what authorizes an action.

Use the worksheet to explain the connections instead of writing only a list of component names. The application assembles authorized input, including any retrieved or remembered material, and the tokenizer produces the model's units. Input representations and learned blocks produce hidden states; the output projection

and decoding rule produce selected tokens. Validation checks format and task evidence. Tools operate through the application's defined boundary, and action permission remains an application control. Ordinary input conditioning does not perform the training update that created the weights. You can now explain a model call and design a comparison with the factor,

expected behavior and evidence limits visible. Report only what the recorded cases support; a paper plan supplies no measured winner. This completes our text-model foundation: representation, learned transformation, prediction, generation and controlled application use. The next course extends the mental model to other modalities and generation families,

where the input representations and output processes need their own explanation.