How to test AI models on your own marketing brief, in 5 steps

Every new AI model arrives with a chart showing it beat the others on a benchmark. Then you give it your product launch email and it writes something that sounds like a bank. The benchmark measured what the lab cares about. You care about whether it can write for your customers in your voice, without making things up.

The only way to know is to test the models yourself on your own work. That sounds like a research project, but it takes about an hour if you follow a fixed method. Here is the one I use, with a scoring sheet you can copy.

Why public benchmarks don't answer your question

A benchmark is a standard test set that labs run their models on so they can compare scores. They are useful for spotting big jumps. Anthropic, for example, reported that Claude Sonnet 5.5, released on 28 September, scored 70.6% on Terminal-Bench 4.0, a test of coding tasks done in a terminal, against 10.3% for Sonnet 5. That tells a developer something real.

It tells a marketer almost nothing. No public benchmark scores "does this caption sound like my brand" or "did it invent a discount we never offered". Those are your criteria, and only your own test can measure them.

Anthropic's own documentation says much the same thing to developers. It advises defining success criteria that are specific, measurable, achievable and relevant, and building tests that "mirror the real-world task distribution", including awkward edge cases. That advice works just as well for a marketing team.

Step 1: Write one real brief and decide what "good" means

Pick a task you do every week. A product launch post, a Meta ad with three headline options, a follow-up email after a free consultation. Use a real past brief, so you already know what a good answer looks like.

Then, before you run anything, write down four to six criteria and what each one means in a sentence. Doing this first matters. If you choose criteria after reading the outputs, you will reward whichever one you liked first.

Typical criteria for marketing work:

• Accuracy: uses only the facts in the brief. No invented prices, features, dates or claims.
• Brand voice: sounds like us. Compare against two past posts you liked.
• Audience fit: speaks to the right customer's problem in their words.
• Follows the format: right length, right number of options, platform rules respected.
• Persuasion: a clear hook and a clear next step.
• Edit time: how many minutes until you would publish it.

Give each criterion a weight. Accuracy should usually carry the most, because one invented claim in an ad costs more than a dull headline.

Step 2: Choose three models and give them identical conditions

Three is enough. Fewer and you learn little, more and the scoring takes too long. Right now a sensible set would be one each from the big labs, for example Claude Sonnet 5.5 from Anthropic, Gemini 4 Argon from Google and GPT-6 Sol from OpenAI, or the versions your plan includes.

Make the test fair:

• Paste the exact same brief into each one.
• Start a fresh chat for each run, so no earlier conversation leaks in.
• Turn off saved memory or custom instructions for the test, or set them the same in every tool.
• Attach the same files, if any.

Write the brief once in a document and copy it from there every time. Retyping it introduces small differences that change the output.

Step 3: Run each model three times and hide the names

AI models give a different answer each time you ask. One great answer could be luck. So run the brief three times in each model, in three fresh chats. That gives you nine outputs.

Now blind them, which means hiding where each one came from: copy each output into a document labelled only with a letter (A to I) and keep a separate key of which letter came from which model. Better still, ask a colleague to do the labelling so you don't know either.

Blinding matters more than people expect. Most of us already have a favourite AI, and if you know which output is from it you will score it higher without meaning to.

Step 4: Score every output on the sheet

Score each blind output from 1 to 5 on each criterion. Then multiply by the weight and add up. Here is the sheet. Copy it into a spreadsheet with one row per output.

The weighted total is (Accuracy × 3) + (Voice × 2) + (Fit × 2) + Format + Persuasion. With these weights the maximum is 45.

To show how the arithmetic works, here is one invented row. These numbers are not a test result. Accuracy 4, voice 3, fit 4, format 5, persuasion 3 gives (4 × 3) + (3 × 2) + (4 × 2) + 5 + 3 = 34 out of 45.

Two rules while scoring:

• Any invented fact sets Accuracy to 1, however good the rest is. Write the invented claim in the Notes column.
• Time the edit. Edit minutes is often the most honest number on the sheet. An output that scores well but needs twenty minutes of fixing is worse than a plainer one you can post in two.

If you want a second opinion on tone, you can ask a different AI to score it against your written criteria. Anthropic's documentation recommends using a different model from the one that wrote the output. Treat that score as a helper. Your own score decides.

Step 5: Unblind, average, and check the cost

Now open the key. For each model, average its three weighted totals and its three edit times. Also look at the spread: a model that scored 40, 41 and 39 is more useful than one that scored 44, 30 and 42, because you can rely on it.

Then add the practical columns:

Price matters less than it used to. API prices dropped sharply this autumn: Anthropic's Claude Haiku 5.5, released 7 October, costs $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100,000 tokens, and Google's Gemini 4 Argon launched on 13 October at an introductory $2 and $10. If you use the chat apps on a subscription, the question is simply which plan you already have.

Pick the winner for this task only. It is normal for one model to win your ad copy and another to win your long-form emails. Write the result down with the date, because models change and you will want to rerun the test when a new version arrives.

Common mistakes

• Testing with "write me an Instagram post about shoes". A vague brief tests nothing. Use a real brief with real facts.
• Scoring after reading. Criteria written afterwards bend to fit your favourite.
• One run per model. You are measuring luck.
• Ignoring invented facts because the writing is good. In marketing, a false claim is the worst failure.
• Never rerunning. A test from three months ago is about an older model.