LLM benchmark for SEO: what AI agents cost and how often they finish
By Deian Isac · Last updated
Definition:
An LLM benchmark for SEO measures how language models do when an agent runs SEO work. This one is small and dated, and covers SEO and client-work tasks in two parts: how often each agent role and model finished real jobs, and what a run cost in model spend, data fees, minutes and requests; and controlled tests where several models did the same task, scored by two LLM judges. Every figure comes from Takibi Base, the agent workspace the author builds, dated and downloadable.
- Content writer
- SEO analyst
- Editor QA
- Data analyst
- Lead researcher
- Link builder
- Technical SEO auditor
- Runs counted
- 271
- Ended done
- 82%
- Role and model pairs
- 10 7 roles, 4 models
- Runs between
- – 2026
Runs on real work
Runs the workspace recorded on client and own-site work (SEO, content and research roles), for each role and model pair with at least 3 runs that recorded a model and an end time. It is a filtered subset, not every run; the method lists the filters. Each model got different tasks in a different period, so these rows say what a setup cost and how often it finished, not which model is better.
How often runs finish
Share of runs that ended done. Read the run count before the percentage: a few runs move the share a lot.
- Content writer meta/muse-ember91% of 53
- Content writer deepseek/deepseek-v4.1-flash-101784% of 19
- Data analyst meta/muse-ember78% of 32
- Editor QA xiaomi/mimo-v2.6-pro72% of 57
- Lead researcher xiaomi/mimo-v2.6-pro63% of 16
- Lead researcher tencent/hy4-flash100% of 10
- Link builder meta/muse-ember82% of 11
- SEO analyst meta/muse-ember91% of 46
- SEO analyst xiaomi/mimo-v2.6-pro82% of 22
- Technical SEO auditor xiaomi/mimo-v2.6-pro40% of 5
What a run costs
Median model spend plus median data fees per run, on a shared scale. The two medians are taken separately, so their sum is a guide, not the median total of a run. Roles that pull keyword, SERP or contact data pay for the calls.
- Content writer meta/muse-ember$0.0252 $0.0252 model + $0.0000 data medians
- Content writer deepseek/deepseek-v4.1-flash-1017$0.0270 $0.0270 model + $0.0000 data medians
- Data analyst meta/muse-ember$0.0485 $0.0301 model + $0.0184 data medians
- Editor QA xiaomi/mimo-v2.6-pro$0.1242 $0.1242 model + $0.0000 data medians
- Lead researcher xiaomi/mimo-v2.6-pro$0.0863 $0.0836 model + $0.0027 data medians
- Lead researcher tencent/hy4-flash$0.0085 $0.0085 model + $0.0000 data medians
- Link builder meta/muse-ember$0.0248 $0.0248 model + $0.0000 data medians
- SEO analyst meta/muse-ember$0.0906 $0.0330 model + $0.0576 data medians
- SEO analyst xiaomi/mimo-v2.6-pro$0.1116 $0.1010 model + $0.0106 data medians
- Technical SEO auditor xiaomi/mimo-v2.6-pro$0.3309 $0.2686 model + $0.0623 data medians
Full table
Every figure from the two charts, plus median minutes and model requests per run. Each row has an anchor you can link to.
| Role and model | Runs | Done | Model $ | Data $ | Minutes | Requests | Runs (2026) |
|---|---|---|---|---|---|---|---|
| Content writer meta/muse-ember | 53 | 91% | 0.0252 | 0.0000 | 1.1 | 15 | – |
| Content writer deepseek/deepseek-v4.1-flash-1017 | 19 | 84% | 0.0270 | 0.0000 | 2.9 | 29 | – |
| Data analyst meta/muse-ember | 32 | 78% | 0.0301 | 0.0184 | 1.2 | 15 | – |
| Editor QA xiaomi/mimo-v2.6-pro | 57 | 72% | 0.1242 | 0.0000 | 24.3 | 26 | – |
| Lead researcher xiaomi/mimo-v2.6-pro | 16 | 63% | 0.0836 | 0.0027 | 15.6 | 23 | – |
| Lead researcher tencent/hy4-flash | 10 | 100% | 0.0085 | 0.0000 | 3.9 | 23 | – |
| Link builder meta/muse-ember | 11 | 82% | 0.0248 | 0.0000 | 1.1 | 10 | – |
| SEO analyst meta/muse-ember | 46 | 91% | 0.0330 | 0.0576 | 1.6 | 18 | – |
| SEO analyst xiaomi/mimo-v2.6-pro | 22 | 82% | 0.1010 | 0.0106 | 12.2 | 23 | – |
| Technical SEO auditor xiaomi/mimo-v2.6-pro | 5 | 40% | 0.2686 | 0.0623 | 39.2 | 36 | – |
Measured single runs
Individual runs that the guides on this site use as field records, with what each one covered.
| Job and what it covered | Date | Model | Model $ | Data $ | Minutes |
|---|---|---|---|---|---|
| Keyword map, first version 11 seed keywords plus 4 related terms: volume, difficulty and 4 SERP reads from DataForSEO; includes one oversized call that was billed and returned nothing | meta/muse-ember | 0.0150 | 0.3190 | 2 | |
| Keyword map, revision adding Search Console 4 Search Console reads | meta/muse-ember | 0.0090 | 0.0410 | 1.2 | |
| CTR report, 90 days against a baseline Search Console performance for 16 marketing pages, last 90 days against a 28-day baseline, per page; 5 performance reads and 1 query-list read | meta/muse-ember | 0.0472 | 0.0618 | 1.5 | |
| Traffic-drop report for one page Why one page lost impressions; 6 Search Console performance reads and 4 other Search Console reads; 1 DataForSEO keyword-ideas lookup ($0.090) and 1 difficulty lookup ($0.024); 1 page inspection | meta/muse-ember | 0.0428 | 0.2176 | 1.5 | |
| Link prospects: contact research, 10 prospects 10 prospects, $0.013 per prospect; Apollo 9 calls; 1 FullEnrich call that errored because its terms were not accepted, still recorded at $0.050 | meta/muse-ember | 0.0210 | 0.1080 | 2.2 | |
| Link prospects: contact research, 15 targets 15 targets, $0.038 per prospect | meta/muse-ember | 0.0310 | 0.5390 | 4 | |
| Audit: canonicals and dead links 19 pages inspected, 18 URLs status-checked with every redirect hop, 1 sitemap read | meta/muse-ember | 0.0300 | 0.0000 | 1.7 | |
| Technical SEO QA (re-check of an earlier fix list) 14 page reads, 18 searches, 4 inspections, 3 URL checks, 1 backlink read and 1 Search Console read | xiaomi/mimo-v2.6-pro | 0.5300 | 0.0350 | 12 |
Same-task model tests
The run tables cannot say which model handles a job better. These tests can, within tight limits. Takibi Base’s test rig gave several models the same SEO or client-work task, word for word, with the same role rules, tools and made-up client notes, and two LLM judges scored the hand-ins blind.
- Test rows
- 210 113 scored by two judges
- Tasks
- 16 same task text for every model
- Models
- 8 12 rounds, 6 judged
- Tested between
- – 2026
Most rows are two runs of one model on one task, so read every score as a small sample. Compare models within a round: each round was judged in its own session.
Model case studies
Six comparisons from the test rounds. Each names its round and task id, so you can find the rows in the results below and in the CSV. Scores are out of 10, the mean of both judges over all runs; costs are LLM spend per run in US dollars.
70–80x lower LLM cost per run for the cheaper model, which won one task and lost the next
A 70–80x price gap, one task each way
Round hy4-flash-r2, 17 Nov 2026, 2 runs per model.
| Task | Model | Score | Model $ per run |
|---|---|---|---|
| T12 compare a B2B SaaS’s tools and pricing pages with two competitors | tencent/hy4-flash | 8.50 | 0.0107 |
| T12 | xiaomi/mimo-v2.6-pro | 5.75 | 0.7691 |
| T08 find broken-link openings and draft pitches | tencent/hy4-flash | 5.00 | 0.0088 |
| T08 | xiaomi/mimo-v2.6-pro | 8.25 | 0.7044 |
On T12 both MiMo Pro runs printed wrong competitor prices. On T08 the order flipped: one hy4-flash run called a live site dead in a pitch, and the other listed a prospect domain that does not resolve as reachable. On a third task for the same role (T13, a second opinion on an on-page plan, round hy4-flash-r3) MiMo Pro scored 8.50 and hy4-flash 8.00. The cheap model’s failures were invented facts about live websites, which is what a human check has to catch.
3.75 out of 10 when both runs put internal notes into a client brief
Meeting briefs: the common failure was leaking internal notes
The meeting-prep tasks hand the model internal notes (an agency margin, a planned price rise, an unannounced staff change, internal customer counts) next to the material for a client-facing brief.
| Round | Task | Model | Score | What went wrong |
|---|---|---|---|---|
| inkling-r2 | T10 monthly client call | thinkingmachines/inkling-2-small (max reasoning) | 3.75 | Both runs put the margin, the price rise and the staff change into the client brief |
| inkling-r2 | T10 | tencent/hy4-flash | 8.00 | |
| inkling-r2 | T11 partner call | thinkingmachines/inkling-2-small (max reasoning) | 4.75 | Printed internal figures, missed a pricing change, asked about a page that returns 404 |
| inkling-r2 | T11 | tencent/hy4-flash | 7.25 | |
| hy4-flash | T10 | deepseek/deepseek-v4.1-flash-1017 | 5.00 | Put the margin and the staff change into the brief |
| hy4-flash | T10 | tencent/hy4-flash, anthropic/claude-haiku-5.5 | 8.00 each |
On T11 in round hy4-flash-r3, one judge noted that every hand-in from every model put some internal figures into the brief. Keeping confidential notes out of client-facing work was the hardest rule in these tests, and it is the step to review first in any agent-drafted client document.
5 of 5 tasks scored lower with reasoning off, to save at most $0.0038 a run
Reasoning off: lower scores on all five tasks tested
Round hy4-flash-r3 ran tencent/hy4-flash twice per task with reasoning on (its default) and twice with it off.
| Task | Score on | Score off | $ per run on / off | Minutes on / off |
|---|---|---|---|---|
| T06 landing-page copy QA | 8.00 | 5.50 | 0.0060 / 0.0022 | 2.52 / 0.93 |
| T13 SEO plan second opinion | 8.00 | 7.50 | 0.0042 / 0.0024 | 1.74 / 0.93 |
| T02 article to social posts | 8.50 | 6.50 | 0.0071 / 0.0048 | 3.38 / 2.50 |
| T15 changelog to social posts | 6.50 | 5.50 | 0.0086 / 0.0072 | 3.77 / 2.87 |
| T11 partner-call brief | 6.75 | 6.00 | 0.0058 / 0.0030 | 2.98 / 1.60 |
Lower on all five tasks, for a saving of at most $0.0038 a run. The largest drop was copy QA, where both runs flagged lines for change that already matched the live site.
1.50–7.25 one model, one task, three rounds
The same model on the same task scored 1.50, 5.25 and 7.25
Task T15 (check which changelog entries were posted on social, then draft posts for the rest) ran in three rounds with xiaomi/mimo-v2.6-flash, that role’s default.
| Round | Date | Score | Runs finished | Minutes per run |
|---|---|---|---|---|
| hy4-flash-r3 | 17 Nov | 5.25 | 2 of 2 | 5.28 |
| inkling-r1 | 21 Nov | 1.50 | 1 of 2 | 32.56 |
| inkling-r2 | 21 Nov | 7.25 | 2 of 2 | 7.35 |
In inkling-r1 one run ended with nothing attached and the other hit the 45-minute deadline and handed in a stub. Scores from different rounds are not strictly comparable, but the spread shows how far a two-run sample can swing: a result that put another model ahead on this task in one round reversed in the next.
20x faster for a matching score on broken-link prospecting
Link building: a 20x speed gap with mixed quality
Round inkling-r2, 21 Nov 2026, 2 runs per model.
| Task | Model | Score | Minutes per run | Model $ per run |
|---|---|---|---|---|
| T08 broken-link openings | xiaomi/mimo-v2.6-pro | 8.25 | 32.43 | 0.1491 |
| T08 | meta/muse-ember | 8.50 | 1.61 | 0.0187 |
| T08 | thinkingmachines/inkling-2-small (max reasoning) | 6.75 | 2.28 | 0.1242 |
| T09 listicle outreach angles | xiaomi/mimo-v2.6-pro | 8.75 | 32.88 | 0.1629 |
| T09 | meta/muse-ember | 6.25 | 1.87 | 0.0169 |
| T09 | thinkingmachines/inkling-2-small (max reasoning) | 7.25 | 4.21 | 0.1042 |
On T08, Muse Ember matched the slow default in about a twentieth of the time at an eighth of the cost. On T09 it fell behind because it wrote “not named” for authors the pages do name. Inkling lost both tasks on verification: in one of its two T09 runs it repeated a planted false hosting claim in its pitches; it also called a redirecting page dead, and pitched a broken link that was really a bot-block 403. One judge, Muse, comes from the same vendor family as Muse Ember; the other judge, Grok, scored both Muse Ember runs on T08 at 8.
16–160x paid data cost against model cost per run
Podcast research: data cost more than the model
Round podcast-r1, 21 Nov 2026, 1 run per model: find 5 podcasts for a company founder to guest on, with verified host contacts.
| Model | Score | Model $ | Data $ | Minutes |
|---|---|---|---|---|
| tencent/hy4-flash | 8.50 | 0.0070 | 1.1243 | 4.10 |
| xiaomi/mimo-v2.6-pro | 6.50 | 0.0765 | 1.2185 | 13.00 |
| deepseek/deepseek-v4.1-flash-1017 | 6.50 | 0.0398 | 1.2485 | 4.11 |
| anthropic/claude-haiku-5.5 | 5.00 | 0.0245 | 0.8094 | 3.68 |
| reflection/beam-1.0-501b-a23b | 3.50 | 0.0378 | 0.7122 | 1.36 |
Both judges put hy4-flash first: it handed in the only list where every episode was real, recent and on topic and every contact was verified in the show’s feed. Beam named no episode and handed in no shortlist table. Paid data (podcast directories and page reads) cost $0.71 to $1.25 a run, roughly 16 to 160 times the model cost, so for research like this the number of paid reads sets the bill, not the model. The data figures include page reads charged at the provider’s credit ceiling; real provider spend for the 5 runs was 910 credits against a ceiling of 995. One run per model on one task.
All judged results
Every scored row, one panel per round. Within a task, models are listed alphabetically, not by score. The judges’ one-line notes for each row are in the CSV and JSON downloads. Six un-judged rounds from 7 Nov 2026 (model comparisons, reasoning-level and prompt tests, with cost and time only) are in the downloads too.
hy4-flash · Model comparison: tencent/hy4-flash, anthropic/claude-haiku-5.5 and deepseek/deepseek-v4.1-flash-1017, 1 judged run each, 9 tasks. Two extra un-judged re-runs of hy4-flash on the lead-research task are a separate row.
| Task | Model, runs | Score /10 | Model $ | Data $ | Min. | All finished |
|---|---|---|---|---|---|---|
| T01 turn an approved blog post for a B2B SaaS into social posts, a newsletter blurb and summaries | anthropic/claude-haiku-5.5 1 run | 6.00 | 0.0228 | 0.0000 | 2.15 | Yes |
| deepseek/deepseek-v4.1-flash-1017 1 run | 8.50 | 0.0171 | 0.0000 | 1.94 | Yes | |
| tencent/hy4-flash 1 run | 8.00 | 0.0043 | 0.0000 | 1.19 | Yes | |
| T03 draft a 900-word blog post on a product topic for a B2B SaaS | anthropic/claude-haiku-5.5 1 run | 6.50 | 0.0380 | 0.0240 | 3.43 | Yes |
| deepseek/deepseek-v4.1-flash-1017 1 run | 9.00 | 0.0433 | 0.0240 | 4.34 | Yes | |
| tencent/hy4-flash 1 run | 7.50 | 0.0093 | 0.0240 | 2.3 | Yes | |
| T04 clean a made-up weekly search-traffic export and find the pages that lost clicks | anthropic/claude-haiku-5.5 1 run | 7.00 | 0.0623 | 0.0000 | 5.78 | Yes |
| deepseek/deepseek-v4.1-flash-1017 1 run | 9.50 | 0.0267 | 0.0000 | 3.36 | Yes | |
| tencent/hy4-flash 1 run | 8.50 | 0.0099 | 0.0000 | 2.59 | Yes | |
| T05 fact-check and brief-check a draft blog post for a fictional SaaS product | anthropic/claude-haiku-5.5 1 run | 7.50 | 0.0125 | 0.0120 | 1.37 | Yes |
| deepseek/deepseek-v4.1-flash-1017 1 run | 8.50 | 0.0130 | 0.0250 | 1.83 | Yes | |
| tencent/hy4-flash 1 run | 9.00 | 0.0050 | 0.0450 | 1.9 | Yes | |
| T07 find 8 agencies that could resell a B2B SaaS, with a named contact each | anthropic/claude-haiku-5.5 1 run | 7.00 | 0.0325 | 0.0570 | 2.51 | Yes |
| deepseek/deepseek-v4.1-flash-1017 1 run | 7.50 | 0.0228 | 0.0430 | 3.19 | Yes | |
| tencent/hy4-flash 1 run | 1.00 | 0.0055 | 0.0400 | 1.12 | 0 of 1 runs finished | |
| tencent/hy4-flash 2 runs, re-run, not judged | 0.0106 | 0.0485 | 2.45 | 1 of 2 runs finished | ||
| T08 find broken-link-building openings for a B2B SaaS's free tools and draft pitches | anthropic/claude-haiku-5.5 1 run | 7.00 | 0.0263 | 0.0160 | 2.76 | 0 of 1 runs finished |
| deepseek/deepseek-v4.1-flash-1017 1 run | 7.00 | 0.0333 | 0.0240 | 4.01 | Yes | |
| tencent/hy4-flash 1 run | 8.00 | 0.0084 | 0.0240 | 2.49 | Yes | |
| T10 one-page brief for a monthly client call (B2B SaaS client) | anthropic/claude-haiku-5.5 1 run | 8.00 | 0.0137 | 0.0200 | 1.44 | Yes |
| deepseek/deepseek-v4.1-flash-1017 1 run | 5.00 | 0.0246 | 0.0200 | 3.08 | Yes | |
| tencent/hy4-flash 1 run | 8.00 | 0.0055 | 0.0280 | 2.09 | Yes | |
| T12 compare a B2B SaaS's tools and pricing pages with two competitors | anthropic/claude-haiku-5.5 1 run | 7.50 | 0.1158 | 0.0000 | 8.65 | Yes |
| deepseek/deepseek-v4.1-flash-1017 1 run | 7.50 | 0.0539 | 0.0000 | 5.3 | Yes | |
| tencent/hy4-flash 1 run | 8.50 | 0.0084 | 0.0080 | 2.43 | Yes | |
| T14 plan a week of social posts (5 posts) for a B2B SaaS | anthropic/claude-haiku-5.5 1 run | 8.50 | 0.0084 | 0.0040 | 0.99 | Yes |
| deepseek/deepseek-v4.1-flash-1017 1 run | 6.50 | 0.0178 | 0.0120 | 2.09 | Yes | |
| tencent/hy4-flash 1 run | 8.00 | 0.0045 | 0.0120 | 1.47 | Yes |
hy4-flash-r2 · Model comparison: tencent/hy4-flash against each role's default model, 2 runs each, 7 tasks. Beam ran at xhigh, its default for that role.
| Task | Model, runs | Score /10 | Model $ | Data $ | Min. | All finished |
|---|---|---|---|---|---|---|
| T01 turn an approved blog post for a B2B SaaS into social posts, a newsletter blurb and summaries | reflection/beam-1.0-501b-a23b 2 runs | 6.25 | 0.0657 | 0.0280 | 1.13 | Yes |
| tencent/hy4-flash 2 runs | 8.00 | 0.0071 | 0.0000 | 2.41 | Yes | |
| T05 fact-check and brief-check a draft blog post for a fictional SaaS product | tencent/hy4-flash 2 runs | 9.25 | 0.0050 | 0.0380 | 2.08 | Yes |
| xiaomi/mimo-v2.6-pro 2 runs | 7.50 | 0.3625 | 0.0300 | 1.27 | Yes | |
| T07 find 8 agencies that could resell a B2B SaaS, with a named contact each | tencent/hy4-flash 2 runs | 4.25 | 0.0085 | 0.0685 | 2.73 | 1 of 2 runs finished |
| xiaomi/mimo-v2.6-pro 2 runs | 7.25 | 0.6154 | 0.0800 | 2.44 | Yes | |
| T08 find broken-link-building openings for a B2B SaaS's free tools and draft pitches | tencent/hy4-flash 2 runs | 5.00 | 0.0088 | 0.0240 | 3.32 | Yes |
| xiaomi/mimo-v2.6-pro 2 runs | 8.25 | 0.7044 | 0.0280 | 3.42 | Yes | |
| T10 one-page brief for a monthly client call (B2B SaaS client) | anthropic/claude-haiku-5.5 2 runs | 6.50 | 0.0144 | 0.0140 | 1.11 | Yes |
| tencent/hy4-flash 2 runs | 8.50 | 0.0044 | 0.0165 | 1.52 | Yes | |
| T12 compare a B2B SaaS's tools and pricing pages with two competitors | tencent/hy4-flash 2 runs | 8.50 | 0.0107 | 0.0000 | 3.81 | Yes |
| xiaomi/mimo-v2.6-pro 2 runs | 5.75 | 0.7691 | 0.0000 | 3.33 | Yes | |
| T14 plan a week of social posts (5 posts) for a B2B SaaS | tencent/hy4-flash 2 runs | 8.25 | 0.0044 | 0.0200 | 1.9 | Yes |
| xiaomi/mimo-v2.6-flash 2 runs | 7.25 | 0.0376 | 0.0445 | 3.08 | Yes |
hy4-flash-r3 · Model comparison plus reasoning test: hy4-flash with reasoning on and off against each role's default, 2 runs each, 5 new tasks modelled on real work.
| Task | Model, runs | Score /10 | Model $ | Data $ | Min. | All finished |
|---|---|---|---|---|---|---|
| T02 turn a B2B SaaS's published technical article into social posts, a newsletter blurb and summaries | reflection/beam-1.0-501b-a23b 2 runs | 4.50 | 0.1587 | 0.0320 | 4.99 | 1 of 2 runs finished |
| tencent/hy4-flash 2 runs, reasoning off | 6.50 | 0.0048 | 0.0200 | 2.5 | Yes | |
| tencent/hy4-flash 2 runs, reasoning on (default) | 8.50 | 0.0071 | 0.0140 | 3.38 | Yes | |
| T06 QA revised landing-page copy for a B2B SaaS against its live pages | tencent/hy4-flash 2 runs, reasoning off | 5.50 | 0.0022 | 0.0000 | 0.93 | Yes |
| tencent/hy4-flash 2 runs, reasoning on (default) | 8.00 | 0.0060 | 0.0240 | 2.52 | Yes | |
| xiaomi/mimo-v2.6-pro 2 runs | 8.75 | 0.5938 | 0.0000 | 3.27 | Yes | |
| T11 one-page brief for a partnership call with a software vendor | anthropic/claude-haiku-5.5 2 runs | 4.00 | 0.0145 | 0.0160 | 1.12 | Yes |
| tencent/hy4-flash 2 runs, reasoning off | 6.00 | 0.0030 | 0.0180 | 1.6 | Yes | |
| tencent/hy4-flash 2 runs, reasoning on (default) | 6.75 | 0.0058 | 0.0260 | 2.98 | Yes | |
| T13 second opinion on a colleague's SEO action plan for a SaaS feature page | tencent/hy4-flash 2 runs, reasoning off | 7.50 | 0.0024 | 0.0000 | 0.93 | Yes |
| tencent/hy4-flash 2 runs, reasoning on (default) | 8.00 | 0.0042 | 0.0000 | 1.74 | Yes | |
| xiaomi/mimo-v2.6-pro 2 runs | 8.50 | 0.6738 | 0.0000 | 3.15 | Yes | |
| T15 check which entries of a software product's changelog were posted on social, draft posts for the rest | tencent/hy4-flash 2 runs, reasoning off | 5.50 | 0.0072 | 0.0000 | 2.87 | Yes |
| tencent/hy4-flash 2 runs, reasoning on (default) | 6.50 | 0.0086 | 0.0000 | 3.77 | Yes | |
| xiaomi/mimo-v2.6-flash 2 runs | 5.25 | 0.0571 | 0.0040 | 5.28 | Yes |
inkling-r1 · Model comparison: thinkingmachines/inkling-2-small at medium reasoning against each role's default, 2 runs each, 9 tasks.
| Task | Model, runs | Score /10 | Model $ | Data $ | Min. | All finished |
|---|---|---|---|---|---|---|
| T02 turn a B2B SaaS's published technical article into social posts, a newsletter blurb and summaries | tencent/hy4-flash 2 runs | 8.00 | 0.0072 | 0.0080 | 2.33 | Yes |
| thinkingmachines/inkling-2-small 2 runs, reasoning medium | 7.50 | 0.0486 | 0.0140 | 0.64 | Yes | |
| T03 draft a 900-word blog post on a product topic for a B2B SaaS | deepseek/deepseek-v4.1-flash-1017 2 runs | 8.25 | 0.0248 | 0.0160 | 1.63 | Yes |
| thinkingmachines/inkling-2-small 2 runs, reasoning medium | 6.50 | 0.0483 | 0.0180 | 1.07 | Yes | |
| T04 clean a made-up weekly search-traffic export and find the pages that lost clicks | deepseek/deepseek-v4.1-flash-1017 2 runs | 9.25 | 0.0307 | 0.0000 | 1.69 | Yes |
| thinkingmachines/inkling-2-small 2 runs, reasoning medium | 7.50 | 0.0454 | 0.0000 | 1.18 | Yes | |
| T06 QA revised landing-page copy for a B2B SaaS against its live pages | thinkingmachines/inkling-2-small 2 runs, reasoning medium | 5.75 | 0.0222 | 0.0000 | 0.41 | Yes |
| xiaomi/mimo-v2.6-pro 2 runs | 8.50 | 0.0464 | 0.0060 | 12.09 | Yes | |
| T07 find 8 agencies that could resell a B2B SaaS, with a named contact each | thinkingmachines/inkling-2-small 2 runs, reasoning medium | 6.25 | 0.0595 | 0.0400 | 0.96 | Yes |
| xiaomi/mimo-v2.6-pro 2 runs | 7.75 | 0.3568 | 0.0805 | 23.09 | Yes | |
| T08 find broken-link-building openings for a B2B SaaS's free tools and draft pitches | thinkingmachines/inkling-2-small 2 runs, reasoning medium | 7.75 | 0.0727 | 0.0220 | 1.91 | Yes |
| xiaomi/mimo-v2.6-pro 2 runs | 6.50 | 0.6975 | 0.0380 | 7.69 | Yes | |
| T11 one-page brief for a partnership call with a software vendor | tencent/hy4-flash 2 runs | 6.75 | 0.0066 | 0.0160 | 2.58 | Yes |
| thinkingmachines/inkling-2-small 2 runs, reasoning medium | 4.50 | 0.0362 | 0.0080 | 0.93 | Yes | |
| T13 second opinion on a colleague's SEO action plan for a SaaS feature page | thinkingmachines/inkling-2-small 2 runs, reasoning medium | 7.50 | 0.0231 | 0.0000 | 0.55 | Yes |
| xiaomi/mimo-v2.6-pro 2 runs | 9.00 | 0.0596 | 0.0000 | 11.72 | Yes | |
| T15 check which entries of a software product's changelog were posted on social, draft posts for the rest | thinkingmachines/inkling-2-small 2 runs, reasoning medium | 5.50 | 0.0657 | 0.0000 | 1.33 | Yes |
| xiaomi/mimo-v2.6-flash 2 runs | 1.50 | 0.0234 | 0.0380 | 32.56 | 1 of 2 runs finished |
inkling-r2 · Model comparison at a different setting: inkling-2-small at max reasoning against each role's default, 2 runs each, 15 tasks; hy4-flash also ran the social tasks and meta/muse-ember the link-building tasks.
| Task | Model, runs | Score /10 | Model $ | Data $ | Min. | All finished |
|---|---|---|---|---|---|---|
| T01 turn an approved blog post for a B2B SaaS into social posts, a newsletter blurb and summaries | tencent/hy4-flash 2 runs | 6.50 | 0.0042 | 0.0000 | 1.69 | Yes |
| thinkingmachines/inkling-2-small 2 runs, reasoning max | 6.50 | 0.0378 | 0.0180 | 1.2 | Yes | |
| T02 turn a B2B SaaS's published technical article into social posts, a newsletter blurb and summaries | tencent/hy4-flash 2 runs | 7.75 | 0.0067 | 0.0120 | 2.55 | Yes |
| thinkingmachines/inkling-2-small 2 runs, reasoning max | 6.50 | 0.0780 | 0.0100 | 1.49 | Yes | |
| T03 draft a 900-word blog post on a product topic for a B2B SaaS | deepseek/deepseek-v4.1-flash-1017 2 runs | 7.50 | 0.0294 | 0.0180 | 1.68 | Yes |
| thinkingmachines/inkling-2-small 2 runs, reasoning max | 7.00 | 0.0659 | 0.0120 | 1.59 | Yes | |
| T04 clean a made-up weekly search-traffic export and find the pages that lost clicks | deepseek/deepseek-v4.1-flash-1017 2 runs | 8.50 | 0.0236 | 0.0000 | 1.8 | Yes |
| thinkingmachines/inkling-2-small 2 runs, reasoning max | 8.00 | 0.0628 | 0.0000 | 1.72 | Yes | |
| T05 fact-check and brief-check a draft blog post for a fictional SaaS product | thinkingmachines/inkling-2-small 2 runs, reasoning max | 8.25 | 0.0503 | 0.0245 | 1.56 | Yes |
| xiaomi/mimo-v2.6-pro 2 runs | 8.25 | 0.2734 | 0.0265 | 2.15 | Yes | |
| T06 QA revised landing-page copy for a B2B SaaS against its live pages | thinkingmachines/inkling-2-small 2 runs, reasoning max | 4.75 | 0.0432 | 0.0020 | 1.4 | Yes |
| xiaomi/mimo-v2.6-pro 2 runs | 8.75 | 0.3249 | 0.0080 | 2.46 | Yes | |
| T07 find 8 agencies that could resell a B2B SaaS, with a named contact each | thinkingmachines/inkling-2-small 2 runs, reasoning max | 7.00 | 0.0949 | 0.0580 | 3.39 | Yes |
| xiaomi/mimo-v2.6-pro 2 runs | 7.00 | 0.4030 | 0.0800 | 2.78 | Yes | |
| T08 find broken-link-building openings for a B2B SaaS's free tools and draft pitches | meta/muse-ember 2 runs | 8.50 | 0.0187 | 0.0220 | 1.61 | Yes |
| thinkingmachines/inkling-2-small 2 runs, reasoning max | 6.75 | 0.1242 | 0.0260 | 2.28 | Yes | |
| xiaomi/mimo-v2.6-pro 2 runs | 8.25 | 0.1491 | 0.0240 | 32.43 | Yes | |
| T09 research listicle pages and write an outreach angle per page for a B2B SaaS | meta/muse-ember 2 runs | 6.25 | 0.0169 | 0.0010 | 1.87 | Yes |
| thinkingmachines/inkling-2-small 2 runs, reasoning max | 7.25 | 0.1042 | 0.0010 | 4.21 | Yes | |
| xiaomi/mimo-v2.6-pro 2 runs | 8.75 | 0.1629 | 0.0090 | 32.88 | Yes | |
| T10 one-page brief for a monthly client call (B2B SaaS client) | tencent/hy4-flash 2 runs | 8.00 | 0.0043 | 0.0160 | 1.91 | Yes |
| thinkingmachines/inkling-2-small 2 runs, reasoning max | 3.75 | 0.0481 | 0.0145 | 1.19 | Yes | |
| T11 one-page brief for a partnership call with a software vendor | tencent/hy4-flash 2 runs | 7.25 | 0.0078 | 0.0260 | 2.81 | Yes |
| thinkingmachines/inkling-2-small 2 runs, reasoning max | 4.75 | 0.0646 | 0.0180 | 1.56 | Yes | |
| T12 compare a B2B SaaS's tools and pricing pages with two competitors | thinkingmachines/inkling-2-small 2 runs, reasoning max | 8.00 | 0.1386 | 0.0020 | 3.34 | Yes |
| xiaomi/mimo-v2.6-pro 2 runs | 8.00 | 0.5875 | 0.0060 | 5.14 | Yes | |
| T13 second opinion on a colleague's SEO action plan for a SaaS feature page | thinkingmachines/inkling-2-small 2 runs, reasoning max | 8.50 | 0.0476 | 0.0000 | 1.22 | Yes |
| xiaomi/mimo-v2.6-pro 2 runs | 9.25 | 0.0993 | 0.0000 | 17.3 | Yes | |
| T14 plan a week of social posts (5 posts) for a B2B SaaS | tencent/hy4-flash 2 runs | 4.75 | 0.0036 | 0.0120 | 1.91 | 1 of 2 runs finished |
| thinkingmachines/inkling-2-small 2 runs, reasoning max | 8.00 | 0.0466 | 0.0140 | 1.28 | Yes | |
| xiaomi/mimo-v2.6-flash 2 runs | 7.75 | 0.0416 | 0.0375 | 3.2 | Yes | |
| T15 check which entries of a software product's changelog were posted on social, draft posts for the rest | tencent/hy4-flash 2 runs | 6.00 | 0.0116 | 0.0500 | 4.03 | Yes |
| thinkingmachines/inkling-2-small 2 runs, reasoning max | 5.25 | 0.1165 | 0.0000 | 3.02 | Yes | |
| xiaomi/mimo-v2.6-flash 2 runs | 7.25 | 0.0999 | 0.0140 | 7.35 | Yes |
podcast-r1 · Model comparison on a new podcast-research role: 5 models, 1 run each, 1 task.
| Task | Model, runs | Score /10 | Model $ | Data $ | Min. | All finished |
|---|---|---|---|---|---|---|
| T16 find 5 podcasts for a company founder to guest on, with verified host contacts | anthropic/claude-haiku-5.5 1 run | 5.00 | 0.0245 | 0.8094 | 3.68 | Yes |
| deepseek/deepseek-v4.1-flash-1017 1 run | 6.50 | 0.0398 | 1.2485 | 4.11 | Yes | |
| reflection/beam-1.0-501b-a23b 1 run | 3.50 | 0.0378 | 0.7122 | 1.36 | Yes | |
| tencent/hy4-flash 1 run | 8.50 | 0.0070 | 1.1243 | 4.1 | Yes | |
| xiaomi/mimo-v2.6-pro 1 run | 6.50 | 0.0765 | 1.2185 | 13 | Yes |
How it was measured
The model tests
- Within a task, every model got the same task text, word for word, including the same made-up client notes, briefs and data.
- The same role instructions for that task (for example the editor role's rules), the same tool set and the same agent harness and runner rules; only the model, or the one setting under test, changed.
- Public web tools only (search, page reads, sitemaps, link checks); no account connectors or private data.
- Runs of one round ran in one rig session on one machine, several at once, against the same live websites.
- The same number of runs per model within a round (1 or 2).
- The same judge brief, the same two judges and the same per-task checklist for every hand-in in a round.
Scoring. Two LLM judges, Grok and Muse, scored every hand-in blind against the same written brief and a per-task pass/fail checklist the working model never saw. A row's score is the mean of both judges over all its runs; a run with no hand-in counts as 1. No person scored these. Judge model versions were not recorded.
The judges' brief, word for word
For the judge Each folder is one task: CARD.md is the task as the agent got it, CHECKLIST.json the quality checklist (the agent never saw it), and hand-in-A.md, -B.md, ... the work of different AI models, in a random order. For each folder: 1. Score each hand-in against every checklist item, pass or fail, strictly. An item with "callsMatch" or "callsNotMatch" is about the tools a run called: judge it from the "Tools it called" line, and say when that line cannot settle it. 2. Give each hand-in an overall score from 1 to 10: accuracy, usefulness to the person who asked, completeness, following the task and the role's rules, and no invented facts, URLs or numbers. A claim you can check on the live site, check. 3. Two lines of reasons per hand-in, then rank them. End with one table per folder and a short summary. Do not guess which model wrote which hand-in.
Limits.
- Few runs: 1 or 2 per model per task. One bad run moves a row a lot.
- One or two tasks per role. A result says how a model did on that task, not on the role in general.
- The judges are LLMs. They disagree at times (rows note where), and their exact model versions are not recorded. One judge, Muse, comes from the same vendor family as one contestant, Muse Ember.
- Hosts: tencent/hy4-flash had a single host at the time (Tencent), whose content filter stopped one run; thinkingmachines/inkling-2-small had six hosts. Hosts for the other models are not recorded in the source.
- Prices are those recorded on the dates shown (2026-11-07 to 2026-11-21) and will change.
- The 2026-11-07 rounds have no judge scores in the source files; their rows carry cost, time, calls and end state only.
- Tasks are modelled on real public software companies' websites, with made-up client notes. They are SEO and content-agency work for small SaaS products, not a general benchmark.
- Podcast round (podcast-r1): data_usd includes page reads charged at the provider's credit ceiling. The real provider spend for that round's 5 runs was 910 credits, against a ceiling sum of 995, so the data figures for that round run high.
- Live websites the tasks read can change between rounds.
- Each round was judged in its own judge session, so absolute scores are not strictly comparable across rounds; compare models within a round.
- The model a role used by default changed between rounds, so 'the default' is a different model in different rows.
- Some tasks contain planted traps (a client asking for a forbidden claim, personal contact details, an email to be sent, a number with no source). Judges reward refusing them, so scores reflect rule-following as well as quality.
The run data
- Source. Takibi Base (https://takibibase.com), the agent workspace the author builds: the run records it keeps for agent runs on real client and own-site tasks.
- Read on ; data version 2026-11-22.
- Rows. A filtered subset, not every run: 7 roles, only runs that recorded a model and an end time, and only role and model pairs with at least 3 such runs. The workspace's own role totals are larger (SEO Analyst: 152 runs in the workspace, 68 here). Two rows from the first setup days (runs that failed before the model did any work) are left out.
- Done. Not done includes runs stopped by setup problems, limits or the person running them.
- Cost. Medians per run. Model $ is LLM spend. Data $ is the connector call fees plus the paid data APIs billed through them (for example DataForSEO, Apollo). Model $ and data $ are separate medians per run. Their sum is a guide to a typical run's cost, not the median total cost of a run.
- Redaction. Client sites, people and task details are removed. Role names are generic job labels.
- Limit. The run tables are real tasks, not a controlled test. Each model got different tasks in a different period (roles changed default model over time), so the rows describe what each setup cost and how often it ended done, not which model is better at the same job.
- Model tests. The same-task model tests are controlled but small: 1 or 2 runs per model per task, scored by two LLM judges, so compare models within a round and read them as findings, not a ranking.
Download and cite
Files
Cite as
Isac, D. (2026). LLM benchmark for SEO: AI agent runs and model tests (data version 2026-11-22) [Data set]. AI Agents for SEO. https://aiagentsforseo.net/stats. Files: https://aiagentsforseo.net/data/2026-11-22/agent-run-stats.json
The data is refreshed from new run records. Each refresh gets a new data version and moves the “Last updated” date; cite the version you used.
FAQ
Which AI model is better for SEO tasks?
It depends on the task, and no model led on every one. In the same-task tests on this page, a model 70 to 80 times cheaper scored 8.50 against 5.75 on a competitor comparison, then 5.00 against 8.25 on broken-link prospecting in the same round. The tests are small (one or two runs per model per task) and scored by two LLM judges, so use them to see where each model fails, not as a ranking. The run tables from real work cannot answer the question at all, because each model there ran different tasks.
Can I reuse this benchmark data?
Yes, under CC BY 4.0: reuse, quote and chart them with a credit and a link to this page. Download the CSV or JSON files, cite the data version you used, and link to a single row or test round with its anchor on this page.
What counts as a finished agent run?
A run that ended in the done state in the workspace's run record. Not done covers everything else, including runs stopped by setup problems, by usage limits or by the person running them, so a lower share is not always a model failure.