LLM benchmark for SEO: what AI agents cost and how often they finish

By Deian Isac · Last updated

Definition:

An LLM benchmark for SEO measures how language models do when an agent runs SEO work. This one is small and dated, and covers SEO and client-work tasks in two parts: how often each agent role and model finished real jobs, and what a run cost in model spend, data fees, minutes and requests; and controlled tests where several models did the same task, scored by two LLM judges. Every figure comes from Takibi Base, the agent workspace the author builds, dated and downloadable.

Download CSVJSONCite this data

Free to reuse under CC BY 4.0 · data version 2026-11-22

271 counted runs on client and own-site work, one square each
  • Content writer 64 of 72 done
  • SEO analyst 60 of 68 done
  • Editor QA 41 of 57 done
  • Data analyst 25 of 32 done
  • Lead researcher 20 of 26 done
  • Link builder 9 of 11 done
  • Technical SEO auditor 2 of 5 done
Runs counted
271
Ended done
82%
Role and model pairs
10 7 roles, 4 models
Runs between
– 2026

Runs on real work

Runs the workspace recorded on client and own-site work (SEO, content and research roles), for each role and model pair with at least 3 runs that recorded a model and an end time. It is a filtered subset, not every run; the method lists the filters. Each model got different tasks in a different period, so these rows say what a setup cost and how often it finished, not which model is better.

How often runs finish

Share of runs that ended done. Read the run count before the percentage: a few runs move the share a lot.

  • Content writer meta/muse-ember91% of 53
  • Content writer deepseek/deepseek-v4.1-flash-101784% of 19
  • Data analyst meta/muse-ember78% of 32
  • Editor QA xiaomi/mimo-v2.6-pro72% of 57
  • Lead researcher xiaomi/mimo-v2.6-pro63% of 16
  • Lead researcher tencent/hy4-flash100% of 10
  • Link builder meta/muse-ember82% of 11
  • SEO analyst meta/muse-ember91% of 46
  • SEO analyst xiaomi/mimo-v2.6-pro82% of 22
  • Technical SEO auditor xiaomi/mimo-v2.6-pro40% of 5

What a run costs

Median model spend plus median data fees per run, on a shared scale. The two medians are taken separately, so their sum is a guide, not the median total of a run. Roles that pull keyword, SERP or contact data pay for the calls.

  • Content writer meta/muse-ember$0.0252 $0.0252 model + $0.0000 data medians
  • Content writer deepseek/deepseek-v4.1-flash-1017$0.0270 $0.0270 model + $0.0000 data medians
  • Data analyst meta/muse-ember$0.0485 $0.0301 model + $0.0184 data medians
  • Editor QA xiaomi/mimo-v2.6-pro$0.1242 $0.1242 model + $0.0000 data medians
  • Lead researcher xiaomi/mimo-v2.6-pro$0.0863 $0.0836 model + $0.0027 data medians
  • Lead researcher tencent/hy4-flash$0.0085 $0.0085 model + $0.0000 data medians
  • Link builder meta/muse-ember$0.0248 $0.0248 model + $0.0000 data medians
  • SEO analyst meta/muse-ember$0.0906 $0.0330 model + $0.0576 data medians
  • SEO analyst xiaomi/mimo-v2.6-pro$0.1116 $0.1010 model + $0.0106 data medians
  • Technical SEO auditor xiaomi/mimo-v2.6-pro$0.3309 $0.2686 model + $0.0623 data medians

Full table

Every figure from the two charts, plus median minutes and model requests per run. Each row has an anchor you can link to.

Runs by role and model, medians per run. Data version 2026-11-22.
Role and modelRunsDoneModel $Data $MinutesRequestsRuns (2026)
Content writer meta/muse-ember5391%0.02520.00001.115–
Content writer deepseek/deepseek-v4.1-flash-10171984%0.02700.00002.929–
Data analyst meta/muse-ember3278%0.03010.01841.215–
Editor QA xiaomi/mimo-v2.6-pro5772%0.12420.000024.326–
Lead researcher xiaomi/mimo-v2.6-pro1663%0.08360.002715.623–
Lead researcher tencent/hy4-flash10100%0.00850.00003.923–
SEO analyst meta/muse-ember4691%0.03300.05761.618–
SEO analyst xiaomi/mimo-v2.6-pro2282%0.10100.010612.223–
Technical SEO auditor xiaomi/mimo-v2.6-pro540%0.26860.062339.236–

Measured single runs

Individual runs that the guides on this site use as field records, with what each one covered.

Single measured runs in 2026, as recorded. Data version 2026-11-22.
Job and what it coveredDateModelModel $Data $Minutes
Keyword map, first version 11 seed keywords plus 4 related terms: volume, difficulty and 4 SERP reads from DataForSEO; includes one oversized call that was billed and returned nothingmeta/muse-ember0.01500.31902
Keyword map, revision adding Search Console 4 Search Console readsmeta/muse-ember0.00900.04101.2
CTR report, 90 days against a baseline Search Console performance for 16 marketing pages, last 90 days against a 28-day baseline, per page; 5 performance reads and 1 query-list readmeta/muse-ember0.04720.06181.5
Traffic-drop report for one page Why one page lost impressions; 6 Search Console performance reads and 4 other Search Console reads; 1 DataForSEO keyword-ideas lookup ($0.090) and 1 difficulty lookup ($0.024); 1 page inspectionmeta/muse-ember0.04280.21761.5
Audit: canonicals and dead links 19 pages inspected, 18 URLs status-checked with every redirect hop, 1 sitemap readmeta/muse-ember0.03000.00001.7
Technical SEO QA (re-check of an earlier fix list) 14 page reads, 18 searches, 4 inspections, 3 URL checks, 1 backlink read and 1 Search Console readxiaomi/mimo-v2.6-pro0.53000.035012

Same-task model tests

The run tables cannot say which model handles a job better. These tests can, within tight limits. Takibi Base’s test rig gave several models the same SEO or client-work task, word for word, with the same role rules, tools and made-up client notes, and two LLM judges scored the hand-ins blind.

Test rows
210 113 scored by two judges
Tasks
16 same task text for every model
Models
8 12 rounds, 6 judged
Tested between
– 2026

Most rows are two runs of one model on one task, so read every score as a small sample. Compare models within a round: each round was judged in its own session.

Model case studies

Six comparisons from the test rounds. Each names its round and task id, so you can find the rows in the results below and in the CSV. Scores are out of 10, the mean of both judges over all runs; costs are LLM spend per run in US dollars.

70–80x lower LLM cost per run for the cheaper model, which won one task and lost the next

A 70–80x price gap, one task each way

Round hy4-flash-r2, 17 Nov 2026, 2 runs per model.

Task Model Score Model $ per run
T12 compare a B2B SaaS’s tools and pricing pages with two competitors tencent/hy4-flash 8.50 0.0107
T12 xiaomi/mimo-v2.6-pro 5.75 0.7691
T08 find broken-link openings and draft pitches tencent/hy4-flash 5.00 0.0088
T08 xiaomi/mimo-v2.6-pro 8.25 0.7044

On T12 both MiMo Pro runs printed wrong competitor prices. On T08 the order flipped: one hy4-flash run called a live site dead in a pitch, and the other listed a prospect domain that does not resolve as reachable. On a third task for the same role (T13, a second opinion on an on-page plan, round hy4-flash-r3) MiMo Pro scored 8.50 and hy4-flash 8.00. The cheap model’s failures were invented facts about live websites, which is what a human check has to catch.

3.75 out of 10 when both runs put internal notes into a client brief

Meeting briefs: the common failure was leaking internal notes

The meeting-prep tasks hand the model internal notes (an agency margin, a planned price rise, an unannounced staff change, internal customer counts) next to the material for a client-facing brief.

Round Task Model Score What went wrong
inkling-r2 T10 monthly client call thinkingmachines/inkling-2-small (max reasoning) 3.75 Both runs put the margin, the price rise and the staff change into the client brief
inkling-r2 T10 tencent/hy4-flash 8.00
inkling-r2 T11 partner call thinkingmachines/inkling-2-small (max reasoning) 4.75 Printed internal figures, missed a pricing change, asked about a page that returns 404
inkling-r2 T11 tencent/hy4-flash 7.25
hy4-flash T10 deepseek/deepseek-v4.1-flash-1017 5.00 Put the margin and the staff change into the brief
hy4-flash T10 tencent/hy4-flash, anthropic/claude-haiku-5.5 8.00 each

On T11 in round hy4-flash-r3, one judge noted that every hand-in from every model put some internal figures into the brief. Keeping confidential notes out of client-facing work was the hardest rule in these tests, and it is the step to review first in any agent-drafted client document.

5 of 5 tasks scored lower with reasoning off, to save at most $0.0038 a run

Reasoning off: lower scores on all five tasks tested

Round hy4-flash-r3 ran tencent/hy4-flash twice per task with reasoning on (its default) and twice with it off.

Task Score on Score off $ per run on / off Minutes on / off
T06 landing-page copy QA 8.00 5.50 0.0060 / 0.0022 2.52 / 0.93
T13 SEO plan second opinion 8.00 7.50 0.0042 / 0.0024 1.74 / 0.93
T02 article to social posts 8.50 6.50 0.0071 / 0.0048 3.38 / 2.50
T15 changelog to social posts 6.50 5.50 0.0086 / 0.0072 3.77 / 2.87
T11 partner-call brief 6.75 6.00 0.0058 / 0.0030 2.98 / 1.60

Lower on all five tasks, for a saving of at most $0.0038 a run. The largest drop was copy QA, where both runs flagged lines for change that already matched the live site.

1.50–7.25 one model, one task, three rounds

The same model on the same task scored 1.50, 5.25 and 7.25

Task T15 (check which changelog entries were posted on social, then draft posts for the rest) ran in three rounds with xiaomi/mimo-v2.6-flash, that role’s default.

Round Date Score Runs finished Minutes per run
hy4-flash-r3 17 Nov 5.25 2 of 2 5.28
inkling-r1 21 Nov 1.50 1 of 2 32.56
inkling-r2 21 Nov 7.25 2 of 2 7.35

In inkling-r1 one run ended with nothing attached and the other hit the 45-minute deadline and handed in a stub. Scores from different rounds are not strictly comparable, but the spread shows how far a two-run sample can swing: a result that put another model ahead on this task in one round reversed in the next.

20x faster for a matching score on broken-link prospecting

Round inkling-r2, 21 Nov 2026, 2 runs per model.

Task Model Score Minutes per run Model $ per run
T08 broken-link openings xiaomi/mimo-v2.6-pro 8.25 32.43 0.1491
T08 meta/muse-ember 8.50 1.61 0.0187
T08 thinkingmachines/inkling-2-small (max reasoning) 6.75 2.28 0.1242
T09 listicle outreach angles xiaomi/mimo-v2.6-pro 8.75 32.88 0.1629
T09 meta/muse-ember 6.25 1.87 0.0169
T09 thinkingmachines/inkling-2-small (max reasoning) 7.25 4.21 0.1042

On T08, Muse Ember matched the slow default in about a twentieth of the time at an eighth of the cost. On T09 it fell behind because it wrote “not named” for authors the pages do name. Inkling lost both tasks on verification: in one of its two T09 runs it repeated a planted false hosting claim in its pitches; it also called a redirecting page dead, and pitched a broken link that was really a bot-block 403. One judge, Muse, comes from the same vendor family as Muse Ember; the other judge, Grok, scored both Muse Ember runs on T08 at 8.

16–160x paid data cost against model cost per run

Podcast research: data cost more than the model

Round podcast-r1, 21 Nov 2026, 1 run per model: find 5 podcasts for a company founder to guest on, with verified host contacts.

Model Score Model $ Data $ Minutes
tencent/hy4-flash 8.50 0.0070 1.1243 4.10
xiaomi/mimo-v2.6-pro 6.50 0.0765 1.2185 13.00
deepseek/deepseek-v4.1-flash-1017 6.50 0.0398 1.2485 4.11
anthropic/claude-haiku-5.5 5.00 0.0245 0.8094 3.68
reflection/beam-1.0-501b-a23b 3.50 0.0378 0.7122 1.36

Both judges put hy4-flash first: it handed in the only list where every episode was real, recent and on topic and every contact was verified in the show’s feed. Beam named no episode and handed in no shortlist table. Paid data (podcast directories and page reads) cost $0.71 to $1.25 a run, roughly 16 to 160 times the model cost, so for research like this the number of paid reads sets the bill, not the model. The data figures include page reads charged at the provider’s credit ceiling; real provider spend for the 5 runs was 910 credits against a ceiling of 995. One run per model on one task.

All judged results

Every scored row, one panel per round. Within a task, models are listed alphabetically, not by score. The judges’ one-line notes for each row are in the CSV and JSON downloads. Six un-judged rounds from 7 Nov 2026 (model comparisons, reasoning-level and prompt tests, with cost and time only) are in the downloads too.

hy4-flash · Model comparison: tencent/hy4-flash, anthropic/claude-haiku-5.5 and deepseek/deepseek-v4.1-flash-1017, 1 judged run each, 9 tasks. Two extra un-judged re-runs of hy4-flash on the lead-research task are a separate row.
TaskModel, runsScore /10Model $Data $Min.All finished
T01 turn an approved blog post for a B2B SaaS into social posts, a newsletter blurb and summariesanthropic/claude-haiku-5.5 1 run6.000.02280.00002.15Yes
deepseek/deepseek-v4.1-flash-1017 1 run8.500.01710.00001.94Yes
tencent/hy4-flash 1 run8.000.00430.00001.19Yes
T03 draft a 900-word blog post on a product topic for a B2B SaaSanthropic/claude-haiku-5.5 1 run6.500.03800.02403.43Yes
deepseek/deepseek-v4.1-flash-1017 1 run9.000.04330.02404.34Yes
tencent/hy4-flash 1 run7.500.00930.02402.3Yes
T04 clean a made-up weekly search-traffic export and find the pages that lost clicksanthropic/claude-haiku-5.5 1 run7.000.06230.00005.78Yes
deepseek/deepseek-v4.1-flash-1017 1 run9.500.02670.00003.36Yes
tencent/hy4-flash 1 run8.500.00990.00002.59Yes
T05 fact-check and brief-check a draft blog post for a fictional SaaS productanthropic/claude-haiku-5.5 1 run7.500.01250.01201.37Yes
deepseek/deepseek-v4.1-flash-1017 1 run8.500.01300.02501.83Yes
tencent/hy4-flash 1 run9.000.00500.04501.9Yes
T07 find 8 agencies that could resell a B2B SaaS, with a named contact eachanthropic/claude-haiku-5.5 1 run7.000.03250.05702.51Yes
deepseek/deepseek-v4.1-flash-1017 1 run7.500.02280.04303.19Yes
tencent/hy4-flash 1 run1.000.00550.04001.120 of 1 runs finished
tencent/hy4-flash 2 runs, re-run, not judgednot scored0.01060.04852.451 of 2 runs finished
T08 find broken-link-building openings for a B2B SaaS's free tools and draft pitchesanthropic/claude-haiku-5.5 1 run7.000.02630.01602.760 of 1 runs finished
deepseek/deepseek-v4.1-flash-1017 1 run7.000.03330.02404.01Yes
tencent/hy4-flash 1 run8.000.00840.02402.49Yes
T10 one-page brief for a monthly client call (B2B SaaS client)anthropic/claude-haiku-5.5 1 run8.000.01370.02001.44Yes
deepseek/deepseek-v4.1-flash-1017 1 run5.000.02460.02003.08Yes
tencent/hy4-flash 1 run8.000.00550.02802.09Yes
T12 compare a B2B SaaS's tools and pricing pages with two competitorsanthropic/claude-haiku-5.5 1 run7.500.11580.00008.65Yes
deepseek/deepseek-v4.1-flash-1017 1 run7.500.05390.00005.3Yes
tencent/hy4-flash 1 run8.500.00840.00802.43Yes
T14 plan a week of social posts (5 posts) for a B2B SaaSanthropic/claude-haiku-5.5 1 run8.500.00840.00400.99Yes
deepseek/deepseek-v4.1-flash-1017 1 run6.500.01780.01202.09Yes
tencent/hy4-flash 1 run8.000.00450.01201.47Yes
hy4-flash-r2 · Model comparison: tencent/hy4-flash against each role's default model, 2 runs each, 7 tasks. Beam ran at xhigh, its default for that role.
TaskModel, runsScore /10Model $Data $Min.All finished
T01 turn an approved blog post for a B2B SaaS into social posts, a newsletter blurb and summariesreflection/beam-1.0-501b-a23b 2 runs6.250.06570.02801.13Yes
tencent/hy4-flash 2 runs8.000.00710.00002.41Yes
T05 fact-check and brief-check a draft blog post for a fictional SaaS producttencent/hy4-flash 2 runs9.250.00500.03802.08Yes
xiaomi/mimo-v2.6-pro 2 runs7.500.36250.03001.27Yes
T07 find 8 agencies that could resell a B2B SaaS, with a named contact eachtencent/hy4-flash 2 runs4.250.00850.06852.731 of 2 runs finished
xiaomi/mimo-v2.6-pro 2 runs7.250.61540.08002.44Yes
T08 find broken-link-building openings for a B2B SaaS's free tools and draft pitchestencent/hy4-flash 2 runs5.000.00880.02403.32Yes
xiaomi/mimo-v2.6-pro 2 runs8.250.70440.02803.42Yes
T10 one-page brief for a monthly client call (B2B SaaS client)anthropic/claude-haiku-5.5 2 runs6.500.01440.01401.11Yes
tencent/hy4-flash 2 runs8.500.00440.01651.52Yes
T12 compare a B2B SaaS's tools and pricing pages with two competitorstencent/hy4-flash 2 runs8.500.01070.00003.81Yes
xiaomi/mimo-v2.6-pro 2 runs5.750.76910.00003.33Yes
T14 plan a week of social posts (5 posts) for a B2B SaaStencent/hy4-flash 2 runs8.250.00440.02001.9Yes
xiaomi/mimo-v2.6-flash 2 runs7.250.03760.04453.08Yes
hy4-flash-r3 · Model comparison plus reasoning test: hy4-flash with reasoning on and off against each role's default, 2 runs each, 5 new tasks modelled on real work.
TaskModel, runsScore /10Model $Data $Min.All finished
T02 turn a B2B SaaS's published technical article into social posts, a newsletter blurb and summariesreflection/beam-1.0-501b-a23b 2 runs4.500.15870.03204.991 of 2 runs finished
tencent/hy4-flash 2 runs, reasoning off6.500.00480.02002.5Yes
tencent/hy4-flash 2 runs, reasoning on (default)8.500.00710.01403.38Yes
T06 QA revised landing-page copy for a B2B SaaS against its live pagestencent/hy4-flash 2 runs, reasoning off5.500.00220.00000.93Yes
tencent/hy4-flash 2 runs, reasoning on (default)8.000.00600.02402.52Yes
xiaomi/mimo-v2.6-pro 2 runs8.750.59380.00003.27Yes
T11 one-page brief for a partnership call with a software vendoranthropic/claude-haiku-5.5 2 runs4.000.01450.01601.12Yes
tencent/hy4-flash 2 runs, reasoning off6.000.00300.01801.6Yes
tencent/hy4-flash 2 runs, reasoning on (default)6.750.00580.02602.98Yes
T13 second opinion on a colleague's SEO action plan for a SaaS feature pagetencent/hy4-flash 2 runs, reasoning off7.500.00240.00000.93Yes
tencent/hy4-flash 2 runs, reasoning on (default)8.000.00420.00001.74Yes
xiaomi/mimo-v2.6-pro 2 runs8.500.67380.00003.15Yes
T15 check which entries of a software product's changelog were posted on social, draft posts for the resttencent/hy4-flash 2 runs, reasoning off5.500.00720.00002.87Yes
tencent/hy4-flash 2 runs, reasoning on (default)6.500.00860.00003.77Yes
xiaomi/mimo-v2.6-flash 2 runs5.250.05710.00405.28Yes
inkling-r1 · Model comparison: thinkingmachines/inkling-2-small at medium reasoning against each role's default, 2 runs each, 9 tasks.
TaskModel, runsScore /10Model $Data $Min.All finished
T02 turn a B2B SaaS's published technical article into social posts, a newsletter blurb and summariestencent/hy4-flash 2 runs8.000.00720.00802.33Yes
thinkingmachines/inkling-2-small 2 runs, reasoning medium7.500.04860.01400.64Yes
T03 draft a 900-word blog post on a product topic for a B2B SaaSdeepseek/deepseek-v4.1-flash-1017 2 runs8.250.02480.01601.63Yes
thinkingmachines/inkling-2-small 2 runs, reasoning medium6.500.04830.01801.07Yes
T04 clean a made-up weekly search-traffic export and find the pages that lost clicksdeepseek/deepseek-v4.1-flash-1017 2 runs9.250.03070.00001.69Yes
thinkingmachines/inkling-2-small 2 runs, reasoning medium7.500.04540.00001.18Yes
T06 QA revised landing-page copy for a B2B SaaS against its live pagesthinkingmachines/inkling-2-small 2 runs, reasoning medium5.750.02220.00000.41Yes
xiaomi/mimo-v2.6-pro 2 runs8.500.04640.006012.09Yes
T07 find 8 agencies that could resell a B2B SaaS, with a named contact eachthinkingmachines/inkling-2-small 2 runs, reasoning medium6.250.05950.04000.96Yes
xiaomi/mimo-v2.6-pro 2 runs7.750.35680.080523.09Yes
T08 find broken-link-building openings for a B2B SaaS's free tools and draft pitchesthinkingmachines/inkling-2-small 2 runs, reasoning medium7.750.07270.02201.91Yes
xiaomi/mimo-v2.6-pro 2 runs6.500.69750.03807.69Yes
T11 one-page brief for a partnership call with a software vendortencent/hy4-flash 2 runs6.750.00660.01602.58Yes
thinkingmachines/inkling-2-small 2 runs, reasoning medium4.500.03620.00800.93Yes
T13 second opinion on a colleague's SEO action plan for a SaaS feature pagethinkingmachines/inkling-2-small 2 runs, reasoning medium7.500.02310.00000.55Yes
xiaomi/mimo-v2.6-pro 2 runs9.000.05960.000011.72Yes
T15 check which entries of a software product's changelog were posted on social, draft posts for the restthinkingmachines/inkling-2-small 2 runs, reasoning medium5.500.06570.00001.33Yes
xiaomi/mimo-v2.6-flash 2 runs1.500.02340.038032.561 of 2 runs finished
inkling-r2 · Model comparison at a different setting: inkling-2-small at max reasoning against each role's default, 2 runs each, 15 tasks; hy4-flash also ran the social tasks and meta/muse-ember the link-building tasks.
TaskModel, runsScore /10Model $Data $Min.All finished
T01 turn an approved blog post for a B2B SaaS into social posts, a newsletter blurb and summariestencent/hy4-flash 2 runs6.500.00420.00001.69Yes
thinkingmachines/inkling-2-small 2 runs, reasoning max6.500.03780.01801.2Yes
T02 turn a B2B SaaS's published technical article into social posts, a newsletter blurb and summariestencent/hy4-flash 2 runs7.750.00670.01202.55Yes
thinkingmachines/inkling-2-small 2 runs, reasoning max6.500.07800.01001.49Yes
T03 draft a 900-word blog post on a product topic for a B2B SaaSdeepseek/deepseek-v4.1-flash-1017 2 runs7.500.02940.01801.68Yes
thinkingmachines/inkling-2-small 2 runs, reasoning max7.000.06590.01201.59Yes
T04 clean a made-up weekly search-traffic export and find the pages that lost clicksdeepseek/deepseek-v4.1-flash-1017 2 runs8.500.02360.00001.8Yes
thinkingmachines/inkling-2-small 2 runs, reasoning max8.000.06280.00001.72Yes
T05 fact-check and brief-check a draft blog post for a fictional SaaS productthinkingmachines/inkling-2-small 2 runs, reasoning max8.250.05030.02451.56Yes
xiaomi/mimo-v2.6-pro 2 runs8.250.27340.02652.15Yes
T06 QA revised landing-page copy for a B2B SaaS against its live pagesthinkingmachines/inkling-2-small 2 runs, reasoning max4.750.04320.00201.4Yes
xiaomi/mimo-v2.6-pro 2 runs8.750.32490.00802.46Yes
T07 find 8 agencies that could resell a B2B SaaS, with a named contact eachthinkingmachines/inkling-2-small 2 runs, reasoning max7.000.09490.05803.39Yes
xiaomi/mimo-v2.6-pro 2 runs7.000.40300.08002.78Yes
T08 find broken-link-building openings for a B2B SaaS's free tools and draft pitchesmeta/muse-ember 2 runs8.500.01870.02201.61Yes
thinkingmachines/inkling-2-small 2 runs, reasoning max6.750.12420.02602.28Yes
xiaomi/mimo-v2.6-pro 2 runs8.250.14910.024032.43Yes
T09 research listicle pages and write an outreach angle per page for a B2B SaaSmeta/muse-ember 2 runs6.250.01690.00101.87Yes
thinkingmachines/inkling-2-small 2 runs, reasoning max7.250.10420.00104.21Yes
xiaomi/mimo-v2.6-pro 2 runs8.750.16290.009032.88Yes
T10 one-page brief for a monthly client call (B2B SaaS client)tencent/hy4-flash 2 runs8.000.00430.01601.91Yes
thinkingmachines/inkling-2-small 2 runs, reasoning max3.750.04810.01451.19Yes
T11 one-page brief for a partnership call with a software vendortencent/hy4-flash 2 runs7.250.00780.02602.81Yes
thinkingmachines/inkling-2-small 2 runs, reasoning max4.750.06460.01801.56Yes
T12 compare a B2B SaaS's tools and pricing pages with two competitorsthinkingmachines/inkling-2-small 2 runs, reasoning max8.000.13860.00203.34Yes
xiaomi/mimo-v2.6-pro 2 runs8.000.58750.00605.14Yes
T13 second opinion on a colleague's SEO action plan for a SaaS feature pagethinkingmachines/inkling-2-small 2 runs, reasoning max8.500.04760.00001.22Yes
xiaomi/mimo-v2.6-pro 2 runs9.250.09930.000017.3Yes
T14 plan a week of social posts (5 posts) for a B2B SaaStencent/hy4-flash 2 runs4.750.00360.01201.911 of 2 runs finished
thinkingmachines/inkling-2-small 2 runs, reasoning max8.000.04660.01401.28Yes
xiaomi/mimo-v2.6-flash 2 runs7.750.04160.03753.2Yes
T15 check which entries of a software product's changelog were posted on social, draft posts for the resttencent/hy4-flash 2 runs6.000.01160.05004.03Yes
thinkingmachines/inkling-2-small 2 runs, reasoning max5.250.11650.00003.02Yes
xiaomi/mimo-v2.6-flash 2 runs7.250.09990.01407.35Yes
podcast-r1 · Model comparison on a new podcast-research role: 5 models, 1 run each, 1 task.
TaskModel, runsScore /10Model $Data $Min.All finished
T16 find 5 podcasts for a company founder to guest on, with verified host contactsanthropic/claude-haiku-5.5 1 run5.000.02450.80943.68Yes
deepseek/deepseek-v4.1-flash-1017 1 run6.500.03981.24854.11Yes
reflection/beam-1.0-501b-a23b 1 run3.500.03780.71221.36Yes
tencent/hy4-flash 1 run8.500.00701.12434.1Yes
xiaomi/mimo-v2.6-pro 1 run6.500.07651.218513Yes

Data $ includes page reads charged at the provider's credit ceiling: real provider spend for the 5 runs was 910 credits against a ceiling sum of 995, so these data figures run high.

How it was measured

The model tests

  • Within a task, every model got the same task text, word for word, including the same made-up client notes, briefs and data.
  • The same role instructions for that task (for example the editor role's rules), the same tool set and the same agent harness and runner rules; only the model, or the one setting under test, changed.
  • Public web tools only (search, page reads, sitemaps, link checks); no account connectors or private data.
  • Runs of one round ran in one rig session on one machine, several at once, against the same live websites.
  • The same number of runs per model within a round (1 or 2).
  • The same judge brief, the same two judges and the same per-task checklist for every hand-in in a round.

Scoring. Two LLM judges, Grok and Muse, scored every hand-in blind against the same written brief and a per-task pass/fail checklist the working model never saw. A row's score is the mean of both judges over all its runs; a run with no hand-in counts as 1. No person scored these. Judge model versions were not recorded.

The judges' brief, word for word
For the judge

Each folder is one task: CARD.md is the task as the agent got it, CHECKLIST.json the quality checklist (the agent never saw it), and hand-in-A.md, -B.md, ... the work of different AI models, in a random order. For each folder:

1. Score each hand-in against every checklist item, pass or fail, strictly. An item with "callsMatch" or "callsNotMatch" is about the tools a run called: judge it from the "Tools it called" line, and say when that line cannot settle it.
2. Give each hand-in an overall score from 1 to 10: accuracy, usefulness to the person who asked, completeness, following the task and the role's rules, and no invented facts, URLs or numbers. A claim you can check on the live site, check.
3. Two lines of reasons per hand-in, then rank them.

End with one table per folder and a short summary. Do not guess which model wrote which hand-in.

The same judge brief was used, unchanged, in every judged round. The per-task checklists are not published because they name the real websites the tasks used.

Limits.

  • Few runs: 1 or 2 per model per task. One bad run moves a row a lot.
  • One or two tasks per role. A result says how a model did on that task, not on the role in general.
  • The judges are LLMs. They disagree at times (rows note where), and their exact model versions are not recorded. One judge, Muse, comes from the same vendor family as one contestant, Muse Ember.
  • Hosts: tencent/hy4-flash had a single host at the time (Tencent), whose content filter stopped one run; thinkingmachines/inkling-2-small had six hosts. Hosts for the other models are not recorded in the source.
  • Prices are those recorded on the dates shown (2026-11-07 to 2026-11-21) and will change.
  • The 2026-11-07 rounds have no judge scores in the source files; their rows carry cost, time, calls and end state only.
  • Tasks are modelled on real public software companies' websites, with made-up client notes. They are SEO and content-agency work for small SaaS products, not a general benchmark.
  • Podcast round (podcast-r1): data_usd includes page reads charged at the provider's credit ceiling. The real provider spend for that round's 5 runs was 910 credits, against a ceiling sum of 995, so the data figures for that round run high.
  • Live websites the tasks read can change between rounds.
  • Each round was judged in its own judge session, so absolute scores are not strictly comparable across rounds; compare models within a round.
  • The model a role used by default changed between rounds, so 'the default' is a different model in different rows.
  • Some tasks contain planted traps (a client asking for a forbidden claim, personal contact details, an email to be sent, a number with no source). Judges reward refusing them, so scores reflect rule-following as well as quality.

The run data

  • Source. Takibi Base (https://takibibase.com), the agent workspace the author builds: the run records it keeps for agent runs on real client and own-site tasks.
  • Read on ; data version 2026-11-22.
  • Rows. A filtered subset, not every run: 7 roles, only runs that recorded a model and an end time, and only role and model pairs with at least 3 such runs. The workspace's own role totals are larger (SEO Analyst: 152 runs in the workspace, 68 here). Two rows from the first setup days (runs that failed before the model did any work) are left out.
  • Done. Not done includes runs stopped by setup problems, limits or the person running them.
  • Cost. Medians per run. Model $ is LLM spend. Data $ is the connector call fees plus the paid data APIs billed through them (for example DataForSEO, Apollo). Model $ and data $ are separate medians per run. Their sum is a guide to a typical run's cost, not the median total cost of a run.
  • Redaction. Client sites, people and task details are removed. Role names are generic job labels.
  • Limit. The run tables are real tasks, not a controlled test. Each model got different tasks in a different period (roles changed default model over time), so the rows describe what each setup cost and how often it ended done, not which model is better at the same job.
  • Model tests. The same-task model tests are controlled but small: 1 or 2 runs per model per task, scored by two LLM judges, so compare models within a round and read them as findings, not a ranking.

Download and cite

Files

These links always serve the latest version. A frozen copy of each version stays at /data/<version>/, for example /data/2026-11-22/agent-run-stats.json.

Cite as

Isac, D. (2026). LLM benchmark for SEO: AI agent runs and model tests (data version 2026-11-22) [Data set]. AI Agents for SEO. https://aiagentsforseo.net/stats. Files: https://aiagentsforseo.net/data/2026-11-22/agent-run-stats.json

Source of the run records and model tests: Takibi Base, the agent workspace the author builds.

Licence: CC BY 4.0. Link to a row with its anchor, for example /stats#run-content-writer-meta-muse-ember.

The data is refreshed from new run records. Each refresh gets a new data version and moves the “Last updated” date; cite the version you used.

FAQ

Which AI model is better for SEO tasks?

It depends on the task, and no model led on every one. In the same-task tests on this page, a model 70 to 80 times cheaper scored 8.50 against 5.75 on a competitor comparison, then 5.00 against 8.25 on broken-link prospecting in the same round. The tests are small (one or two runs per model per task) and scored by two LLM judges, so use them to see where each model fails, not as a ranking. The run tables from real work cannot answer the question at all, because each model there ran different tasks.

Can I reuse this benchmark data?

Yes, under CC BY 4.0: reuse, quote and chart them with a credit and a link to this page. Download the CSV or JSON files, cite the data version you used, and link to a single row or test round with its anchor on this page.

What counts as a finished agent run?

A run that ended in the done state in the workspace's run record. Not done covers everything else, including runs stopped by setup problems, by usage limits or by the person running them, so a lower share is not always a model failure.