AI-assisted technical SEO audit: crawl data in, prioritised findings out
By Deian Isac · Last updated
Definition:
An AI-assisted technical SEO audit uses an agent to crawl a site, group the findings by type and impact, and draft a prioritised list for a person to check. The agent speeds up the first pass, and false positives still have to be ruled out by someone who knows the site.
Crawl data goes in; a prioritised, verified findings list comes out. The centrepiece of this page is the false-positive checklist.
Inputs
- A crawler export from any desktop or cloud crawler (Screaming Frog SEO Spider and Sitebulb are common examples; no ranking implied).
- The GSC Page indexing report (help) and the URL Inspection API for samples. The API has a per-property daily quota, so inspect a sample rather than every URL.
- Core Web Vitals field data (CrUX; background at web.dev), alongside any lab runs.
- Sitemaps and robots.txt.
- Search Console data to join against, by URL.
The workflow
-
Normalise the crawl export and join it to GSC data by URL (see below).
-
Group the issues by type (taxonomy below).
-
Score impact: affected pages × those pages’ traffic or importance. A technical SEO audit done well ranks findings by this score instead of by raw issue count.
-
Run the false-positive pass (next section).
-
Draft a prioritised fix list with an evidence URL per finding.
Field record: canonical and dead-link check, November 2026. Raw-HTML page inspection (title, canonical, full link list), a status check of every canonical target and outbound link with every redirect hop followed, and a sitemap read. The finding types are shown; the site, its pages and the totals are left out so the client cannot be identified. From the author’s own agent workspace (related row on the statistics page). The author builds an agent workspace; see About. Finding Evidence the agent recorded Fix Canonical points at a URL that now redirects to the live article on another domain 301 → 200 on the final URL Point the canonical (and the “Read more” link) at the final URL Canonical points at a URL that redirects twice before it lands 301 → 301 → 200 Point the canonical at the final URL Canonical target is gone 301 → 410, or 410 directly Remove the page, after checking its backlinks; redirect it instead if it earns links Outbound link to a page that is gone 301 → 410, or 410 directly Remove the link; no live replacement was found Canonical loop The canonical target 301s back to the page itself Make the page self-canonical No readable sitemap sitemap.xml answers 404 Publish a sitemap; until then, “still listed” cannot be checked Not checked: a full crawl (there was no readable sitemap), indexation in Google, and the pages’ backlinks. The audit says so.
-
Human review before anything reaches the client (human-in-the-loop).
Sampling large sites
Full crawls of very large sites hit crawler limits and token budgets. Sample instead: the sitemap’s highest-traffic sections first (join sitemap URLs to GSC clicks), then a random sample of the long tail, and say in the audit notes which sections were sampled and at what rate. The URL Inspection API is the right tool for spot-checking individual URLs. It has a per-property daily quota (usage limits), so budget it for the samples that matter.
Re-auditing after fixes
An audit is not done when the list is delivered; it is done when the fixes are verified. After the client ships fixes, re-crawl the affected URL groups only, confirm each finding’s evidence URL now passes, and close the ticket. The agent can run that verification pass and draft the “fixed / still broken / regressed” summary, and the human confirms it before it goes in the next client report.
Presenting the audit to the client
Lead with the summary: the three findings that matter most and why, backed by the impact formula, then the full list as an appendix with evidence URLs. Every finding in the appendix must trace to a row in the crawl or GSC export. If the client spot-checks one and it does not hold up, the whole audit loses credibility. That is why the evidence-URL rule exists.
Normalising URLs before the join
A “duplicate” finding is often a join failure in the data, with nothing wrong on the site. Before joining crawl rows to GSC rows, normalise both sides: strip trailing slashes, lowercase the host, decide on www vs non-www once, drop tracking parameters the site itself ignores, and keep the path case-sensitive. Record the mapping rules in the audit notes. They are the first thing to check when GSC shows clicks on a URL the crawl reports as “not found”.
Issue taxonomy
Grouping by type is what turns a crawl dump into an audit. A working taxonomy:
| Category | Examples | Typical impact pattern |
|---|---|---|
| Indexation | noindex on indexable pages, canonical conflicts, excluded-by-robots pages that should rank | Lost visibility on pages that should earn it |
| Titles and meta | duplicate, missing, truncated titles | Lower CTR on pages that already rank |
| Canonicals and redirects | redirect chains, canonical pointing elsewhere, mixed signals | Diluted signals across URL variants |
| Internal links | orphan pages, broken internal links, links to redirects | Crawl budget wasted; orphan pages rarely rank |
| Rendering and JS | content only in JS, blocked resources, wrong status codes for bots | Googlebot sees a different page than users |
| Speed (field data) | LCP/INP/CLS failures in CrUX | UX and ranking factor; failures usually span whole templates |
From findings to tickets
The agent can draft one ticket per finding: title, evidence URLs, affected-URL count, suggested fix, and acceptance criteria (“URL returns 200, canonical self-referencing, present in sitemap”). Tickets go to the client’s tracker only after human review. The review is where false positives get caught and where the client sees the reasoning.
False-positive checklist
Agents produce false positives at scale, so each check below names how the agent tests the finding in the data, and when a human must decide anyway:
| Check | Why it happens | How the agent tests it | Human decides when |
|---|---|---|---|
| Staging, test or parameter URLs crawled by mistake | Crawl config follows every link | Cross-check URL against robots.txt, sitemap, and whether GSC shows it indexed | The URL is linked internally on purpose |
| Intentional noindex or canonical tags flagged as errors | Crawler reports the tag, not the intent | Check the tag against the page’s role (e.g. thank-you pages) | The “intentional” is undocumented |
| Redirect chains that are fine in practice | 301 chains of length 2+ look wasteful | Count hops and check the final URL returns 200 and is canonical | Chain crosses domains or loses equity |
| JS-rendered content missing from a non-rendering crawl | Crawler saw the HTML shell | Compare rendered vs raw HTML for a sample | The rendering is genuinely broken |
| “Thin” pages that are intentionally short (utility, legal) | Word count below a threshold | Check page type and whether it ranks/serves a query | Content should actually be expanded |
| Duplicate titles across paginated or faceted URLs already handled | Facets reuse the title | Check canonical and pagination tags | Handling is wrong |
| Lab vs field Core Web Vitals mismatch | Lab (synthetic) vs field (real users) disagree | Compare CrUX field data to the lab run | Field data is missing |
| Issues on URLs with no traffic and no links | Crawl reports everything | Join to GSC and backlinks; zero-traffic, zero-link URLs drop out | The URL matters for other reasons |
| Round, task | What the agent reported | What was true | Check that catches it |
|---|---|---|---|
| inkling-r2, T08 | A broken link worth pitching a replacement for | The 403 came from the site blocking bots, which does not show the page is gone | Re-check as a normal browser before calling a page dead |
| inkling-r2, T08 | A dead tool page | The page redirects; the agent read the redirect as dead | Follow every hop and judge the final status |
| hy4-flash-r2, T08 | A dead site, in a draft pitch | The site was up | Re-check the same day before a finding leaves the agent |
| hy4-flash-r3, T06 (reasoning off) | Landing-page lines that needed changing | The live page already matched both runs’ suggested lines | Compare each suggestion with the live page first |
Prioritisation logic
Use a simple, explainable formula instead of a raw issue count: run the false-positive pass first, then rank what survives by impact = affected pages × average monthly clicks (or an importance weight for key templates). A title-tag issue on 2,000 indexed pages outranks a broken image on 3 unlinked pages. Explainable matters because the client will ask why a finding ranks first.
Should the agent fix things itself?
Not on client sites without approval. Drafting tickets is fine.
What breaks
- Crawl limits that hide whole sections (pagination, JS-only nav).
- JS rendering differences between the crawler and Googlebot.
- Stale exports presented as current.
- The agent inventing issues that are not in the data. Require an evidence URL for every finding, and drop any finding without one.
What it costs
What a run is made of: crawler licence, API calls (URL Inspection is quota-limited), and LLM tokens per 1,000 URLs audited.
| Run | Model | Model $ | Data $ | Time | What it covered |
|---|---|---|---|---|---|
| Canonical and dead-link audit (the record above) | meta/muse-ember | 0.030 | 0.000 | 100 s | 15 requests: 19 pages inspected, 18 URLs checked with every hop, 1 sitemap read |
| Technical SEO QA (re-check of an earlier fix list) | xiaomi/mimo-v2.6-pro | 0.530 | 0.035 | 12 min | 51 requests: 14 page reads, 18 searches, 4 inspections, 3 URL checks, 1 backlink and 1 Search Console read |
No audit of 1,000 URLs or more has run in this workspace yet, so there is no per-1,000-URL figure here; the two runs above covered 19 and 14 pages.
FAQ
Can an AI agent run a technical SEO audit?
Yes for the first pass: it can crawl a site, group the findings by type and impact, and draft a prioritised fix list with evidence URLs. That is an AI-assisted technical SEO audit. What it cannot do is rule out false positives on a site it has never seen, so a person who knows the site reviews the list before anything reaches the client. That review is the difference between an audit and a raw crawl export.
Which crawl data does an audit agent need?
A crawler export (any desktop or cloud crawler; Screaming Frog and Sitebulb are common examples, no ranking), the GSC Page indexing report and URL Inspection API for samples, Core Web Vitals field data from web.dev's vitals guidance, plus the sitemaps and robots.txt. The agent joins crawl data to Search Console data by URL so impact can be scored against real traffic.
How do you stop an audit agent reporting false positives?
Two controls: the agent runs the false-positive checklist itself (testing each finding against the data, e.g. "is this URL actually linked or indexed?"), and a human makes the final call on anything ambiguous. Every finding must carry an evidence URL. A finding without evidence is treated as a false positive until proven otherwise.
Should an audit agent fix the issues it finds?
Not on client sites without approval. Drafting tickets or a fix list is fine; deploying changes is a human-in-the-loop decision, because a wrong "fix" (say, noindexing a revenue page) is worse than no fix. On your own staging site, let it iterate.