Screaming Frog Tutorial: How to Audit Your Site Like an SEO Agency

Screaming Frog is the desktop crawler that every reputable SEO agency runs, including ours. The Screaming Frog SEO Spider crawls your website the way Google does, then surfaces the technical issues that hurt rankings — broken links, redirect chains, missing meta data, indexability problems, schema errors and slow pages. This tutorial covers how Rank Haus uses Screaming Frog day-to-day to audit Australian client sites in 2026: the configuration that matters, the crawls to run, and the exports that translate into action.

What Is Screaming Frog?

The Screaming Frog SEO Spider is a desktop application (Windows, Mac, Linux) that crawls websites and reports on technical SEO data. The free version crawls up to 500 URLs per crawl, which is enough for very small sites. The paid licence (around £199/year as of 2026) unlocks unlimited URLs, JavaScript rendering, scheduled crawls, custom extraction, API connections to GSC and GA4, and integration with the major LLMs for analysis. It is the industry-standard crawler for in-house teams, agencies and freelancers.

Installing and Licensing

Download the installer from the Screaming Frog website and run it. The default installation is fine for most users. On first launch the free version is active; paste your licence key under Licence → Enter Licence to unlock the paid features. The paid version pays for itself the first time you audit a site with more than 500 URLs, which is almost every commercial site in 2026.

Configuration Before the First Crawl

The default settings are too generous for real client audits. The configuration changes we make before every crawl:

• Configuration → Spider → Crawl → JavaScript Rendering. Switch to “JavaScript” if the site is built on React, Vue, Next.js or any modern JS framework, or uses heavy JavaScript for content. For static WordPress and Shopify sites the “Text Only” mode is faster and sufficient.
• Configuration → Spider → Crawl → Crawl All Subdomains. Enable this if the site spans multiple subdomains (blog.example.com.au, shop.example.com.au).
• Configuration → Robots.txt. Choose whether to ignore or respect robots.txt. We respect it by default to see the same crawl Google sees, but switch to ignore when investigating accidentally-blocked sections.
• Configuration → User-Agent. Default is the Screaming Frog agent. For sites with WAF rules that block crawlers, switch to Googlebot.
• Configuration → Speed. Default is 5 threads at 2 URLs/second. Reduce to 1-2 threads on small hosts that 429 (rate-limit) the crawler.
• Configuration → API Access. Connect Google Search Console and GA4 to pull impression, click and engagement data alongside the crawl.

For multi-day audits on large sites, switch to Database Storage mode under Configuration → System → Storage Mode → Database. This lets you crawl millions of URLs without running out of memory.

Running Your First Crawl

Type or paste the site URL into the bar at the top and click Start. Screaming Frog crawls the homepage, follows every internal link, and discovers URLs the way Google does. Progress shows in the bottom-right corner. For a typical AU service business site of 50-200 pages, expect 5-15 minutes; ecommerce sites with 5,000+ products can take an hour or more.

While the crawl runs, the tabs at the top start populating. Internal, External, Response Codes, URL, Page Titles, Meta Description, H1, H2, Images, Canonicals, Pagination, Directives, Hreflang, AMP, Structured Data, Sitemaps. Each tab is one slice of the audit.

The Tabs to Audit First

For a fast pass on any new client site, the tabs we open in order:

1. Response Codes → Client Error (4xx). Every internal 404 is a broken link or a missing page. Filter, export, fix.
2. Response Codes → Redirection (3xx). Look for redirect chains (multiple hops) and redirect loops. The “Redirect Chains” report under Reports → Redirects gives the full chain.
3. Page Titles → Missing / Duplicate / Over 60 Characters. Missing titles are an emergency; duplicates split ranking signal; over-60 truncate in SERPs.
4. Meta Description → Missing / Duplicate. Missing descriptions let Google write its own, which often beats yours but loses your CTA. Duplicates suggest template copy across many pages.
5. H1 → Missing / Multiple / Duplicate. A page without an H1 is missing its strongest topical signal.
6. Canonicals → Self Referencing / Canonicalised / Missing. Misconfigured canonicals are one of the most damaging silent SEO problems.
7. Directives → Noindex. Find pages accidentally marked noindex (especially after WordPress theme updates).
8. Images → Missing Alt Text / Over 100KB. Accessibility, image search and page speed all hinge on these.
9. Structured Data → Validation Errors / Warnings. Pre-validates your schema before it fails in Google’s Rich Results Test.

JavaScript Rendering: When It Matters

Per the Screaming Frog configuration documentation, the SEO Spider can render JavaScript using a headless Chrome browser, mirroring how Google’s Web Rendering Service sees the page. Switch on JavaScript Rendering whenever you audit a site built on React, Vue, Angular, Next.js, Nuxt, Gatsby, or any heavy client-side framework. The crawl will be 5-10x slower but you will see what Google actually indexes, not the raw HTML shell.

The most common JS-rendering issue we find: content that appears in the rendered HTML but not the raw HTML, with no internal links pointing to it. Google can usually index it but treats it as lower priority. For SPAs and headless commerce sites, server-side rendering (SSR) or static generation (SSG) is the fix.

Version 24 and AI Integration (2026)

Screaming Frog 24, released in 2025, added native AI integrations with OpenAI, Claude and Gemini, plus an MCP (Model Context Protocol) server that lets AI assistants query crawl data directly. The practical use cases:

• Bulk meta description generation via OpenAI/Claude over a list of crawled URLs with their page content as input.
• Topic classification of every URL in a crawl using LLM tagging.
• Content gap analysis by querying the crawl through Claude or Gemini in natural language (“which blog posts cover topic X but don’t mention competitor Y”).

The AI integration is the biggest workflow upgrade in years for technical SEO audits and is the main reason agencies that delayed upgrading to 24 are catching up in 2026.

Connecting Google Search Console and GA4

Under Configuration → API Access, connect Google Search Console and GA4 with read access. After the crawl, the Internal tab gains columns for impressions, clicks, CTR and position (from GSC) and sessions, engagement rate and conversions (from GA4). The pattern that surfaces the most opportunity: pages with high GSC impressions but low CTR (meta description / title rewrites needed) and pages with high GA4 traffic but low conversion (CRO opportunities). See our guide to Google Search Console for more on what each metric means.

Exports That Translate to Action

Crawling is the easy part; making the data actionable is what separates agencies from amateurs. The exports we hand to clients (or use ourselves):

• Bulk Export → Response Codes → Client Error (4xx) Inlinks. Every broken link with the page it sits on. Send this to the dev team.
• Reports → Redirects → All Redirects. Full redirect chains with hop counts.
• Bulk Export → Page Titles → Missing. Pages needing titles, with their current H1 and content snippet for context.
• Reports → Structured Data → Validation Errors & Warnings. Every schema error grouped by type.
• Reports → Canonicals → Non-Indexable Canonicals. Pages canonicalising to non-200 URLs.
• Bulk Export → Images → Missing Alt Text Inlinks. Every image without alt text with the page it appears on.

Common Screaming Frog Mistakes

• Crawling without setting JS rendering correctly. Static sites in JS mode are slow; SPA sites in Text mode are inaccurate.
• Ignoring the speed setting. Aggressive crawling on small hosts triggers 429s and skews response code reports.
• Skipping API connections. Without GSC and GA4 connected, the crawl is technical-only and misses traffic context.
• Crawling once and never re-crawling. Sites change weekly; a stale crawl is a misleading audit.
• Treating warnings as urgent. Not every warning needs fixing; prioritise by impact, not volume.

Frequently Asked Questions

Is Screaming Frog free?

The free version crawls up to 500 URLs per crawl with limited features. The paid licence is around £199/year and unlocks unlimited URLs, JavaScript rendering, scheduling, custom extraction and API integrations. For any commercial site of more than a few hundred pages, the paid version is essential.

Does Screaming Frog work like Googlebot?

Yes, very closely. With JavaScript rendering enabled and the user agent set to Googlebot, Screaming Frog crawls and renders the way Google’s Web Rendering Service does. The main differences: Screaming Frog respects (or ignores) robots.txt deterministically, and does not factor in any signal beyond the crawl itself.

Can Screaming Frog audit a Shopify or WordPress site?

Yes — Screaming Frog is platform-agnostic. It crawls any public URL. For Shopify stores it surfaces the same issues an in-platform SEO app would (missing alt text, duplicate titles, redirect chains) plus deeper technical data the in-app tools miss.

How long does a Screaming Frog crawl take?

Small AU service sites (50-200 URLs): 5-15 minutes. Mid-sized ecommerce (1,000-5,000 URLs): 30-90 minutes. Enterprise sites (50,000+ URLs): hours to days, with database storage mode mandatory.

Should I run Screaming Frog on my client’s production site?

Yes, with the speed setting reduced (1-2 threads, 1-2 URLs/second) on shared hosting or unknown infrastructure. Crawl during off-peak hours if possible. Notify the client beforehand so they do not panic when Google Analytics shows a traffic spike from your IP.

What is the alternative to Screaming Frog?

Sitebulb is the closest direct competitor, with a friendlier interface but less raw flexibility. For cloud-based crawling, OnCrawl, JetOctopus and Botify are the enterprise options. Free alternatives like Visual SEO Studio exist but lack the depth and integration. For most agencies, Screaming Frog remains the best value.

Want Us to Run the Audit?

Crawling a site is easy; interpreting the data and turning it into a prioritised action plan is the work. Rank Haus runs full Screaming Frog audits as part of every technical SEO engagement, including GSC + GA4 cross-referencing and a one-page issue summary your dev team can action. Get in touch for a quote.