This guide covers who these crawlers are, how to control them, what llms.txt is (and is not), and how to make sensible decisions instead of blanket ones.

Three kinds of AI bots

Not every AI crawler does the same job. A useful mental model:

There are also robots.txt tokens that are not separate crawlers. Google-Extended controls whether content crawled by Google is used for Gemini model training and grounding; it does not affect Google Search crawling or ranking. Applebot-Extended plays a similar role for Apple.

Names and behaviors change, so check each vendor's current documentation before relying on any specific token.

GPTBot and friends in robots.txt

GPTBot is OpenAI's training crawler and a good example of how these controls work. Major AI vendors say their crawlers respect robots.txt:

# Allow AI search and user-triggered fetches, opt out of training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Allow: /

A few caveats:

• robots.txt is voluntary. Reputable crawlers follow it; bad actors do not. Enforcement needs a WAF or bot management at the CDN.
• Blocking a search-type crawler can remove you from that assistant's citations. Decide based on whether you want visibility there.
• Verify bots by IP ranges (most vendors publish them) before acting on user agents, since they are easy to spoof.

Deciding what to allow

There is no universally right answer. Questions to ask:

1. Do you want to be cited in AI answers? If AI assistants are becoming a discovery channel for your audience, blocking their search crawlers works against you.
2. Is your content your product? Publishers and paid-data businesses may reasonably opt out of training while allowing search-style access.
3. What is the server cost? Logs will tell you how much traffic each bot generates. Aggressive crawlers can be rate-limited rather than blocked.
4. Are there legal or licensing constraints? Some content may carry obligations about reuse.

For many marketing sites, a reasonable default is to allow search and user-triggered fetchers, then make a deliberate choice about training crawlers.

Google AI Overviews are part of Search

AI Overviews in Google Search use Google's normal search index, crawled by Googlebot. Blocking Google-Extended does not remove you from AI Overviews, and blocking Googlebot removes you from Search entirely. If you want to limit how your content appears in AI Overviews, the available controls are the same snippet controls used for regular results, such as nosnippet and max-snippet, which also affect your normal listings.

What is llms.txt?

llms.txt is a proposed convention, introduced in 2024, for a Markdown file at /llms.txt that gives language models a curated overview of a site: what it is, and links to the most useful pages, often as clean Markdown versions.

# Example Docs

> Example is an open-source job queue for Node.js.

## Docs
- [Quickstart](https://example.com/docs/quickstart.md): install and run your first job
- [API reference](https://example.com/docs/api.md): all methods and options

## Optional
- [Changelog](https://example.com/changelog.md)

Be clear-eyed about its status:

• It is not a standard and not an access-control mechanism. It does not block or allow anything.
• Google has said it does not use llms.txt for Search or AI Overviews.
• Evidence that major AI assistants consult it when answering questions is limited. Some developer tools and IDE assistants do use it to pull documentation.

For documentation-heavy developer products, publishing one is cheap and may help coding assistants. For a typical marketing site, treat it as optional and low priority compared with fundamentals.

What actually improves AI visibility

AI visibility, meaning how often your brand and pages are mentioned or cited in AI answers, mostly rests on the same foundations as search:

• Be crawlable without JavaScript. Many AI fetchers do not render JS. Server-rendered HTML matters even more here than for Google.
• Clear entities. Consistent organization, product and author information, backed by structured data, supports entity SEO: helping machines understand who you are and what you offer.
• Answer questions directly. Concise definitions, comparison tables and step-by-step sections are easy to extract and cite.
• Be cited elsewhere. Mentions on reputable third-party sites influence what models and AI search systems know about you.
• Keep facts accurate and current. Outdated pricing or specs get repeated.

This overlap is why many teams fold LLM SEO into their existing SEO process rather than treating it as a separate discipline. Our AI search topic hub and answer engine optimization topic hub gather the related terminology.

Measuring AI referrals

You can see some AI traffic directly:

• Referrers such as chatgpt.com, perplexity.ai, gemini.google.com and copilot.microsoft.com show up in analytics. Create a channel group for them.
• Server logs show which AI fetchers request which pages, a proxy for what is being cited.
• Many AI clicks arrive without referrers, so treat measured numbers as a floor.

In one of our projects, a new SEO SaaS grew from roughly 15 to about 2,500 monthly Google clicks in six months, and the content software case study includes lessons on how AI search changed click-through behavior along the way.

FAQ

Does blocking GPTBot remove me from ChatGPT answers? Not necessarily. GPTBot is for training. ChatGPT search and user-triggered browsing use other agents. Check OpenAI's current documentation.

Should every site add llms.txt? It is optional. Developer documentation sites are the most likely to benefit.

Will blocking Google-Extended hurt my Google rankings? Google states it does not affect Search inclusion or ranking.

How do I stop bots that ignore robots.txt? Use IP verification plus WAF rules or rate limiting at your CDN.

Wrapping up

Treat AI bots as distinct agents with distinct purposes, decide per category, verify before blocking, and invest in the fundamentals that make your content easy to understand and cite.

Want a plan for search and AI visibility together? Get a free growth plan from Top SEO Experts.

Published by Top SEO Experts.

This guide covers what to collect, how to verify real search engine bots, the questions logs answer best, and a simple pipeline you can run with standard tools.

Why logs matter for SEO

Log file analysis answers questions that crawlers and Search Console only approximate:

• Which URLs does Googlebot request most, and which does it never touch?
• How much crawl activity goes to parameters, faceted URLs, or old redirects?
• Are bots hitting 404s, 5xx errors or slow responses?
• How quickly are new pages discovered after publishing?
• Which AI and third-party bots are crawling, and how often?

For most small sites, crawling is not a constraint. For large sites (tens of thousands of URLs or more), ecommerce catalogs and sites with heavy parameter use, crawl budget becomes real: if bots spend their time on junk URLs, important pages are crawled less often.

What to collect

A standard combined log format line contains most of what you need:

66.249.66.1 - - [10/Oct/2026:06:14:22 +0000] "GET /blog/solar-incentives/ HTTP/1.1" 200 18432 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Fields to keep: client IP, timestamp, method, path with query string, status, bytes, user agent, and ideally response time. If you are behind a CDN, collect logs at the edge, because many requests never reach your origin. Most CDNs offer log push to object storage.

Retain at least 30 to 90 days so you can see patterns rather than a single snapshot.

Verify bots before trusting them

User agents are trivially spoofed. Scrapers routinely claim to be Googlebot. Verify with reverse and forward DNS:

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The hostname should end in googlebot.com or google.com and resolve back to the same IP. Google also publishes its crawler IP ranges as JSON, which is faster for bulk verification. Bing, OpenAI and others publish similar lists.

A minimal pipeline

You do not need a dedicated platform to start. Here is a pandas approach for a few million lines:

import re, pandas as pd

pat = re.compile(
    r'(?P<ip>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) \S+" '
    r'(?P<status>\d{3}) (?P<bytes>\S+) "[^"]*" "(?P<ua>[^"]*)"'
)

rows = (m.groupdict() for line in open("access.log") if (m := pat.match(line)))
df = pd.DataFrame(rows)
df["ts"] = pd.to_datetime(df["ts"], format="%d/%b/%Y:%H:%M:%S %z")
df["status"] = df["status"].astype(int)

gbot = df[df["ua"].str.contains("Googlebot", na=False)]   # then filter by verified IPs
gbot["section"] = gbot["path"].str.extract(r"^/([^/?]+)")
print(gbot.groupby("section").size().sort_values(ascending=False).head(15))
print(gbot["status"].value_counts())

For larger volumes, load logs into BigQuery, ClickHouse or DuckDB and run the same queries in SQL.

The questions worth asking

1. Where is crawl activity going?

Group verified Googlebot hits by site section and by URL type (canonical page, parameter URL, redirect, asset). If a large share goes to ?sort= and ?filter= combinations, you have a parameter problem to address with canonicalization, robots rules or better link hygiene.

2. Which important pages are rarely crawled?

Join the log data with your list of canonical URLs (from the sitemap or CMS). Pages with zero or very few Googlebot hits over 30 days are candidates for better internal links. Pages that are crawled heavily but are low value may be candidates for content pruning or consolidation.

3. What errors are bots seeing?

SELECT path, status, COUNT(*) AS hits
FROM logs
WHERE is_verified_googlebot AND status >= 400
GROUP BY path, status
ORDER BY hits DESC
LIMIT 50;

Spikes of 5xx responses to Googlebot can cause it to slow down crawling. Repeated 404s on URLs that used to exist point to missing redirects.

4. How fast is discovery?

Measure the time between publishing a URL and the first verified Googlebot request. If it is slow, improve internal linking from frequently crawled pages and keep sitemaps current with accurate lastmod values. For Bing, Yandex and other participating engines, IndexNow lets you notify them of changed URLs directly; Google does not currently use IndexNow.

5. Which other bots are crawling?

Logs reveal AI crawlers, SEO tool bots and scrapers. Knowing their volume helps you make informed robots.txt decisions rather than guessing.

Connecting logs to outcomes

Crawl data is a means, not an end. The goal is more of the right pages indexed and earning organic traffic. Join your crawl frequency data with Search Console clicks and with landing-page sessions and conversions from Google Analytics 4. Patterns that often emerge:

• Pages crawled frequently and earning traffic: protect them.
• Pages crawled frequently with no traffic: improve or consolidate.
• Valuable pages rarely crawled: fix internal links and discovery.

When you need to justify the engineering time to clean up crawl waste, framing it in business terms helps. Our SEO ROI calculator is a simple way to model what additional organic traffic might be worth.

Common pitfalls

• Analyzing unverified bots. Spoofed Googlebot traffic can be a large share of "Googlebot" hits on some sites.
• Origin-only logs. If your CDN serves cached pages, origin logs undercount bot activity.
• One-day snapshots. Crawl patterns vary daily; use weeks of data.
• Ignoring response time. Slow responses to bots can reduce crawl rate. Log $request_time in Nginx or the equivalent.
• Privacy. Logs contain IP addresses. Follow your retention and anonymization policies; for SEO analysis you rarely need user IPs beyond bot verification.

Where logs fit in the toolkit

Logs complement crawlers and Search Console rather than replacing them. Our technical SEO topic hub and search engines topic hub cover related concepts if you want to go deeper.

FAQ

How often should I analyze logs? A monthly review works for most sites; after migrations or big launches, check daily for a few weeks.

Is crawl budget a concern for small sites? Rarely. Under a few thousand URLs with healthy servers, focus on content and links first.

Can Search Console replace log analysis? The Crawl Stats report is a useful summary, but it is sampled and aggregated. Logs give URL-level detail.

Which log fields matter most? Path, status, user agent, IP (for verification), timestamp and response time.

Wrapping up

Logs are ground truth. Verify the bots, segment by URL type, and connect crawl behavior to indexing and traffic. It is one of the highest-signal analyses a developer can run for an SEO team.

Want help turning log data into an action plan? See our SEO services or get a free growth plan.

Published by Top SEO Experts.

This guide covers what hreflang does, the three ways to implement it, the rules that cause most bugs, and how to validate it automatically.

What hreflang does (and does not do)

hreflang annotations tell search engines that a set of URLs are equivalent versions of the same content in different languages or for different regions. When someone searches in Spanish from Mexico, a correct hreflang setup helps Google show your es-MX page instead of the English one.

It does not:

• Translate anything.
• Act as a ranking boost.
• Prevent a page from being indexed.
• Override strong signals like a canonical pointing elsewhere.

For broader context on language, region and domain strategy, see our international SEO topic hub.

Choosing a URL structure first

hreflang works with any structure, but decide this before implementing:

A ccTLD such as .de or .fr signals a country, which is great when you truly serve one market per domain. For most teams, subdirectories are easier to run.

Three ways to implement

1. HTML link elements

<link rel="alternate" hreflang="en-US" href="https://example.com/en-us/pricing/" />
<link rel="alternate" hreflang="en-GB" href="https://example.com/en-gb/pricing/" />
<link rel="alternate" hreflang="es"    href="https://example.com/es/pricing/" />
<link rel="alternate" hreflang="x-default" href="https://example.com/pricing/" />

2. HTTP headers

Useful for non-HTML files such as PDFs:

Link: <https://example.com/en/guide.pdf>; rel="alternate"; hreflang="en",
      <https://example.com/de/guide.pdf>; rel="alternate"; hreflang="de"

3. XML sitemap

Best for large sites, since it keeps page HTML lean and centralizes the logic:

<url>
  <loc>https://example.com/en-us/pricing/</loc>
  <xhtml:link rel="alternate" hreflang="en-US" href="https://example.com/en-us/pricing/"/>
  <xhtml:link rel="alternate" hreflang="es" href="https://example.com/es/pricing/"/>
  <xhtml:link rel="alternate" hreflang="x-default" href="https://example.com/pricing/"/>
</url>

Pick one method per page set. Mixing them increases the chance of contradictions.

The rules that cause most bugs

Return links are mandatory

If page A lists page B as an alternate, page B must list page A. Without reciprocity, search engines may ignore the annotation. This is the single most common hreflang error.

Self-reference

Each page should include itself in its own hreflang set.

Valid codes

Language is ISO 639-1 (en, es, de). Region is optional and uses ISO 3166-1 alpha-2 (US, GB, MX). Common mistakes:

• en-UK is invalid; the code is en-GB.
• es-LA is not a country code, and regional groupings such as es-419 are not supported in hreflang either. Use es as a generic Spanish version, plus specific countries such as es-MX where you truly localize.
• Region alone (hreflang="US") is invalid.

Point only at canonical, indexable URLs

Every URL in an hreflang set should return 200, be self-canonical, and be indexable. If an alternate URL redirects, is noindexed, or canonicalizes elsewhere, the cluster breaks.

This is where Search Console statuses get confusing. Seeing localized URLs under Alternate page with proper canonical tag is often fine and expected if they canonicalize to another version intentionally, but for hreflang you usually want each language version to be its own canonical. If you see Duplicate without user-selected canonical for regional variants, the pages may be too similar (for example, en-US and en-GB with identical text), and Google is choosing one for you.

Use x-default deliberately

x-default is the fallback for users who match none of your listed locales, often a language selector or your main English page.

Generating hreflang from data

Do not hand-write hreflang. Generate it from a translation map in your CMS or i18n config:

const locales = ['en-us', 'en-gb', 'es', 'de'];

function hreflangLinks(slugByLocale, defaultLocale = 'en-us') {
  const links = locales
    .filter(l => slugByLocale[l])
    .map(l => ({ hreflang: l.replace(/-(\w+)$/, (_, r) => '-' + r.toUpperCase()),
                 href: `https://example.com/${l}/${slugByLocale[l]}/` }));
  links.push({ hreflang: 'x-default',
               href: `https://example.com/${defaultLocale}/${slugByLocale[defaultLocale]}/` });
  return links;
}

Because every page in the group is rendered from the same slugByLocale map, reciprocity is automatic. When a translation is missing, the locale is simply omitted for the whole group.

Content quality still matters

hreflang tells search engines which version to show; it does not make a weak translation rank. Raw machine translation without review often produces awkward pages that underperform and can look like low-value scaled content. Have native speakers review at least your key pages, and localize the details: currency, units, examples, legal text, and calls to action.

Localize keyword research too. A direct translation of an English keyword is often not what people actually search. Run a separate keyword mapping per language so each localized landing page targets the terms its market uses, then check the local SERP to confirm the intent matches.

A bilingual example

We built a bilingual California personal-injury site with English and Spanish versions, structured around practice areas and cities, with consent-aware lead capture connected to the client's CRM. The Devoted Injury Lawyers portfolio entry shows how the two languages share one architecture rather than living as separate sites.

Validating automatically

Add a test that crawls the built site and checks, for every hreflang group:

1. Every URL returns 200 and is self-canonical.
2. Every URL lists every other URL in the group (reciprocity).
3. Every code is valid ISO language plus optional region.
4. Exactly one x-default per group.

Run it in CI so a new locale or slug change cannot quietly break the cluster.

FAQ

Do I need hreflang if I only have one language? No.

Does hreflang work for Bing? Bing has historically relied more on the content-language meta tag and other signals; set both if Bing matters to you.

Should en-US and en-GB pages have different content? They should differ meaningfully (spelling, pricing, examples). Identical pages may be clustered as duplicates.

Can hreflang point across domains? Yes, as long as reciprocity holds on both domains.

Wrapping up

Generate hreflang from one source of truth, keep every alternate canonical and indexable, and validate reciprocity in CI. Most hreflang bugs disappear once humans stop writing it by hand.

Expanding into new markets? Get a free growth plan from Top SEO Experts.

Published by Top SEO Experts.

This guide covers how to decide whether a programmatic project is worth doing, how to design templates that add value, and the technical controls that keep a large page set healthy.

What programmatic SEO actually is

Programmatic SEO means generating many pages from structured data, each targeting a specific query pattern. Familiar examples:

• "[Tool] integrations with [app]"
• "[Product] price in [city]"
• "[Term] definition" (glossaries)
• "[Currency A] to [currency B]"
• "Weather in [place]"

The pattern works when each page answers a real question with data the searcher cannot easily get elsewhere. Our programmatic SEO topic hub collects the related concepts if you want a primer before you start.

Where it goes wrong

Google's spam policies describe scaled content abuse: producing many pages primarily to manipulate rankings rather than help users, regardless of whether they are made by people, templates or AI. The policy is about intent and value, not the method.

Related failure patterns to recognize:

• Doorway pages: near-identical pages for each city or keyword variant that funnel users to the same place.
• Thin content: pages with a heading, a sentence of swapped variables, and nothing else.
• Duplicate content: thousands of pages that differ only by a single token, which search engines will cluster and mostly ignore.

The local-services version of this problem is especially common. We wrote about building location pages without creating doorways, which applies equally to any template keyed on place names.

The value test: run it before writing code

For a candidate page pattern, answer honestly:

1. Is there real search demand for the pattern? Check a sample of 20 to 50 variants, not just the head term.
2. Do you have unique data per page? Proprietary numbers, first-party research, inventory, pricing, reviews, specifications, local regulations. If the only thing that changes is the keyword, stop.
3. Would the page be useful with the keyword removed? If a user landed on it from a bookmark, would it still help?
4. Can you keep it accurate? Stale data at scale is worse than no page.

If you cannot answer yes to the second question, the project is a content strategy problem, not a template problem.

Designing a template that earns its place

A good programmatic template has a fixed skeleton and variable substance:

/integrations/{app}/
  H1: {Product} + {App} integration
  - What the integration does (specific to {app}'s API capabilities)
  - Setup steps (actual fields and auth method for {app})
  - Supported triggers/actions table (from integration metadata)
  - Limits and pricing notes (from data)
  - 2-3 real use cases (curated, not generated filler)
  - Related integrations (same category)

Practical rules:

• Conditional sections. If a field is empty, hide the section rather than showing "N/A." A page with three strong sections beats one with eight empty ones.
• Minimum data thresholds. Do not generate a page until the record has enough data to be useful. Put the rule in code.
• Human-written components. Introductions, comparisons and FAQs that a subject-matter expert writes per category, then the template reuses where they genuinely apply.
• Unique titles and descriptions built from the meaningful data, not just the keyword.

function shouldPublish(record) {
  const filled = ['summary', 'specs', 'pricing', 'faqs']
    .filter(k => record[k] && record[k].length > 0).length;
  return record.verified && filled >= 3;
}

Technical controls for large page sets

Index only what is ready

Generate pages, but let a quality flag decide whether they get index or noindex, and whether they appear in the sitemap. This lets you ship the skeleton and promote pages as data improves.

Watch for index bloat

Index bloat is when search engines hold many low-value URLs from your site. Faceted filters, sort parameters and empty combinations are common sources. Block or canonicalize parameter combinations that do not create meaningful pages, and keep the sitemap limited to indexable, canonical URLs.

Link the set into the site

Programmatic pages need internal links from category hubs and related pages, not just a sitemap. A flat list of 10,000 links on one page is not a structure. Use hierarchical hubs (country → region → city, or category → subcategory → item).

Roll out in batches

Publish a sample of a few hundred, measure indexing and engagement in Search Console over several weeks, then expand. If most of the sample sits in "Crawled - currently not indexed," that is a quality signal, not a crawl budget problem.

A real example: scale with substance

For a solar installer, we rebuilt the site and built a content architecture that included a 161-term glossary, a large solar content library and 50 researched city pages. The city pages were researched individually rather than spun from a single template. The solar installer content architecture case study shows monthly Google clicks going from roughly 1–2k to more than 20k. The scale helped, but only because each page had a reason to exist.

Programmatic pages and helpful content

Google's helpful content guidance asks whether content is made primarily for people. For programmatic projects, a useful internal check is: would we build this page set if search engines did not exist, for example as an internal tool or a product feature? If yes, you are probably on the right side of the line.

FAQ

Is programmatic SEO against Google's guidelines? No. Generating pages from data is fine. Producing low-value pages at scale to manipulate rankings is not.

How many pages should I launch at once? Start with a representative batch, measure indexing and engagement, then scale.

Can I use AI to write the variable text? You can, but the policy question is the same: does the result add real, accurate value per page? AI filler around a keyword does not.

What should I do with pages that never get indexed? Improve them, merge them, or noindex and remove them from the sitemap.

Wrapping up

Programmatic SEO is a data product. Start from the dataset, not the keyword list; set quality thresholds in code; roll out in batches; and be willing to delete pages that do not earn their place.

If you are considering a programmatic build, get a free growth plan and we will help you pressure-test the idea before you generate a single page.

Published by Top SEO Experts.

This playbook covers redirect fundamentals, building a redirect map you can test, and the checks to run before and after launch.

Redirect status codes, briefly

A 301 redirect tells search engines the old URL has been replaced and that the new URL should be treated as canonical. A 302 redirect says the move is temporary. Google has said that it handles both as redirects and that a long-standing 302 may eventually be treated like a 301, but the clearest signal for a migration is still a permanent redirect. Do not rely on the search engine guessing your intent.

Meta refresh and JavaScript redirects are followed by Google in many cases, but server-side redirects are faster, more reliable, and understood by every client.

What redirects preserve, and what they do not

When a URL is properly redirected to a relevant equivalent, the link equity earned by the old URL is largely consolidated on the new one. The key word is relevant. Redirecting hundreds of old product pages to the homepage is usually treated as a soft 404, which means the value is not passed.

Redirects do not preserve anything if:

• The target is unrelated to the source.
• The redirect is removed too soon (keep them for at least a year; ideally indefinitely).
• The chain is so long that crawlers give up.

Avoid chains and loops

A redirect chain is A → B → C → D. Each hop adds latency for users and a chance for crawlers to stop following. Googlebot follows up to around ten hops, but you should aim for one.

Chains build up over successive migrations. When you launch a new redirect map, update existing redirects so they point straight to the final destination.

# See every hop
curl -sIL http://example.com/old-page | grep -E "^HTTP|^location"

The small things that create duplicate URLs

Before you write a single redirect, decide on canonical URL rules:

• Protocol: HTTPS only.
• Host: www or apex, not both.
• Case: lowercase paths.
• Trailing slash: pick one convention and redirect the other.
• Index files: /about/index.html → /about/.

Combine these into a single normalization step so http://Example.com/About reaches https://example.com/about/ in one hop, not four.

# Nginx: force HTTPS + apex in one hop
server {
  listen 80;
  server_name example.com www.example.com;
  return 301 https://example.com$request_uri;
}

Use the migration to clean up your URL slugs only if the benefit is real. Every changed URL is a redirect you have to maintain and a small risk. Short, descriptive, stable slugs beat clever restructuring.

Building the redirect map

1. Collect every old URL. Combine a full crawl, XML sitemaps, analytics landing pages, Search Console top pages, and backlink exports. Crawls alone miss orphaned pages that still have links and traffic.
2. Map each URL to its closest equivalent. Use rules for patterns (/blog/:year/:slug → /blog/:slug/) and a lookup table for the rest.
3. Decide what dies. Content with no replacement and no value can return 404 or 410. A genuine 404 error for removed content is fine; a 404 for a page that has a clear successor is a wasted asset.
4. Store the map in version control as CSV or JSON so it can be reviewed and tested.

source,target,status
/services/web-design.html,/services/website-design/,301
/blog/2019/07/seo-tips,/blog/seo-tips/,301
/old-promo,,410

Testing the map before launch

Write a script that reads the CSV and asserts each source returns the right status and Location on staging:

import { readFileSync } from 'node:fs';

const rows = readFileSync('redirects.csv', 'utf8').trim().split('\n').slice(1);
for (const row of rows) {
  const [source, target, status] = row.split(',');
  const res = await fetch(`https://staging.example.com${source}`, { redirect: 'manual' });
  const loc = res.headers.get('location') || '';
  const ok = String(res.status) === status && (!target || loc.endsWith(target));
  if (!ok) console.error(`FAIL ${source}: ${res.status} -> ${loc}`);
}

Then check that every target returns 200 and is not itself redirected.

Canonicals during and after migration

Update every canonical tag to the new URLs. A surprisingly common bug is a new site whose canonicals still point at the old domain, sending conflicting signals that slow down consolidation. Good canonicalization means redirects, canonicals, internal links, hreflang and sitemaps all agree on the same preferred URL.

Launch day checklist

• Redirects deployed and tested against the map.
• Internal links updated to final URLs (do not rely on redirects internally).
• New XML sitemap submitted; keep the old sitemap briefly so Google recrawls the old URLs and sees redirects.
• robots.txt on production allows crawling; no staging noindex.
• Analytics and conversion tracking verified on new templates.
• For domain moves, use Search Console's Change of Address tool.

After launch

Monitor 404s in server logs and Search Console daily for the first few weeks. Some fluctuation in rankings after a large migration is normal. Persistent drops usually trace back to missing redirects, changed content, or lost internal links.

Also review your backlink profile. Valuable external links pointing at URLs that now 404 are worth fixing, either by adding a redirect or by asking the linking site to update. That is link reclamation, and after a migration it is often the highest-return outreach you can do.

FAQ

How long should I keep redirects? At least a year, and preferably permanently. They are cheap to keep and expensive to lose.

Will I lose rankings after a migration? A short period of fluctuation is common. A well-planned migration with complete redirects usually recovers; a poorly mapped one may not.

Should I redirect deleted pages to the homepage? Only if the homepage is genuinely the best replacement. Otherwise, redirect to the closest relevant page or return 404/410.

Is a 308 OK for SEO? Google treats 308 like 301.

Wrapping up

A migration is a data problem: inventory the old URLs, map them deliberately, test the map like code, and keep every signal pointing at the same final URL.

Planning a rebuild or domain move? Request a free growth plan and we will help you scope the migration.

Published by Top SEO Experts.

This guide explains what each control does, how they conflict, and a repeatable debugging workflow.

Crawling vs indexing: the distinction that matters

• Crawling is whether a bot is allowed to fetch a URL.
• Indexing is whether a fetched URL can be stored and shown in search results.

robots.txt controls crawling. The robots meta tag and the X-Robots-Tag header control indexing. Mixing them up causes most of the bugs in this article.

robots.txt: a crawl gate, not an index gate

User-agent: *
Disallow: /admin/
Disallow: /search?
Allow: /search?page=

Sitemap: https://example.com/sitemap.xml

Key behaviors:

• A disallowed URL can still be indexed (usually without a snippet) if other pages link to it. robots.txt does not remove pages from search.
• Most specific rule wins. For Google, the longest matching path takes precedence, and Allow wins ties.
• Rules are case-sensitive and match from the start of the path.
• The file applies per host and protocol. blog.example.com needs its own.
• A 5xx response on robots.txt can cause Google to pause crawling the site; a 404 is treated as "no restrictions."

The meta robots tag

<meta name="robots" content="noindex, follow">

A noindex directive tells search engines not to show the page in results. For Google to see it, the page must be crawlable. This is the most important interaction in the whole topic:

If a page is disallowed in robots.txt, Google cannot fetch it, so it never sees the noindex.

If you want a page out of the index, allow crawling and serve noindex. Once it has dropped out, you can decide whether to block crawling.

X-Robots-Tag: noindex in the HTTP header

The header does everything the meta tag does, but it works for any file type, including PDFs, images and JSON.

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nofollow

Nginx example for PDFs:

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex" always;
}

The danger is that headers are invisible when you view the page source. A header set at the CDN, load balancer, framework middleware or hosting platform can silently noindex the entire site, and nobody notices because the HTML looks fine. We wrote a short guide on how to check for a noindex header because this is so easy to miss.

A real example of a hidden header

In one project for a solar design SaaS, we found a site-wide noindex header combined with an over-broad content gate that was hiding more than 1,000 pages from search. The HTML had no noindex tag at all. The SaaS indexation recovery case study walks through how it was found and how indexation was restored. The lesson: always check headers, not just markup.

Common conflicts and what wins

Directives that do not exist

• noindex inside robots.txt: Google stopped supporting it in 2019.
• <meta name="robots" content="index"> does nothing useful; indexable is the default.
• nofollow in robots meta does not prevent indexing.

A step-by-step debugging workflow

1. Start in Search Console

The Page indexing report in Google Search Console groups non-indexed URLs by reason: "Excluded by 'noindex' tag," "Blocked by robots.txt," "Crawled - currently not indexed," and so on. The reason tells you which layer to check.

2. Inspect the URL

URL Inspection shows whether the page is allowed by robots.txt, whether indexing is allowed, the canonical Google selected, and the rendered HTML. "Test live URL" fetches the current version.

3. Check headers directly

curl -sI https://example.com/page | grep -i -E "x-robots|location|cache|HTTP/"
# Follow redirects and show headers for every hop
curl -sIL https://example.com/page

Run it with a Googlebot user agent too. If the response differs, find out why; serving different content to bots and users can drift into cloaking territory even when it is accidental.

curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/page

4. Check the rendered DOM

A noindex might be injected by JavaScript (a consent tool, an A/B testing script, a staging flag). Compare raw HTML from curl with the rendered HTML in URL Inspection.

5. Test robots.txt rules

Use Search Console's robots.txt report to see the fetched version and any parse errors. For local testing, Google's open-source robots.txt parser lets you check rules against paths in CI.

6. Align your sitemap

Your XML sitemap should list only canonical, indexable, 200-status URLs. Generate it from the same logic that decides indexability so the two cannot disagree.

Guardrails for staging and deploys

The classic disaster is a staging environment's Disallow: / or global noindex header shipped to production. Prevent it:

• Drive robots rules from an environment variable, and assert in a post-deploy smoke test that production returns no X-Robots-Tag: noindex on key templates.
• Protect staging with HTTP auth instead of relying on robots.txt.
• Monitor a handful of important URLs daily for status, canonical and robots signals.

FAQ

How long does noindex take to work? Until the page is recrawled and processed. Important pages may be days; deep pages can take weeks.

Can I use robots.txt to remove a page from Google? No. Use noindex (with crawling allowed), a 404/410, or Search Console's temporary removal tool.

Does X-Robots-Tag work for HTML pages? Yes, it applies to any response.

Should I noindex paginated pages? Usually not. Paginated pages help discovery of deeper items.

Wrapping up

Remember the layers: robots.txt decides fetching, meta and header directives decide indexing, and a blocked page cannot show its noindex. Check headers every time. If indexing issues are costing you traffic, our technical SEO services start with exactly this kind of diagnosis.

Need a fast diagnosis? Get a free growth plan from Top SEO Experts.

Published by Top SEO Experts.

This guide covers how to generate JSON-LD from your existing data, which types are worth implementing, and how to test it in CI.

Why JSON-LD

Structured data can be expressed as Microdata, RDFa or JSON-LD. Google supports all three but recommends JSON-LD, and for developers it is the obvious choice: it lives in a single <script type="application/ld+json"> block, it does not tangle with your HTML templates, and it can be built from the same data objects that render the page.

The vocabulary comes from Schema.org. When people say schema markup, they mean Schema.org types and properties, regardless of syntax.

What structured data can and cannot do

Structured data can make a page eligible for rich results like product price and availability, review snippets, breadcrumbs, and event details. It also helps search engines connect your pages to real-world entities: your organization, products, people and places.

It does not guarantee rich results, and it is not a direct ranking factor in the "add schema, rank higher" sense. Google decides whether to show enhancements based on eligibility, quality and the query.

Some features have also narrowed over time. Since 2023, Google shows FAQ rich results mainly for well-known government and health sites, and HowTo rich results were removed. FAQPage markup is still valid Schema.org and other consumers may read it, but do not promise a client FAQ snippets in Google.

Types worth implementing first

Organization

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "@id": "https://example.com/#org",
  "name": "Example Co",
  "url": "https://example.com/",
  "logo": "https://example.com/logo-512.png",
  "sameAs": [
    "https://www.linkedin.com/company/example",
    "https://github.com/example"
  ]
}

The @id lets other blocks reference this organization instead of repeating it. That is how you build a connected graph rather than isolated islands of markup. Consistent organization data, combined with mentions elsewhere on the web, is one of the inputs that can contribute to a knowledge panel, though there is no markup that triggers one directly.

Product

Product schema should be generated from the same record that renders the price on screen:

function productJsonLd(p) {
  return {
    '@context': 'https://schema.org',
    '@type': 'Product',
    name: p.title,
    sku: p.sku,
    image: p.images.map(i => i.url),
    description: p.summary,
    brand: { '@type': 'Brand', name: p.brand },
    offers: {
      '@type': 'Offer',
      price: p.price.toFixed(2),
      priceCurrency: p.currency,
      availability: p.inStock
        ? 'https://schema.org/InStock'
        : 'https://schema.org/OutOfStock',
      url: p.canonicalUrl,
    },
  };
}

Generating from one source of truth avoids the most common problem we find: markup that says $49 while the page says $59 because someone edited one and not the other.

BreadcrumbList

Breadcrumbs help users and give search engines a clear hierarchy. Build the JSON-LD from the same route data that renders the visible trail:

{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    { "@type": "ListItem", "position": 1, "name": "Docs", "item": "https://example.com/docs/" },
    { "@type": "ListItem", "position": 2, "name": "API", "item": "https://example.com/docs/api/" },
    { "@type": "ListItem", "position": 3, "name": "Auth" }
  ]
}

Rendering JSON-LD in modern frameworks

Server-render the script tag. Google can read JSON-LD injected by JavaScript after rendering, but other consumers often cannot, and you lose nothing by sending it in the initial HTML.

export default function ProductPage({ product }) {
  return (
    <>
      <script
        type="application/ld+json"
        dangerouslySetInnerHTML={{ __html: JSON.stringify(productJsonLd(product)) }}
      />
      {/* page content */}
    </>
  );
}

One security note: if any field can contain user input, escape < as < in the serialized JSON so a stray </script> cannot break out of the tag.

Rules that keep you out of trouble

1. Mark up what is visible. Structured data must describe content users can see on the page. Hidden reviews or invented ratings violate Google's guidelines.
2. One primary type per page. Supporting types (breadcrumbs, organization) are fine; a product page that also claims to be a recipe is not.
3. Use absolute URLs and the canonical version of each.
4. Keep it current. Out-of-stock products marked InStock, or old event dates, erode trust.
5. Do not mark up self-serving reviews of your own business on your own site for review snippets.

Testing and monitoring

• Rich Results Test shows which Google features a page is eligible for.
• Schema Markup Validator (validator.schema.org) checks general Schema.org validity.
• Search Console enhancement reports show errors and warnings across the site over time.
• CI checks: parse the JSON-LD from built pages and validate required properties with a JSON Schema or a small test suite, so a template change cannot silently strip offers.

An example from client work

For a B2B solar design services firm that had seen a year-long click decline, schema was one part of a broader programme that also covered content, internal linking and on-page fixes. The B2B website search growth case study describes how those changes coincided with a return to growth. We are careful not to credit schema alone; it works best as part of a coherent technical foundation.

FAQ

Is JSON-LD better than Microdata for SEO? Search engines treat valid markup equivalently. JSON-LD is simply easier to maintain and less error-prone.

Can I put JSON-LD in the body? Yes. Google reads it in the head or body.

Should every page have structured data? Most pages benefit from at least BreadcrumbList and an Organization or WebSite reference. Add specific types where they truly describe the content.

Why did my rich results disappear? Check for new errors, policy changes for that feature, and whether the visible content still matches the markup.

Wrapping up

Treat structured data as a typed API for search engines: generate it from your real data, render it on the server, validate it in CI, and only describe what users can see.

If you want help designing a schema graph for your site, get a free growth plan from Top SEO Experts.

Published by Top SEO Experts.

This guide covers what each metric measures, how to debug it with tools you already have, and fixes that tend to move the numbers.

Field data vs lab data

Before touching code, understand where the numbers come from.

• Field data comes from real Chrome users (the Chrome UX Report). This is what Google uses for its page experience signals. It is assessed at the 75th percentile over a rolling 28-day window.
• Lab data comes from a synthetic run (Lighthouse, WebPageTest). It is reproducible and great for debugging, but it is not what Google evaluates.

PageSpeed Insights shows both on one screen. If field data is missing, your URL or origin does not have enough traffic in CrUX, and lab data is your best proxy.

The current "good" thresholds, at the 75th percentile:

A note on expectations: Core Web Vitals are a ranking factor, but a modest one. Relevance and content quality still dominate. The bigger payoff is usually in user behavior, conversions and the ability to crawl and render your pages efficiently.

Largest Contentful Paint (LCP)

Largest Contentful Paint measures when the largest image or text block in the viewport finishes rendering. Break it into four parts:

1. Time to First Byte: server response time.
2. Resource load delay: time before the browser starts fetching the LCP resource.
3. Resource load duration: time to download it.
4. Element render delay: time from download to paint.

Start with TTFB

Time to First Byte is not a Core Web Vital, but every millisecond of it is added to LCP. Cache HTML at the edge where possible, avoid uncached database calls on hot templates, and watch for redirect hops before the final URL.

Then fix discovery of the LCP resource

The most common LCP problem we see is a hero image the browser discovers late: set as a CSS background, injected by JavaScript, or lazy-loaded.

<!-- Good: discoverable in HTML, prioritized, not lazy -->
<img src="/hero-1200.avif"
     srcset="/hero-800.avif 800w, /hero-1200.avif 1200w"
     sizes="(max-width: 800px) 100vw, 1200px"
     width="1200" height="630"
     fetchpriority="high"
     alt="Installer mounting solar panels on a pitched roof">

Never put loading="lazy" on the LCP image. Use fetchpriority="high" on it and modern formats (AVIF/WebP) at sensible dimensions. Good image SEO and performance overlap a lot here: correct sizing, compression and descriptive alt text help both users and search.

Reduce render delay

Render-blocking CSS and synchronous scripts in the <head> push LCP back. Inline critical CSS, defer non-critical scripts, and preload web fonts used in the LCP text with font-display: swap or optional.

Interaction to Next Paint (INP)

Interaction to Next Paint replaced First Input Delay in March 2024. Instead of measuring only the first interaction's delay, it looks at the latency of interactions across the page visit and reports roughly the worst one.

Each interaction has three phases: input delay (main thread busy), processing time (your event handlers), and presentation delay (style, layout, paint).

Find the slow interactions

• In Chrome DevTools, open the Performance panel and record while clicking through the page. Long tasks are flagged in red.
• Use the web-vitals library to log INP with attribution in production:

import { onINP } from 'web-vitals/attribution';

onINP(({ value, attribution }) => {
  navigator.sendBeacon('/rum', JSON.stringify({
    inp: value,
    target: attribution.interactionTarget,
    type: attribution.interactionType,
  }));
});

Fixes that work

• Yield to the main thread. Break long handlers into chunks with await scheduler.yield() (where supported) or a setTimeout fallback.
• Show feedback first, compute second. Update the UI state immediately, then do expensive work.
• Audit third-party scripts. Tag managers, chat widgets and A/B tools are frequent culprits. Load them after interaction or idle.
• Reduce hydration cost. Partial hydration or islands architecture means less JavaScript competing for the main thread.
• Avoid layout thrashing. Batch DOM reads before writes.

Cumulative Layout Shift (CLS)

Cumulative Layout Shift measures unexpected movement of visible content. It is usually the easiest of the three to fix.

• Always set width and height (or aspect-ratio) on images, videos and iframes.
• Reserve space for ads, embeds and cookie banners with fixed-height containers.
• Do not insert content above existing content after load unless it is in response to user input.
• Use font-display: optional or size-adjusted fallback fonts to reduce text reflow.
• Animate with transform and opacity, not properties that trigger layout.

.embed-slot { aspect-ratio: 16 / 9; width: 100%; }
.ad-slot    { min-height: 250px; }

Build it into your workflow

Performance regresses quietly. A few habits keep it in check:

1. Add Lighthouse CI or a performance budget to pull requests.
2. Collect real-user metrics in production so you see what CrUX will see, weeks earlier.
3. Track vitals per template, not per URL. One bad component usually affects hundreds of pages.
4. Re-check after adding any third-party script.

For a quick first look across a site, our free website SEO checker flags common technical issues alongside performance, which helps you prioritize.

FAQ

Will passing Core Web Vitals boost my rankings? It can help at the margin, especially when competing pages are otherwise similar. It will not make up for weak content.

Why do my Lighthouse scores and field data disagree? Lighthouse runs one simulated device and network. Field data reflects real devices, networks and interactions over 28 days.

Do Core Web Vitals affect bounce rate? Slow, jumpy pages tend to lose users. Bounce rate is not a direct ranking signal, but it is a useful business metric to watch alongside vitals.

How long until improvements show up in Search Console? Field data is a 28-day rolling window, so expect several weeks.

Wrapping up

Treat each vital as an engineering problem with a known shape: LCP is the critical path, INP is main-thread budget, CLS is reserved space. Measure in the field, debug in the lab, and fix at the template level.

If you want help prioritizing performance against other technical SEO work, request a free growth plan from Top SEO Experts.

Published by Top SEO Experts.

How Google actually processes a JavaScript page

Googlebot works in three broad stages: crawl, render, index. It fetches the raw HTML, parses it for links and directives, and queues the URL for rendering in a headless, evergreen Chromium. After rendering, it indexes the resulting DOM.

The important detail for developers is that the raw HTML is evaluated before rendering. If your initial response contains a noindex meta tag that JavaScript later removes, Google may never render the page at all. If the raw HTML has no links, discovery of deeper pages depends entirely on the render step succeeding.

Other crawlers are less patient. Many web crawlers, including several AI and social preview bots, do not execute JavaScript at all. For them, a client-side-rendered page is an empty <div id="root"></div>.

That is the core of JavaScript SEO: making sure the content and signals you care about exist in a form every important crawler can read.

Rendering strategies compared

For content that should rank, prefer SSR or SSG. Keep CSR for authenticated dashboards and app views that should not be indexed anyway. Frameworks like Next.js, Nuxt, SvelteKit and Astro make this a per-route decision, so you rarely need to choose one model for the whole site.

The five failure modes we see most often

1. Metadata injected only on the client

Single-page apps often set the title tag and meta description with a head manager after hydration. Google can pick these up after rendering, but social scrapers and many other bots will not. Render head tags on the server.

<!-- What the server should send, not what JS patches in later -->
<head>
  <title>Solar Panel Cost Calculator | Example</title>
  <meta name="description" content="Estimate system size, cost and payback in two minutes.">
  <link rel="canonical" href="https://example.com/tools/solar-cost/">
</head>

2. Links that are not links

Google discovers URLs from <a href="..."> elements. A <div onClick={() => router.push('/pricing')}> is invisible as a link. So is <a href="javascript:void(0)">. Use real anchors and let your router intercept the click.

// Crawlable: real href, client-side navigation still works
<Link href="/pricing">Pricing</Link>

// Not crawlable
<span onClick={() => navigate('/pricing')}>Pricing</span>

3. Soft 404s from catch-all routes

A SPA with a catch-all route usually returns HTTP 200 for every path, then renders a "Not found" component. Google may classify these as a soft 404, and worse, they can pile up as junk URLs. Return a real 404 status from the server for unknown routes. If you cannot, add <meta name="robots" content="noindex"> to the not-found view as a fallback.

4. Content behind interaction

Tabs, accordions and "load more" buttons are fine if the content is in the DOM on load. Content fetched only after a click or scroll will not be seen, because Googlebot does not click or scroll like a user. Infinite scroll needs paginated, linkable URLs behind it.

5. Blocked resources

If robots.txt disallows /static/ or your API domain, the renderer cannot fetch the JS or JSON it needs, and the rendered page comes out empty. Check that rendering-critical resources are crawlable.

Headless CMS setups

A headless CMS separates content from presentation, which is great for developer velocity but moves SEO responsibility into your frontend code. Things the old monolithic CMS handled for free now need to be built: canonical tags, redirects managed by editors, XML sitemaps generated from the content API, structured data templates and preview URLs that do not leak into the index. Put these on the build checklist before launch, not after the first traffic dip. Our website development topic hub collects related definitions if you need to brief a team.

A practical testing workflow

1. Compare raw vs rendered HTML. curl -s https://example.com/page | grep -i "<title" shows what the server sends. Compare with the rendered DOM in Search Console's URL Inspection tool ("View crawled page").
2. Disable JavaScript in DevTools and reload. Whatever disappears is at risk with non-rendering crawlers.
3. Check status codes for nonsense paths. curl -I https://example.com/this-should-404 should return 404, not 200.
4. Crawl with and without rendering. Most desktop crawlers support both modes. A big difference in discovered URLs points to JS-dependent links.
5. Inspect live URLs in Search Console after deploys that touch routing or head management.

A real-world example

When we rebuilt a solar CRM's marketing site on Astro, a static-first framework, part of the job was fixing broken forms and navigation and unifying content that had drifted across templates. The estimation software case study describes the project; impressions grew roughly 21x year over year after the rebuild and regional content work. We would not attribute that to the framework alone, but shipping full HTML on first response removed a whole class of rendering questions.

FAQ

Does Google index client-side rendered content? Usually, yes, if rendering succeeds and resources are not blocked. The risk is in delays, render failures, and the many non-Google crawlers that do not execute JavaScript.

Is dynamic rendering still recommended? Google describes it as a workaround rather than a long-term solution. SSR, SSG or hydration-based approaches are the better path.

Do I need SSR for every route? No. Index-worthy pages need server-rendered HTML. Logged-in app views generally should not be indexed at all.

How do I know if JS is hurting my rankings? Compare raw and rendered HTML, check indexing coverage for templates that rely on client rendering, and look for soft 404s and missing titles in Search Console.

Wrapping up

Rendering is an architecture decision with long-lived SEO consequences. Ship real HTML for pages you want found, use real links, return real status codes, and test the raw response, not just what you see in your browser. If you are planning a rebuild or migration, our website design and development team builds SEO requirements into the framework choice from day one.

Want a second pair of eyes on your stack? Get a free growth plan from Top SEO Experts.

Published by Top SEO Experts.