How LLMs Fetch Web Pages: The Full Chain of Crawling, Fetching, and Parsing

Summary: Large models don't "browse" web pages — they run a mechanical chain: a crawler discovers and fetches the page HTML, parses out the body and structure, strips boilerplate like nav and ads, chunks it into semantic blocks, embeds and stores them, and finally retrieves the most relevant block and attaches a citation when answering. This article strings those six steps into one readable picture and draws the GEO lesson: get crawled first, then get cited.

Many people imagine ChatGPT, Claude, or Perplexity "seeing" the web like a human: scrolling, clicking tabs, reading images. The reality is closer to a barebones downloader from 2010: it sends one HTTP request, reads back the bytes, parses the HTML, and leaves. Understanding this chain is the precondition for understanding why some content makes it into AI answers and some disappears forever. Let's break it down from discovery to citation.

Key Takeaways

• An AI crawler is an HTTP client, not a browser: it sends a request, reads HTML bytes, parses, follows links, and mostly does not run JavaScript.
• Pure JS rendering is the number-one trap: Vercel measured ChatGPT's 404 rate on pure-JS pages at 34.82%; critical content must live in server-side HTML (SSR/SSG).
• Parsing = extracting body + structure: text, heading levels, JSON-LD, meta, internal links; boilerplate (nav/ads/footer) is 3–5x the body HTML and must be cleaned.
• Cleaning has hard evidence: trafilatura with Markdownify cuts token cost by up to 67%; Readability's median body extraction is 0.970; keeping hierarchical structure beats plain-text chunking by 30%+ on retrieval precision.
• Chunking decides retrieval quality: fixed-size / sentence / heading-based / semantic are the four strategies; production defaults to recursive splitting + parent-child chunks (small chunk for precise search, parent for context).
• Embedding + retrieval is what produces citations: body → embeddings → vector store; retrieval crawlers (OAI-SearchBot / Claude-SearchBot / PerplexityBot) fetch the freshest page live and attach citations.
• The three crawler classes are managed differently: blocking training crawlers (GPTBot / ClaudeBot) does not hurt AI-search visibility (search uses a separate path); mistakenly blocking retrieval crawlers is what makes you vanish from answers.

1. Discovery and Crawling: Starting from Seed URLs

An AI crawler's starting point is the same as a traditional search engine's: a set of seed URLs — your sitemap.xml, links discovered from other pages, or a direct request from an AI feature. It sends an HTTP GET with a recognizable user-agent (e.g., GPTBot, ClaudeBot, PerplexityBot), gets the response body, and parses the HTML in it.

Before actually crawling, a compliant crawler reads robots.txt to judge whether it has permission for that path. Here is the key distinction: AI crawlers come in three classes, managed differently (see section seven):

• Training crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini training token), CCBot (Common Crawl) — they pull content into future training sets, producing no citation or traffic today.
• Retrieval / search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot — they build the index and, when answering in real time, pull from it and attach citations.
• User-triggered agents: ChatGPT-User, Claude-User, Perplexity-User — they fetch a page on the spot when a human asks the AI to open it.

2. Parsing HTML: Most Crawlers Don't Run JavaScript

After getting the HTML, the crawler parses text, heading levels (H1–H6), metadata (title/description/OG), JSON-LD structured data, image alt text, and internal links. But it almost never runs JavaScript — GPTBot, ClaudeBot, and PerplexityBot are plain HTTP clients that read the initial response and leave; there is no DOM, no event loop.

This is the number-one reason content "disappears." Vercel's measurement: ChatGPT's 404 rate on pure-JS rendered pages hits 34.82% — the page looks normal to a human, but the crawler gets only an empty shell. Product specs, prices, and inventory that hydrate in via a client-side API are, to an LLM crawler, nonexistent. Googlebot is a rare exception (it has a second headless-Chromium render pass), but that is a separate, lagging pass that can be seconds to weeks late — never treat it as the primary delivery path.

Conclusion: critical content must appear in the server-side HTML response — use SSR, SSG, or prerendering; at minimum give an SPA a server-rendered fallback.

3. Cleaning and Converting to Markdown: Pulling the Body Out of the Noise

A typical page's structure is hostile to text extraction: in one news article, nav, related recommendations, comments, cookie popups, and structured-data scripts often make up 3–5x the body's HTML. Feeding the model uncleaned is like polluting the embedding vectors with nav text, causing wrong matches at retrieval.

Production-grade parsing usually uses dedicated libraries rather than hand-written selectors:

• trafilatura: the default choice for article extraction; it combines HTML structure with text-density scoring to isolate body blocks, auto-removes boilerplate, and returns clean text/XML (with author, date). Paired with Markdownify to map HTML tags into Markdown headings, token cost drops by up to 67%.
• Readability.js: picks out the body by text/link density; benchmark median performance is 0.970, steadily stripping non-essential elements.
• newspaper4k / Jina Reader: the former layers keywords and summaries on top of extraction; the latter is an API-style "pass a URL, get clean Markdown," good for low-volume prototypes.
• Plain BeautifulSoup get_text is the anti-pattern — it loses hierarchy and has no boilerplate detection.

Key evidence: a 2024 comparison study found that methods keeping document hierarchy (headings, paragraph relations) beat plain-text chunking by over 30% on retrieval precision. So "convert to Markdown, not plain text" is not fastidiousness — it is accuracy.

4. Chunking: Split by Meaning, Not by Character Count

The cleaned body still has to be split into "chunks" before it enters the vector store. How you split directly decides retrieval quality:

The production default is recursive character splitting (first by paragraph, then sentence, then word; LangChain's RecursiveCharacterTextSplitter is the most common), with a token cap recursively applied to over-long passages. More advanced is parent-child chunking: a small chunk (128–256 tokens) for precise search, falling back to the parent chunk (1024–2048 tokens) for model context — getting both precision and completeness.

5. Embedding and Storage: Making "Similar Meaning" Retrievable

Each text chunk is turned into a vector by an embedding model and stored in a vector database (Milvus, Elasticsearch, Pinecone, etc.). Embeddings let the system compare meaning rather than keywords — a user's phrasing can differ from the original yet still recall the relevant chunk. This is the first step of RAG (retrieval-augmented generation): turning the entire web, or your site, into a knowledge base that can be pulled by semantics at any time.

6. Retrieval and Citation: Pulling the Most Relevant Chunk at Answer Time

When a user asks, the AI also embeds the question, finds the nearest chunks in the store, assembles them into context to generate an answer, and (in retrieval systems) attaches the source link. PerplexityBot, OAI-SearchBot, and Claude-SearchBot do exactly this — they fetch the freshest page live, build the index, and cite you in the answer. This is also where "being cited by AI" actually happens: content enters the retrieval index AND a chunk semantically matches the question, and only then can it be cited.

7. Three Crawler Classes and the robots.txt Decision

Many sites mistakenly think "block all AI crawlers" is safe, and end up vanishing entirely from AI answers. The correct move is to distinguish by class:

Key fact (CrawlForge and others measure consistently): blocking GPTBot / ClaudeBot does not affect your visibility in AI search — because ChatGPT search uses OAI-SearchBot and Claude search uses Claude-SearchBot, with independent settings. The one you mistakenly block is the retrieval class, which is what makes you "vanish from answers." Note that Google-Extended is only a robots.txt token, not a real crawler; blocking it does not affect Google Search ranking, only opts you out of Gemini training.

8. What This Means for You: Get Crawled Before You Can Be Cited

Stringing the above together, the first principle of GEO (generative engine optimization) is plain — for AI to cite you, it must first crawl you and understand you. In practice:

1. Critical content into server-side HTML: use SSR/SSG; don't leave specs, prices, and conclusions living only in JS (or face the 34.82% 404 risk).
2. Semantic HTML + JSON-LD: use
   
   , clear H1–H6, lists, tables; add Article/FAQPage/BreadcrumbList structured data so AI doesn't guess "is this a price or a phone number."
3. Sitemap with lastmod, internal links as a web: let crawlers find every important page; don't leave orphans.
4. Add llms.txt: use the llms.txt spec to give AI a site guide; forward-looking sites already do.
5. Fast load: slow pages may be skipped on timeout.
6. Don't mistakenly block retrieval crawlers: allow OAI-SearchBot / Claude-SearchBot / PerplexityBot in robots.txt; block only the training class.

9. Common Misconceptions

• "AI sees web pages like a human": wrong. It is an HTTP client that reads HTML bytes, runs no JS, scrolls no pages, clicks no tabs.
• "My SPA is fine, humans can see it": the crawler sees an empty shell; critical content must be server-rendered.
• "Blocking all AI crawlers is safest": mistakenly blocking the retrieval class makes you vanish from AI answers; only blocking the training class leaves search visibility untouched.
• "Longer body is better": uncleaned long HTML is noise, and boilerplate pollutes embeddings; clean structure + reasonable chunking beats piling up words.
• "Feeding raw HTML as plain text is easier": you lose hierarchy and retrieval precision drops 30%+; convert to Markdown to keep structure.
• "Blocking Google-Extended will drop my ranking": no, it only controls Gemini training use and is unrelated to Search ranking.

FAQ

Q: How is a large-model crawler different from a normal search-engine crawler?

A: Different goals. Googlebot / Bingbot build a retrievable index and send humans back to your page; training-class AI crawlers (GPTBot / ClaudeBot) "absorb" the page into model knowledge with no traffic today; retrieval-class AI crawlers (OAI-SearchBot / PerplexityBot) build the index and cite you with links in real-time answers. In behavior, most AI crawlers also don't run JavaScript.

Q: My site is a React/Vue single-page app — can AI read it?

A: Probably not the critical content. Most LLM crawlers don't run JS and only take the initial HTML shell. Vercel measured ChatGPT's 404 rate on pure-JS pages at 34.82%. You must do SSR/SSG or prerendering so specs, prices, and conclusions appear in server-side HTML.

Q: Why can't I just feed the HTML to the model as plain text?

A: A typical page's nav/ads/footer is 3–5x the body HTML; plain text embeds the noise together and causes wrong matches at retrieval. Clean first (trafilatura / Readability), then convert to Markdown to keep hierarchy — evidence shows over 30% higher retrieval precision and up to 67% lower token cost.

Q: Will blocking GPTBot make my content disappear from ChatGPT search?

A: No. ChatGPT search uses OAI-SearchBot, independent of GPTBot settings; blocking the training class only opts you out of future training sets, not real-time search citations. What actually makes you vanish from AI answers is mistakenly blocking the retrieval class (OAI-SearchBot / Claude-SearchBot / PerplexityBot).

Q: How do I size chunks?

A: Most production systems land at 256–1024 tokens per chunk. Default to recursive character splitting (paragraph, then sentence, then word), with a token cap for over-long passages; more advanced is parent-child chunking — small chunk for precise search, parent for context, getting both precision and completeness.

Q: Is llms.txt the same as robots.txt?

A: No. robots.txt controls "can it crawl, and where"; llms.txt (the llmstxt.org spec) is a site guide for AI, telling it which pages matter and how to understand your site — a nice-to-have, not mandatory. Configure both.

About Tengence

Tengence GEO helps enterprises claim a position in the "content crawled → understood → cited by AI" chain: while keeping SEO fundamentals (SSR/SSG friendliness, semantic HTML, structured data, sitemap), it adds JSON-LD, answer-block restructuring, and per-engine AI-citation-rate monitoring, so the brand stays in both traditional search and AI answers. Tengence (TENGENCE) is an enterprise growth engine for the AI era, focused on public-traffic acquisition and private-domain conversion; its core products are Tengence GEO (the GEO + SEO dual engine) and Tengence Search.

Related Reading

• What Is GEO (Generative Engine Optimization)? Definition, How It Works, and How It Differs from SEO
• Whom Do ChatGPT, Perplexity, and Gemini Cite? A Breakdown of the Three AI Platforms' Source Preferences
• How Structured Data Helps GEO: The Complete Schema Markup Guide

Data Sources

1. anglera: How LLM Crawlers Fetch Pages: GPTBot/ClaudeBot/PerplexityBot are ordinary HTTP clients, run no JS, read only the initial HTML bytes; JS-only SPA product data is invisible to LLM crawlers.
2. trackmyvisibility: How AI Crawlers Work and What They Look For: the four-step fetch-parse-chunk-embeddings/vector-store; Vercel measured ChatGPT's 404 rate on pure-JS pages at 34.82%; SSG/SSR required.
3. aisearchdynamics: How AI Models Crawl Websites: training crawlers vs retrieval crawlers, two-way split; parsing takes text/heading/JSON-LD/meta/internal links; no JS; step-by-step action items.
4. tubrief: Why Scraping Markdown Instead of HTML Makes RAG Answers More Accurate: keeping hierarchical structure beats plain-text chunking by over 30% on retrieval precision (2024 study); trafilatura with Markdownify cuts tokens 67%; Readability median 0.970.
5. CrawlForge: AI Crawler List 2026: blocking GPTBot/ClaudeBot does not affect AI-search visibility (search uses OAI-SearchBot/Claude-SearchBot), with independent settings.
6. bestseo.sg: GPTBot, ClaudeBot and PerplexityBot 2026 Configuration Guide: complete reference table and default actions for the three AI crawler classes (training block / retrieval allow / user-triggered allow).
7. dev.to: Web Scraping Pipeline for RAG: trafilatura extracts the cleanest body (better than readability-lxml/newspaper3k); five-stage pipeline (fetch → extract → normalize → chunk → embed).

Get Started

• Try Tengence free
• Book a free expert consultation and diagnostic
• Learn about Tengence GEO