Overseas Site Technical SEO & AI Crawlers: Get Found by Global Search and Generative Engines

Summary: An overseas site's technical SEO cannot focus only on Googlebot — the same content must also be crawled and cited by AI engines like ChatGPT, Perplexity, and Claude. The biggest gap is rendering: Googlebot executes JavaScript, but most AI crawlers do not, so front-end-rendered pages appear blank to AI.

Teams expanding internationally often hit a paradox: a page ranks well on Google yet vanishes from ChatGPT's answers. The cause is rarely content quality — it is a technical layer that treats "traditional search crawling" and "AI crawling" as the same problem. This article separates the two and gives concrete robots.txt, international-structure, CDN, and rendering configurations so both global search and generative engines can reliably find and cite your content.

Key Takeaways

• Crawlers fall into three classes: Googlebot (indexing, drives clicks), AI training crawlers (GPTBot / Google-Extended, feed model weights), and AI retrieval crawlers (OAI-SearchBot / PerplexityBot / ClaudeBot, real-time citations — the only bots that drive AI traffic).
• The rendering gap is the top risk: GPTBot, ClaudeBot, and PerplexityBot do not execute JavaScript; a front-end-rendered international SPA is effectively blank to AI engines.
• Separate "training" from "retrieval / citation" in robots.txt: blocking GPTBot affects model training but not Google rankings; blocking PerplexityBot drops citations to zero within 48 hours.
• For international structure, prefer subdirectories (/de/, /fr/) over subdomains or ccTLDs, with bidirectional hreflang — consolidating authority without losing geo signals.
• Server physical location is not a ranking factor, but a CDN that caches languages together or strips hreflang will cut off AI and regional reach.
• Focus on three actions: put content in first-paint HTML, explicitly allow retrieval crawlers, and verify each crawler's HTTP status in server logs.

1. Two Crawling Problems Stack on Overseas Sites

Traditional search optimization solves one thing: get Googlebot to index the page into search results so users click through. Generative engine optimization (GEO) solves another: get AI crawlers to cite your content when answering user questions — and the user may never leave the chat interface. A tactic that works for one does not automatically work for the other; the two differ in capability, frequency, and intent.

On an overseas site this problem is amplified. You must use hreflang, subdirectories, and CDN to deliver the right content to the right language and region, while ensuring that content is visible to both the JavaScript-rendering Googlebot and the non-rendering AI crawlers. If any link in that chain shuts AI crawlers out, the brand disappears from AI recommendations overnight.

2. Know the Three Crawler Classes: Googlebot, AI Training, AI Retrieval

Sorting crawlers into three classes is the prerequisite before any configuration:

Two conclusions matter. First, AI retrieval crawlers are the only bots that bring your brand into AI answers — they should always be allowed; whether to allow training crawlers depends on whether you want your content fed into model weights. Second, Cloudflare's January 2026 analysis shows Googlebot fetched 1.7 times more unique URLs than ClaudeBot, confirming traditional search still leads in raw crawl volume even as AI crawler traffic grows fast.

3. The Rendering Gap: AI Crawlers Do Not Run JavaScript, and International SPAs Are Most Exposed

This is the blind spot overseas sites overlook most often. Googlebot runs a full Chrome rendering pipeline and executes JavaScript, rendering a single-page application (SPA) into what the user sees. But GPTBot, ClaudeBot, and PerplexityBot largely do not execute JavaScript — they parse only the raw HTML returned by the server. If your product descriptions, pricing tables, and FAQs load via client-side JavaScript, AI crawlers see an empty shell.

The evidence is hard: Vercel analyzed 1.3 billion AI crawler fetches from ChatGPT, Claude, and Perplexity and found zero JavaScript execution. Google also enforces a 2 MB HTML threshold — pages over it are truncated and bottom content never gets indexed; AI crawlers are stricter, working on token budgets where framework code crowds out real text. There are only three fixes: server-side rendering (SSR), static site generation (SSG), or dynamic rendering for AI crawlers that returns complete HTML on the first HTTP response.

A page ranking #1 on Google can be completely invisible to ChatGPT because it depends on client-side rendering — this is not hypothetical, it is a high-frequency incident for overseas SPA sites.

4. robots.txt: Separate "Training" from "Retrieval / Citation"

In 2026, robots.txt manages more than Googlebot — you must configure GPTBot, ClaudeBot, PerplexityBot, and others individually. The most dangerous mistake is a single User-agent: * Disallow: / that blocks everyone, including Google. The correct approach is tiered by crawler type:

# Allow real-time retrieval / citation crawlers (these bring AI traffic — must stay open)
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /

# Training crawlers are a strategic choice (blocking them does not hurt Google rankings)
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /

# Search engines always allowed
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
Sitemap: https://www.example.com/sitemap.xml

Three principles. First, blocking GPTBot (training) is unrelated to Google rankings — Google-Extended is a token separate from Googlebot, and Google explicitly documents that blocking it does not affect search inclusion or ranking. Second, blocking PerplexityBot is expensive: monitoring shows a site's citations in that engine dropped to zero within 48 hours of being blocked. Third, do not assume one user-agent name covers everything — OpenAI, Anthropic, and Google each split training and retrieval into different tokens, so a complete policy names each one. Also note Bytespider (ByteDance) has uneven robots.txt compliance; hard-blocking it requires user-agent or IP rules at the CDN / WAF, not just a text file.

5. International Structure: Subdirectories First, Bidirectional hreflang, No CDN Mishaps

Overseas sites have three URL structures, with very different implications for crawlability and authority:

For most international teams, subdirectories (/de/, /fr/) are the default recommendation: all backlinks accrue to one domain, you manage one Search Console property, and you inherit the authority the main domain already earned. ccTLDs fit long-term investment in specific markets with local teams and compliance needs; subdomains are only worth it when the tech stack must be isolated.

Whatever you choose, get hreflang right: every language / region variant must self-reference and bidirectionally reference all other variants (A links to B, B must link back to A), using ISO codes (en, de, fr, or region-qualified en-us); implement it in the HTML head, HTTP headers, or XML sitemap, and validate with Search Console's International Targeting report. Two CDN pitfalls: do not let the CDN strip hreflang tags or key technical elements on fetch; and cache each language's URLs separately so a French user never sees a cached English page.

6. Server Location and CDN: Not a Ranking Factor, but They Decide Crawl and Speed

Many teams agonize over "Germany or US for the server." The fact is: the physical location of the server IP is not a ranking factor — Google confirmed this years ago. What actually affects ranking is user experience, especially page-load speed, plus explicit geo signals via ccTLD, hreflang, or Search Console settings.

But "not a ranking factor" does not mean "unimportant." Every 100 ms of delay cuts conversion by roughly 7 percent (consistent across multiple studies); an overseas user hitting a US-origin server directly will lose conversions to latency. A CDN (Cloudflare, Fastly, AWS CloudFront) caches content at the edge node closest to the user; the origin can sit anywhere, and users rarely hit it directly. Track Core Web Vitals separately per target region, targeting LCP under 2.5 seconds — slow or resource-heavy pages are more likely to be skipped outright by AI crawlers. A frequently missed trap: a CDN or WAF (such as Cloudflare Bot Management) may misclassify AI crawlers as malicious scrapers and return 403; BrightEdge data shows that in January 2026, server errors accounted for nearly 25 percent of ChatGPT enterprise requests. After configuring, check server logs to confirm each crawler receives 200, not silent failures.

7. An AI-Crawlability Checklist for Overseas Sites

Turn the above into actionable steps:

1. Put content in first-paint HTML: use SSR or SSG so AI crawlers get the full body, headings, and FAQ on the first response without depending on JavaScript.
2. Explicitly allow retrieval crawlers: write explicit Allow rules in robots.txt for OAI-SearchBot, ChatGPT-User, PerplexityBot, and ClaudeBot — never let a wildcard block them.
3. Decide training crawlers by strategy: to keep your brand's semantic presence in models, allow GPTBot / Google-Extended; to block only training, Disallow them individually without affecting search.
4. Verify status codes: check the HTTP status each AI user-agent receives in server logs; fix 403 / soft-404 immediately — GA4 cannot see these visits, so you must read raw logs.
5. International structure + bidirectional hreflang: prefer subdirectories, self-reference hreflang and cross-link every pair, and include all language versions in the XML sitemap.
6. CDN: no language mixing, no tag stripping: cache per language, preserve hreflang, and point the Sitemap so AI crawlers prioritize important pages.
7. Quarterly review: new AI crawlers keep launching; re-audit robots.txt and CDN rules each quarter, evaluating every new crawler by its retrieval / citation role.

FAQ

Q: Will blocking GPTBot hurt my Google ranking?

A: No. GPTBot is OpenAI's training crawler, completely separate from Googlebot; Google-Extended is also a token independent of search crawling, and Google explicitly documents that blocking it does not affect search inclusion or ranking. Blocking GPTBot costs your content a place in model training, not search visibility.

Q: My page ranks #1 on Google but I can't find it in ChatGPT — why?

A: The most common cause is the rendering gap: the page depends on client-side JavaScript rendering. Googlebot executes JS and eventually indexes it, but ChatGPT / Perplexity / Claude crawlers do not, seeing only an empty shell. Switch to SSR or SSG so the body appears in first-paint HTML. It can also be a robots.txt rule that accidentally blocks OAI-SearchBot or PerplexityBot.

Q: Are subdirectories or subdomains better for an international site?

A: For most international teams, subdirectories (example.com/de/) win: they share the main domain's authority, are simple to operate, and need only one Search Console property; hreflang compensates for their weaker geo signal. ccTLDs give the strongest geo signal but each domain starts from zero; subdomains are treated as semi-independent sites with partially fragmented authority.

Q: Does the server have to be in the target country?

A: No. Server physical location is not a ranking factor; use a CDN to cache content at the edge closest to users, so the origin's location is irrelevant. What matters is Core Web Vitals per region (LCP under 2.5 seconds) and clear geo signals (hreflang / ccTLD / Search Console International Targeting).

Q: Can I just use User-agent: * Disallow: / to block all bots at once?

A: Absolutely not. That rule also blocks Googlebot and Bingbot, wiping out search traffic instantly. Configure by training / retrieval class individually, and always Allow the retrieval / citation crawlers.

About Tengence

Tengence is an AI-driven growth engine that uses a dual GEO + SEO engine to help cross-border businesses acquire customers through public-traffic channels and convert them in private domains. The scenario this article addresses — making content reliably crawlable and citable by both global search and generative engines — is exactly where Tengence GEO delivers: we fold the rendering gap, crawler tiering, international structure, and CDN traps into one integrated technical-SEO and GEO program, so brands gain visibility across Google and AI engines like ChatGPT, Perplexity, and Claude.

Related Reading

• Conversion Rate Optimization Guide
• How Do LLMs Get Web Content? The Complete Crawling and Parsing Pipeline
• How to Measure GEO Performance? Core Metrics and Tool Selection (2026)

Data Sources

1. What Is an AI Bot Crawler and How Is It Different From Googlebot? — Mersel AI: three crawler classes and key metrics (Googlebot 53 KB / 14:1, AI training 134 KB, ClaudeBot peak ~500,000:1).
2. Robots.txt and AI: Should You Block AI Bots? — AISOSystem: 35% of top global sites block GPTBot (42% in Europe), PerplexityBot citations hit zero within 48 hours, Google-Extended separate from Googlebot.
3. JavaScript SEO for AI Crawlers — AISEOShift: GPTBot / ClaudeBot / PerplexityBot do not run JS; Vercel's 1.3 billion fetches show zero JS execution; fix = SSR / SSG.
4. The Rendering Gap: Why AI Can't See Your JavaScript Website — Indexable AI: Google's 2 MB HTML truncation threshold and token-budget constraint.
5. AI Crawlers vs Search Engine Crawlers — Presenc AI: comparison table of rendering, frequency, content unit, and robots.txt compliance.
6. International SEO Strategy: Hreflang, CDN & Localization — Markana Media: subdirectories first, server location not a ranking factor, 100 ms ≈ 7% conversion, CDN language-mixing pitfall.
7. International SEO for SaaS: Hreflang, CDN Configuration — SaaSSEO: bidirectional hreflang implementation and per-region Core Web Vitals (LCP < 2.5s).
8. robots.txt for AI Crawlers — The Website Specification: 2026 major AI user-agent list and per-host configuration notes.