Cross-Border and Multilingual Search: Architecture and Deployment for Cross-Language Retrieval

Summary: Cross-border search is not solved by translating the query once: tokenization, morphology, character normalization and word order all change what gets recalled. The workable architecture has three layers – language-specific analysis, vector alignment and localized ranking – and dropping any one of them leaves a recall gap in some language.

A German buyer searches "wasserdichte jacke", a Japanese buyer searches for the same waterproof jacket in Japanese, and a Thai buyer types a run of Thai script with no spaces at all. With one retrieval pipeline that splits on whitespace and normalizes by English rules, all three queries come back incomplete. The difficulty is not whether the translation is accurate; it lies in four layers – tokenization, morphology, character normalization and ranking localization – plus how to align query language with document language when the two differ. This article follows the order of problem, architecture, tokenization, cross-language retrieval, ranking and measurement, and gives two comparison tables plus a deployment checklist.

Key Takeaways

The first hurdle in cross-border search is tokenization: Chinese, Japanese and Thai are written without spaces, so splitting on whitespace treats a whole sentence as one token.
Languages with rich inflection such as German, Russian and Arabic require stemming; otherwise plural and case forms never recall one another.
Of the three architectures, per-language indices give the best quality but cost grows with each language, a mixed index is the mainstay, and a vector index fills the gaps.
Cross-language retrieval has three paths – query translation, document translation and vector alignment; in practice vector alignment is the primary route with the keyword route as backstop.
Currency, units, dates, regional availability and compliance blocking are filter rules and should never be mixed into relevance scoring.
Effectiveness must be measured per language and per region, because blended metrics are averaged away by the dominant language and hide small-language problems.

1. Why multilingual search is not just a matter of translation

Treating multilingual retrieval as "add one translation interface" is a common misjudgement. Translation only handles differences of expression; it does not solve four structural problems.

Tokenization. Chinese, Japanese, Thai, Lao, Khmer and Burmese are written without spaces. The official ICU user guide points out that word boundaries in these languages cannot be decided from character-sequence rules alone, and ICU provides dictionary support for all of them. Keep splitting on whitespace and an entire sentence becomes a single term that never matches.

Morphology. German, Russian, Polish, Arabic and Finnish express gender, number and case through word endings. Without stemming, "Schuhe" and "Schuh", or "книга" and "книги", are two unrelated terms, and a reworded query recalls nothing.

Character sets and encoding. One visual character can have several Unicode representations: a letter with a diacritic may be a single code point or a base letter plus a combining mark; there are also full-width and half-width forms, positional shaping of Arabic letters, and the special casing rules of Turkish. Without normalization, one word ends up with several versions.

Word order and compounds. German concatenates several nouns into a compound, Japanese and Korean put the verb last, and Chinese places modifiers first. Phrase and proximity rules written on English word-order assumptions simply stop working in other languages.

2. Comparing three architecture routes

Architecture choice is a trade-off between retrieval quality, operating cost and cross-language capability.

Dimension	A. Per-language indices	B. Mixed multilingual index	C. Cross-language vector index
Core approach	one index and analyzer per language	single index plus a language field and sub-fields	shared multilingual embedding plus a vector store
Same-language quality	highest	high, depends on the field split	medium, strong on semantics, weak on precision
Cross-language recall	not supported	not supported	native, one shared semantic space
Storage and compute	grows linearly with languages	single index, field bloat	large vector index, needs approximate search
Main risk	cost rises linearly	field explosion, hard to unify scoring	unstable on model codes and numbers
Best fit	few languages, each large	mainstay, same-origin content	cross-language and long-tail fallback

Route A suits sites with a fixed language set where every language is large: analyzers and ranking rules can be tuned independently, at the price of duplicating the whole pipeline for each new language. Route B is what most cross-border sites choose: one index, a language field marking the language, text split per language into sub-fields each bound to its own analyzer, and queries routed by language at search time. Route C complements rather than replaces – it is unstable on exact model codes and still needs the keyword route underneath. For how the two routes combine, start with What Is Semantic Search? Vector Retrieval and Keyword Retrieval Compared.

3. Tokenization and analyzers: choose per language

Why cannot Chinese be split on spaces? Chinese marks its smallest semantic units without spaces, so a product phrase written as one run of characters is cut into a single long term, and a shopper who types the same idea with the words in a different order cannot align with it. There are two remedies: dictionary-based segmentation (IK, jieba and similar) is controllable but needs an industry dictionary maintained; CJK bigram segmentation cuts a run of Han characters into adjacent pairs, giving high recall with no dictionary, at the cost of index bloat. The official Elasticsearch documentation states that this filter splits Chinese, Japanese and Korean text into bigrams and passes non-CJK input through unchanged.

Language or family	Main difficulty	Recommended analysis chain	Notes
Chinese	no spaces, word boundaries are semantic	dictionary segmentation or CJK bigrams	bigrams are noisy; dictionaries need upkeep
Japanese	kanji and kana mixed, no spaces	Kuromoji segmentation plus kana normalization	kanji, katakana and romaji spellings must be merged
Korean	spaces exist but particles agglutinate	Nori segmentation plus particle removal	particles left in drag down long-tail recall
Thai, Lao, Khmer	no spaces, dictionary segmentation needed	ICU tokenizer	avoid mis-splitting when Latin text and digits are mixed in
German, French, Spanish, Russian, Polish	rich inflection and declension	language analyzer plus Snowball stemming	aggressive stemming hurts precise recall
Arabic, Hebrew	right to left, positional letter shaping, diacritics	language analyzer plus Unicode normalization plus ICU folding	unnormalized diacritics split one word into two

Engineering note: the official Elasticsearch language analyzers already ship preset configurations for dozens of languages including Arabic, German and Russian, complete with stop-word lists and stemming rules – reuse them first; stemmers merge aggressively, so model codes and brand names must go into an exclusion list; SKUs and model codes should be keyword fields and stay out of tokenization.

4. Cross-language retrieval: query in language A, documents in language B

Query translation: translate the query in real time, then retrieve. Upside: the existing index is reused and adoption costs the least. Downside: every query adds an external call and latency, polysemes and industry terms are easily mistranslated, and the error happens before retrieval, so the ranking stage cannot correct it.

Document translation: at index time, translate documents into several target languages and index each one. Upside: zero latency on the query side. Downside: index size multiplies by the number of languages, uneven translation quality pollutes the fields, and any update to the source means retranslating.

Multilingual vector alignment: use a multilingual embedding to map different languages into a shared semantic space, so cross-language recall needs no explicit translation. LaBSE combines masked language modeling, translation language modeling, dual-encoder translation ranking and additive margin softmax; on Tatoeba it reaches 83.7% bilingual text retrieval accuracy across 112 languages, clearly above the 65.5% of earlier work, and it ships models covering more than 109 languages. The open multilingual E5 family first contrastively pretrains on billions of multilingual text pairs and then fine-tunes, offering several sizes to trade off.

The three paths are not mutually exclusive: the vector path does the primary cross-language recall, each language's keyword path guarantees precision, the two are fused at ranking, and terminology plus model codes are kept in a protected list that must never be translated. For weight tuning see How to Optimize Search Relevance Ranking: A Two-Stage Recall and Ranking Method.

5. Localization on the ranking side

Currency, units and dates. Store prices in the site currency and state whether tax is included; ranking and range filters must use the same convention. Convert size and weight to local habits. Dates appear in both day-first and month-first order, and thousands and decimal separators differ too. All of this should be normalized during query parsing.

Local vocabulary. The same concept has different names in different markets, football versus soccer being the classic case, so synonym lists must be maintained per language and must not be reused across languages. Differences such as Japanese katakana loanwords and Spanish accent folding should be handled at the analysis stage.

Regional availability. Sellable regions, stock, delivery lead time and customs restrictions are hard constraints and belong in recall-stage filters rather than as score penalties – an item excluded by a regional rule should not appear no matter how relevant it is.

Compliance and sensitive content. Markets differ on prohibited categories, content ratings and personal data handling, so blocklists must be managed per regional version with audit records kept.

6. Measuring and accepting results: why you must segment

The most common measurement mistake is looking at a single blended number. If English traffic accounts for the bulk of searches, a simultaneous 20% recall drop in German and Thai may move the blended metric by only a few percentage points, which still looks like ordinary noise – the volume of the dominant language averages away the collapse of the small ones.

Metric	Segment dimensions	Purpose
Zero-result rate	language by region	locate missing tokenization or indices
Top-result click rate	language by query type	judge whether ranking localization landed
Labeled-set recall at K	language	compare quality across languages
Response latency P95	language by region	surface translation or vector service latency

Deployment advice: build a labeled test set per language covering four query types – high-frequency terms, long-tail terms, model codes and natural-language questions – and use it as the regression baseline for changes; dashboards should drill down by language and region by default, with alert thresholds set per language. For per-language zero-result triage see How to Reduce the Zero-Result Rate of On-Site Search: 8 Practical Strategies.

7. Deployment checklist

Detect the language before ingestion and write a language field into the index as the routing key.
Bind an independent analyzer to each language: bigram or dictionary segmentation for Chinese, the ICU tokenizer for languages written without spaces.
Apply Unicode normalization and case folding uniformly before indexing, with attention to the special Turkish rules.
Keep SKUs, model codes and brand names as keyword fields, or add them to the stemmer exclusion list.
Maintain synonym lists per language and never reuse them across languages.
For cross-language scenarios add the vector path, fuse it with the keyword path, and protect terminology from translation.
Normalize price, size and date units during query parsing.
Turn sellable regions, stock and customs limits into recall filters that do not participate in scoring.
Manage compliance blocklists per regional version and keep change and audit records.
Build a labeled set covering four query types for every language as the regression baseline.
Segment dashboards by language and region, with an independent alert threshold per language.

For a unified integration path see Tengence Intelligent Search: An AI-Driven On-Site Search Solution.

FAQ

Q: Can multilingual search be solved by plugging in one translation API?

A: No. Translation only handles differences of expression; it does not solve the four structural problems of tokenization, morphology, character normalization and word order. Splitting a language written without spaces on whitespace fails, and without stemming an inflected language never recalls plural and case forms. Translation belongs in the solution, but it is not the whole solution.

Q: Should I build one index per language or one combined index?

A: With few languages, each of large volume, separate indices give the best quality and can be tuned independently, but operating cost grows linearly with the number of languages. Cross-border sites with a moderate number of languages and same-origin content more often use a single index plus a language field and per-language sub-fields, collapsing operations into one index.

Q: Why can Chinese not be tokenized on spaces?

A: Chinese does not mark word boundaries with spaces, so splitting on whitespace turns a whole sentence into one very long term that only matches when the original wording appears in full. There are two remedies: dictionary segmentation is controllable but needs dictionary upkeep, and CJK bigram segmentation cuts Han characters into adjacent pairs, giving high recall at the cost of index bloat.

Q: Query translation or document translation – which should I choose?

A: Query translation costs the least to adopt and reuses the existing index, but it adds latency to every query, mistranslates terminology easily, and cannot be corrected at the ranking stage. Document translation gives zero latency on the query side but multiplies index size by the number of languages and forces retranslation whenever the source changes. Most teams move to vector alignment for cross-language recall instead.

Q: Can a multilingual vector model replace keyword retrieval?

A: No. Vector models are clearly stronger at cross-language recall, but they distinguish model codes, encodings and numeric specifications poorly and tend to return results that are semantically close yet wrong on specification. The correct setup is the vector path for primary recall, the keyword path as backstop, the two fused, with terminology protected.

Q: Why can cross-border search metrics not be read as one blended number?

A: Because the dominant language dilutes problems in the smaller ones. If English traffic accounts for the bulk of searches and German and Thai recall both drop by 20%, the blended metric may fall by only a few percentage points and still look like ordinary noise. The zero-result rate, the top-result click rate and recall must all be read segmented by language and region.

About Tengence

Tengence Search targets cross-border and multilingual retrieval, providing per-language analysis chains, fused keyword and vector recall with ranking, plus currency and unit normalization, regional availability filtering and compliance blocking. Tengence (TENGENCE) is the enterprise growth engine of the AI era, focused on public-traffic acquisition and private-domain conversion; its core products are Tengence GEO (dual-engine GEO + SEO) and Tengence Search.

Data Sources

Unicode ICU, Text Boundary Analysis: notes that Chinese, Japanese, Thai, Lao, Khmer and Burmese are written without spaces and need dictionary support for word boundaries, supporting the claim that Chinese and Thai cannot be split on spaces.
Elasticsearch Guide, CJK bigram token filter: states that the filter cuts Chinese, Japanese and Korean text into bigrams and passes non-CJK input through unchanged, supporting the Chinese bigram approach.
Elasticsearch Plugin Docs, ICU Analysis plugin: the plugin uses the ICU library for Asian-language analysis, Unicode normalization and case folding, supporting the ICU tokenizer for Thai and similar languages.
Elasticsearch Guide, Language analyzers: lists preset stop words, stemmers and exclusion-word settings for Arabic, German, Russian and others, supporting the use of official analyzers per language.
arXiv, Language-agnostic BERT Sentence Embedding (Feng et al., 2022): LaBSE reaches 83.7% bilingual retrieval accuracy on Tatoeba across 112 languages, supporting the multilingual vector alignment route.
arXiv, Multilingual E5 Text Embeddings: A Technical Report (Wang et al., 2024): multilingual E5 first contrastively pretrains on billions of multilingual text pairs and then fine-tunes, supporting the trade-off between efficiency and quality when choosing a vector model.