Why Do Different AIs See Text Differently?

TL;DR

The same text can be “read” differently by three different LLMs. The reason lies in three technical layers: how the text is split into tokens, which vector space it then lives in, and which filter (top‑k) selects documents before the generative part even gets a chance to evaluate quality. If you write without accounting for this mechanics, an article may physically fail to reach the final generation in one system while working perfectly in another.

I’m writing an article, optimising the headline, inserting keywords where classical SEO taught me five years ago – and getting two opposite results. Yandex cites the material in the top three sources with a link right in the answer. Google AI Overview demonstratively ignores the same text, pulling up a competitor’s article that is worse written, shorter, and without a single practical example.

My first thought was a markup error or an indexing problem. I check: the page is indexed, load speed is fine, everything looks good. The problem is not with the site’s tech side. The problem is that the text has to pass through three completely different “perception filters”, and each AI system is built differently at every one of these levels.

What follows is what I’ve learned from dozens of such experiments: why tokenizers (algorithms that break words into pieces for the model) cut the same word differently, why embedding spaces (numeric representations of text meaning) are incompatible between models, and why top‑k is literally the grid through which your text either passes or doesn’t.

What “AI sees text” actually means

The phrase “AI sees text” sounds as if there’s a little gnome sitting inside the model who opens your article and forms an opinion. In reality, of course, it’s nothing like that.

For a language model, text does not exist as text. It exists as a sequence of numbers. The path from letters to numbers passes through three stations, and at each station the model makes decisions that you will never see in a chat interface or search result:

First, the text is split into tokens – the smallest pieces the model can work with directly. Then these tokens are turned into vectors (sets of numbers describing meaning) in an embedding space unique to each model. And only then, if we’re talking about search or RAG systems (generating an answer by loading documents found on the Internet), the document selection mechanism – top‑k (a filter that decides which materials actually reach the generative part) – kicks in.

Three stations – three points where a text can “break” or, conversely, pass the test perfectly. And at each station, Yandex, Google, OpenAI and Perplexity have different engineers with different solutions. Hence the difference in how they perceive the same text.

Let’s break down each station separately, starting with the very first and most underestimated – tokenization.

Tokenizers – different “alphabets” for the same text

Let me start with what surprised me personally when I first delved deeper into this topic instead of just thinking “well, there’s some tokenizer, it splits text into words”.

A tokenizer does not split text into words. It splits text into frequent substrings found statistically in the training data of a particular LLM. The algorithms are called BPE (Byte Pair Encoding) and SentencePiece (an open‑source library from Google for fast tokenization and detokenization) – these methods find the most frequent character combinations in a huge text corpus and record them as separate “meaning units” for the model.

Each company trains its own token vocabulary on its own data. Yandex focuses on Russian because its main traffic and training corpora are Russian‑language. Google and its Gemini train a multilingual vocabulary optimised for dozens of languages at once, where Russian is just one among many.

The result of such different approaches to vocabulary training is a different number of tokens for the same Russian word. For Yandex, a common Russian word can fit into one token. For Gemini, the same word can break down into three or four sub‑tokens (parts of the word, each counted as a separate token).

It would seem, what difference does it make to the model whether it sees the word “пылесос” (vacuum cleaner) as a whole or in pieces “пыле” + “сос”? The meaning is the same. But that’s the catch: before the model reaches the level of semantics (meaning), it already works with these tokens as basic units. If a word is fragmented into atypical pieces, there is more “noise” at the input, and the statistical connection between the parts of the word and the rest of the context becomes weaker.

Let’s look at a specific example. A user enters the query “робот‑пылесос для шерсти” (robot vacuum for pet hair). Yandex, with its emphasis on the Russian vocabulary, will most likely cut this phrase into three whole semantic blocks: “робот‑пылесос”, “для”, “шерсти”. Each block is a ready, recognisable unit.

Gemini with its multilingual vocabulary may go the other way and break the compound word into less obvious pieces: “роб”, “от”, “пыле”, “сос”. The model will, of course, reconstruct the meaning of the word “пылесос” at deeper layers – the transformer architecture (the type of neural network underlying most modern language models) is sufficiently robust to such fragmentation. But the process of meaning reconstruction itself requires more computational “effort” from the model, and in borderline cases – for instance, with typos, slang, or non‑standard spelling – the probability of losing connection increases.

What this means for your text

If an article says “робот пылесос” without a hyphen, or uses a colloquial abbreviation like “пылик”, for one tokenizer it will be a whole, recognisable unit, but for another it will be a set of scattered sub‑tokens that the model will have to glue back together, losing some accuracy in the process.

Natural, standard word forms without slang abbreviations and with correct punctuation (hyphens, quotation marks, capital letters where needed) pass through any tokenizer more cleanly. This doesn’t mean you have to write in bureaucratic language – lively speech also works fine as long as it is grammatically correct. Problems start where authors chase “trendy” spelling, merged compound words, or transliteration instead of Cyrillic.

It’s easy to check: take a key phrase from your article and see how public tokenizers from different companies break it down (most major models have open tools for visualising tokenization). If the phrase breaks into an abnormally high number of pieces compared to ordinary words in the same language, that’s a sign that you should rephrase the headline or key phrase into a more standard word form.

Embedding spaces – different geometry of meaning

Tokenization is only the first step. After the text is split into tokens, the model turns each token into a vector – a set of numbers that describes the meaning of that token relative to all the other tokens the model saw during training. This is called the embedding space (literally “space of embeddings”).

It’s important to understand one thing that sounds counter‑intuitive at first: each company – and often even within the same company – has its own separate embedding space. The model that searches for documents (the retriever) and the model that generates the final answer text (the generator) at Google, Yandex, and most other players are often different neural networks with different weights, trained on different data with different objective functions.

Different weights mean different meanings. Words that are close to each other in the vector space for one model may be far apart for another. It’s like two people who learned a foreign language from different textbooks: the basic words match, but the associative chains that come to mind first differ.

The puddle example that explains everything

Take the word “лужа” (puddle) in the context of an article about a robot vacuum that sometimes runs over a puddle left by a pet and smears it across the floor.

Model A was trained mainly on technical reviews, instructions, and specifications. In its embedding space, the word “лужа” in this context will be close to vectors for “cliff sensor” (the sensor that should prevent driving over obstacles and liquids), “virtual wall” (a software boundary the robot does not cross), and “dust bin capacity”.

Model B was trained on user forums, reviews, and social media discussions. For it, “лужа” will connect to a completely different cluster: “smeared dirt all over the apartment”, “unpleasant smell throughout the house”, “nightmare cleaning after that”. Zero technical terms, but maximum emotion and everyday experience.

Both models are right, each in its own space. Both describe the same real situation. But if your article about robot vacuums covers only the technical angle – sensor specs, suction power, container volume – it fits perfectly into Model A’s cluster and barely overlaps with Model B’s cluster. Conversely, a purely emotional review text without a single number resonates beautifully with Model B but looks semantically “empty” to Model A.

Why this is critical for content that needs to work everywhere

Before, when it was about two search engines with two ranking algorithms, the task was clear: learn what this particular algorithm likes and write for it. Now, dozens of models with different embedding spaces read the text simultaneously – and what resonates perfectly with one may seem incomplete to another.

An article that needs to work across different AI systems must cover all types of clusters at the same time. Dry technical specs – power, battery life, dust bin size, presence of a cliff sensor – and right next to them, real‑life scenarios with specific details: how the robot behaved when encountering a cat’s puddle, what happened to the carpet, how long it took to clean up the mess.

If an article covers only one of many clusters, it risks being classified by one of the models as “semantically incomplete” – not because the text is bad, but because in that particular model’s vector space, it simply doesn’t reach the zone of relevance.

Texts where the technical block and the user block are written in separate sections (not mixed in one paragraph, but clearly separated – e.g., a separate subsection with specs and a separate one with real user experience) pass through both types of embedding spaces noticeably more smoothly than texts where everything is blended into a single unstructured flow.

Top‑k and reranking – the coarse filter almost no one writes about

Now we get to the most practical part of the whole story, and it explains why the same article can shine in Yandex and completely disappear from Google AI Overview answers.

There are two fundamentally different approaches to how an AI system forms an answer to a user query.

The first approach – the generative model answers either directly from the data it was trained on (training data) or through its own built‑in retrieval layer (a mechanism for searching relevant documents), which for major players is integrated right with their search index. This is how Google AI Overview, Gemini and YandexGPT work in pure mode. Their search system is essentially an extension of the company’s regular search index, but with a generative layer on top.

The second approach is classic RAG (Retrieval‑Augmented Generation). Here, top‑k is literally an engineering parameter of a specific software pipeline.

How the RAG pipeline works

In a RAG system, the user query is first encoded into a vector via an embedding model. All documents in the database (or in the index the system has access to) are also encoded into vectors. Then the retriever (the search module) calculates the mathematical similarity between the query vector and the document vectors and selects the top‑k documents with the highest similarity score – literally “k documents whose score is higher than the rest”.

After this coarse selection, a reranker steps in – a separate model, most often a cross‑encoder (a type of model that processes the pair “query plus document” jointly rather than separately, and therefore gives a more accurate relevance assessment, albeit slower and more computationally expensive). The reranker reorders the candidates selected in the first coarse step and chooses the final, much narrower group of documents that actually make it into the generative part of the answer.

At Perplexity, according to descriptions of its architecture in open sources (not official technical specifications, which the company does not publish in full), the pipeline is multi‑stage. First, a broad selection through a combination of BM25 (a classic ranking algorithm based on word matching and frequency, without considering meaning) and semantic embedding search – the emphasis is on recall (coverage), i.e. the system tries not to miss potentially relevant documents, even if there are hundreds of them. Then cross‑encoder reranking for precision – a much narrower and higher‑quality selection. And the final layer is an ML reranker with additional signals like domain authority and publication freshness.

The exact k values at each layer are not publicly disclosed. Reconstructions and analyses by enthusiasts give different estimates, and these numbers should not be taken as official data.

An example that makes this whole mechanics clear

Imagine this scenario: you wrote a review “Top 5 robot vacuum cleaners with mopping for 2026”, where you tested each model, including how it handles puddles from pets. A user asks an AI: “Which vacuum doesn’t smear dog puddles?”

In Perplexity, the following happens. The retriever, using embeddings and BM25, roughly selects the top 100 articles that contain words about dogs, puddles and cleaning. Your article makes it into this broad hundred – because recall at this stage is high, the system tries to cover as much potentially relevant content as possible without much filtering for quality. Then the cross‑encoder reranker steps in. It reads your article paragraph by paragraph and sees: here is actually a test of the puddle‑avoidance function, there is concrete information, there are details specifically about this problem. The reranker pushes your article into the top 3. The result – Perplexity cites you with a link right in the answer.

In Google AI Overview or Yandex’s AlisaAI, the situation is different. They have closed internal retrieval layers built into the company’s overall search index. If your site has not passed their primary filter – for example, because of a weak link profile, UX issues, or lack of sufficient E‑E‑A‑T signals – the article physically never reaches the generative part of the system. The AI simply generates an answer based on competitors’ materials that passed their internal top‑k selection, even if your text would have been more useful to the user.

The main practical takeaway from this whole mechanics

Top‑k is about your content physically being unable to pass the very first, coarsest mathematical selection – selection by embedding similarity – before any model capable of evaluating the real quality of the writing even gets to it.

Hence the difference in behaviour of different LLMs towards the same text. Each system has its own retriever, its own set of k parameters at the input, its own signals for filtering. Material that easily passes one system’s filter may be rejected by another system’s filter – and not because of the text’s quality, but because of the technical parameters of that specific pipeline.

Optimising for AI search is no longer just a battle for “tasty”, well‑readable text for the final generation. It is also a battle to pass the first coarse mathematical filter. If your content or trust (the search engine’s level of trust in your domain) doesn’t break through that threshold, the text simply won’t be seen – no matter how useful it actually is.

Why the same article can “win” with one system and “lose” with another

If you put all three mechanics together – tokenization, embedding spaces and top‑k filters – it becomes clear why a universal recipe “how to write a text that AI will love” simply does not exist in principle. Every system at every one of the three levels makes its own decisions, and the cumulative effect of those decisions can either amplify the visibility of the text or completely extinguish it.

Let me analyse a specific case from my own practice to make the mechanics really clear. I was preparing material on how to properly reject candidates – a topic that seems simple at first glance, from the soft skills category. The text contained both practical phrasing (templates for polite rejection, what to write after the third interview, not just after a resume) and technical nuances (response deadlines according to labour law, the difference between rejecting by email and by phone).

In Perplexity, the material appeared in cited sources almost immediately after publication – the reranker, apparently, appreciated precisely the combination of concrete phrasing and practical cases, which a RAG system hunts for by its very nature: it needs documents from which it can actually assemble a detailed answer with ready‑made examples.

In Google AI Overview, the same material appeared in the results noticeably later, after weeks – even though the content hadn’t changed. A possible explanation (just possible, not a confirmed fact) is that the domain needed time to accumulate sufficient trust signals for the thematic cluster “HR and recruiting” before the internal retriever began consistently passing this site’s materials into its closed index’s top‑k.

In YandexGPT, the material didn’t appear in extended answers at all at first, even though the page was ranking decently in Yandex’s classic search. Here, most likely, the effect of tokenization and the embedding space of that particular model played a role: part of the terminology in the text (some HR terms like “оффер”, “фидбэк”, “reject‑письмо”) was written in a mixed English‑Russian variant, which could lead to more fragmented tokenization and, consequently, weaker vector connection to typical Russian‑language user queries.

The conclusion from this case is simple and not particularly pleasant for those looking for one universal checklist: different AI systems may require different amounts of time to “recognise” the same text, and the reasons for the delay lie at different levels – sometimes tokenization, sometimes domain trust, sometimes simply retriever parameters that you cannot directly control.

What can you control?

Since the three levels of mechanics – tokenization, embedding, top‑k – work differently in each system, it’s logical to ask: what then makes sense for an optimizer/author to do if you can’t directly control all three levels?

The answer – control the input data for all three levels simultaneously, even without knowing the exact internal parameters of each specific system. Let’s break down what this means in practice.

For the tokenization level

Use standard, grammatically correct word forms instead of trendy abbreviations and slang. Put hyphens in compound words where the rules of the language require them – “робот‑пылесос”, not “робот пылесос” written together or separated by a space without a hyphen. Avoid mixing Latin and Cyrillic in the same word without necessity – if a term can be written entirely in Russian, it’s better to do so, and add the Latin name in parentheses on first mention.

Write headings and subheadings as whole semantic phrases, without artificially breaking stable collocations. For example, if you’re writing about “умный дом” (smart home), don’t break this stable phrase with extra words between “умный” and “дом” in the heading – this increases the chance of anomalous tokenization for models whose vocabulary is not well acquainted with that specific context.

For the embedding spaces level

Cover several types of content simultaneously within a single article, even if the topic seems purely technical or purely emotional at first glance.

Use synonyms and different phrasings for the key concepts of the article, not the same phrase throughout the whole text. Different embedding spaces link synonyms with varying strength, and the more variations of semantic formulations you have in the text, the higher the chance of landing in the relevance zone of several models at once, not just one specific one.

For the top‑k and retriever level

This is where direct control is smallest, because the retrieval‑layer parameters of each company are closed and not fully published. But some things are still in the hands of the author and site owner:

• Concreteness and factual content in the text – rerankers, especially in RAG systems like Perplexity, based on observed behaviour, seem to prefer materials with verifiable details over general reasoning without facts.
• A text structure with clear semantic blocks – makes the work easier both for the coarse retriever at the embedding comparison stage and for the precise reranker at the cross‑encoder analysis stage.
• E‑E‑A‑T signals at the domain level – link profile, authorship with real expertise, history of publications on the topic – these are things that accumulate over months and years, not something you can fix in one day of editing.
• Freshness of publication and regular updates – some retrievers, based on observations, seem to take the last page update date as a relevance signal, especially for topics where timeliness changes quickly.

None of these points guarantees passing the top‑k filter of a specific system – there are too many hidden variables. But the combination of these factors noticeably increases the probability of landing in the narrow circle of documents that actually reach generation.

Common mistakes when trying to “please all AIs at once”

Let’s look at a few misconceptions I regularly encounter among authors trying to optimise text for artificial intelligence in general, without separating out specific systems.

The first misconception – the idea that it’s enough to stuff the text with keywords in different variations, and that will automatically cover all embedding spaces at once. In practice, excessive repetition of the same phrase without changing the meaning of the context around it does not increase coverage of different vector spaces – it just makes the text less natural, and some modern rerankers, judging by observed behaviour, penalise for artificial keyword density because it is recognised as a signal of low‑quality content rather than a relevance signal.

The second misconception – the conviction that if an article ranks well in classic search (the usual top ten blue links), it will automatically pass through the top‑k filters of generative AI systems. Classic ranking and the retrieval layer of a generative system are often different mechanisms even within the same company, with different weights and different signals. A good position in classic search increases the chances, but does not guarantee passing through a separate AI‑oriented retriever.

The third misconception – ignoring the fact that embedding models change over time. Companies periodically retrain or replace their search and generation models. A text that passed through filters perfectly six months ago may suddenly lose visibility not because of changes in the text itself, but because the model through which the text is now being evaluated has changed. This is not a reason to panic at every fluctuation in visibility, but it is a reason to periodically re‑check old materials rather than consider a once‑written text as forever optimised.

The fourth misconception – trying to write a text “equally neutrally” for all clusters in embedding space, avoiding both purely technical and purely emotional language, hoping to land somewhere “in the middle”. Based on observations, such an averaged approach often loses to both poles at once – the text ends up being insufficiently technical for one cluster of models and insufficiently lively for another. It is much more effective to explicitly cover both poles with separate blocks within a single article than to blur them into one neutral stream.

What does all this mean?

The mechanics described above imply, among other things, that a search optimisation specialist now has to understand not only link profiles and keyword density, but also the architecture of retrievers, the logic of tokenization, the principles of how embedding spaces work – that very layer which three years ago seemed exclusively the internal kitchen of ML engineers. This layer is no longer foreign; it has become a working context, and ignoring it means voluntarily taking yourself out of the process.

To be honest, the amount of information one needs to keep in mind has long crossed into the exhausting category. Forty studies a day just to keep up with the current version of someone’s pipeline; simultaneously monitoring how reranker parameters change, how the geometry of vector indices is being rebuilt, how another assistant is chewing through and reassembling the search results. There is objectively not enough energy for this, and the feeling that the feed will not end but only accelerate does not dull over time – quite the opposite.

At the same time, the alternative of “not reading and optimising the old way” doesn’t exactly leave room for manoeuvre. A text built according to five‑year‑old logic simply does not pass through the filters of a generative overview, does not surface in a RAG answer, does not withstand the test of factual density. A specialist who continues to work as if nothing has changed does not get a soft transition or a second chance – they gradually fall out of queries, out of search results, out of the profession.

Returning to the question that started this entire analysis – why different AIs see text differently – the answer consists of three layers of decisions made by engineers at different companies independently of each other: how to split text into tokens, in which space to consider meaning close or distant, and which coarse filter to apply before the model even gets a chance to evaluate the quality of what’s written. Three layers, three different sets of rules – and the same text at the output of this system can be both a cited source and a material invisible to a particular user, depending on which AI they asked.