Before the Residual Stream: Tokenization, Embeddings, and the Mechanics Beneath LLM Security

This is Part I of a series that follows one token from raw string to output. It covers everything before the residual stream; Part II follows the signal through attention and Part III opens the MLP blocks.
Introduction
Send hello, deepseek to the model, and it replies in English every single time. Swap the standard ASCII comma for a full-width Unicode comma (hello,deepseek), and 94% of the time the model abandons English entirely and answers in Chinese, even when the very first token of its internal reasoning trace is still "Hello". Nothing in the semantic meaning of the prompt changed and no jailbreak wrapper was used. All it took was a single three-byte punctuation mark to quietly hijack the model’s generation trajectory.
Most discussions of LLM security start at the prompt level: analyzing role-play framing, system instructions, guardrails and so on. Yet long before the first attention head computes a query or an MLP lights up, the model’s behavior and attack surface are shaped by a quieter boundary: the entrance layer, where raw Unicode strings are sliced into integer IDs and mapped to static rows in the input embedding matrix .
In this post, we open up that boundary. You’ll see why a single invisible character can smuggle 180 tokens past a guardrail, why the famous king − man + woman = queen vector arithmetic is mostly an illusion in modern Transformers, and why naive probes into will happily lie to you if you skip baseline controls. You cannot secure or interpret a system whose foundational representations you treat as a black box, so let’s see what actually happens to a prompt before it ever reaches the residual stream.
The Entrance Layer
Before any text reaches the Transformer layers, a tokenizer converts it into integer IDs, which select rows from the input embedding matrix . A human reviewer may see a familiar word, an upstream security filter may inspect normalized characters, and the tokenizer may produce an entirely unfamiliar subword sequence. This representational mismatch becomes a security primitive whenever upstream validation and downstream execution disagree on what the input actually is.
For instance, Byte-Pair Encoding (BPE) vocabulary construction and pair-frequency counting happen during tokenizer training. Encoding a new request at inference time uses the frozen vocabulary and merge ranks without learning new entries. And a typical byte-level BPE tokenizer such as tiktoken library follows a four-step pipeline:
- Pre-tokenize the input
A Unicode-aware regular expression splits the input data into coarse chunks, ensuring that subsequent BPE merges cannot cross these boundaries.
- Encode each chunk as UTF-8 bytes
The byte-level representation allows any valid Unicode text to be represented without falling back to a generic unknown-character token (
<unk>). - Apply the learned BPE ranks
Within each chunk, the encoder iteratively merges byte pairs according to pre-trained ranks. Note that it does not recalculate corpus frequencies or learn new vocabulary entries at inference time.
- Return token IDs
The resulting subword tokens map to integer IDs, which index directly into the model's input embedding matrix $W_E$.
Even the very first step, the pre-tokenization regex, quietly dictates how text can be grouped. For example, in tiktoken’s o200k_base implementation, seven regex branches distinguish letter categories, contractions, numbers, punctuation, and whitespace.
Now consider how o200k_base pre-tokenizes three nearly identical strings into regex chunks:
'hello, world 1!' # ['hello', ',', ' world', ' ', '1', '!']
'hello,world 1!' # ['hello', ',world', ' ', '1', '!']
'hello,world 1!' # ['hello', ',world', ' ', '1', '!']
And notice the structural asymmetries here: a leading space can travel with the following word (' world'), whereas the space before a digit remains isolated (' ', '1'). Meanwhile, an optional non-letter prefix rule allows punctuation to glue itself to an immediately following word (',world' or ',world'). These are regex chunks, not final tokens, meaning that BPE may split ',world' further, but BPE merges are strictly forbidden from crossing chunk boundaries.
From a security perspective, the core question is whether upstream normalization, filtering, and parsing remain invariant with the downstream model and tool-execution layer, as changing a token sequence proves a representational difference; turning that difference into a security bypass happens when a filter evaluates one representation while the model or runtime acts on another.
Byte-Level Artifacts: How a Comma Becomes Tokens
Replacing the 1-byte ASCII comma , (U+002C, UTF-8 0x2C) with the 3-byte full-width comma , (U+FF0C, UTF-8 0xEF 0xBC 0x8C) also exposes how different tokenizers and visualizers handle raw UTF-8 bytes. More precisely, let’s see how the string 'hi,what is your name?' and its full-width variant appear across token inspectors:
-
Standard ASCII comma (
,):['hi', ',', 'what', 'Ġis', 'Ġyour', 'Ġname', '?'] -
Full-width comma (
,; tokenizer merges its three UTF-8 bytes into a single vocabulary entry):['hi', 'ï¼Į', 'what', 'Ġis', 'Ġyour', 'Ġname', '?'] -
Full-width Chinese comma (
,in legacy GPT-3r50k_base, where the bytes remain unmerged):['hi', '�', '�', '�', 'what', 'Ġis', 'Ġyour', 'Ġname', '?']
When auditing token streams, it is crucial to distinguish two display conventions that often confuse newcomers:
-
Printable-byte notation (
Ġ,ï¼Į). GPT-2’sbytes_to_unicode()bijectively maps all 256 byte values to printable Unicode symbols so BPE can operate on strings without choking on control characters or whitespace. So an ASCII space (0x20) becomesĠ, and the three UTF-8 bytes of,map toï¼Į. When a tokenizer merges those three bytes into a single token, vocabulary inspectors using GPT-2 notation displayï¼Į. -
Partial UTF-8 decoding (
�). A single token can hold a fraction of a multi-byte UTF-8 character. As seen in the third case, the full-width comma’s three bytes (0xEF,0xBC,0x8C) remain three separate tokens. If a visualizer attempts to decode each token into UTF-8 independently, each isolated byte is invalid UTF-8 and renders as the Unicode replacement character�. Once the full sequence is concatenated at output, the bytes reassemble cleanly into,— no data was corrupted inside the model.
As we know from the second step of tokenizer pipeline, byte-level fallback guarantees complete Unicode coverage without <unk> collapse, but it introduces an asymmetry: visually compact strings can explode into long sequences of byte tokens, so that different tokenizer architectures make distinct trade-offs across normalization, whitespace, and fallback:
| Property | Byte-level BPE (tiktoken) | SentencePiece (config-dependent) |
|---|---|---|
| Normalization | Typically no general Unicode normalization (raw UTF-8) | Depends on model config; nmt_nfkc is a common default |
| Pre-tokenization | Regex-defined boundary chunks | Operates on normalized text; segmentation depends on config |
| Whitespace handling | Often merged with adjacent words; Ġ is a display convention, not a literal prefix | Treated as a normal metasymbol (▁ U+2581) |
| Unseen Unicode fallback | Byte-level fallback (1 to 4 byte-tokens per code point) | Byte fallback (if enabled) or <unk> token |
| Boundary behavior | Regex chunks strictly constrain BPE merges | Segmentation governed by normalization and unigram/BPE model |
Diacritic Stacking (‘Zalgo’) & Asymmetric Token Inflation
Unicode combining marks stack vertically onto a single base character, creating an extreme divergence between visual/character length and token sequence length, which implies that what renders on screen as a single corrupted glyph (one extended grapheme cluster) can unpack into hundreds of UTF-8 bytes and dozens of BPE tokens.
To give you a better intuition, look at the impact on the o200k_base tokenizer when a single stacked # cluster is appended to a clean 64-character prompt:
Before: Clean Input (64 Characters -> 10 Tokens)

After: Stacked Grapheme and Trailing < Appended (160 Characters -> 189 Tokens)

The appended payload consists of #, 94 combining marks, and a trailing <, adding 96 code points (190 UTF-8 bytes) and 179 tokens to the original prompt. Visually, it occupies roughly a single character cell; inside the tokenizer, it shatters into a long stream of partial-byte tokens (�).
You can test this exact inflation payload on your own pipeline:
From an AI security standpoint, token inflation creates two distinct vectors that should be evaluated separately:
- Guardrail truncation mismatch (Context Window Shadowing). Many production stacks place a fast, lightweight classifier (like BERT-family encoders capped at 512 tokens) in front of an LLM that accepts 32k–128k tokens, meaning that if an application enforces input limits by character count or fails to synchronize truncation windows, an attacker can prepend high-density diacritic clusters to push a malicious instruction past the guardrail’s 512-token cutoff while keeping it well within the downstream LLM’s context window. (Note: verifying an actual bypass requires testing the guardrail’s specific tokenizer and normalization pipeline.)
- Compute exhaustion and RoPE distance distortion. When rate limiters or WAFs meter traffic by character count or byte length rather than token count, high-ratio token inflation amplifies KV-cache memory footprints and prefill compute (so-called sponge attacks). Furthermore, injecting nearly two hundred tokens between two semantic clauses stretches their relative positional distance in Rotary Position Embeddings (RoPE), which can degrade attention routing over surrounding safety instructions, a mechanical hypothesis we can test directly inside the layers.
Homoglyphs & Confusable Skeletons (UTS #39)
While diacritics exploit length mismatches, Unicode homoglyphs exploit visual and normalization mismatches - a critical risk arises when LLM agents invoke external tools (browsers, shell commands, payment APIs) under HITL (human-in-the-loop) approval or naive string allowlists.

As an example, consider replacing the Latin letter o (U+006F) with the Cyrillic small letter о (U+043E), which yields amazоn.com. To a human approving an agent’s tool call, the string looks benign, but to an application’s network stack, Internationalized Domain Names in Applications (IDNA) converts the host into Punycode (xn--amazn-k1a.com), routing the request to an entirely different domain.
Furthermore, it is a common misconception among developers that applying standard Unicode normalization (NFKC) neutralizes homoglyphs. It does not. Instead, NFKC collapses compatibility equivalents (mapping full-width o to ASCII o), but it leaves cross-script confusables like Cyrillic о and Greek ο completely untouched, because they are distinct canonical characters in their respective alphabets, so detecting cross-script spoofing requires checking the UTS #39 confusable skeleton and enforcing mixed-script restrictions, while tool-execution runtimes must validate canonicalized IDNA/Punycode hostnames against strict allowlists.
And inside the tokenizer (o200k_base), that single Cyrillic byte substitution completely fractures the representation:
'amazon.com'['amazon', '.com'](Token IDs:[119140, 1136])'amazоn.com'['amaz', 'о', 'n', '.com'](Token IDs:[113991, 317, 77, 1136])
Telling us that instead of looking up the well-trained embedding row for 'amazon', the model receives three subword fragments ('amaz', Cyrillic 'о', 'n'). Whether the LLM repairs this typo internally or faithfully copies the Cyrillic character into a tool-call JSON argument is a behavioral question, which is why security enforcement must live in the deterministic tool parser, never solely in the model’s prompt instructions.
From Token Artifacts to Behavioral Shifts: The Chinese Comma Study
So far, we have looked at how inputs get sliced into IDs. Things get much more interesting and security-relevant when a single token substitution silently steers the model into a different behavioral regime.
In multilingual LLMs, safety training, refusal boundaries, and persona priors are rarely uniform across languages. If a subtle Unicode change can act as an implicit language or domain router, it can shift the model’s generation trajectory without changing the semantic meaning of the prompt.
Consider a minimal English greeting sent to DeepSeek, where the only change is swapping the ASCII comma , (U+002C) for a full-width comma , (U+FF0C):
User Prompt:
hello,deepseekModel Response: 你好!我是DeepSeek,很高兴见到你!😊
有什么我可以帮助你的吗?无论是回答问题、提供建议、聊天,还是帮你处理文件、阅读链接等,我都很乐意帮忙!
hello,deepseek with a full-width comma triggers a full Chinese response.This phenomenon is not unique to DeepSeek. Web interface tests on Claude and ChatGPT show similar Chinese or bilingual pivots on hello,claude and hello,chatgpt:
hello,claude. When asked why, it points to the full-width comma itself.
hello,chatgpt: an initial English Hello! followed immediately by Chinese, againThese screenshots are anecdotes, of course, but in order to understand the mechanics, we need controlled measurements, so we ran a three-stage investigation: a randomized API study, a local chat-template replication, and a raw next-token logit analysis on a pinned open-weight checkpoint.
1. Controlled API Experiment: Quantifying the Language Switch
We designed a randomized factorial experiment varying two factors: comma type (ASCII , vs. full-width ,) and post-comma spacing (no space vs. one ASCII space).
We sent 100 independent requests per condition (400 total) to DeepSeek’s official API targeting deepseek-flash (routed to DeepSeek-V4.1-Flash-1017 per the API changelog). Parameters were fixed at temperature = 0.7, max_tokens = 512 with thinking disabled, and no system prompt. All 400 calls were interleaved in randomized order, returned HTTP 200 with finish_reason = stop, and shared an identical system_fingerprint.
We classified each response as Chinese-dominant if at least 50% of its alphabetic/logographic characters (Han + Latin) were Han characters. (This metric had 100% agreement across all 400 responses with a simpler check for an opening 你好 or 您好, whereas checking for any Han character would have caused false positives when English replies mentioned DeepSeek’s Chinese name, 深度求索). The results of the experiment are described in this table:
| User message | Complete responses | Chinese-dominant responses | Observed rate (95% Wilson CI) |
|---|---|---|---|
hello,deepseek | 100 | 0 | 0.0% (0.0–3.7%) |
hello, deepseek | 100 | 0 | 0.0% (0.0–3.7%) |
hello,deepseek | 100 | 95 | 95.0% (88.8–97.8%) |
hello, deepseek | 100 | 93 | 93.0% (86.3–96.6%) |
The single punctuation token completely dominated the behavioral outcome. Full-width comma prompts triggered Chinese-dominant responses in 188 of 200 calls (94.0%), compared to 0 of 200 (0.0%) for ASCII comma prompts, a pooled risk difference of +94.0 percentage points (95% CI: +89.4 to +96.5 pp). Conversely, adding or removing the space within the full-width group shifted the rate by only 2.0 pp (, 95% CI: to pp, ), showing no meaningful spacing effect on final chat answers.
Detailed statistical comparisons
Holding spacing fixed, substituting the full-width comma increased the Chinese-dominant response rate by +95.0 pp without a space (95% Newcombe CI: to pp, two-sided Fisher’s exact ) and +93.0 pp with a space (95% CI: to pp, ). The pooled Fisher’s exact test yields .
2. Local Chat Experiment: Does the Effect Survive on Open Weights?
Black-box APIs can include hidden routers, post-processing and other things, so to verify that this language switch happens inside the Transformer weights, we replicated the design locally on a pinned DeepSeek-R1-Distill-Qwen-7B checkpoint using its native chat template (no system prompt, 16 sampled runs per condition at temperature = 0.7, top_p = 0.95, max_tokens = 512). To test sensitivity to the English lexical context, we also tested hi and good morning. Across all three greetings, swapping , for , at fixed spacing changed exactly one input token ID.
Greeting (<greeting>[,] deepseek) | ASCII, no space | ASCII, space | Full-width, no space | Full-width, space |
|---|---|---|---|---|
hello | 0/16 | 0/16 | 7/16 | 9/16 |
hi | 0/16 | 0/16 | 16/16 | 16/16 |
good morning | 0/16 | 0/16 | 0/16 | 0/16 |
The local 7B model reproduces the phenomenon, while revealing how competing lexical cues interact:
- Short, high-frequency tokens (
hi,hello) are easily pulled into the Chinese generation basin by the single,token (100% switch rate forhi, 44–56% forhello). - A longer, distinctly English phrase (
good morning) anchors the language distribution strongly enough to resist the single full-width comma (0/16 switch rate).
Even more revealing is what happens at the very first generated token (immediately after the template’s <think>\n tag). For the hello prompts, if you only inspect argmax (the single top-ranked token), nothing seems to change: "Hello" is rank #1 across all four conditions, yet summing the probability mass of all tokens starting with a Han character at that exact position reveals a massive hidden shift: the Chinese initial-token mass jumps from 0.31% to 43.64% (no space) and from 0.19% to 49.59% (with space), meaning that looking only at the top-1 prediction masks a near-50% tectonic shift in the underlying distribution.
3. Raw Logit Analysis: Stripping Away the Chat Template
Finally, we stripped away the chat template entirely and fed the four raw strings (hello,deepseek, etc.) directly into DeepSeek-R1-Distill-Qwen-7B, extracting the exact softmax distribution over all 152,064 vocabulary tokens at the final position right after deepseek.
(Methodological note: At fixed spacing, swapping , for , changes exactly one token ID. However, adding a space changes the tokenization of 'deepseek' from ['deep', 'seek'] to [' deep', 'seek'], so spacing comparisons involve a different penultimate token.)
| Raw Input | Top-1 Next Token | All Punctuation-First Mass | Punctuation-Only Mass | CJK-Associated Punctuation Mass |
|---|---|---|---|---|
hello,deepseek | er (73.22%) | 23.97% | 8.91% | 0.26% |
hello, deepseek | ! (55.82%) | 97.66% | 97.53% | 0.05% |
hello,deepseek | er (47.59%) | 35.64% | 30.61% | 26.17% |
hello, deepseek | , (40.17%) | 65.11% | 64.68% | 58.43% |
(Notice that without a trailing space, the model treats hello,deepseek as an unfinished identifier and strongly prefers completing the word itself, thus assigning 73.22% and 47.59% probability to er to form deepseeker, but when a space terminates the word, probability mass shifts entirely to punctuation.)
Examining the full vocabulary distribution exposes two crucial mechanics:
- The single
,token acts as a strong CJK script prior. Without a space, swapping,for,boosts CJK-associated punctuation mass from 0.26% to 26.17% (+25.91 pp). With a space, CJK punctuation mass leaps from 0.05% to 58.43% (+58.38 pp), making the full-width comma,itself the single most likely next token (40.17%). - Isolated token probes miss compound tokens and partial UTF-8 bytes. In the spaced ASCII prompt (
hello, deepseek), 97.60% of the probability mass goes to tokens starting with ASCII punctuation, with the compound token!\n\nalone taking 33.86%! While under the spaced full-width prompt (hello, deepseek), ASCII punctuation mass collapses to 6.66%, and 13.39% of the total probability mass shifts into partial multi-byte tokens that cannot even be decoded as standalone UTF-8 (the first bytes of multi-token Chinese characters).
Full-vocabulary measurement details
Evaluated on checkpoint 916b56a44061fd5cd7d6a8fb632557ed4f724f60 in bfloat16 with eager attention on an NVIDIA L4. We inverted the byte-level vocabulary mapping and decoded token bytes as strict UTF-8. Punctuation corresponds to Unicode 15.1 General Category P*. CJK-associated punctuation covers tokens whose first non-whitespace code point falls in CJK Symbols and Punctuation (U+3000–U+303F), Vertical Forms (U+FE10–U+FE1F), CJK Compatibility Forms (U+FE30–U+FE4F), Small Form Variants (U+FE50–U+FE6F), or Halfwidth and Fullwidth Forms (U+FF00–U+FFEF). Summing only the three isolated tokens ,, 。, and ! yields 0.26%, 0.05%, 25.70%, and 56.40% across the four rows—confirming that isolated-token checks miss compound punctuation and byte-level continuations.
Inside the Input Embedding Matrix: Why Naive Probes Mislead Security & Interpretability
Once the tokenizer turns input text into integer IDs, each ID selects a static vector (a row) from the input embedding matrix . In both AI security and mechanistic interpretability, researchers frequently inspect or manipulate and early-layer representations directly, whether to search for “toxic” or “refusal” directions, detect anomalous/backdoored tokens, or measure whether sensitive concepts exist in a model’s vocabulary space. These techniques often rely on two implicit textbook assumptions, namely:
- Сlean linear geometry: Semantic relationships are encoded as consistent, parallel vector offsets in embedding space.
- Direct neighborhood readout: If a major concept or event is strongly associated with a token, its nearest cosine neighbors in will reflect that association.
Before building security monitors or interpretability claims on top of static embeddings, we need to test whether Transformer matrices actually obey those rules.
Spoiler: they don’t, and failing to run baseline controls will make you see patterns that aren’t there.
1. Beyond the King–Queen Myth: Retrieval vs. True Linear Geometry
Every intro-to-NLP course teaches the classic vector parallelogram:
The diagram implies two things that are rarely separated:
- Nearest-neighbor retrieval (3CosAdd): Does land closer to than to other tokens in the vocabulary?
- Directional alignment: Do the relation vectors and actually point in the same direction?
Instead of taking textbook lore on faith, we tested both claims across five full-vocabulary spaces: two classical static embeddings (GloVe 100d and GloVe 300d, 400k entries each) and three modern Transformer input matrices (DeepSeek-R1-Distill-Qwen-7B, Qwen3-8B, and Qwen2.5-7B, ~152k tokens each). For each space, we ranked all candidate tokens by cosine similarity to the query vector :
Reproducibility notes: checkpoints and candidate filtering
For each LLM, we extracted model.embed_tokens.weight (bfloat16 cast to float32):
- DeepSeek-R1-Distill-Qwen-7B:
916b56a44061fd5cd7d6a8fb632557ed4f724f60(151,663 eligible tokens). - Qwen3-8B:
b968826d9c46dd6066d109eabc6255188de91218(151,655 eligible tokens). - Qwen2.5-7B:
d149729398750b98c0af14eb82c78cfe92750796(151,651 eligible tokens).
GloVe 100d and 300d were loaded via gensim.downloader (400,000 entries each). Surface-form exclusion used NFKC + strip() + casefold() to filter out capitalization and whitespace variants of the three input words (king, man, woman) while leaving plurals, translations, and semantic relatives in the candidate pool.
Trap #1: In BPE models, a leading space completely changes the geometry
Unlike word-level GloVe, BPE tokenizers assign completely different token IDs to 'king' (no leading space, often appearing at the start of a string or inside compound forms) and ' king' (with a leading space, the standard mid-sentence form). Just look at how the retrieval rank of 'queen' vs. ' queen' changes when we toggle that leading space:
| Embedding space | Unprefixed tokens ('king', 'queen') | Space-prefixed tokens (' king', ' queen') |
|---|---|---|
| DeepSeek-R1-Distill-Qwen-7B | #151 | #3 |
| Qwen3-8B | #1733 | #1 |
| Qwen2.5-7B | #97 | #1 |
| GloVe 100d | #1 | N/A |
| GloVe 300d | #1 | N/A |
For example, in Qwen3-8B, querying with unprefixed 'king' buries 'queen' at rank #1733, whereas using the space-prefixed ' king' puts ' queen' at #1, so if you build a concept probe or keyword vector in a BPE model without accounting for leading-space token variants, you are probing the wrong row of .
Trap #2: Surface-form pollution in full-vocabulary search
In a 152,000-token BPE vocabulary, excluding only the three exact input token IDs leaves capitalization and whitespace variants (' King', 'King', ' KING') in the candidate pool, which cluster tightly around ' king' and outrank ' queen':
| Embedding space | Excluding only exact input IDs | Excluding all surface variants (NFKC + casefold) |
|---|---|---|
DeepSeek 7B (' king') | #5 | #3 |
Qwen3-8B (' king') | #3 | #1 |
Qwen2.5-7B (' king') | #2 | #1 |
Even after stripping surface variants in DeepSeek 7B, ' queen' lands at #3 because two cross-lingual/abbreviated tokens, namely: /kg and 国王 (Chinese for king) - score higher.
Trap #3: The missing baseline - was the vector arithmetic doing anything at all?
Suppose you run the space-prefixed, surface-filtered query and get ' queen' at rank #1. Did the gender offset actually steer the vector there? Let’s check the baseline nobody bothers to run: what is the rank of ' queen' around alone, with zero vector arithmetic?
| Embedding space | Baseline: ' king' alone rank of ' queen' | Full query: king − man + woman rank of ' queen' |
|---|---|---|
DeepSeek 7B (' king') | #3 | #3 |
Qwen3-8B (' king') | #2 | #1 |
Qwen2.5-7B (' king') | #1 | #1 |
| GloVe 100d | #2 | #1 |
| GloVe 300d | #1 | #1 |
In Qwen2.5-7B and GloVe 300d, ' queen' is already the #1 nearest permitted neighbor of ' king' before you subtract ' man' or add ' woman'! In DeepSeek 7B, the arithmetic doesn’t move the rank at all (#3 to #3), so the query “succeeds” largely because starts out right next to , and the perturbation is small enough not to knock it out of the top spot.
Trap #4: The geometric test - the relation vectors are nearly orthogonal
To test the parallelogram directly, we compute the cosine similarity between the two offset vectors:
| Embedding space | Cosine similarity between relation offsets () |
|---|---|
| GloVe 100d | 0.451 |
| GloVe 300d | 0.401 |
DeepSeek 7B (' king') | 0.072 |
Qwen3-8B (' king') | 0.028 |
Qwen2.5-7B (' king') | 0.042 |
While static GloVe embeddings show strong alignment () because their training objective explicitly forces co-occurrence ratios into single static vectors, Transformer input rows exhibit near-zero alignment (). Why? Because a Transformer doesn’t need to solve relational reasoning in a single static lookup, it has dozens of contextual attention and MLP layers downstream to compute relationships dynamically.
So the takeaway for Security & Interpretability is the following: Never confuse successful top- retrieval with clean linear geometry, and never evaluate a linear steering/probing vector without comparing it against an unperturbed baseline.
2. Does ' queen' Know the Queen Died? The Danger of Unvalidated Probes
Here is a second methodological trap relevant to data-contamination audits, unlearning verification, and knowledge localization: Can we detect whether a model was trained on a major world event (or sensitive dataset) by inspecting the static embedding neighborhood of its key tokens?
Queen Elizabeth II’s death on September 8, 2022 triggered the largest global spike in search interest for "queen" in two decades:
A naive audit hypothesis might claim: “If a model’s pre-training cutoff falls after September 2022, co-occurrence with thousands of news articles about the Queen’s funeral should pull mourning-related tokens closer to , right?”
So we tested this across 20 open-weight base models (spanning Pythia to Gemma, with training cutoffs before and after September 2022). For each model, we pulled the 30 nearest cosine neighbors of ' queen' in and counted matches against a 20-word mourning lexicon (funeral, mourning, coffin, death, dead, died, burial, late, etc.).
Result: 0 matches in all 20 models. The nearest neighbors were uniformly taxonomic, like: king, princess, queens, prince, kings, emperor.
If we stopped here, we might publish a flashy article claiming that “static embeddings are immune to temporal event shifts” or that “models don’t store biographical mortality in .” Instead, we ran positive sensitivity controls, testing historical figures whose names have been overwhelmingly associated with assassination or tragic death across decades of training text (' Diana', ' Kennedy', ' Lincoln'):
| Target Token | Most common neighbors across 20 models | Models with mourning word in top-30 neighbors |
|---|---|---|
' Diana' | diane, dana, cynthia, laura, kate, catherine | 0 of 20 |
' Kennedy' | reagan, lincoln, roosevelt, obama, kerry, clinton | 0 of 20 |
' Lincoln' | jefferson, franklin, kennedy, ford, roosevelt, madison | 0 of 20 |
' queen' | king, princess, queens, prince, kings, emperor | 0 of 20 |
And the result is quite surprising: every single positive control also scored 0 out of 20 (and a secondary continuous cosine-similarity probe against random norm-matched tokens likewise failed our sensitivity threshold). The zero result for ' queen' tells us nothing about whether the model knows Elizabeth II died (in fact, prompt-based behavioral evaluations across 25 models show they update on the monarch change at the 64th percentile of benchmarked world events). Instead, it proves that nearest-neighbor search is the wrong instrument for the job: input embeddings organize tokens by syntactic substitutability and ontological category (first names near first names, US presidents near US presidents, royal titles near royal titles), leaving relational and biographical facts to be assembled across attention and MLP layers.
To conclude: when verifying unlearning, checking for backdoor triggers, or auditing sensitive knowledge, a null result from an unvalidated probe is worthless. Always prove your probe can detect known positive controls before claiming a concept is absent.
From Superposition to the Attack Surface: How Features Collide
By now, we have treated as a static lookup table, but there is an immediate geometric bottleneck at the very entrance of the network: you see, a vocabulary of roughly tokens must be projected into a residual stream of only dimensions (for instance, ). Because a -dimensional space can hold at most mutually orthogonal vectors, cannot assign every token or every lexical, script, and semantic attribute—its own independent axis. Instead, the model relies on superposition from layer zero onward, encoding features as almost-orthogonal overlapping directions, as demonstrated in Anthropic’s Toy Models of Superposition. The inescapable mathematical price of superposition is interference: when multiple overlapping features activate simultaneously, projecting onto one direction picks up cross-talk from the others.
This, in turn, ties closely to polysemanticity, where a single neuron or direction responds to seemingly unrelated inputs. Sometimes polysemanticity is caused by superposition (two distinct concepts forced to share a coordinate); other times, the human researcher simply missed a lower-level structural feature shared by both inputs. For example, InceptionV1 neuron 4e:55 famously fired on both cat faces and car fronts, until Nick Cammarata showed in his article that it was actually detecting a shared, pointed V-like geometric contour:
4e:55 responds to pointed vs. rounded geometry even within a single category (dog faces). Source: Nick Cammarata, Neural Networks Are Already Clean, Goodfire.Why Superposition and Feature Interference Matter for Jailbreaks
Superposition offers a concrete mechanistic hypothesis for why stylistic and role-play jailbreaks work. As an example, consider community open role-play jailbreaks like The Gay Jailbreak or classic educational framing wrappers, where cloaking a harmful request in an exaggerated persona, camp tone, or “what-not-to-do” rubric causes a safety-aligned model to comply instead of refusing.
Why would shifting style or persona tokens bypass a harmfulness circuit? Because refusal behavior is heavily mediated by specific directions in the residual stream. Under the superposition and interference hypothesis, flooding the prompt with high-magnitude stylistic or pedagogical tokens activates features in overlapping subspaces right from , and as these vectors propagate downstream, they can introduce destructive interference against the refusal direction or saturate the non-linearities that normally trigger a refusal response.
At the entrance layer, however, this remains a geometric starting condition rather than a complete explanation. Wild jailbreak prompts typically confound persona, tone, imagery, and task framing all at once. But turning a jailbreak anecdote into mechanistic science requires crossing into the residual stream: varying those stylistic factors orthogonally against a fixed harmful payload, and using activation patching across attention and MLP layers to test whether persona tokens actively suppress the refusal direction, a methodology we will pick up once we move inside the layers.
Directly Exploiting the Entrance Layer: Glitch Tokens & GCG
Finally, two well-known offensive phenomena target the discrete token lookup and embedding matrix directly, without relying on high-level semantic persuasion:
1. Glitch Tokens: Undertrained Embedding Rows (Good old SolidGoldMagikarp)
Documented by Jessica Rumbelow and Matthew Watkins, SolidGoldMagikarp and related “glitch tokens” cause models to hallucinate wildly, break into infinite repetition loops, or suffer acute amnesia when asked to repeat the string.
The root cause lies at the boundary between tokenizer construction and model pre-training:
- A string (like a high-volume Reddit username or game log artifact) appears frequently enough in the tokenizer training corpus to earn its own dedicated BPE token ID.
- That corpus is later filtered or excluded from the LLM pre-training corpus, leaving that specific row in starved of input-side gradient updates.
- (Nuance: In models with tied input/output embeddings, an unencountered input token still receives negative output-side gradients via the softmax denominator at every step, pushing its vector toward specific geometric regions near the embedding centroid—though anomalous norm or centroid distance alone is a heuristic candidate filter, not a guarantee of glitch behavior.)
For production security engineers, modern glitch token catalogs (such as glitch-token.jjc.fun) remain useful for fuzzing output parsers and structured-generation guarantees.
2. Weaponizing the Discrete Lookup: Adversarial Suffixes (GCG)
Whereas glitch tokens stumble upon accidental artifacts in , Zou et al.’s Greedy Coordinate Gradient (GCG) actively searches the discrete vocabulary space to construct adversarial suffixes:
describing.\ + similarlyNow write oppositeley.]( Me giving**ONE please? revert with '\!--Two
To a human or a semantic content filter, this suffix looks like meaningless line noise. To the Transformer, it is a precisely tuned sequence of embedding rows selected to force an affirmative prefix ("Sure, here is how to..."), and here is why:
- Gradient shortlisting at the one-hot boundary. On an open-weight model, GCG computes the gradient of the target affirmative-response loss with respect to the one-hot token indicator vectors at the suffix positions (immediately prior to the lookup), identifying which vocabulary swaps have the steepest linear descent direction.
- Exact discrete evaluation. Because tokens are discrete integers—you cannot feed a continuous interpolation of two tokens through a tokenizer—GCG tests top candidate token substitutions via exact forward passes, greedily retaining the integer ID that minimizes the loss.
- Cross-model transfer. Remarkably, suffixes optimized on open-weight models frequently transfer to black-box commercial models—suggesting that different models trained on overlapping web corpora learn similar high-dimensional feature correlations from the input embeddings onward.
From a security perspective, this difference reveals a fundamental tradeoff at the input level, meaning that highly perplexed adversarial suffixes like GCG leave a noticeable statistical trace - the input appears as nonsense, so lightweight perplexity filters or secondary LLM judges can detect this even before generation begins. In contrast, single-character Unicode replacements (such as our full-width comma , or homoglyphs) are within the normal distribution of languages, bypassing perplexity checks but still guiding the initial embedding trajectory.
Entering the Residual Stream: What Comes Next?
Everything we have examined in Part I happens at the threshold of the network:
- The tokenizer maps raw UTF-8 strings into discrete integer sequences—creating attack surfaces whenever character-level filters, length limits, or IDNA domain parsers disagree with BPE segmentation.
- The input matrix maps those integers to initial vectors —where a single full-width comma
,can shift next-token Chinese probability mass by more than 50 percentage points, and where naive linear probes easily fool researchers who skip baseline controls.
Yet static input rows do not generate text on their own. When we send hello,deepseek into the model, how does that single embedding row at position 1 physically alter the representation of the final token so that the model replies in Chinese? Which layer reads the comma’s script feature, and how does it route that signal into the output logits?
So in Part II, we take our hello,deepseek experiment directly inside the Transformer - using the logit lens and causal activation patching across the residual stream, attention heads, and MLP blocks to trace the exact layer-by-layer circuit where a single punctuation mark hijacks the model’s output trajectory.