[Bug] Fast Tokenizer (ByteLevel BPE) incorrectly decodes added_tokens containing specific Unicode characters (e.g., 'č' becomes '\r')

[zidsi]

System Info

• transformers version: 4.57.6
• Platform: Linux-6.8.0-90-generic-x86_64-with-glibc2.35
• Python version: 3.12.13
• Huggingface_hub version: 0.36.2
• Safetensors version: 0.7.0
• Accelerate version: 1.13.0
• Accelerate config: - compute_environment: LOCAL_MACHINE
  - distributed_type: MULTI_GPU
  - mixed_precision: bf16
  - use_cpu: False
  - debug: False
  - num_processes: 8
  - machine_rank: 0
  - num_machines: 1
  - gpu_ids: all
  - rdzv_backend: static
  - same_network: True
  - main_training_function: main
  - enable_cpu_affinity: False
  - downcast_bf16: no
  - tpu_use_cluster: False
  - tpu_use_sudo: False
  - tpu_env: []
• DeepSpeed version: not installed
• PyTorch version (accelerator?): 2.10.0+cu129 (CUDA)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed

Who can help?

Who can help?

Tagging the tokenizers maintainers: @ArthurZucker @itazap

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder
• My own task or dataset (Tokenizer Expansion / Added Tokens)

Reproduction

Bug Description:
When using a Fast Tokenizer with a ByteLevel BPE base (such as gpt2, qwen, roberta), adding new tokens (added_tokens) that contain specific Unicode characters results in corrupted decoding.

Specifically, characters whose lower byte matches control characters (0-32) are stripped of their upper byte during the fast decoding process.
For example:

• The character č (U+010D) is incorrectly decoded as \r (0x0D / Carriage Return).
• The character ć (U+0107) is incorrectly decoded as \a (0x07 / Bell).
• The character đ (U+0111) is incorrectly decoded as \x11 (0x11 / Device Control 1).

This only happens when use_fast=True. The slow Python tokenizer (use_fast=False) handles these added tokens perfectly. The bug seems to lie in how the Rust backend's ByteLevel decoder processes added_tokens_decoder items, likely applying a bitwise & 0xFF operation (or similar) that strips the U+0100 offset.

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Minimal Reproducible Example (MRE):

from transformers import AutoTokenizer

# Load standard ByteLevel BPE tokenizer
tokenizer_fast = AutoTokenizer.from_pretrained("gpt2", use_fast=True)
tokenizer_slow = AutoTokenizer.from_pretrained("gpt2", use_fast=False)

# Add words containing the problematic characters
new_tokens = ["Začnimo", "kuća", "međa"]

tokenizer_fast.add_tokens(new_tokens)
tokenizer_slow.add_tokens(new_tokens)

print("--- FAST TOKENIZER (BUGGY) ---")
for word in new_tokens:
    ids = tokenizer_fast.encode(word)
    decoded = tokenizer_fast.decode(ids)
    print(f"Original: {word} -> Decoded: {repr(decoded)}")

print("\n--- SLOW TOKENIZER (CORRECT) ---")
for word in new_tokens:
    ids = tokenizer_slow.encode(word)
    decoded = tokenizer_slow.decode(ids)
    print(f"Original: {word} -> Decoded: {repr(decoded)}")

Expected Output:

The Fast Tokenizer should output the original words ("Začnimo", "kuća", "međa").

Actual Output:

--- FAST TOKENIZER (BUGGY) ---
Original: Začnimo -> Decoded: 'Za\rnimo' <-- 'č' get corrupted? 'č' becomes \r
Original: kuća -> Decoded: 'ku\x07a'
Original: međa -> Decoded: 'me\x11a'

--- SLOW TOKENIZER (CORRECT) ---
Original: Začnimo -> Decoded: 'Začnimo'
Original: kuća -> Decoded: 'kuća'
Original: međa -> Decoded: 'međa'

Expected behavior

Added tokens containing any valid Unicode characters should be safely decoded by the Fast Tokenizer without their byte-values being stripped or downcast to control characters.

[ArthurZucker]

Hey! this was fixed by: #1555

[zidsi]

So this shouldn't be happening?

transformers version 4.57.1
--- FAST TOKENIZER (BUGGY) ---
Original: Začnimo -> Decoded: 'Za\rnimo'
Original: kuća -> Decoded: 'ku\x07a'
Original: međa -> Decoded: 'me\x11a'

--- SLOW TOKENIZER (CORRECT) ---
Original: Začnimo -> Decoded: 'Začnimo'
Original: kuća -> Decoded: 'kuća'
Original: međa -> Decoded: 'međa'

and in 5.4 even "slow" fails:

transformers version 5.4.0
--- FAST TOKENIZER (BUGGY) ---
Original: Začnimo -> Decoded: 'Za\rnimo'
Original: kuća -> Decoded: 'ku\x07a'
Original: međa -> Decoded: 'me\x11a'

--- SLOW TOKENIZER (BUGGY as well) ---
Original: Začnimo -> Decoded: 'Za\rnimo'
Original: kuća -> Decoded: 'ku\x07a'
Original: međa -> Decoded: 'me\x11a'

The code used

from transformers import AutoTokenizer,__version__
print(f"transformers version {__version__}")
# Load standard ByteLevel BPE tokenizer
tokenizer_fast = AutoTokenizer.from_pretrained("gpt2", use_fast=True)
tokenizer_slow = AutoTokenizer.from_pretrained("gpt2", use_fast=False)

# Add words containing the problematic characters
new_tokens = ["Začnimo", "kuća", "međa"]

tokenizer_fast.add_tokens(new_tokens)
tokenizer_slow.add_tokens(new_tokens)

print("--- FAST TOKENIZER (BUGGY) ---")
for word in new_tokens:
    ids = tokenizer_fast.encode(word)
    decoded = tokenizer_fast.decode(ids)
    print(f"Original: {word} -> Decoded: {repr(decoded)}")

print("\n--- SLOW TOKENIZER (CORRECT) ---")
for word in new_tokens:
    ids = tokenizer_slow.encode(word)
    decoded = tokenizer_slow.decode(ids)
    print(f"Original: {word} -> Decoded: {repr(decoded)}")

[zidsi]

@ArthurZucker tested with tokenizers-0.22.1 and tokenizers-0.22.2, so I'd guess the issue persists.

Tokenizers version 0.22.2
Transformers version 4.57.6
--- FAST TOKENIZER (BUGGY) ---
Original: Začnimo -> Decoded: 'Za\rnimo'
Original: kuća -> Decoded: 'ku\x07a'
Original: međa -> Decoded: 'me\x11a'

--- SLOW TOKENIZER (CORRECT) ---
Original: Začnimo -> Decoded: 'Začnimo'
Original: kuća -> Decoded: 'kuća'
Original: međa -> Decoded: 'međa'

[matdou]

I tried to dig into this, here's what I found.

This looks related to #1392, which was supposedly fixed in #1555 (July 2024) by introducing a ByteLevel normalizer. The idea was to pre-encode added tokens through the byte-level alphabet at store time so the decoder always receives them in the right form.

But I think that fix only works if the tokenizer has the ByteLevel normalizer configured but GPT-2 doesn't seem to?

>>> import json
>>> with open("tokenizer.json") as f:
...     cfg = json.load(f)
>>> print(cfg["normalizer"])
None

If that's correct, it would explain why add_tokens stores the token as raw UTF-8 with no byte-level encoding applied. Then when decode_chain runs, it looks up every character in CHAR_BYTES. And characters like č (U+010D) happen to be in the byte-level remapping range (control bytes 0x00–0x1F get mapped to U+0100–U+011F), so they get mapped back to control bytes:

• č (U+010D) → 0x0D → \r
• ć (U+0107) → 0x07 → \a
• đ (U+0111) → 0x11 → \x11

The fallback in decode_chain (which would correctly use raw UTF-8) only triggers if all charatcers in the token fail the lookup, but since most chars in a word like Začnimo are plain ASCII and succeed, the fallback is not called.

Happy to be corrected if I'm misreading the code!

[matdou]

I think the real fix needs to happen in huggingface/tokenizers, add_tokens in added_vocabulary.rs doesn't seem to apply the normalizer before storing the token, so the byte-level encoding never happens.

[zidsi]

@matdou Thank you for the quick insight! You are absolutely right. I checked the tokenizer.json for Qwen (and GPT-2), and they do not define a ByteLevel normalizer in their pipeline (they only use a ByteLevel pre-tokenizer and decoder). Because the normalizer is missing, add_tokens("čnim") stores the literal Unicode string "čnim" into the added_tokens array in tokenizer.json.

When the Fast Tokenizer attempts to decode this later, the ByteLevel decoder encounters the literal č (U+010D), applies the bitmask & 0xFF, and erroneously outputs \r (0x0D).

While adding a ByteLevel normalizer before adding tokens would fix the mapping at store-time, this is highly problematic for users fine-tuning or expanding vocabularies on existing base models (like Qwen, LLaMA-3, GPT-2) that don't ship with a normalizer. If users add a normalizer just to fix added_tokens, they risk altering the tokenization behavior of the entire base model.

@ArthurZucker Perhaps the Fast Decoder's logic could be updated to gracefully handle (pass-through) characters in added_tokens that haven't been byte-level encoded, rather than indiscriminately stripping their upper bytes?

[ArthurZucker]

Not sure if you read the PR description I sent.
Here we can't choose what the qwen team does or not. The best way to fix the issue is to submit one with this:

If users add a normalizer just to fix added_tokens, they risk altering the tokenization behavior of the entire base model.

This is just wrong and baseless. The normalizer is a drop in replacement for special tokens. The string is always split anyways.

[ArthurZucker]

If you want a "fix" that works without doing anything its not really possible due to how tokenizers works itself. What we can do however is change in transformers places where we use the pre_tokenizer in favor of the normalizer!

(fyi @itazap) not super high prio so will give it a good second issue tag

[zidsi]

Not sure if you read the PR description I sent. Here we can't choose what the qwen team does or not. The best way to fix the issue is to submit one with this:

If users add a normalizer just to fix added_tokens, they risk altering the tokenization behavior of the entire base model.

This is just wrong and baseless. The normalizer is a drop in replacement for special tokens. The string is always split anyways.

Thank you so much for taking the time to explain this, and I apologize if my initial report caused any frustration. I really appreciate the context regarding PR huggingface/transformers#1555.

I completely agree with your point—neither Hugging Face nor the community can dictate how upstream model providers (like the Qwen, Mistral, or Nvidia teams) configure their base tokenizer.json files.

If I understand your explanation correctly, the intended mechanism is that by introducing the ByteLevel normalizer, the added tokens are pre-encoded through the byte-level alphabet at store time, ensuring the decoder receives them safely. Please let me know if I am misunderstanding how this is supposed to work under the hood!

I tried to apply this workaround by injecting the normalizer into the backend before calling add_tokens() via the transformers API (which is the standard pipeline most users use for vocabulary expansion). However, it seems the bug might still persist. It looks like the transformers API might still be passing the literal Unicode strings to the backend, bypassing the store-time encoding.

I wrote a quick test script to check this behavior across several modern Tiktoken/BPE-based models, and the output still comes out corrupted:

from transformers import AutoTokenizer, __version__
import tokenizers
from tokenizers import normalizers

print(f"Tokenizers version: {tokenizers.__version__}")
print(f"Transformers version: {__version__}")

models = [
    'Qwen/Qwen3.5-35B-A3B-Base',
    'Qwen/Qwen3.5-9B-Base',
    'mistralai/Ministral-3-8B-Base-2512',
    'nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16',
    'LiquidAI/LFM2.5-1.2B-Base'
]

new_tokens = ["Začnimo", "kuća", "međa"]

for name in models:
    tokenizer_fast = AutoTokenizer.from_pretrained(name, use_fast=True)
    
    # Applying the suggested ByteLevel normalizer fix
    tokenizer_fast._tokenizer.normalizer = normalizers.ByteLevel()
    
    tokenizer_fast.add_tokens(new_tokens)
    
    print(f"\n--- TOKENIZER {name} ---")
    for word in new_tokens:
        ids = tokenizer_fast.encode(word, add_special_tokens=False)
        decoded = tokenizer_fast.decode(ids)
        status = "✅" if decoded == word else f"❌ (Got: {repr(decoded)})"
        print(f"Original: {word} -> {status}")

Output:

Tokenizers version: 0.22.2
Transformers version: 5.4.0

--- TOKENIZER Qwen/Qwen3.5-35B-A3B-Base ---
Original: Začnimo -> ❌ (Got: 'Za\rnimo')
Original: kuća -> ❌ (Got: 'ku\x07a')
Original: međa -> ❌ (Got: 'me\x11a')

--- TOKENIZER Qwen/Qwen3.5-9B-Base ---
Original: Začnimo -> ❌ (Got: 'Za\rnimo')
Original: kuća -> ❌ (Got: 'ku\x07a')
Original: međa -> ❌ (Got: 'me\x11a')

--- TOKENIZER mistralai/Ministral-3-8B-Base-2512 ---
Original: Začnimo -> ❌ (Got: 'Za\rnimo')
Original: kuća -> ❌ (Got: 'ku\x07a')
Original: međa -> ❌ (Got: 'me\x11a')

--- TOKENIZER nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16 ---
Original: Začnimo -> ❌ (Got: 'Za\rnimo')
Original: kuća -> ❌ (Got: 'ku\x07a')
Original: međa -> ❌ (Got: 'me\x11a')

--- TOKENIZER LiquidAI/LFM2.5-1.2B-Base ---
Original: Začnimo -> ❌ (Got: 'Za\rnimo')
Original: kuća -> ❌ (Got: 'ku\x07a')
Original: međa -> ❌ (Got: 'me\x11a')

Since use_fast=False is not (going to be) an option in transformers v5, could you kindly point me in the right direction? Is there a different, officially supported way to ensure these AddedToken objects are properly byte-level encoded at store-time when using the transformers API?

Thanks again for your patience and for the amazing work you do on this library!

[ArthurZucker]

You are right my bad