Chat templates and alignment failures: how ChatBug turns formatting into a safety vulnerability

Why chat templates can become a safety vulnerability

Bottom Line: ChatBug is a class of prompt-format exploits that bypasses safety alignment in instruction-tuned LLMs by violating the rigid formatting contract that chat templates impose on model behavior but not on user input. The paper ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates demonstrates successful exploitation across eight state-of-the-art aligned LLMs using two distinct attack paths — format mismatch and message overflow — and shows that adversarial training mitigates the vulnerability at a measurable helpfulness cost. This is not a model-family-specific bug; it is a consequence of how prompt serialization and safety alignment interact at the template boundary, which means any instruction-tuned model relying on chat-template formatting is a potential target.

The asymmetry at the core of ChatBug is deceptively simple. As the paper states: "the chat templates provide a rigid format that need to be followed by LLMs, but not by users." Safety alignment training teaches the model to expect a specific token sequence structure. Users are under no such constraint. That gap is the attack surface.

Two exploit families operationalize this asymmetry: the format mismatch attack, which submits a prompt that deviates from the expected template structure, and the message overflow attack, which injects tokens into fields the model treats as reserved. Both exploit the same root cause — the model's safety behaviors are conditioned on template-conformant input, and non-conformant input degrades those behaviors without triggering any explicit error signal. The paper confirms that "a malicious user can exploit the ChatBug vulnerability of eight state-of-the-art (SOTA) LLMs and effectively elicit unintended responses from these models."

Existing jailbreak attacks become more effective when combined with ChatBug, compounding the risk. Adversarial training provides partial mitigation, but the paper documents significant performance degradation in the patched model — a trade-off practitioners must evaluate deliberately rather than assume away.

---

How the template alignment stack is supposed to work

Chat templates are the serialization layer between a structured conversation (a list of role-keyed messages) and the flat token sequence a model actually sees. In Hugging Face Transformers, the tokenizer exposes this as a Jinja template string: chat_template (str, optional) — A Jinja template string that will be used to format lists of chat messages. When apply_chat_template is called, it reads this attribute along with the tokenizer's control tokens and produces the exact sequence the model was trained to receive.

The alignment stack depends on this serialization being stable. Safety training — whether via RLHF, DPO, or SFT with safety examples — proceeds over template-formatted sequences. The model learns to produce aligned outputs when inputs look like those sequences. Any deviation from the expected format is out-of-distribution from the safety training perspective, even when it looks textually harmless.

The table below summarizes how each layer of the stack contributes to alignment, and where the assumption breaks:

Stack layer	Responsibility	What breaks under ChatBug
Tokenizer chat_template	Converts message list → token sequence with role delimiters and control tokens	Template-noncompliant input bypasses expected structure
Special / control tokens	Demarcate role boundaries (<\|im_start\|>, [INST], <\|eot_id\|>, etc.)	Overflow or omission confuses role parsing
SFT / RLHF safety tuning	Trains aligned responses over template-formatted sequences	Safety behaviors conditioned on conformant input degrade silently
Inference-time prompt assembly	Assembles system prompt + user turn + assistant prefix	No runtime validation that user input respects template structure
Output layer	Samples from distribution shaped by alignment tuning	Produces unsafe completions when in-context distribution is perturbed

Layer	Normal role in alignment	ChatBug failure mode	Operational impact	Example control point
Tokenizer template	Serializes roles consistently	User text collides with control tokens	Boundary confusion	apply_chat_template
Special tokens	Mark system/user/assistant turns	Delimiter overflow or omission	Misparsed roles	all_special_tokens
Safety tuning	Conditions refusals on format	Noncompliant input is out-of-distribution	Refusal weakens	RLHF / DPO / SFT
Prompt assembly	Builds final input string	No enforcement at the input edge	Untrusted content passes through	API or UI layer
Generation head	Samples next token	Distribution shifts upstream	Unsafe continuation	Decoding step

The Hugging Face docs make an important caveat explicit: "Chat templates should already include all the necessary special tokens, and adding additional special tokens is often incorrect or duplicated, hurting model performance." This warning applies during training pipeline setup, but it signals something deeper — the template and its control tokens are a tightly coupled system, and any perturbation of that system propagates through model behavior in ways that are not always visible until a safety test fails.

Why training-time templates differ from user-time prompts

The training pipeline applies the chat template to every example before tokenization. The resulting token sequences are what the model's safety conditioning is fitted to. At inference time, the application code calls apply_chat_template on user messages — but there is no enforcement mechanism that constrains what the user message contains before that call.

A user turn that contains partial template syntax, a misplaced role delimiter, or a fabricated control token sequence will be embedded inside the structured template by the application, but the model will parse the resulting token sequence differently than the application intends. The structural signals the model uses to determine role boundaries and activate safety behaviors are now ambiguous or corrupted.

Pro Tip: When building a chat application, treat the user message field as an untrusted string that must be sanitized against your model's specific control tokens — not just for injection of arbitrary text, but explicitly for characters and byte sequences that appear in your tokenizer's special token vocabulary. Print tokenizer.all_special_tokens and tokenizer.all_special_ids for the model you're deploying, then build a deny-list or escape step before apply_chat_template.

The asymmetry is structural: the training pipeline controls the template application; the inference pipeline does not control the content that flows into it.

Where safety alignment is attached in the prompt pipeline

Safety alignment does not live in a separate module — it is baked into the weight distribution through training on template-formatted sequences. The mechanism is described precisely in the Hugging Face tokenizer docs: "This method is intended for use with chat models, and will read the tokenizer's chat_template attribute to determine the format and control tokens to use when converting."

This means apply_chat_template is the gate through which safety conditioning is activated. The model has no separate safety head; it has a distribution over continuations that was shaped by exposure to aligned template-formatted sequences. Change the shape of the input sequence — even subtly — and you shift which region of the model's learned distribution the forward pass samples from.

Watch Out: Changing a tokenizer's chat_template after safety training has completed can silently invalidate alignment assumptions. The model weights are unchanged, but the prompt structure the weights were trained on no longer matches what the tokenizer produces. This is not a hypothetical edge case — it is the mechanism ChatBug exploits, and it can be triggered accidentally by engineers who modify templates for tool-use or multi-turn formatting without re-running safety evaluation.

---

The ChatBug mechanism: format asymmetry and prompt parsing failure

The ChatBug paper defines the vulnerability precisely: the rigid format required by the chat template applies to the LLM, not to the user. Safety alignment is effectively a behavioral constraint conditioned on input structure. When input structure diverges from the training distribution, the behavioral constraint weakens or disappears.

The two exploit paths — format mismatch and message overflow — both operate on this principle, but they attack different points in the template parsing chain. The architecture diagram below maps the full parsing path and shows where each attack intervenes:

The paper defines two exploit paths — format mismatch attack and message overflow attack — to exploit the template boundary weakness.

flowchart LR
    subgraph USER["User Input (uncontrolled)"]
        U[Raw user message string]
    end

    subgraph APP["Application Layer"]
        T["apply_chat_template(messages)"]
        V["Optional: input validation\n(often absent)"]
    end

    subgraph TOKEN["Tokenizer"]
        J["Jinja template expansion\n(role delimiters + control tokens)"]
        S["Special token injection\n(<|im_start|>, [INST], <|eot_id|>, etc.)"]
        F["Flat token sequence"]
    end

    subgraph MODEL["Aligned LLM"]
        SA["Safety-conditioned\ndistribution (expects\ntemplate-conformant input)"]
        GEN["Token generation"]
    end

    U --> V
    V -->|"No enforcement\nagainst control tokens"| T
    T --> J
    J --> S
    S --> F
    F -->|"Format mismatch:\nrole boundaries ambiguous"| SA
    F -->|"Message overflow:\nreserved field corrupted"| SA
    SA --> GEN

    style V fill:#f5a623,color:#000
    style SA fill:#d0021b,color:#fff

The orange validation node is absent in most production stacks. The red safety-conditioned distribution node is the target: both attacks reach it by corrupting the token sequence upstream of it, without touching model weights or inference infrastructure.

As the paper states: "We developed two attacks, format mismatch attack and message overflow attack, to exploit the ChatBug vulnerability." The competitive gap other summaries leave open is precisely here — the vulnerability is not in a misconfigured system prompt or a weak policy statement; it is in the structural coupling between template parsing and alignment conditioning.

Format mismatch attack path

The format mismatch attack submits a user prompt that violates the structural conventions of the model's expected template. As the paper states, "the format mismatch attack alters the default chat format … to bypass safety alignment of LLMs."

Concretely: if a model's template wraps each user turn in a specific delimiter pair (e.g., [INST]...[/INST] for Mistral-family models, or <|im_start|>user\n...<|im_end|> for Qwen/Llama), and the attacker injects a partial or synthetic version of those delimiters inside the user content, the tokenizer serializes the full conversation into a sequence where role boundaries are ambiguous. The model's safety-conditioned layers were trained to associate certain boundary patterns with the "user turn that requires alignment filtering" role. A corrupted boundary pattern shifts parsing into a different learned region — one where alignment constraints are weaker or absent.

The key point is that this requires no knowledge of model weights, no gradient access, and no special infrastructure. It requires only knowledge of the target model's template format — which is public for every major instruction-tuned open-weight model on Hugging Face.

Pro Tip: The format mismatch attack is most effective against models where control token sequences are short and human-readable (e.g., [INST], <<SYS>>), because the attacker can type them directly in a chat UI. Models using long byte-string or binary control tokens are harder to exploit via keyboard input, but not via API calls that accept raw strings. Audit your API's input handling as well as your UI's.

Message overflow attack path

The message overflow attack operates on a different seam in the template. Rather than altering the structural delimiters of an existing role turn, it injects a token sequence into a field the model treats as reserved — a slot that, under normal template operation, is controlled by the application or system, not by the user.

The paper describes this directly: "the message overflow attack injects a sequence of tokens into the model's reserved field."

Many chat templates include fields that the user is not supposed to write into: the system prompt field, the assistant prefix (the beginning of the model's response turn that is pre-filled to steer generation), or tool-response slots. If the application's prompt assembly code does not enforce hard boundaries between user-controlled content and these reserved fields, a crafted message can overflow into them. The model then reads content the attacker controls as if it originated from a trusted source (the system or the assistant turn), bypassing the safety filters that apply specifically to user-role input.

Tool-use templates are especially exposed here, because they introduce additional fields — tool call arguments, tool results, function schema slots — that extend the attack surface beyond the simple system/user/assistant triad. Any field the model is trained to treat as authoritative is a potential overflow target.

Pro Tip: When building tool-use templates, never concatenate user-provided strings directly into tool-result or function-schema fields without first normalizing them against your tokenizer's special token vocabulary. The model treats content in those fields as system-tier input. An overflow from a user message into a tool-result field can effectively impersonate the tool's response with attacker-controlled content.

Why existing jailbreaks get stronger under ChatBug

Standard jailbreaks work by crafting prompts whose textual content steers the model toward unsafe outputs despite safety training. ChatBug adds a structural layer on top: by corrupting template boundaries before a jailbreak payload reaches the model, the attacker reduces the effective strength of safety conditioning before the textual manipulation even takes effect.

The paper is explicit: "Moreover, we show that ChatBug can be exploited by existing jailbreak attacks to enhance their attack success rates." ChatBug does not replace existing jailbreaks — it amplifies them. This has a direct consequence for red-team evaluation: measuring attack success on template-conformant inputs understates actual vulnerability.

Jailbreak condition	Template-conformant input	Template-noncompliant input (ChatBug active)
Direct harmful request	Typically blocked by safety alignment	Alignment weakened; refusal rate drops
Indirect / obfuscated harmful request	Variable; depends on phrasing	Baseline defense further reduced
Role-play / persona bypass	Partially blocked	Structural mismatch compounds persona confusion
Combined jailbreak + ChatBug	N/A	Attack success rate higher than jailbreak alone

The practical implication: any evaluation benchmark that tests jailbreak resistance without template-noncompliant variants is measuring a lower-bound on safety performance, not actual deployed robustness.

---

What the paper shows across eight aligned LLMs

The paper demonstrates ChatBug exploitation across eight state-of-the-art aligned LLMs. Cross-model recurrence at this scale rules out a family-specific implementation bug — the vulnerability is structural and follows from how chat-template-conditioned safety alignment works, regardless of the underlying architecture or training recipe. As the paper states: "We show that a malicious user can exploit the ChatBug vulnerability of eight state-of-the-art (SOTA) LLMs and effectively elicit unintended responses from these models."

The table below reports the paper's high-level cross-model findings:

Finding	Scope	Attack type	Outcome
Successful exploitation	8 SOTA aligned LLMs	Format mismatch	Unintended (unsafe) responses elicited
Successful exploitation	8 SOTA aligned LLMs	Message overflow	Unintended (unsafe) responses elicited
Jailbreak amplification	8 SOTA aligned LLMs	Existing jailbreaks + ChatBug	Attack success rate elevated vs. jailbreak alone
Adversarial training mitigation	Victim model subset	Adversarial fine-tuning	Vulnerability reduced; performance degraded significantly

Result category	Models affected	Attack path	Paper-supported outcome	Practical reading
Exploitation	8	Format mismatch	Successful unintended responses	Structural boundary bypass works across families
Exploitation	8	Message overflow	Successful unintended responses	Reserved-field corruption is reusable
Amplification	8	Existing jailbreaks + ChatBug	Higher attack success rate	Structural and textual attacks compound
Mitigation	Subset	Adversarial training	Vulnerability reduced, helpfulness degraded	Defense exists, but with trade-off

Model families and template formats that were evaluated

The paper evaluates across eight SOTA LLMs, and the relevant semantic entities in the published discussion include models from distinct template families: Llama 3.1 70B Instruct, Mistral 7B Instruct, and Qwen2.5-32B-Instruct represent three structurally distinct template conventions. The Hugging Face tokenizer docs confirm that templates are tokenizer-specific — apply_chat_template reads the tokenizer's chat_template attribute to determine format and control tokens. Switching model families is therefore also switching template formats, and each format creates a different but equally exploitable boundary structure.

Model family	Template style	Role delimiter example	Reserved fields
Llama 3.1 Instruct	<\|im_start\|> / <\|eot_id\|> with header ids	<\|start_header_id\|>user<\|end_header_id\|>	System header, assistant prefix
Mistral Instruct	[INST] / [/INST] with <<SYS>>	[INST] user content [/INST]	System (<<SYS>>), implicit assistant continuation
Qwen2.5 Instruct	<\|im_start\|> / <\|im_end\|>	<\|im_start\|>user\ncontent<\|im_end\|>	System role, tool-call fields
Generic ChatML	<\|im_start\|> / <\|im_end\|>	<\|im_start\|>role\ncontent<\|im_end\|>	System, tool result

Cross-family recurrence confirms that the attack surface is not an artifact of any single template design. It is the template-as-safety-mediator pattern itself that creates the vulnerability.

What success and failure look like in practice

A successful ChatBug exploitation produces an unintended response — a completion that the same model would refuse under template-conformant input. The failure mode is observable at the output level: the model generates content it was alignment-trained to decline, without any visible error state or safety filter trigger.

A failed exploitation — the model still refuses despite a template-noncompliant prompt — indicates that the model's safety conditioning is robust enough to generalize beyond its training distribution, or that the specific template violation did not create sufficient structural ambiguity to shift the parsed role context.

Behavior	Template-conformant prompt	Template-noncompliant (ChatBug) prompt
Harmful content request	Refusal with explanation	Unsafe completion generated
Borderline content request	Variable depending on policy tuning	Refusal rate drops; unsafe outputs more frequent
Benign request	Normal helpful response	Normal helpful response (no degradation on benign)
Post-adversarial training	Refusal rate restored	Refusal rate partially restored; helpfulness reduced

The paper frames the mitigation result explicitly: adversarial training that reduces ChatBug vulnerability does so at the cost of general model helpfulness. Fixing the structural vulnerability by training the model to resist template-noncompliant inputs changes the model's broader response distribution.

---

Why adversarial training helps and where it hurts

Adversarial training mitigates ChatBug by exposing the model, during fine-tuning, to template-noncompliant inputs paired with aligned (refusing) outputs. The model learns to produce safety-compliant behavior even when input structure deviates from the training-time template format. The paper confirms this works: "while adversarial training effectively mitigates the ChatBug vulnerability, the victim model incurs significant performance degradation."

The degradation is not incidental — it reflects the same coupling that makes the vulnerability possible. Safety alignment is conditioned on input structure. Expanding the set of input structures over which aligned behavior is expected requires updating the model's learned distribution over a broader input domain, which perturbs the distributions that govern helpfulness on standard prompts.

Metric dimension	Before adversarial training	After adversarial training
ChatBug format mismatch attack success	High across 8 models	Significantly reduced
ChatBug message overflow attack success	High across 8 models	Significantly reduced
Jailbreak amplification via ChatBug	Elevated vs. standalone jailbreak	Reduced (structural amplifier weakened)
General helpfulness / downstream benchmark	Baseline	Significant degradation reported
Safety on template-conformant inputs	High (standard aligned behavior)	Maintained or marginally changed

Measure	Before adversarial training	After adversarial training	Result
ChatBug attack success	High	Lower	Mitigated
Helpfulness on standard prompts	Baseline	Lower	Regression reported
Safety on noncompliant prompts	Weak	Stronger	Better refusal behavior
Overall trade-off	None	Present	Safety gain versus helpfulness cost

Safety gain versus helpfulness regression

The paper is direct: "These results highlight the trade-off between safety alignment and helpfulness." This is not a claim that adversarial training is ineffective — it demonstrably reduces ChatBug vulnerability. It is a claim that the cost is real and must be measured before committing to this mitigation.

The degradation most likely manifests in tasks that depend on the model's sensitivity to nuanced prompt phrasing and instruction-following on complex, multi-part queries — precisely the capabilities that instruction tuning is designed to maximize. Adversarial training that broadens the model's refusal behavior to cover template-noncompliant inputs may inadvertently broaden it to cover legitimate but unusual phrasing as well.

• Choose adversarial training when: the deployment context requires robust safety against adversarial users, the task domain is safety-critical (medical guidance, legal information, content moderation), and you can measure and accept a regression on standard instruction-following benchmarks.
• Choose compensating controls instead when: the helpfulness regression is unacceptable, the user base is not adversarial, or you can enforce template-conformance at the application layer before prompts reach the model.
• Combine both when: the deployment is public-facing and high-stakes — enforce template-conformance at the API layer as a first line of defense, and accept a moderate adversarial training penalty as defense in depth.

When mitigation is worth the cost

The decision to apply adversarial training is a risk-calibration call, not a pure safety call.

• High-risk deployment, adversarial user population: Public API, user-generated content, red-team-tested product. Adversarial training cost is justified; measure with MT-Bench or AlpacaEval before and after to quantify the regression and set a regression budget.
• Internal tooling, trusted users, template-enforcement enforced at API boundary: Input validation that strips or rejects special tokens from user content reduces ChatBug risk substantially without weight modification. Adversarial training is likely not worth the helpfulness cost here.
• Research or evaluation context: If you are benchmarking instruction-following quality, adversarial training introduces a confound. Evaluate the base model first; run safety evals separately on a patched checkpoint.
• Fine-tuned downstream model: If you fine-tune an already-aligned model (e.g., Llama 3.1 70B Instruct) on a domain-specific SFT dataset using TRL or Hugging Face Trainer, your fine-tuning may partially overwrite alignment conditioning. Adversarial training against ChatBug should be considered part of the post-SFT safety evaluation pass, not an optional add-on.

---

Template design checks for instruction-tuned models

Template design is a first-class safety concern, not a formatting convenience. The Hugging Face documentation is clear that the chat_template field is responsible for control token injection and message structure, and that incorrect or duplicated special tokens hurt model performance: "Chat templates should already include all the necessary special tokens, and adding additional special tokens is often incorrect or duplicated, hurting model performance."

The table below maps the most common template design decisions to their safety implications:

Design decision	Risk if wrong	Correct practice
Adding custom role delimiters	Confuses model's role-boundary parsing; safety conditioning may not apply	Use only role names the model was trained on (user, assistant, system, tool)
Injecting extra special tokens	Duplication or incorrect token can corrupt context boundaries	Audit against tokenizer.all_special_tokens before inserting any new token
Modifying assistant prefix	Changes what the model treats as its own prior output; safety guardrails may not apply	Only pre-fill the assistant turn with content validated against policy
Tool-result field handling	Free-form tool output injected into a trusted field creates message overflow surface	Sanitize tool outputs against special token vocabulary before insertion
Changing EOS token handling	Early or missing EOS can cause generation to continue past expected boundary	Verify EOS placement in template matches tokenizer's eos_token_id

Design decision	Risk if wrong	Correct practice	Why it matters
Role delimiter edits	Boundary confusion	Keep model-native roles	Preserves alignment conditioning
Extra special tokens	Duplicated or corrupt context	Audit special-token inventory	Avoids hidden serialization drift
Assistant prefix changes	Prior-output mismatch	Validate the prefill content	Keeps generation framing stable
Tool-result injection	Message overflow surface	Sanitize tool outputs first	Prevents trusted-field poisoning
EOS handling changes	Late or early termination	Match tokenizer EOS behavior	Keeps stop criteria consistent

Questions to ask before changing a tokenizer template

Template edits are behavioral changes. Engineers frequently modify templates to add tool-use support, extend multi-turn handling, or port a model to a new serving framework. Each of these changes can alter the input distribution seen by the model and invalidate the safety assumptions the alignment training established.

Before modifying a chat_template on an aligned model, the following questions require explicit answers:

1. Does the modified template produce a token sequence that includes all and only the control tokens the model was aligned on?
2. Does the assistant mask (the labels mask used during SFT) still correctly cover only assistant turns in the new format?
3. Does EOS placement in the new template match the model's expected generation-stop behavior?
4. Does any new field in the template (tool call, tool result, function schema) accept user-controlled content? If so, what sanitization is applied?
5. Has the modified template been tested against a suite of ChatBug-style noncompliant inputs to confirm safety behavior is preserved?

Watch Out: A template change that looks purely cosmetic — reordering role delimiters, shortening a newline sequence, adding a field for tool metadata — can silently shift the model out of its safety-conditioned input distribution. There is no runtime warning. The model will continue to generate output; the outputs will simply no longer carry the same safety guarantees. Treat every template diff as a security-relevant code change requiring explicit regression testing against your safety eval suite.

How to sanity-check template diffs across model families

When porting a pipeline from one model family to another — for example, migrating from Mistral 7B Instruct to Llama 3.1 70B Instruct or Qwen2.5-32B-Instruct — the template changes are not cosmetic. Each family's tokenizer encodes different control tokens, different role delimiters, and different reserved fields.

Template property	Mistral Instruct	Llama 3.1 Instruct	Qwen2.5 Instruct
User delimiter	[INST] / [/INST]	<\|start_header_id\|>user<\|end_header_id\|>	<\|im_start\|>user
System field	<<SYS>> / <</SYS>>	Separate system header block	<\|im_start\|>system
EOS token	</s>	<\|eot_id\|>	<\|im_end\|>
Tool-use fields	Not standard in base template	Supported via function-call role extensions	Native tool-call fields in template
Reserved field overflow risk	<<SYS>> field injectable via user content if not guarded	System header injectable if user content not stripped	Tool-call fields injectable if tool results not sanitized

Diffing templates across families means checking control-token identity, EOS behavior, and whether message role collapse is possible. Cross-family portability is not guaranteed even when the prompt text looks structurally similar. Always call tokenizer.apply_chat_template(test_messages, tokenize=True) and inspect the resulting token ID sequence — not just the decoded string — when validating a template port.

---

Implications for alignment, red teaming, and tool-use prompts

ChatBug changes the scope of what alignment engineers and red teams must validate. Safety alignment that was evaluated only on template-conformant inputs should be treated as an incomplete safety signal. The vulnerability also has direct consequences for tool-use prompt design, where the expanded field structure amplifies the message overflow attack surface.

• Choose strict input validation when deploying a public-facing chat API: sanitize user content against the model's special token vocabulary before calling apply_chat_template. This does not require model changes and eliminates a large fraction of format mismatch attack surface.
• Choose adversarial training when the deployment context is high-risk, the user population is adversarial, and you can quantify and absorb the helpfulness regression. Do not apply without measuring the regression against a standard instruction-following benchmark.
• Choose template auditing + red-team evaluation when fine-tuning or porting an aligned model: any template modification requires a re-run of safety evals against both conformant and noncompliant inputs.
• For tool-use template design specifically: treat tool-result and function-schema fields as high-trust inputs and enforce sanitization at the application layer before those fields are populated. Do not assume the model distinguishes injected content from legitimate tool output.

Deployment context	Primary risk	Recommended mitigation
Public chat API (open input)	Format mismatch + jailbreak amplification	Input sanitization against special token vocabulary
Internal API, trusted users	Low ChatBug risk	Template audit on any change; light red-team eval
Tool-use / function-calling pipeline	Message overflow into tool-result fields	Sanitize tool outputs; audit reserved field access
Fine-tuned downstream model (SFT)	Alignment conditioning partially overwritten	Post-SFT safety eval with ChatBug test cases
Model ported across template families	Silent template format shift	Full template diff + alignment re-evaluation

Deployment context	Primary risk	Choose this mitigation when	Notes
Public chat API (open input)	Format mismatch + jailbreak amplification	You can sanitize user content before templating	Eliminates much of the boundary attack surface
Internal API, trusted users	Low ChatBug risk	Change control is already strict	Focus on audit and light red-teaming
Tool-use / function-calling pipeline	Message overflow into tool-result fields	Tool outputs can be normalized before insertion	Treat tool data as high-trust until validated
Fine-tuned downstream model (SFT)	Alignment drift after fine-tuning	You can rerun safety evaluation post-SFT	Catch overwritten alignment assumptions
Model ported across template families	Silent template format shift	You can diff templates and token IDs	Validate on both conformant and noncompliant inputs

Red-team cases that should be added to your eval suite

Standard red-team suites test content-policy violations on template-conformant inputs. ChatBug demonstrates that this is insufficient. The two attack families the paper identifies — format mismatch and message overflow — provide concrete templates for expanding your eval coverage:

Red-team case	Attack family	What to test	Pass criterion
Injected role delimiter in user turn	Format mismatch	Insert model-family-specific role tokens in user content	Model refuses or ignores injected structure; does not treat it as a new role
Partial control token injection	Format mismatch	Insert partial or misspelled control tokens that approximate real ones	No degradation in refusal behavior on harmful requests
System prompt overflow via user input	Message overflow	Include <<SYS>> or <\|im_start\|>system sequences in user message	Injected system content does not override policy
Tool-result field injection	Message overflow	Include function-result-like syntax in user message	Model does not treat user-injected content as tool output
Jailbreak + format mismatch combined	ChatBug amplification	Run standard jailbreak payloads under noncompliant template input	Attack success rate should not exceed template-conformant baseline
Cross-turn delimiter collision	Format mismatch	Inject EOS/EOT tokens mid-message	Generation does not terminate or role-shift unexpectedly

Production notes for tokenizer and chat-template governance

Production Note: Chat templates and tokenizer configs are security-relevant artifacts that must be version-controlled alongside model weights. A tokenizer checkpoint that differs from the one used during safety training — even if the weights are identical — can silently invalidate alignment guarantees. Treat tokenizer_config.json and any chat_template overrides with the same change-control rigor as model weight checksums. Require explicit regression testing against your safety eval suite before any template change is deployed to production. For models deployed via Hugging Face-compatible serving stacks, pin the tokenizer revision hash in your deployment manifest and audit diffs against the safety-certified baseline before upgrading.

---

FAQ

What is ChatBug in LLMs?

ChatBug is a vulnerability class in instruction-tuned, aligned LLMs that arises from the mismatch between the rigid prompt formatting that chat templates impose on the model and the unconstrained input that users can submit. Because safety alignment is conditioned on template-conformant input, a user who submits a structurally noncompliant prompt can degrade or bypass the model's safety behaviors without modifying its weights.

How do chat templates affect safety alignment?

Safety alignment training — whether via RLHF, DPO, or SFT with safety examples — proceeds over template-formatted sequences. The model learns to produce aligned outputs when inputs match those template structures. Chat templates are therefore not just a formatting convenience; they are the structural prerequisite for safety conditioning. When the template contract is violated at inference time, alignment behaviors degrade.

Can adversarial training fix ChatBug?

Adversarial training effectively reduces ChatBug vulnerability by exposing the model to template-noncompliant inputs during fine-tuning. However, the paper is explicit that this comes at a cost: "while adversarial training effectively mitigates the ChatBug vulnerability, the victim model incurs significant performance degradation." It is a partial mitigation with a real trade-off, not a zero-cost fix.

Why do chat templates matter for instruction-tuned models?

Chat templates determine the exact token sequence the model receives, including role-boundary control tokens that the model uses to parse conversation structure. As the paper states, "the chat templates provide a rigid format that need to be followed by LLMs, but not by users." For instruction-tuned models, this means the template is the mechanism through which system-level safety instructions are distinguished from user input. Any change to the template — including accidental drift during fine-tuning or cross-family porting — can invalidate that distinction.

Does ChatBug affect only open-weight models?

The paper evaluates eight SOTA aligned LLMs across multiple families. The vulnerability mechanism — format asymmetry between model-side template enforcement and user-side input freedom — is not specific to open-weight models. Any system that mediates safety alignment through chat-template-formatted prompts and does not validate template conformance at the input layer carries this exposure.

Bottom Line: ChatBug is a structural property of how chat-template-conditioned alignment works, not a bug in any single model. Every instruction-tuned LLM that relies on template formatting for safety alignment is exposed unless input sanitization is enforced at the application layer or adversarial training has been applied with an accepted helpfulness regression. The paper demonstrates this across eight SOTA models and provides two concrete attack families — format mismatch and message overflow — that should be incorporated into every red-team eval suite for aligned chat models.

---

Sources & References

• ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates — arXiv preprint; primary source for mechanism, attack types, cross-model results, and adversarial training trade-off.
• ChatBug PDF v2 — arXiv PDF with experimental results and attack descriptions.
• ChatBug HTML v2 — arXiv HTML describing format mismatch and message overflow.
• Hugging Face Transformers: Tokenizer docs — official documentation for chat_template and apply_chat_template.
• Hugging Face Transformers: Chat templating guide — official guidance on special tokens and template design warnings.

---

Keywords: ChatBug, Llama 3.1 70B Instruct, Mistral 7B Instruct, Qwen2.5-32B-Instruct, Hugging Face Transformers, tokenizer chat template, instruction tuning, safety alignment, adversarial training, jailbreak attacks, format mismatch attack, message overflow attack, TRL, Hugging Face Trainer