When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer

The background question

LayerNorm is one of the most standard building blocks in a transformer: it rescales activations at each layer to keep training stable. A recent line of work has proposed replacing it with Dynamic Tanh (DyT), a much simpler squashing function, on the grounds that most of what LayerNorm does can be approximated with a learned tanh curve.

The open question this paper asks: does removing LayerNorm this way actually behave like a free simplification, or does it change training in ways that depend on scale and data?

How to read the figure

Each cell compares Dynamic Tanh against the LayerNorm baseline for a model/data regime. Blue cells mean the replacement helped in that tested setting; red cells mean it hurt. The important pattern is the diagonal: the same architectural change can move from helpful to harmful as the token-to-parameter ratio changes.

That is why I do not frame this as "DyT wins" or "LayerNorm wins." The figure is more useful as a warning label: the regime decides whether the simplification behaves like regularization or like damage.

What we found

DyT's effect flips sign depending on the regime. At some combinations of model size and training-token count, it behaves like a helpful implicit regularizer, similar to how weight decay can prevent overfitting. At other combinations, it hurts training relative to LayerNorm. The controlling variable is the token-to-parameter ratio, not model size or data size in isolation.

We tested this across 5 GPT-2-family scales spanning 64M to 3.78B parameters, at both a 1M-token and a 118M-token regime, with additional checks on Llama-style models and Vision Transformers (ViT), plus RMSNorm and HardTanh baselines and attention-mechanism ablations (DiffAttn V1, a V2-inspired sigmoid-lambda variant, and gated-attention controls) to separate the normalization effect from other architecture changes.

What I would do before using DyT

I would run a small calibration grid before replacing LayerNorm in a real training run. At minimum, I would test the target model scale at a small and medium token budget, log validation loss and stability metrics, and compare against LayerNorm and RMSNorm rather than only against a single baseline. If the goal is speed or simplicity, the replacement still has to earn that change in the target regime.

The mistake would be to treat a normalization swap as a local code change. It changes the training dynamics. That can be useful, but it is not free.

Why "regime-dependent" is the headline, not "DyT wins" or "DyT loses"

This is an empirical regime study, not a theory paper and not a claim that one normalization choice is universally better. The practical takeaway for anyone considering DyT: check where your training run sits on the token-to-parameter axis before assuming the swap is free. A change that helps a small model trained on little data can hurt a larger model trained on more data, or vice versa.

Frequently asked questions

Links

@article{verma2026removinglayernorm,
  title={When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer},
  author={Verma, Lucky},
  journal={arXiv preprint arXiv:2604.23434},
  year={2026}
}