Abstract
This paper audits a transferred top-1-frequency collapse warning for LoRA fine-tuning of masked diffusion language models. In the reported cohorts, the top-1 warning fires broadly while observed collapse is zero at the audited horizons, motivating family-local monitoring with max LoRA gradient norm instead.
TL;DR: A widely-used warning signal for LoRA fine-tuning of masked diffusion language models fires on every tested run, whether or not the model is actually collapsing. We recommend watching max LoRA gradient norm instead, and checking any transferred monitor against the specific model family before trusting it.
If you're skimming
- Question: can a collapse warning from one LoRA setting be trusted for masked diffusion language models?
- Finding: in the audited cohorts, top-1 token frequency fires on every run while measured collapse is zero at the audited horizons.
- Replacement signal: max LoRA gradient norm stays useful after top-1 frequency has already saturated.
- Limit: this is a monitor-transfer result, not a claim about every possible top-1 use in every model family.
The problem
When you fine-tune a large model with LoRA, you want an early-warning signal that tells you something has gone wrong before you waste a full training run. One common signal watches the "top-1 token frequency": if the model starts predicting the same token over and over, that usually means training has collapsed.
That heuristic was built and validated in one setting. This paper asks a simple question: does it still work when you move it to masked diffusion language models (DLMs), a different model family with a different training objective?
What we found
Across the DLM cohorts tested, the top-1 warning fires on every configuration. If you trusted it literally, you would conclude every run was collapsing.
But actual collapse, measured directly, was zero at the training horizons we audited. The warning was a false alarm, consistently. We treat this as a pre-equilibrium artifact of how top-1 frequency behaves early in DLM-LoRA training, not evidence of a real failure.
How to read the figure
The left panel is the failure of the transferred detector: the top-1 warning crosses the threshold so early and so broadly that it stops separating ordinary training from real collapse. The middle panel shows max LoRA gradient separation, which keeps moving after the top-1 signal has saturated. The right panel is the practical check: the replacement signal gives better precision for the audited failure mode.
If you only remember one thing from the figure, remember the mismatch: a useful training monitor should reduce uncertainty. In these DLM-LoRA runs, the transferred top-1 warning increases it.
What to use instead
Max LoRA gradient norm, monitored per model family, gave a more useful signal in every tested cohort. It does not saturate the same way, so it keeps distinguishing healthy runs from unhealthy ones past the point where the top-1 warning has already maxed out.
What I would do in a training pipeline
I would not delete top-1 logging. I would demote it from "stop the run" to "debug context" unless it has been recalibrated for the model family. For DLM-LoRA work, I would log max LoRA gradient norm by layer or adapter group, set thresholds from a small family-local calibration sweep, and keep the raw top-1 trace beside it so postmortems can still see when token concentration started.
The boring part matters here: a monitor is an operational decision rule, not just a plotted statistic. If the rule pages you at 2 a.m. or kills a job, it needs evidence in the setting where it will be used.
Why this matters beyond this one warning
The broader point is not about this specific metric. It's that a monitoring heuristic validated in one model family is not automatically valid in another, even when the two families look superficially similar. Before wiring an early-warning signal into a training pipeline for a new architecture, it's worth testing whether the signal actually correlates with the failure it claims to detect in that setting.
What is public
- Paper on arXiv with the full framing, cohorts, and result tables.
- GitHub repository with reference logging/checking scripts and sanitized aggregate artifacts.
- Publication entry that keeps the paper, DOI, code, and this project page linked from one canonical page.
Frequently asked questions
What does "top-1 fails" mean in the title?
It refers to the top-1 token frequency warning. We show it fires on every tested masked-diffusion-LM configuration even when no collapse is actually happening at the audited horizons.
What should I watch instead?
Max LoRA gradient norm, tracked per model family. It behaved as a more informative signal for actual training health in every cohort we tested.
Does this mean top-1 monitoring is broken in general?
No. We only tested transferring the warning into masked diffusion LM LoRA fine-tuning. A heuristic built for a different setting does not automatically transfer - check any transferred monitor against your specific model family first.
Is the code and data public?
Yes: reference logging/checking scripts and sanitized aggregate result artifacts on GitHub, paper on arXiv.
Links
BibTeX
@article{verma2026top1fails,
title={When Top-1 Fails: Calibrating LoRA Monitors for Masked Diffusion LMs},
author={Verma, Lucky and Yadav, Pratik},
journal={arXiv preprint arXiv:2606.24119},
year={2026}
}