Llama — Field Evidence

GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries — Transformers are known to have internal continuous symmetries that leave outputs invariant, while modifying quantization. GaugeQuant leverages this in-training by introducing a LogSumExp term to the l