Draft:Grogging
Submission declined on 15 September 2026 by ChrysGalley (talk). This draft's references do not show that the neologism meets Wikipedia's criteria for inclusion. The draft requires multiple published secondary sources that:
Please add references that meet all three of these criteria. If none exist, the subject is not yet suitable for Wikipedia. Wikipedia is not a dictionary and we do not accept articles that are simply definitions of neologisms.
Where to get help
How to improve a draft
You can also browse Wikipedia:Featured articles and Wikipedia:Good articles to find examples of Wikipedia's best writing on topics similar to your proposed article. Improving your odds of a speedy review To improve your odds of a faster review, tag your draft with relevant WikiProject tags using the button below. This will let reviewers know a new draft has been submitted in their area of interest. For instance, if you wrote about a female astronomer, you would want to add the Biography, Astronomy, and Women scientists tags. Editor resources
|  |
Comment: Clearly AI assisted. ChrysGalley (talk) 08:48, 15 September 2026 (UTC)
Template:Infobox machine learning
Grokking is a phenomenon observed in machine learning in which a model initially memorizes the training data and shows poor generalization, but after extended training suddenly transitions to a state of high test performance. The term became widely used in 2021–2022 following research on small transformer models trained on algorithmic tasks such as modular arithmetic.[1]
The name derives from the verb to grok, introduced by Robert A. Heinlein in Stranger in a Strange Land (1961), meaning “to understand something profoundly.”[2]
Description
[edit]Grokking typically appears in settings where the dataset is small or synthetic, the task has an underlying algebraic structure, and the model has sufficient capacity to memorize the training set. During early training, the model achieves near‑perfect accuracy on the training data but poor accuracy on the test set. After prolonged optimization, the model abruptly discovers a more generalizable internal representation, leading to a sudden improvement in test performance.[1]
This transition is often described as a “phase change” in validation accuracy.
Proposed mechanisms
[edit]Several hypotheses have been proposed to explain grokking:
- Representation phase transition: the model shifts from a memorization-based solution to a structure-based solution.[3]
- Implicit regularization: optimization dynamics such as stochastic gradient descent may favor simpler or more generalizable solutions over time.
- Weight decay effects: strong regularization can encourage the model to abandon memorized patterns and discover algorithmic structure.[4]
- Emergence of sparse internal circuits: studies of small transformers show interpretable circuits for arithmetic operations.
Examples
[edit]Grokking has been documented in tasks such as:
- modular addition and multiplication,
- parity and logical operations,
- synthetic algorithmic datasets,
- small-scale transformer models trained on structured patterns.
In many cases, grokking occurs only after thousands of additional training steps beyond the point where the model has already overfit.
Research history
[edit]The phenomenon was first widely discussed after a 2021 OpenAI post analyzing small transformer models trained on modular arithmetic.[1] Subsequent work by interpretability groups expanded the concept and provided mechanistic explanations for the delayed generalization transition.
See also
[edit]References
[edit]- 1 2 3 "Grokking: Generalization Beyond Overfitting". OpenAI. 2021. Retrieved 2026-09-15.
- ↑ Heinlein, Robert A. (1961). Stranger in a Strange Land. G. P. Putnam's Sons.
- ↑ "Mechanistic Interpretability: Modular Addition". Anthropic. 2022.
- ↑ Power, Alethea; Burda, Yuri; Edwards, Harri; Babuschkin, Igor; Misra, Vedant (2022). "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets". arXiv:2201.02177 [cs.LG].
Category:Machine learning Category:Artificial intelligence Category:Neural networks Category:Deep learning
Please delete the portions which were LLM-generated, and summarize in your own words a range of independent, reliable, published sources that discuss the subject. If you have used an LLM but you believe that your use is covered by one of the two exceptions (basic copyediting and translation between two languages in which you are fully fluent), please provide a transcript of the prompt(s) you used while writing your draft (see the how-to guide). Reviewers can make mistakes. If you have not received any LLM assistance, you may seek a second opinion at the AfC Help Desk.
See the advice page on large language models for more information.