The wikipedia page also has an article on large language models[0] that includes a section on emergent behaviour.

> While it is generally the case that performance of large models on various tasks can be extrapolated based on the performance of similar smaller models, sometimes large models undergo a "discontinuous phase shift" where the model suddenly acquires substantial abilities not seen in smaller models. These are known as "emergent abilities", and have been the subject of substantial study. Researchers note that such abilities "cannot be predicted simply by extrapolating the performance of smaller models".

[0]: https://en.wikipedia.org/wiki/Large_language_model

The reference/source to that Wikipedia paragraph is: https://openreview.net/forum?id=yzkSU5zdwD

The owner of the link is listed on that page, so I'm guessing it's no coincidence.

To me, it seems like a good definition.

If scaling the size of the model leads to a linear increase in capability, then nothing is emergent.

If scaling the size of the model accretes almost no gains, and then there is a huge unpredicted step in capabilities, a capability has "emerged".

I thought the thing that is scaring people is that there isn’t a defined relationship between model scale and capabilities. You may triple the model size and get no increase but then quadruple it and get 10,000% increase in capability. So no one knows where these massive models are headed and people want to pause until they understand the relationship.

Emergence doesn’t need to depend on scale. For example, gliders in the Game of Life are emergent behavior, and they don’t depend on scale. The way proteins fold is emergent from the physical forces, but is not a function of scale, just of particular configurations. Emergence is about the level on which an explanation or description works. You only “see” the glider move at a higher level of description than the cellular rules.

hmm...yes that's perhaps better. Defining it as a bigger improvement than would be expected from the size increase.

Don't think the baseline expectation would be linear per se though

an Emergent behavior is usually defined as a behavior that is not expected from the training or the starting configuration. Their Emergent Definition is absurd. is like saying that understanding Calculus is emergent behavior in 12 grade because in was not present in 3rd grade.

The problem isn't that they should have defined "emergent" better, it's that they should have used a different word that actually means what they intend to communicate, such as "size-based" or "scale-dependent".