Gradient Descent as Natural Selection: A Dangerous Analogy
We keep reaching for evolutionary metaphors to explain how neural networks learn. The analogy is seductive and partially correct. The ways it breaks down are more interesting than the ways it holds.
The Analogy
Gradient descent optimizes a loss function by iteratively adjusting parameters in the direction that reduces loss. Natural selection optimizes fitness by iteratively selecting for traits that improve reproductive success. Both are hill-climbing algorithms (or valley-descending, depending on your framing). Both operate on populations of possibilities. Both produce complex, adapted structures without any designer.
The analogy is so compelling that it's become a cliché. And like most clichés, it obscures as much as it reveals.
Where It Holds
The deep similarity is this: both processes are blind optimizers. They don't know what they're optimizing for in any meaningful sense — they just follow the gradient. Evolution doesn't "want" to produce eyes; it produces eyes because eyes increase fitness in environments where light is informative. Gradient descent doesn't "want" to recognize cats; it adjusts weights because that reduces cross-entropy loss on the training set.
This blindness is a feature, not a bug. It's why both processes can discover solutions that no designer would have anticipated. The AlphaFold breakthrough (DeepMind, 2021) — solving the protein folding problem that had stumped biochemists for 50 years — is a striking example of optimization finding solutions in a space too large for human intuition.
Where It Breaks
Timescale: Evolution operates over millions of generations. Gradient descent operates over millions of parameter updates per hour. The rate difference is so extreme that qualitatively different dynamics emerge. Richard Sutton's famous Bitter Lesson (2019) argues that scale and compute, not clever algorithms, are what drive AI progress — which is more consistent with the evolutionary analogy than most AI researchers would like to admit.
Heritability: In evolution, traits are inherited with variation. In gradient descent, there's no inheritance — each update modifies the same parameter set. There's no population, no recombination, no genetic drift.
The fitness landscape problem: Evolution's fitness landscape changes as the environment changes and as other organisms evolve. The loss landscape in ML is fixed (given a fixed dataset and architecture). This is why evolutionary algorithms can sometimes escape local optima that gradient descent gets stuck in — and why neuroevolution (Kenneth Stanley et al.) remains an active research area.
The crucial difference: Evolution has no training set. It's optimizing against reality, not a proxy. This is why evolved organisms are robust in ways that trained models often aren't — they've been tested against the full complexity of the world, not a curated sample.
The Lesson
The analogy is useful for intuition but dangerous for engineering. When we treat ML systems as "evolved" intelligences, we import assumptions about robustness and generalization that aren't warranted. A model that achieves 99% accuracy on a benchmark hasn't been tested against reality — it's been tested against a sample of reality, which is a very different thing.
The ongoing debate about AI benchmarks and their limitations (see the HELM benchmark paper from Stanford, 2022) is really a debate about this gap. Nature doesn't have a test set. That's its advantage — and our problem.