Frequency-dependent selection

Frequency-dependent selection is a mode of natural selection in which the fitness conferred by a particular phenotype is not fixed but instead varies as a function of that phenotype's frequency in the population. Unlike directional selection, which consistently favours one end of a trait distribution, or stabilizing selection, which favours an intermediate optimum, frequency-dependent selection creates a dynamic fitness landscape that shifts as allele frequencies change from one generation to the next.¹³ This frequency sensitivity means that the selective value of a genotype depends not only on the environment in which an organism lives but also on the composition of the population surrounding it. The concept was first developed formally by Bryan Clarke in the early 1960s in the context of predator-prey interactions, where he demonstrated that predators could impose selection favouring rare prey morphs simply by forming search images for the most common forms.¹⁴

Frequency-dependent selection comes in two fundamentally different forms with opposing evolutionary consequences. Negative frequency-dependent selection, in which rare phenotypes enjoy a fitness advantage precisely because they are rare, acts as a stabilizing force that maintains genetic variation within populations. Positive frequency-dependent selection, in which common phenotypes are favoured, acts as a destabilizing force that drives populations toward fixation of a single variant.¹³ Of the two, negative frequency dependence has received far more attention from evolutionary biologists because of its remarkable power to explain the persistence of genetic diversity that would otherwise be eroded by drift and directional selection.

Negative frequency-dependent selection

Negative frequency-dependent selection occurs when the fitness of a phenotype increases as it becomes rarer in the population. The mechanism is straightforward in principle: whatever ecological or social process generates the fitness advantage does so more effectively when the favoured variant is uncommon, and less effectively as the variant becomes widespread. The result is a built-in regulatory mechanism—when an allele becomes too common, its fitness declines; when it becomes rare, its fitness rises—that pushes allele frequencies toward a stable internal equilibrium rather than allowing fixation of any single variant.¹¹

Clarke coined the term "apostatic selection" to describe one of the most intuitive examples of this process: predator-mediated selection on prey colour morphs.¹¹ When predators develop search images for the most common prey form, they disproportionately consume individuals of that morph. Rare morphs, which the predator has not learned to recognize efficiently, survive at higher rates. As a once-rare morph increases in frequency and becomes the predator's new target, its fitness advantage evaporates and shifts to whichever morph is now least common. This cyclical dynamic can maintain two, three, or more distinct colour morphs indefinitely within a single population, a pattern that would be difficult to explain under any other form of selection.¹⁴

The consequences of negative frequency-dependent selection extend well beyond colour polymorphisms. It is now recognized as one of the principal mechanisms responsible for maintaining balanced polymorphisms across the tree of life, from immune gene diversity in vertebrates to incompatibility alleles in flowering plants to behavioural strategies in lizards and fish.¹³ Wherever the rarity of a phenotype confers an advantage—whether through predator avoidance, pathogen resistance, mating success, or resource partitioning—negative frequency dependence can act as a powerful counterweight to the homogenizing forces of genetic drift and directional selection.

Positive frequency-dependent selection

Positive frequency-dependent selection produces the opposite pattern: the more common a phenotype becomes, the greater its fitness advantage. This form of selection tends to eliminate variation rather than preserve it, driving rare variants to extinction and pushing populations toward monomorphism.¹³ While less celebrated than its negative counterpart, positive frequency dependence plays an important role in a number of biological systems.

One of the best-understood examples involves Mullerian mimicry, in which multiple toxic or distasteful species converge on a shared warning signal. Each species benefits from the shared pattern because predators learn to avoid that signal more quickly when it is common. A rare, divergent warning pattern would be less effective because predators would encounter it too infrequently to learn the association between appearance and unpalatability. The result is strong positive frequency-dependent selection favouring the most common warning pattern within a community, which drives convergence among defended species and can eliminate variant colour forms.¹³

Positive frequency dependence also arises in mate recognition systems, where individuals with common signals may have an easier time finding compatible mates. In species that rely on acoustic, chemical, or visual signals for mate identification, rare signal variants may be ignored or rejected by potential partners, reducing the reproductive success of uncommon phenotypes. This process can reinforce reproductive isolation between diverging populations, contributing to speciation when populations with different signal frequencies come into secondary contact.¹³

MHC diversity and pathogen-mediated selection

The major histocompatibility complex (MHC) provides one of the most compelling examples of negative frequency-dependent selection in vertebrates. MHC molecules are cell-surface glycoproteins that present pathogen-derived peptides to T cells, initiating the adaptive immune response. The MHC genes are among the most polymorphic loci in vertebrate genomes: hundreds of alleles have been documented at individual loci in humans, and similarly extraordinary diversity exists in other mammals, birds, and fish.⁷ This diversity far exceeds what would be expected under neutral evolution or simple directional selection.

The prevailing explanation for MHC polymorphism centres on frequency-dependent interactions between hosts and pathogens. Pathogens are under strong selection to evade recognition by common MHC alleles, because those alleles represent the majority of the host population they encounter. Over time, pathogen populations evolve to escape the most frequent host genotypes, which reduces the fitness of individuals carrying those common alleles. Rare MHC alleles, by contrast, present peptide combinations that local pathogens have not adapted to evade, conferring a survival advantage on their bearers.^{6, 7} As rare alleles increase in frequency due to this advantage, they in turn become targets for pathogen counter-adaptation, and formerly common alleles that have become rare regain their protective value. This perpetual cycling of allele-specific pathogen adaptation and host counter-adaptation maintains extraordinary allelic diversity at MHC loci across generations.¹⁵

The MHC system also interacts with sexual selection through mate choice. Studies across multiple vertebrate species have demonstrated that individuals preferentially choose mates with dissimilar MHC genotypes, a behaviour that may function to produce offspring with broader pathogen resistance.⁸ This MHC-disassortative mating represents an additional mechanism through which frequency-dependent selection can maintain immunogenetic diversity, layering sexual selection on top of pathogen-mediated viability selection.

Classic examples in nature

The side-blotched lizard (Uta stansburiana) of western North America provides one of the most elegant demonstrations of frequency-dependent selection in the wild. Males of this species exhibit three distinct throat colours—orange, blue, and yellow—that correspond to three different mating strategies. Orange-throated males are highly aggressive and maintain large territories, dominating blue-throated males, which defend smaller territories with single mates. Yellow-throated males are "sneakers" that mimic females and infiltrate the territories of orange males to copulate surreptitiously. However, blue males, which guard their mates closely, are resistant to the sneaker strategy of yellow males. The result is a rock-paper-scissors dynamic: orange beats blue, blue beats yellow, and yellow beats orange.³ Sinervo and Lively demonstrated that the frequencies of these three morphs cycle over a period of roughly six years, with each morph gaining a fitness advantage when rare and losing it when common—a textbook case of negative frequency-dependent selection maintaining a stable trimorphism.³

A strikingly parallel example occurs in the scale-eating cichlid Perissodus microlepis of Lake Tanganyika. Individuals of this species have asymmetric mouths, opening either to the left or to the right, which determines the side from which they approach prey fish to scrape scales. Hori demonstrated that the frequency of left-mouthed and right-mouthed morphs oscillates around a 1:1 ratio in the wild population.⁴ When one morph becomes common, prey fish become more vigilant against attacks from that side, giving the rarer morph higher foraging success. This frequency-dependent advantage drives the population back toward equal frequencies, maintaining the mouth-handedness polymorphism indefinitely.⁴

Batesian mimicry is inherently frequency-dependent. A harmless mimic gains protection by resembling a dangerous or unpalatable model species, but this protection erodes as the mimic becomes more common relative to the model. When mimics are rare, predators that have learned to avoid the model extend that avoidance to the few mimics they encounter. As mimics increase in frequency, predators encounter unpunished attacks more often, weaken their learned avoidance, and begin to attack both mimics and models indiscriminately.⁵ Pfennig, Harcombe, and Pfennig confirmed this prediction experimentally, demonstrating that the survival advantage of coral snake mimics (the nonvenomous scarlet kingsnake, Lampropeltis elapsoides) was greatest in areas where the model (the venomous eastern coral snake, Micrurus fulvius) was common relative to the mimic, and weakest or absent where models were rare.⁵

Self-incompatibility in plants

Self-incompatibility (SI) systems in flowering plants represent another canonical example of negative frequency-dependent selection. In gametophytic SI, a pollen grain is rejected if it carries the same S-allele as the pistil it lands on, preventing self-fertilization and mating between close relatives. The frequency-dependent advantage of rare S-alleles is immediate and powerful: a pollen grain carrying a rare S-allele will be compatible with a larger fraction of the female genotypes in the population than a pollen grain carrying a common allele, because fewer pistils share its incompatibility type.⁹

This strong negative frequency dependence generates some of the most extreme allelic diversity observed at any locus in nature. The evening primrose (Oenothera organensis) maintains over 50 S-alleles in a population of only a few hundred individuals, and some populations of Trifolium (clover) harbour over 200 S-alleles.⁹ Coalescent analyses have shown that many S-allele lineages predate the divergence of their host species, with some allelic lineages in the Solanaceae estimated to be over 30 million years old.¹⁰ This "trans-specific polymorphism"—in which allelic lineages persist across speciation events—is a hallmark of ancient balancing selection and provides some of the strongest molecular evidence for the long-term maintenance of variation by negative frequency-dependent selection.

Mathematical framework

The simplest mathematical treatment of frequency-dependent selection modifies the standard population genetic model of selection at a single locus by making fitnesses functions of allele or genotype frequencies rather than constants. In a two-allele system with alleles A and a at frequencies p and q = 1 – p, the fitnesses of the three genotypes AA, Aa, and aa can be written as w_AA(p), w_Aa(p), and w_aa(p). Under negative frequency dependence, the fitness of AA decreases as p increases, and the fitness of aa decreases as q increases. This generates an internal equilibrium frequency p* at which the marginal fitnesses of the two alleles are equal, and this equilibrium is stable: perturbations away from p* produce selection that pushes the population back toward it.¹³

The dynamics become richer with three or more strategies, as illustrated by the side-blotched lizard system. In such cases, the appropriate mathematical framework is the replicator equation from evolutionary game theory, which describes how the frequency of each strategy changes over time as a function of its payoff relative to the population mean payoff.¹ For an n-strategy system, the rate of change in the frequency of strategy i is proportional to the difference between its fitness and the mean population fitness, where each fitness value is itself a function of all strategy frequencies. The equilibria of this system correspond to the Nash equilibria of the underlying game, and a strategy that cannot be invaded by any rare mutant is termed an evolutionarily stable strategy (ESS).²

Connection to evolutionary game theory

Frequency-dependent selection provides the biological foundation for evolutionary game theory, a framework introduced by Maynard Smith and Price in 1973.² In classical population genetics, the fitness of a genotype is typically treated as a property of the genotype itself, determined by its interaction with the physical environment. Game theory recognizes that in many biological contexts—contests over mates, territory, food, or other resources—the payoff of a behavioural strategy depends critically on what strategies other individuals are playing. This is precisely the condition that defines frequency-dependent selection.¹

The central concept of evolutionary game theory is the evolutionarily stable strategy, a strategy that, if adopted by most members of a population, cannot be invaded by any rare alternative strategy. The ESS concept formalizes the intuition that frequency-dependent selection can drive populations to equilibria where no mutant can gain a foothold. In the classic hawk-dove game, for instance, pure "hawk" (always fight) and pure "dove" (always retreat) are not individually stable; a population of all hawks can be invaded by doves, and a population of all doves can be invaded by hawks. The ESS is a mixed strategy or a stable polymorphism in which hawks and doves coexist at a frequency determined by the costs and benefits of fighting.^{1, 2}

Nowak and Sigmund extended this framework to show that frequency-dependent dynamics underpin a wide range of coevolutionary phenomena, from the evolution of cooperation and reciprocal altruism to the maintenance of costly punishment in social groups.¹² In each case, the fitness of a strategy is not intrinsic but contextual, rising or falling depending on the strategies prevalent in the surrounding population. This insight—that fitness is fundamentally relational—represents one of the most important conceptual advances in evolutionary biology since the modern synthesis, and frequency-dependent selection is the mechanism that makes it possible.

Evolutionary significance

Frequency-dependent selection occupies a central place in evolutionary biology because it resolves a fundamental puzzle: why natural populations maintain so much genetic variation. Classical models of natural selection predict that selection should erode variation by fixing the fittest allele, while genetic drift erodes variation by random loss. Yet natural populations are often strikingly polymorphic, harbouring multiple alleles, morphs, and strategies that persist over evolutionary timescales. Negative frequency-dependent selection provides a general mechanism for maintaining this variation, one that operates through a simple and elegant principle—be rare, and you will be favoured.^{11, 13}

The scope of frequency-dependent selection in nature is likely much broader than the well-studied examples suggest. Any ecological interaction in which the payoff depends on the relative abundance of different types—host-pathogen dynamics, pollinator competition, social foraging, warning coloration, mating systems—has the potential to generate frequency-dependent fitness effects. As genomic tools make it increasingly possible to detect the molecular signatures of balancing selection across entire genomes, the number of loci identified as targets of frequency-dependent selection continues to grow.^{7, 10} Far from being a special case applicable to a handful of textbook examples, frequency-dependent selection appears to be a pervasive force shaping the genetic architecture of natural populations, one whose full importance evolutionary biology is only beginning to appreciate.