Hybrid speciation
Hybrid speciation is a form of speciation where hybridization between two different species leads to a new species, reproductively isolated from the parent species. Previously, reproductive isolation between hybrids and their parents was thought to be particularly difficult to achieve, and thus hybrid species were thought to be extremely rare. With DNA analysis becoming more accessible in the 1990s, hybrid speciation has been shown to be a fairly common phenomenon, particularly in plants. In botanical nomenclature, a hybrid species is also called a nothospecies. Hybrid species are by their nature polyphyletic.

Ecology

A hybrid may occasionally be better fitted to the local environment than the parental lineage, and as such, natural selection may favor these individuals. If reproductive isolation is subsequently achieved, a separate species may arise. Reproductive isolation may be genetic, ecological, behavioral, spatial, or a combination of these.

If reproductive isolation fails to establish, the hybrid population may merge with either or both parent species. This will lead to an influx of foreign genes into the parent population, a situation called an introgression. Introgression is a source of genetic variation, and can in itself facilitate speciation. There is evidence that introgression is a ubiquitous phenomenon in plants and animals, even in humans, where genetic material from Neanderthals and Denisovans is responsible for much of the immune genes in non-African populations.

Ecological constraints
For a hybrid form to persist, it must be able to exploit the available resources better than either parent species, which, in most cases, it will have to compete with. While grizzly bears and polar bears may have offspring, a grizzly–polar bear hybrid will likely be less suited in either of the ecological roles than the parents themselves. Although the hybrid is fertile, this poor adaptation would prevent the establishment of a permanent population.

Likewise, lions and tigers have historically overlapped in a portion of their range and can theoretically produce wild hybrids: ligers, which are a cross between a male lion and female tiger, and tigons, which are a cross between a male tiger and a female lion; however, tigers and lions have thus far only hybridized in captivity. In both ligers and tigons, the females are fertile and the males are sterile. One of these hybrids (the tigon) carries growth-inhibitor genes from both parents and thus is smaller than either parent species and might in the wild come into competition with smaller carnivores, e.g. the leopard. The other hybrid, the liger, ends up larger than either of its parents: about a thousand pounds (450 kilograms) fully grown.  No tiger-lion hybrids are known from the wild, particularly because each species is confined to geographically separated ranges (tigers are not found in Africa, and the Asiatic lion is only found in the Gir Forest National Park, where tigers also are absent).

Some situations may favor hybrid population. One example is rapid turnover of available environment types, like the historical fluctuation of water level in Lake Malawi, a situation that generally favors speciation.  A similar situation can be found where closely related species occupy a chain of islands. This will allow any present hybrid population to move into new, unoccupied habitats, avoiding direct competition with parent species and giving a hybrid population time and space to establish. Genetics, too, can occasionally favor hybrids. In the Amboseli National Park in Kenya, yellow baboons and anubis baboons regularly interbreed. The hybrid males reach maturity earlier than their pure-bred cousins, setting up a situation where the hybrid population may over time replace one or both of the parent species in the area.

Genetics of hybridization
Genetics are more variable and malleable in plants than in animals, probably reflecting the higher activity level in animals. Hybrids' genetics will necessarily be less stable than those of species evolving through isolation, which explains why hybrid species appear more common in plants than in animals. Many agricultural crops are hybrids with double or even triple chromosome sets. Having multiple sets of chromosomes is called polyploidy. Polyploidy is usually fatal in animals where extra chromosome sets upset fetal development, but is often found in plants. A form of hybrid speciation that is relatively common in plants occurs when an infertile hybrid becomes fertile after doubling of the chromosome number.

Hybridization without change in chromosome number is called homoploid hybrid speciation. This is the situation found in most animal hybrids. For a hybrid to be viable, the chromosomes of the two organisms will have to be very similar, i.e., the parent species must be closely related, or else the difference in chromosome arrangement will make mitosis problematic. With polyploid hybridization, this constraint is less acute.

Super-numerary chromosome numbers can be unstable, which can lead to instability in the genetics of the hybrid. The European edible frog appears to be a species, but is actually a triploid semi-permanent hybrid between pool frogs and marsh frogs. In most populations, the edible frog population is dependent on the presence of at least one of the parent species to be maintained, as each individual need two gene sets from one parent species and one from the other. Also, the male sex determination gene in the hybrids is only found in the genome of the pool frog, further undermining stability. Such instability can also lead to rapid reduction of chromosome numbers, creating reproductive barriers and thus allowing speciation.

Known cases

Animals

Homoploid hybrid speciation

Hybrid speciation in animals is primarily homoploid. While thought not to be very common, a few animal species are the result of hybridization, mostly insects such as tephritid fruitflies that inhabit  Lonicera plants and Heliconius butterflies, as well as some fish, one marine mammal, the clymene dolphin, and a few birds.

One bird is an unnamed form of Darwin's finch from the Galapagos island of Daphne Major, described in 2017 and likely founded in the early 1980s by a male Española cactus finch from Española Island and a female medium ground finch from Daphne Major. Another is the great skua, which has a surprising genetic similarity to the physically very different pomarine skua; most ornithologists now assume it to be a hybrid between the pomarine skua and one of the northern skuas. The golden-crowned manakin was formed 180,000 years ago by hybridization between snow-capped and opal-crowned manakins.

Multiple hybrids during rapid divergence

Rapidly diverging species can sometimes form multiple hybrid species, giving rise to a species complex, like several physically divergent but closely related genera of cichlid fishes in Lake Malawi. The duck genus Anas (mallards and teals) has a very recent divergence history, many of the species are inter-fertile, and quite a few of them are thought to be hybrids. While hybrid species generally appear rare in mammals, the American red wolf appears to be a hybrid species of the Canis species complex, between gray wolf and coyote. Hybridization may have led to the species-rich Heliconius butterflies, though this conclusion has been criticized.

Plants

Since plants are more tolerant of polyploidy, hybrid species are more common than in animals. Estimates indicate as many as 2–4% of all flowering plants and 7% of all fern species are the results of polyploid hybridization. Many crop species such as wheat are hybrids, and hybridization is an important factor in speciation in some plant groups. Garden flowers in the genus Saxifraga are often hybrids, and a tetraploid natural hybrid, Saxifraga osloenis, is thought to have formed at the end of the last ice age.. Homoploid speciation also occurs in plants, and has for example given rise to several species of sunflower.