Coalescent Models

The standard neutral coalescent model and its extensions to include changes in population size over time and population structure are reviewed. Gene genealogies are shown to provide the hidden structure behind patterns of genetic variation. Expressions for expected levels of genetic variation are presented and explained, and tests of the standard neutral model based on the frequencies of mutations at single-nucleotide sites (aka “site frequencies”) are outlined. Several examples of deviations from the standard model are discussed, and their effects on expected site frequencies are illustrated. Some attention is given to the fact that coalescent theory has not fully grappled with the existence of underlying population pedigrees.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Buy Now

eBook EUR 64.19 Price includes VAT (France)

Hardcover Book EUR 79.11 Price includes VAT (France)

Tax calculation will be finalised at checkout

Purchases are for personal use only

Similar content being viewed by others

Chance, Variation and Shared Ancestry: Population Genetics After the Synthesis

The IICR and the non-stationary structured coalescent: towards demographic inference with arbitrary changes in population structure

Evolution within Populations

References

• Achaz G (2009) Frequency spectrum neutrality tests: one for all and all for one. Genetics 183:249–258 ArticlePubMedPubMed CentralGoogle Scholar
• Alvarado-Serrano DF, Hickerson MJ (2016) Spatially explicit summary statistics for historical population genetic inference. Methods Ecol Evol 7:418–427 ArticleGoogle Scholar
• Alvarez G, Ceballos FC, Quinteiro C (2009) The role of inbreeding in the extinction of a European royal dynasty. PLoS One 4(4):e5174 ArticlePubMedPubMed CentralCASGoogle Scholar
• Beaumont MA (2010) Approximate Bayesian computation in evolution and ecology. Annu Rev Ecol Evol Syst 41:379–406 ArticleGoogle Scholar
• Beerli P (2006) Comparison of Bayesian and maximum-likelihood inference of population genetic parameters. Bioinformatics 22:341–345 ArticleCASPubMedGoogle Scholar
• Bycro C et al (2018) The UK biobank resource with deep phenotyping and genomic data. Nature 562:203–209 ArticleCASGoogle Scholar
• Cannings C (1974) The latent roots of certain Markov chains arising in genetics: a new approach. I. Haploid models. Adv Appl Probab 6:260–290 ArticleGoogle Scholar
• Cannings C, Thompson EA, Skolnick MH (1978) Probability functions on complex pedigrees. Adv Appl Probab 10:26–61 ArticleGoogle Scholar
• Cargill M, Altshuler D, Ireland J, Sklar P, Ardlie K, Patil N, Lane CR, Lim EP, Kalyanaraman N, Nemesh J, Ziaugra L, Friedland L, Rolfe A, Warrington J, Lipshutz R, Daly GQ, Lander ES (1999) Characterization of single-nucleotide polymorphisms in coding regions of human genes. Nat Genet 22:231–237 ArticleCASPubMedGoogle Scholar
• Chang JT (1999) Recent common ancestors of all present-day individuals. Adv Appl Probab 31:1002–1026 ArticleGoogle Scholar
• de Iorio M, Griffiths RC, Leblois R, Rousset F (2005) Stepwise mutation likelihood computation by sequential importance sampling in subdivided population models. Theoret Pop Biol 68:41–53 ArticleGoogle Scholar
• Donnelly P, Tavaré S (1995) Coalescents and genealogical structure under neutrality. Annu Rev Genet 29:401–421 ArticleCASPubMedGoogle Scholar
• Drummond AJ, Suchard MA, Xie D, Rambaut A (2012) Bayesian phylogenetics with BEAUti and the BEAST 1.7. Mol Biol Evol 29:1969–1973 ArticleCASPubMedPubMed CentralGoogle Scholar
• Durrett R, Schweinsberg J (2004) Approximating selective sweeps. Theoret Pop Biol 66:129–138 ArticleGoogle Scholar
• Etheridge AM, Pfaffelhuber P, Wakolbinger A (2006) An approximate sampling formula under genetic hitchhiking. Ann Appl Probab 16:685–729 ArticleGoogle Scholar
• Ewens WJ (1972) The sampling theory of selectively neutral alleles. Theoret Pop Biol 3:87–112 ArticleCASGoogle Scholar
• Ewens WJ (1974) A note on the sampling theory for infinite alleles and infinite sites models. Theoret Pop Biol 6:143–148 ArticleCASGoogle Scholar
• Ewens WJ (1990) Population genetics theory—the past and the future. In: Lessard S (ed) Mathematical and statistical developments of evolutionary theory. Kluwer Academic, Amsterdam, pp 177–227 ChapterGoogle Scholar
• Ewens WJ (2004) Mathematical population genetics, vol I: theoretical foundations. Springer, Berlin BookGoogle Scholar
• Fay JC, Wu C-I (2000) Hitchhiking under positive Darwinian selection. Genetics 155:1405–1413 ArticleCASPubMedPubMed CentralGoogle Scholar
• Fearnhead P (2006) Perfect simulation from nonneutral population genetic models: variable population size and population subsdivision. Genetics 174:1397–1406 ArticlePubMedPubMed CentralGoogle Scholar
• Felsenstein J (2006) Accuracy of coalescent likelihood estimates: do we need more sites, more sequences, or more loci? Mol Biol Evol 23:691–700 ArticleCASPubMedGoogle Scholar
• Ferretti L, Perez-Enciso M, Ramos-Onsins S (2010) Optimal neutrality tests based on the frequency spectrum. Genetics 186:353–365 ArticlePubMedPubMed CentralGoogle Scholar
• Fisher RA (1930) The genetical theory of natural selection. Clarendon, Oxford BookGoogle Scholar
• Fu Y-X (1995) Statistical properties of segregating sites. Theoret Pop Biol 48:172–197 ArticleCASGoogle Scholar
• Fu Y-X (1997) Statistical tests of neutrality of mutations against population growth, hitchhiking and background selection. Genetics 147:915–925 ArticleCASPubMedPubMed CentralGoogle Scholar
• Fu Y-X, Li W-H (1993) Statistical tests of neutrality of mutations. Genetics 133:693–709 ArticleCASPubMedPubMed CentralGoogle Scholar
• Griffiths RC, Tavaré S (1994) Simulating probability distributions in the coalescent. Theoret Pop Biol 46:131–159 ArticleGoogle Scholar
• Griffiths RC, Tavaré S (1996) Monte Carlo inference methods in population genetics. Math Comput Modelling 23:141–158 ArticleGoogle Scholar
• Hanski I, Gaggiotti OE (2004) Ecology, genetics, and evolution of metapopulations. Elsevier Academic, London Google Scholar
• Harris K (2019) From a database of genomes to a forest of evolutionary trees. Nat Genet 51:1304–1307 ArticleCASGoogle Scholar
• Hasegawa M, Kishino H, Yano H (1985) Dating of the human-ape splitting by a molecular clock of mitochondrial DNA. J Mol Evol 22:160–174 ArticleCASPubMedGoogle Scholar
• Hawks J. “Coalescent Gene Genealogies” from the Wolfram Demonstrations Project. http://demonstrations.wolfram.com/CoalescentGeneGenealogies/
• Hein J, Schierup MH, Wiuf C (2005) Gene genealogies, variation and evolution: a primer in coalescent theory. Oxford University Press, Oxford Google Scholar
• Herbots HM (1997) The structured coalescent. In: Donnelly P, Tavaré S (eds) Progress in population genetics and human evolution, IMA volumes in mathematics and its applications, vol 87. Springer, New York, pp 231–255 Google Scholar
• Hey J (2010) Isolation with migration models for more than two populations. Mol Biol Evol 27:905–920 ArticleCASPubMedGoogle Scholar
• Hey J, Nielsen R (2004) Multilocus methods for estimating population sizes, migration rates and divergence time, with applications to the divergence of Drosophila pseudoobscura and D. persimilis. Genetics 167:747–760 ArticleCASPubMedPubMed CentralGoogle Scholar
• Hey J, Nielsen R (2007) Integration within the Felsenstein equation for improved Markov chain Monte Carlo methods in population genetics. Proc Natl Acad Sci U S A 104:2785–2790 ArticleCASPubMedPubMed CentralGoogle Scholar
• Hochman A (2019) Race and reference. Biology & Philosophy 34:32 ArticleGoogle Scholar
• Hodgkinson A, Eyre-Walker A (2010) Human triallelic sites: evidence for a new mutational mechanism? Genetics 184:233–241 ArticleCASPubMedPubMed CentralGoogle Scholar
• Hudson RR (1983) Testing the constant-rate neutral allele model with protein sequence data. Evolution 37:203–217 ArticlePubMedGoogle Scholar
• Hudson RR (1990) Gene genealogies and the coalescent process. In: Futuyma DJ, Antonovics J (eds) Oxford surveys in evolutionary biology, vol 7. Oxford University Press, Oxford, pp 1–44 Google Scholar
• Huff CD, Xing J, Rogers AR, Witherspoon D, Jorde LB (2010) Mobile elements reveal small population size in the ancient ancestors of Homo sapiens. Proc Natl Acad Sci USA 107:2147–2152 ArticleCASPubMedPubMed CentralGoogle Scholar
• Keinan A, Clark AG (2012) Recent explosive human population growth has resulted in an excess of rare genetic variants. Science 336:740–743 ArticleCASPubMedPubMed CentralGoogle Scholar
• Kelleher J, Wong Y, Wohns AW, Fadil C, Albers PK, McVean G (2019) Inferring whole-genome histories in large population datasets. Nat Genet 51:1330–1338 ArticleCASPubMedPubMed CentralGoogle Scholar
• Kingman JFC (1982a) On the genealogy of large populations. J Appl Probab 19A:27–43 ArticleGoogle Scholar
• Kingman JFC (1982b) The coalescent. Stoch Process Appl 13:235–248 ArticleGoogle Scholar
• Kingman JFC (1982c) Exchangeability and the evolution of large populations. In: Koch G, Spizzichino F (eds) Exchangeability in probability and statistics. North-Holland, Amsterdam, pp 97–112 Google Scholar
• Ko A, Nielsen R (2019) Joint estimation of pedigrees and effective population size using Markov chain Monte Carlo. Genetics 212:855–868 ArticlePubMedPubMed CentralGoogle Scholar
• Kuhner MK (2006) LAMARC 2.0: maximum likelihood and Bayesian estimation of population parameters. Bioinformatics 22:768–770 ArticleCASPubMedGoogle Scholar
• Kuhner MK, Yamato J, Felsenstein J (1995) Estimating effective population size and mutation rate from sequence data using Metropolis-Hastings sampling. Genetics 140:1421–1430 ArticleCASPubMedPubMed CentralGoogle Scholar
• Leffler EM, Bullaughey K, Matute DR, Meyer WK, Ségurel L, Venkat A, Andolfatto P, Przeworski M (2012) Revisiting an old riddle: what determines genetic diversity levels within species? PLoS Biol 10(9):e1001388 ArticleCASPubMedPubMed CentralGoogle Scholar
• Li W-H (1976) Distribution of nucleotide difference between two randomly chosen cistrons in a subdivided population: the finite island model. Theoret Pop Biol 10:303–308 ArticleCASGoogle Scholar
• Li H, Durbin R (2011) Inference of population history from individual whole-genome sequences. Nature 475:493–496 ArticleCASPubMedPubMed CentralGoogle Scholar
• Li N, Stephens M (2003) Modeling linkage disequilibrium and identifying recombination hotspots using single-nucleotide polymorphism data. Genetics 165:2213–2233 ArticleCASPubMedPubMed CentralGoogle Scholar
• Malécot G (1946) La consaguinite dans une population limitee. Comp Rendus Acad Sci Paris 222:841–843 Google Scholar
• Mallick S et al (2016) The Simons genome diversity project: 300 genomes from 142 diverse populations. Nature 538:201–206 ArticleCASPubMedPubMed CentralGoogle Scholar
• Möhle M (1998a) Robustness results for the coalescent. J Appl Probab 35:438–447 ArticleGoogle Scholar
• Möhle M (1998b) A convergence theorem for Markov chains arising in population genetics and the coalescent with partial selfing. Adv Appl Probab 30:493–512 ArticleGoogle Scholar
• Möhle M (1998c) Coalescent results for two-sex population models. Adv Appl Probab 30:513–520 ArticleGoogle Scholar
• Möhle M (1999) The concept of duality and applications to Markov processes arising in neutral population genetics models. Bernoulli 5:761–777 ArticleGoogle Scholar
• Möhle M, Sagitov S (2001) A classification of coalescent processes for haploid exchangeable population models. Ann Appl Probab 29:1547–1562 Google Scholar
• Notohara M (1990) The coalescent and the genealogical process in geographically structured population. J Math Biol 9:59–75 Google Scholar
• Ott J (1999) Analysis of human genetic linkage, 3rd edn. Johns Hopkins University Press, Baltimore Google Scholar
• Pfaffelhuber P, Wakolbinger A (2005) The process of most recent common ancestors in an evolving coalescent. Stoch Proc App 116:1836–1859 ArticleGoogle Scholar
• Pluzhnikov A, Donnelly P (1996) Optimal sequencing strategies for surveying molecular genetic diversity. Genetics 144:1247–1262 ArticleCASPubMedPubMed CentralGoogle Scholar
• Polanski A, Kimmel M (2003) New explicit expressions for relative frequencies of single-nucleotide polymorphisms with application to statistical inference on population growth. Genetics 165:427–436 ArticleCASPubMedPubMed CentralGoogle Scholar
• Rauch EM, Bar-Yam Y (2004) Theory predicts the uneven distribution of genetic diversity within species. Nature 431:449–452 ArticleCASPubMedGoogle Scholar
• Rohde DLT, Olsen S, Chang JT (2003) Modeling the recent common ancestry of all living humans. Nature 425:798–804 CASGoogle Scholar
• Rosenberg NA (2006) Standardized subsets of the HGDP-CEPH human genome diversity cell line panel, accounting for atypical and duplicated samples and pairs of close relatives. Ann Hum Genet 70:841–847 ArticleCASPubMedGoogle Scholar
• Rosenberg NA, Mahajan S, Ramachandran S, Zhao C, Pritchard JK, Feldman MW (2005) Clines, clusters, and the effect of study design on the inference of human population structure. PLoS Genet 1:e70 ArticlePubMedPubMed CentralCASGoogle Scholar
• Sainudiin R, Véber A (2018) Full likelihood inference from the site frequency spectrum based on the optimal tree resolution. Theoret Pop Biol 124:1–15 ArticleGoogle Scholar
• Sargsyan O, Wakeley J (2008) A coalescent process with simultaneous multiple mergers for approximating the gene genealogies of many marine organisms. Theoret Pop Biol 74:104–114 ArticleGoogle Scholar
• Simonsen KL, Churchill GA, Aquadro CF (1995) Properties of statistical tests of neutrality for DNA polymorphism data. Genetics 141:413–429 ArticleCASPubMedPubMed CentralGoogle Scholar
• Sjödin P, Kaj I, Krone S, Lascoux M, Nordborg M (2005) On the meaningand existence of an effective population size. Genetics 169:1061–1070 ArticlePubMedPubMed CentralGoogle Scholar
• Slatkin M (1987) The average number of sites separating DNA sequences drawn from a subdivided population. Theoret Pop Biol 32:42–49 ArticleCASGoogle Scholar
• Slatkin M (1991) Inbreeding coefficients and coalescence times. Genet Res Camb 58:167–175 ArticleCASGoogle Scholar
• Slatkin M, Hudson RR (1991) Pairwise comparisons of mitochondrial DNA sequences in stable and exponentially growing populations. Genetics 129:555–562 ArticleCASPubMedPubMed CentralGoogle Scholar
• Speidel L, Forest M, Sinan S, Myers SR (2019) A method for genome-wide genealogy estimation for thousands of samples. Nat Genet 51:1321–1329 ArticleCASPubMedPubMed CentralGoogle Scholar
• Spence JP, Steinrücken M, Terhorst J, Song YS (2018) Inference of population history using coalescent HMMs: review and outlook. Curr Op Genet Devel 53:70–76 ArticleCASGoogle Scholar
• Stephens M, Donnelly P (2000) Inference in molecular population genetics. J R Stat Soc Ser B 62:605–655 ArticleGoogle Scholar
• Stephens M, Donnelly P (2003) Ancestral inference in population genetics models with selection. Aust N Z J Stat 45:395–430 ArticleGoogle Scholar
• Stephens JC, Schneider JA, Tanguay DA, Choi J, Acharya T, Stanley SE, Jiang R, Messer CJ, Chew A, Han J-H, Duan J, Carr JL, Lee MS, Koshy B, Kumar AM, Zhang G, Newell W, Windemuth A, Xu C, Kalbfleisch TS, Shaner SL, Arnold K, Schultz V, Drysdale CM, Nandabalan K, Judson RS, Ruaño G, Vovis GF (2001) Haplotype variation and linkage disequilibrium in 313 human genes. Science 293:489–493 ArticleCASPubMedGoogle Scholar
• Strobeck C (1987) Average number of nucleotide differences in a sample from a single subpopulation: a test for population subdivision. Genetics 117:149–153 ArticleCASPubMedPubMed CentralGoogle Scholar
• Tajima F (1983) Evolutionary relationship of DNA sequences in finite populations. Genetics 105:437–460 ArticleCASPubMedPubMed CentralGoogle Scholar
• Tajima F (1989) Statistical method for testing the neutral mutation hypothesis by DNA. Genetics 123:585–595 ArticleCASPubMedPubMed CentralGoogle Scholar
• Tajima F (1997) Estimation of the amount of DNA polymorphism and statistical tests of the neutral mutation hypothesis based on DNA polymorphism. In: Donnelly P, Tavaré S (eds) Progress in population genetics and human evolution. Springer, New York, pp 149–164 ChapterGoogle Scholar
• Takahata N (1988) The coalescent in two partially isolated diffusion populations. Genet Res Camb 53:213–222 ArticleGoogle Scholar
• Takahata N, Nei M (1985) Gene genealogy and variance of interpopulational nucleotide differences. Genetics 110:325–344 ArticleCASPubMedPubMed CentralGoogle Scholar
• Tavaré S (1984) Lines-of-descent and genealogical processes, and theirapplication in population genetic models. Theor Popul Biol 26:119–164 ArticlePubMedGoogle Scholar
• The 1000 Genomes Project Consortium (2015) A global reference for human genetic variation. Nature 526:68–74 ArticleCASGoogle Scholar
• Valdes AM, Slatkin M, Freimer NB (1993) Allele frequencies at microsatellite loci: the stepwise mutation model revisited. Genetics 133:737–749 ArticleCASPubMedPubMed CentralGoogle Scholar
• Wakeley J (1999) Non-equilibrium migration in human history. Genetics 153:1863–1871 ArticleCASPubMedPubMed CentralGoogle Scholar
• Wakeley J (2009) Coalescent theory: an introduction. Macmillan Learning, Macmillan, New York Google Scholar
• Wakeley J, Hey J (1997) Estimating ancestral population parameters. Genetics 145:847–855 ArticleCASPubMedPubMed CentralGoogle Scholar
• Wakeley J, King L, Low BS, Ramachandran S (2012) Gene genealogies within a fixed pedigree, and the robustness of Kingman's coalescent. Genetics 190:1433–1445 ArticlePubMedPubMed CentralGoogle Scholar
• Wakeley J, King L, Wilton P (2016) Effects of the population pedigree on genetic signatures of historical demographic events. Proc Natl Acad Sci USA 113:7994–8001 ArticleCASPubMedPubMed CentralGoogle Scholar
• Watterson GA (1975) On the number of segregating sites in genetical models without recombination. Theoret Pop Biol 7:256–276 ArticleCASGoogle Scholar
• Watterson GA (1982) Mutant substitutions at linked nucleotide sites. Adv Appl Probab 14:166–205 ArticleGoogle Scholar
• Wilkinson-Herbots HM (2008) The distribution of the coalescence time and the number of pairwise nucleotide differences in the “isolation with migration” model. Theoret Pop Biol 73:277–288 ArticleGoogle Scholar
• Wilton PR, Baduel P, Landon MM, Wakeley J (2017) Population structure and coalescence in pedigrees: comparisons to the structured coalescent and a framework for inference. Theoret Pop Biol 115:1–12 ArticleGoogle Scholar
• Winther GW, Giordano R, Edge MD, Nieslen R (2015) The mind, the lab, and the field: three kinds of populations in scientific practice. Stud Hist Phil Biol Biomed Sci 52:12–21 ArticleGoogle Scholar
• Wright S (1931) Evolution in Mendelian populations. Genetics 16:97–159 ArticleCASPubMedPubMed CentralGoogle Scholar
• Wu Y (2010) Exact computation of coalescent likelihood for panmictic and subdivided populations under the infinite sites model. IEEE/ACM Trans Comput Biol Bioinform 7:611–618 ArticleCASPubMedGoogle Scholar

Author information

Authors and Affiliations

1. Department of Organismic and Evolutionary Biology, Harvard University, Cambridge, MA, USA John Wakeley

1. John Wakeley

Corresponding author

Editor information

Editors and Affiliations

1. Ecology and Evolutionary Biology, University of California, Los Angeles, CA, USA Kirk E. Lohmueller
2. Integrative Biology, University of California, Berkeley, CA, USA Rasmus Nielsen

Rights and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Cite this chapter

Wakeley, J. (2021). Coalescent Models. In: Lohmueller, K.E., Nielsen, R. (eds) Human Population Genomics. Springer, Cham. https://doi.org/10.1007/978-3-030-61646-5_1

Download citation

• .RIS
• .ENW
• .BIB

• DOI : https://doi.org/10.1007/978-3-030-61646-5_1
• Published : 14 March 2021
• Publisher Name : Springer, Cham
• Print ISBN : 978-3-030-61644-1
• Online ISBN : 978-3-030-61646-5
• eBook Packages : Biomedical and Life SciencesBiomedical and Life Sciences (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Get shareable link

Sorry, a shareable link is not currently available for this article.

Copy to clipboard

Provided by the Springer Nature SharedIt content-sharing initiative