RT Journal Article
T1 Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology
A1 Gould, Elliot
A1 Fraser, Hannah S.
A1 Parker, Timothy H.
A1 Nakagawa, Shinichi
A1 Griffith, Simon C.
A1 Vesk, Peter A.
A1 Fidler, Fiona
A1 Hamilton, Daniel G.
A1 Abbey Lee, Robin N.
A1 Abbott, Jessica K.
A1 Aguirre, Luis A.
A1 Alcaraz, Carles
A1 Aloni, Irith
A1 Altschul, Drew
A1 Arekar, Kunal
A1 Atkins, Jeff W.
A1 Atkinson, Joe
A1 Baker, Christopher M.
A1 Barrett, Meghan
A1 Bell, Kristian
A1 Bello, Suleiman Kehinde
A1 Beltrán, Iván
A1 Berauer, Bernd J.
A1 Bertram, Michael Grant
A1 Billman, Peter D.
A1 Blake, Charlie K.
A1 Blake, Shannon
A1 Bliard, Louis
A1 Bonisoli Alquati, Andrea
A1 Bonnet, Timothée
A1 Bordes, Camille Nina Marion
A1 Bose, Aneesh P. H.
A1 Botterill James, Thomas
A1 Boyd, Melissa Anna
A1 Boyle, Sarah A.
A1 Bradfer Lawrence, Tom
A1 Bradham, Jennifer
A1 Brand, Jack A.
A1 Brengdahl, Martin I.
A1 Bulla, Martin
A1 Bussière, Luc
A1 Camerlenghi, Ettore
A1 Campbell, Sara E.
A1 Campos, Leonardo L. F.
A1 Caravaggi, Anthony
A1 Cardoso, Pedro
A1 Carroll, Charles J. W.
A1 Catanach, Therese A.
A1 Chen, Xuan
A1 Chik, Heung Ying Janet
A1 Choy, Emily Sarah
A1 Christie, Alec Philip
A1 Chuang, Angela
A1 Chunco, Amanda J.
A1 Clark, Bethany L.
A1 Contina, Andrea
A1 Covernton, Garth A.
A1 Cox, Murray P.
A1 Cressman, Kimberly A.
A1 Crotti, Marco
A1 Crouch, Connor Davidson
A1 D’Amelio, Pietro B.
A1 de Sousa, Alexandra Allison
A1 Döbert, Timm Fabian
A1 Dobler, Ralph
A1 Dobson, Adam J.
A1 Doherty, Tim S.
A1 Drobniak, Szymon Marian
A1 Duffy, Alexandra Grace
A1 Duncan, Alison B.
A1 Dunn, Robert P.
A1 Dunning, Jamie
A1 Dutta, Trishna
A1 Eberhart Hertel, Luke
A1 Elmore, Jared Alan
A1 Elsherif, Mahmoud Medhat
A1 English, Holly M.
A1 Ensminger, David C.
A1 Ernst, Ulrich Rainer
A1 Ferguson, Stephen M.
A1 Fernandez Juricic, Esteban
A1 Ferreira Arruda, Thalita
A1 Fieberg, John
A1 Finch, Elizabeth A.
A1 Fiorenza, Evan A.
A1 Fisher, David N.
A1 Fontaine, Amélie
A1 Forstmeier, Wolfgang
A1 Fourcade, Yoan
A1 Frank, Graham S.
A1 Freund, Cathryn A.
A1 Fuentes Lillo, Eduardo
A1 Gandy, Sara L.
A1 Gannon, Dustin G.
A1 García Cervigón, Ana I.
A1 Garretson, Alexis C.
A1 Ge, Xuezhen
A1 Geary, William L.
A1 Géron, Charly
A1 Gilles, Marc
A1 Girndt, Antje
A1 Gliksman, Daniel
A1 Goldspiel, Harrison B.
A1 Gomes, Dylan G. E.
A1 Good, Megan Kate
A1 Goslee, Sarah C.
A1 Gosnell, J. Stephen
A1 Grames, Eliza M.
A1 Gratton, Paolo
A1 Grebe, Nicholas M.
A1 Greenler, Skye M.
A1 Griffioen, Maaike
A1 Griffith, Daniel M.
A1 Griffith, Frances J.
A1 Grossman, Jake J.
A1 Güncan, Ali
A1 Haesen, Stef
A1 Hagan, James G.
A1 Hager, Heather A.
A1 Harris, Jonathan Philo
A1 Harrison, Natasha Dean
A1 Hasnain, Sarah Syedia
A1 Havird, Justin Chase
A1 Heaton, Andrew J.
A1 Herrera Chaustre, María Laura
A1 Howard, Tanner J.
A1 Hsu, Bin Yan
A1 Iannarilli, Fabiola
A1 Iranzo, Esperanza C.
A1 Iverson, Erik N. K.
A1 Jimoh, Saheed Olaide
A1 Johnson, Douglas H.
A1 Johnsson, Martin
A1 Jorna, Jesse
A1 Jucker, Tommaso
A1 Jung, Martin
A1 Kačergytė, Ineta
A1 Kaltz, Oliver
A1 Ke, Alison
A1 Kelly, Clint D.
A1 Keogan, Katharine
A1 Keppeler, Friedrich Wolfgang
A1 Killion, Alexander K.
A1 Kim, Dongmin
A1 Kochan, David P.
A1 Korsten, Peter
A1 Kothari, Shan
A1 Kuppler, Jonas
A1 Kusch, Jillian M.
A1 Lagisz, Malgorzata
A1 Lalla, Kristen Marianne
A1 Larkin, Daniel J.
A1 Larson, Courtney L.
A1 Lauck, Katherine S.
A1 Lauterbur, M. Elise
A1 Law, Alan
A1 Léandri Breton, Don-Jean
A1 Lembrechts, Jonas J.
A1 L’Herpiniere, Kiara
A1 Lievens, Eva J. P.
A1 de Lima, Daniela Oliveira
A1 Lindsay, Shane
A1 Luquet, Martin
A1 MacLeod, Ross
A1 Macphie, Kirsty H.
A1 Magellan, Kit
A1 Mair, Magdalena M.
A1 Malm, Lisa E.
A1 Mammola, Stefano
A1 Mandeville, Caitlin P.
A1 Manhart, Michael
A1 Manrique-Garzon, Laura Milena
A1 Mäntylä, Elina
A1 Marchand, Philippe
A1 Marshall, Benjamin Michael
A1 Martin, Charles A.
A1 Martin, Dominic Andreas
A1 Martin, Jake Mitchell
A1 Martinig, April Robin
A1 McCallum, Erin S.
A1 McCauley, Mark
A1 McNew, Sabrina M.
A1 Meiners, Scott J.
A1 Merkling, Thomas
A1 Michelangeli, Marcus
A1 Moiron, Maria
A1 Moreira, Bruno
A1 Mortensen, Jennifer
A1 Mos, Benjamin
A1 Muraina, Taofeek Olatunbosun
A1 Murphy, Penelope Wrenn
A1 Nelli, Luca
A1 Niemelä, Petri
A1 Nightingale, Josh
A1 Nilsonne, Gustav
A1 Nolazco, Sergio
A1 Nooten, Sabine S.
A1 Novotny, Jessie Lanterman
A1 Olin, Agnes Birgitta
A1 Organ, Chris L.
A1 Ostevik, Kate L.
A1 Palacio, Facundo Xavier
A1 Paquet, Matthieu
A1 Parker, Darren James
A1 Pascall, David J.
A1 Pasquarella, Valerie J.
A1 Paterson, John Harold
A1 Payo Payo, Ana
A1 Pedersen, Karen Marie
A1 Perez, Grégoire
A1 Perry, Kayla I.
A1 Pottier, Patrice
A1 Proulx, Michael J.
A1 Proulx, Raphaël
A1 Pruett, Jessica L
A1 Ramananjato, Veronarindra
A1 Randimbiarison, Finaritra Tolotra
A1 Razafindratsima, Onja H.
A1 Rennison, Diana J.
A1 Riva, Federico
A1 Riyahi, Sepand
A1 Roast, Michael James
A1 Rocha, Felipe Pereira
A1 Roche, Dominique G.
A1 Román Palacios, Cristian
A1 Rosenberg, Michael S.
A1 Ross, Jessica
A1 Rowland, Freya E.
A1 Rugemalila, Deusdedith
A1 Russell, Avery L.
A1 Ruuskanen, Suvi
A1 Saccone, Patrick
A1 Sadeh, Asaf
A1 Salazar, Stephen M.
A1 Sales, Kris
A1 Salmón, Pablo
A1 Sánchez Tójar, Alfredo
A1 Santos, Leticia Pereira
A1 Santostefano, Francesca
A1 Schilling, Hayden T.
A1 Schmidt, Marcus
A1 Schmoll, Tim
A1 Schneider, Adam C.
A1 Schrock, Allie E.
A1 Schroeder, Julia
A1 Schtickzelle, Nicolas
A1 Schultz, Nick L.
A1 Scott, Drew A.
A1 Scroggie, Michael Peter
A1 Shapiro, Julie Teresa
A1 Sharma, Nitika
A1 Shearer, Caroline L.
A1 Simón, Diego
A1 Sitvarin, Michael I.
A1 Skupien, Fabrício Luiz
A1 Slinn, Heather Lea
A1 Smith, Grania Polly
A1 Smith, Jeremy A.
A1 Sollmann, Rahel
A1 Whitney, Kaitlin Stack
A1 Still, Shannon Michael
A1 Stuber, Erica F.
A1 Sutton, Guy F.
A1 Swallow, Ben
A1 Taff, Conor Claverie
A1 Takola, Elina
A1 Tanentzap, Andrew J.
A1 Tarjuelo, Rocío
A1 Telford, Richard J.
A1 Thawley, Christopher J.
A1 Thierry, Hugo
A1 Thomson, Jacqueline
A1 Tidau, Svenja
A1 Tompkins, Emily M.
A1 Tortorelli, Claire Marie
A1 Trlica, Andrew
A1 Turnell, Biz R.
A1 Urban, Lara
A1 Van de Vondel, Stijn
A1 van der Wal, Jessica Eva Megan
A1 Van Eeckhoven, Jens
A1 van Oordt, Francis
A1 Vanderwel, K. Michelle
A1 Vanderwel, Mark C.
A1 Vanderwolf, Karen J.
A1 Vélez, Juliana
A1 Vergara-Florez, Diana Carolina
A1 Verrelli, Brian C.
A1 Vieira, Marcus Vinícius
A1 Villamil, Nora
A1 Vitali, Valerio
A1 Vollering, Julien
A1 Walker, Jeffrey
A1 Walker, Xanthe J.
A1 Walter, Jonathan A.
A1 Waryszak, Pawel
A1 Weaver, Ryan J.
A1 Wedegärtner, Ronja E. M.
A1 Weller, Daniel L.
A1 Whelan, Shannon
A1 White, Rachel Louise
A1 Wolfson, David William
A1 Wood, Andrew
A1 Yanco, Scott W.
A1 Yen, Jian D. L.
A1 Youngflesh, Casey
A1 Zilio, Giacomo
A1 Zimmer, Cédric
A1 Zimmerman, Gregory Mark
A1 Zitomer, Rachel A.
AB Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small “many analyst” study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.
PB Springer
SN 1741-7007
YR 2025
FD 2025
LK https://hdl.handle.net/20.500.14352/119267
UL https://hdl.handle.net/20.500.14352/119267
LA eng
NO Gould, E., Fraser, H.S., Parker, T.H. et al. Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology. BMC Biol 23, 35 (2025). https://doi.org/10.1186/s12915-024-02101-x
NO EG’s contributions were supported by an Australian Government Research Training Program Scholarship, AIMOS top-up scholarship (2022) and Melbourne Centre of Data Science Doctoral Academy Fellowship (2021). FF’s contributions were supported by ARC Future Fellowship FT150100297.
NO Association for Interdisciplinary Meta-Research and Open Science
NO Australian Government
NO Melbourne Centre for Data Science
NO Australian Research Council
DS Docta Complutense
RD 14 ago 2025