• KELLY FINDLEY The University of Illinois at Urbana-Champaign
  • BREIN MOSELY Harvard University
  • AARON LUDKOWSKI University of Chicago



Statistics education research, Sampling, Generalizability, Power, Causality


Reform efforts in statistics education emphasize the need for students to develop statistical thinking. Critical to this goal is a solid understanding of design in the process of collecting data, evaluating evidence, and drawing conclusions. We collected survey responses from over 700 college students at the start of an introductory statistics course to determine how they evaluated the validity of different designs. Despite preferring different designs, students offered a variety of productive arguments supporting their choices. For example, some students viewed intervention as a weakness that disrupted the ability to generalize results, whereas others viewed intervention as critical for identifying causality. Our results highlight that instruction should frame design as the balancing of different priorities: namely causality, generalizability, and power.


American Statistical Association Undergraduate Guidelines Workgroup. (2014). Curriculum guidelines for undergraduate programs in statistical science. American Statistical Association.

Bennett, K. A. (2015). Using a discussion about scientific controversy to teach central concepts in experimental design. Teaching Statistics, 37(3), 71–77.

Berland, L. K., & Reiser, B. J. (2009). Making sense of argumentation and explanation. Science Education, 93(1), 26–55.

College Board. (2020). AP Statistics course and exam description.

Crowley, M., Thomas, K., & Haberman, M. (2020, April 5). Ignoring expert opinion, Trump again promotes hydroxychloroquine. New York Times.

Cummiskey, K., Adams, B., Pleuss, J., Turner, D., Clark, N., & Watts, K. (2020). Causal inference in introductory statistics courses. Journal of Statistics Education, 28(1), 2–8.

Dasgupta, A. P., Anderson, T. R., & Pelaez, N. (2014). Development and validation of a rubric for diagnosing students’ experimental design knowledge and difficulties. CBE—Life Sciences Education, 13(2), 265–284.

Deane, T., Nomme, K., Jeffery, E., Pollock, C., & Birol, G. (2014). Development of the biological experimental design concept inventory (BEDCI). CBE—Life Sciences Education, 13(3), 540–551.

Derry, S. J., Levin, J. R., Osana, H. P., Jones, M. S., & Peterson, M. (2000). Fostering students’ statistical and scientific thinking: Lessons learned from an innovative college course. American Educational Research Journal, 37(3), 747–773.

Easterling, R. (2004). Teaching experimental design. The American Statistician, 58(3), 244–252.

Fauconnier, G., & Turner, M. (2002). The way we think: Conceptual blending and the mind’s hidden complexities. Basic Books.

Findley, K., & Lyford, A. (2019). Investigating students’ reasoning about sampling distributions through a resource perspective. Statistics Education Research Journal, 18(1), 26–45.

Fry, E. B. (2017). Introductory statistics students’ conceptual understanding of study design and conclusions, [Doctoral dissertation, University of Minnesota].

GAISE College Report ASA Revision Committee. (2016). Guidelines for Assessment and Instruction in Statistics Education college report 2016. American Statistical Association.

Garfield, J., Le, L., Zieffler, A., & Ben-Zvi, D. (2015). Developing students’ reasoning about samples and sampling variability as a path to expert statistical thinking. Educational Studies in Mathematics, 88, 327–342.

Gautret, P., Lagier, J.-C., Parola, P., Hoang, V. T., Meddeb, L., Mailhe, M., Doudier, B., Courjon, J., Giordanengo, V., Vieira, V. E., Dupont, H. T., Honoré, S., Colson, P., Chabrière, E., La Scola, B., Rolain, J.-M., Brouqui, P., & Raoult, D. (2020). Hydroxychloroquine and azithromycin as a treatment of COVID-19: Results of an open-label non-randomized clinical trial. International Journal of Antimicrobial Agents, 56(1), 1–6.

Hammer, D. (1996). Misconceptions or p-prims: How may alternative perspectives of cognitive structure influence instructional perceptions and intentions. The Journal of the Learning Sciences, 5(2), 97–127.

Hiebert, S. M. (2007). Teaching simple experimental design to undergraduates: Do your students understand the basics? Advances in Physiology Education, 31(1), 82–92.

Imai, K., Keele, L., Tingley, D., & Yamamoto, T. (2011). Unpacking the black box of causality: Learning about causal mechanisms from experimental and observational studies. American Political Science Review, 105(4), 765–789.

Kaplan, J. J. (2009). Effect of belief bias on the development of undergraduate students' reasoning about inference. Journal of Statistics Education, 17(1).

Kupferschmidt, K. (2020). Big studies dim hopes for hydroxychloroquine. Science, 368(6496), 1166–1167.

Next Generation Science Standards Lead States. (2013). Next generation science standards: For states, by states. The National Academies Press.

Saldaña, J. (2021). The coding manual for qualitative researchers. SAGE Publications.

Schield, M. (2018). Confounding and cornfield: Back to the future. In M. A. Sorto, A. White, & L. Guyot (Eds.), Looking back, looking forward. Proceedings of the Tenth International Conference on Teaching Statistics (ICOTS10, July, 2018). International Statistical Institute/International Association for Statistical Education.

Smith, C. L., Maclin, D., Houghton, C., & Hennessey, M. G. (2000). Sixth-grade students’ epistemologies of science: The impact of school science experiences on epistemological development. Cognition and Instruction, 18(3), 349–422.

Sotos, A. E. C., Vanhoof, S., Van den Noortgate, W., & Onghena, P. (2007). Students’ misconceptions of statistical inference: A review of the empirical evidence from research on statistics education. Educational Research Review, 2(2), 98–113.

Tintle, N., Chance, B., Cobb, G., Rossman, A., Roy, S., Swanson, T., & VanderStoep, J. (2013). Challenging the state of the art in post-introductory statistics. Proceedings of the 59th World Statistics Congress of the International Statistical Institute (pp. 295–300).

Tintle, N., Chance, B., Cobb, G., Roy, S., Swanson, T., & VanderStoep, J. (2015). Combating anti-statistical thinking using simulation-based methods throughout the undergraduate curriculum. The American Statistician, 69(4), 362–370.

Wagler, A., & Wagler, R. (2013). Randomizing roaches: Exploring the ‘bugs’ of randomization in experimental design. Teaching Statistics, 36(1), 13–20.

Watson, J., Fitzallen, N., & Chick, H. (2020). What is the role of statistics in integrating STEM education? In J. Anderson & Y. Li (Eds.), Integrated Approaches to STEM Education. Advances in STEM Education. Springer, Cham.

Wroughton, J. R., McGowan, H. M., Weiss, L. V., & Cope, T. M. (2013). Exploring the role of context in students’ understanding of sampling. Statistics Education Research Journal, 12(2), 32–58.

Zieffler, A., Garfield, J., delMas, R., & Reading, C. (2008). A framework to support research on informal inferential reasoning. Statistics Education Research Journal, 7(2), 40–58.


Additional Files





Regular Articles