DATA SCIENTISTS’ EPISTEMIC THINKING FOR CREATING AND INTERPRETING VISUALIZATIONS
DOI:
https://doi.org/10.52041/serj.v21i2.21Keywords:
Statistics education research, Data visualization, Epistemic thinking, Data scienceAbstract
The purpose of the study was to understand the experiences of data scientists regarding common skills and strategies of interpreting and creating data visualizations. In this Delphi study, the participants were researchers in Data Science using three rounds of surveys. Skills and strategies were identified after Delphi Panel 1 and then brought back to them in Delphi Panel 2 to rate the level of importance they attribute to those skills/strategies. Consensus was determined using a cutoff for the interquartile range for each skill/strategy, and overall group ratings were presented to researchers in Delphi Panel 3 for them to adjust their ratings if desired. This study provided empirical evidence of skills/strategies that data scientists engage in when interpreting and creating visualizations.
References
Adler, M., & Ziglio, E. (1996). Gazing into the oracle: The Delphi method and its application to social policy and public health. Kingsley.
Alexander, D. R. (2008). A modified delphi study of future crises and required leader competencies. [Doctoral dissertation, University of Phoenix].
Amar, R., Eagan, J., & Stasko, J. (2005). Low-level components of analytic activity in information visualization. In J. Stasko & M. Ward (Eds.), Proceedings of the IEEE Symposium on Information Visualization (InfoVis05), Minneapolis, MN, October 23–25, (pp. 111–117). https://doi.ieeecomputersociety.org/10.1109/INFVIS.2005.1532136
Azzam, T., & Evergreen, S. (2013). Data visualization, Part 1: New directions for evaluation. Jossey-Bass.
Baker, K. (2005). A model for leading online K–12 learning environments [Doctoral dissertation, University of Phoenix].
Barzilai, S., & Chinn, C. A. (2018). On the goals of epistemic education: Promoting apt epistemic performance. Journal of the Learning Sciences, 27(3), 353–389. https://doi.org/10.1080/10508406.2017.1392968
Barzilai, S., & Eilam, B. (2018). Learners' epistemic criteria and strategies for evaluating scientific visual representations. Learning and Instruction, 58, 137–147.
Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online sources Cognition and Instruction, 30(1), 39-85. https://doi.org/10.1080/07370008.2011.636495
Barzilai, S., & Zohar, A. (2014). Reconsidering personal epistemology as metacognition: A multifaceted approach to the analysis of epistemic thinking. Educational Psychologist, 49(1), 13–35. https://doi.org/10.1080/00461520.2013.863265
Barzilai, S., & Zohar, A. (2016). Epistemic (meta)cognition: Ways of thinking about knowledge and knowing. In J. A. Greene, W. A. Sandoval, & I. Bråten (Eds.), Handbook of epistemic cognition. Routledge.
Bernard, H. R. (2002). Research methods in anthropology: Qualitative and quantitative approaches (3rd ed.). Alta Mira Press.
Börner, K., Bueckle, A., & Ginda, M. (2019). Data visualization literacy: Definitions, conceptual frameworks, exercises, and assessments. Proceedings of the National Academy of Sciences, 116(6), 1857–1864. https://doi.org/10.1073/pnas.1807180116
Börner, K., Maltese, A., Balliet, R. N., & Heimlich, J. (2016). Investigating aspects of data visualization literacy using 20 information visualizations and 273 science museum visitors. Information Visualization, 15(3), 198–213.
Bouquin, D., & Epstein, H.-A. B. (2015). Teaching data visualization basics to market the value of a hospital library: An infographic as one example. Journal of Hospital Librarianship, 15(4), 349–364. https://doi.org/10.1080/15323269.2015.1079686
Boy, J., Rensink, R. A., Bertini, E., & Fekete, J.-D. (2014). A principled way of assessing visualization literacy. IEEE Transactions on Visualization and Computer Graphics, 20(12), 1963–1972. https://doi.org/10.1109/TVCG.2014.2346984
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Brehmer, M., & Munzner, T. (2013). A multi-level typology of abstract visualization tasks. IEEE Transactions on Visualization and Computer Graphics, 19, 2376–2385.
Brill, J. M., Bishop, M. J., & Walker, A. E. (2006). The competencies and characteristics required of an effective project manager: A web-based Delphi study. Educational Technology Research and Development, 54(2), 115–140.
Calzada Prado, J., & Marzal Miguel, Á. (2013). Incorporating data literacy into information literacy programs: Core competencies and contents. Libri, 63(2), 123–134. https://doi.org/10.1515/libri-2013-0010
Cleveland, W. S. (2001). Data science: An action plan for expanding the technical areas of the field of statistics. International Statistical Review/Revue Internationale de Statistique, 69(1), 21–26. https://doi.org/10.2307/1403527
Cambridge Cognition. (2015). What is cognition? https://cambridgecognition.com/blog/entry/what-is-cognition
Carrión Pérez, J. C., & Espinel Febles, M. C. (2006). An investigation about translation and interpretation of statistical graphs and tables by students of primary education. In A. Rossman & B. Chance (Eds.), Working cooperatively in statistics education. Proceedings of the 7th International Conference on Teaching Statistics (ICOTS7), Salvador, Bahia, Brazil, July. 2–7. International Statistical Institute. https://iase-web.org/documents/papers/icots7/C332.pdf?1402524967
Cooper, L. L., & Shore, F. S. (2010). The effects of data and graph type on concepts and visualizations of variability. Journal of Statistics Education, 18(2), 1–16. https://doi.org/10.1080/10691898.2010.11889487
Curcio, F. R. (1987). Comprehension of mathematical relationships expressed in graphs. Journal for Research in Mathematics Education, 18(5), 382–393.
De Veaux, R. D., Agarwal, M., Averett, M., Baumer, B. S., Bray, A., Bressoud, T. C., . . . Ye, P. (2017). Curriculum guidelines for undergraduate programs in data science. Annual Review of Statistics and Its Application, 4(1), 15–30. https://doi.org/10.1146/annurev-statistics-060116-053930
Eilam, B. (2015). Teaching, learning, and visual literacy: The dual role of visual representation. Cambridge University Press.
Figueiras, A. (2013). A typology for data visualization on the web. In the Proceedings of the 17th International Conference on Information Visualisation, London, England, July 16–18 (pp. 351–358). https://doi.ieeecomputersociety.org/10.1109/IV.2013.45
Forbes, S., Chapman, J., Harraway, J., Stirling, D., & Wild, C. (2014). Use of data visualizations in the teaching of statistics: A New Zealand perspective. Statistics Education Research Journal, 13(2), 187–201. https://doi.org/10.52041/serj.v13i2.290
Friel, S. N., Curcio, F. R., & Bright, G. W. (2001). Making sense of graphs: Critical factors influencing comprehension and instructional implications. Journal for Research in Mathematics Education, 32(2), 124–158.
Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32, 1008–1015. https://doi.org/10.1046/j.1365-2648.2000.t01-1-01567.x
Hofer, B. K. (2004). Epistemological understanding as a metacognitive process: Thinking aloud during online searching. Educational Psychologist, 39(1), 43–55. https://doi.org/10.1207/s15326985ep3901_5
Hofer, B. K. (2016). Epistemic cognition as a psychological construct: Advancements and challenges. In J. A. Greene, W. A. Sandoval, & I. Bråten (Eds.), Handbook of epistemic cognition (pp. 19–38). Routledge.
Hofer, B. K., & Pintrich, P. R. (1997). The development of epistemological theories: Beliefs about knowledge and knowing and their relation to learning. Review of Educational Research, 67(1), 88–140. https://doi.org/10.3102/00346543067001088
Hsu, C. C., & Sandford, B. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), Article 10.
Hullman, J., & Diakopoulos, N. (2011). Visualization rhetoric: Framing effects in narrative visualization. IEEE Transactions on Visualization and Computer Graphics, 17(12), 2231–2240. https://doi.org/10.1109/TVCG.2011.255
Hussein, M. M. (2010). Corporate social responsibility: Finding the middle ground. Social Responsibility Journal, 6(3), 420–432.
Kandel, S., Paepcke, A., Hellerstein, J. M., & Heer, J. (2012). Enterprise data analysis and visualization: An interview study. IEEE Transactions on Visualization and Computer Graphics, 18(12), 2917–2926.
Kapler, T., & Wright, W. (2004). GeoTime information visualization. In M. Ward & T. Munzer (Eds.), Proceedings of the IEEE Symposium on Information Visualization (InfoVis04), Washington, DC, October 10–12. https://doi.org/10.1109/INFVIS.2004.27
Keeney, S., Hasson, F., & McKenna, H. P. (2001). A critical review of the Delphi technique as a research methodology for nursing. International Journal of Nursing Studies, 38(2), 195–200. https://doi.org/10.1016/S0020-7489(00)00044-4
Keeney, S., Hasson, F., & McKenna, H. P. (2011). The Delphi technique in nursing and health research. Wiley-Blackwell.
Keim, D. A., Kohlammer, J., Mansmann, F., May, T., & Wanner, F. (2010). Visual analytics. In D. A. Keim, J. Kohlhammer, & G. Ellis (Eds.), Mastering the information age: Solving problems with visual analytics. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.363.2661
Kirk, A. (2016). Data visualisation: A handbook for data driven design. SAGE Publications.
Kosslyn, S. M. (1985). Graphics and human information processing: A review of five books. Journal of the American Statistical Association, 80(391), 499–512.
Laina, V., & Wilkerson, M. (2016). Distributions, trends, and contradictions: A case study in sensemaking with interactive data visualizations. In C. K. Looi, J. L., Polman, U.,Cress, & P. Reimann (Eds.). Proceedings of the International Society of the Learning Sciences, Singapore, June 20-24). https://repository.isls.org//handle/1/347
Lam, H., Bertini, E., Isenberg, P., Plaisant, C., & Carpendale, S. (2012). Empirical studies in information visualization: Seven scenarios. IEEE Transactions on Visualization and Computer Graphics, 18(9), 1520–1536.
Lee, S., Kim, S.-H., & Kwon, B. C. (2017). VLAT: Development of a visualization literacy assessment test. IEEE Transactions on Visualization and Computer Graphics, 23(1), 551–560. https://doi.org/10.1109/TVCG.2016.2598920
Lem, S., Kempen, G., Ceulemans, E., Onghena, P., Verschaffel, L., & Van Dooren, W. (2015). Combining multiple external representations and refutational text: An intervention on learning to interpret box plots. International Journal of Science and Mathematics Education, 13(4), 909–926. https://doi.org/10.1007/s10763-014-9604-3
Lima, I., & Selva, A. (2010). Youth and adults students interpreting bar and line graphs. In C. Reading (Ed.), Data and context in statistics education: Towards an evidence-based society. Proceedings of the 8th International Conference on Teaching Statistics (ICOTS8), Ljubljana, Slovenia, July 11–16. https://iase-web.org/documents/papers/icots8/ICOTS8_C134_LIMA.pdf?1402524973
Linstone, H. A. (1978). The Delphi technique. Handbook of futures research. Greenwood.
Linstone, H. A., & Turoff, M. (1975). The Delphi method: Techniques and applications. Addison-Wesley.
Lowrie, T., Diezmann, C. M., & Logan, T. (2012). A framework for mathematics graphical tasks: The influence of the graphic element on student sense making. Mathematics Education Research Journal, 24(2), 169–187.
Mackinlay, J., & Kosara, R. (2013). Storytelling: The next step for visualization. Computer, 46, 44–50.
Maltese, A. V., Svetina, D., & Harsh, J. A. (2015). Data visualization literacy: Investigating data interpretation along the novice-expert continuum. Journal of College Science Teaching, 45(1), 84–90.
Mason, L. (2016). Psychological perspectives on measuring epistemic cognition. In J. A. Greene, W. A. Sandoval, & I. Bråten (Eds.), Handbook of Epistemic Cognition (pp. 375 – 392). Routledge.
Miller, B., & Pasley, J. (2012). What do we know and how well do we know it? Identifying practice-based insights in education. Evidence and Policy, 8(2), 193–212.
Mirel, B., Kumar, A., Nong, P., Su, G., & Meng, F. (2016). Using interactive data visualizations for exploratory analysis in undergraduate genomics coursework: Field study findings and guidelines. Journal of Science Education and Technology, 25, 91–110.
Mosca, A., Robinson, S., Clarke, M., Redelmeier, R., Coates, S., Cashman, D., & Chang, R. (2019). Defining an analysis: A study of client-facing data scientists. In J. Johansson, F. Sadlo, & G. E. Marai (Eds.), Proceedings of the EuroVis 2019: 21th EG/VGTC Conference on Visualization, Short Papers, Porto, Portugal, June 3–7 (pp. 73–77). https://doi.org/10.2312/evs.20191173
Nicolaou, C. T., Nicolaidou, I. A., Zacharia, Z. C., & Constantinou, C. P. (2007). Enhancing fourth graders' ability to interpret graphical representations through the use of microcomputer-based labs implemented within an inquiry-based activity sequence. The Journal of Computers in Mathematics and Science Teaching, 26(1), 75–99.
Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 42, 533–544.
Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). SAGE Publications.
Persai, D., Panda, R., Kumar, R., & Mc Ewen, A. (2016). A Delphi study for setting up tobacco research and practice network in India. Tobacco Induced Diseases, 14, 4–4. https://doi.org/10.1186/s12971-016-0067-x
Pfannkuch, M. (2006). Comparing box plot distributions: A teacher's reasoning. Statistics Education Research Journal, 5(2), 27–45.
Pfannkuch, M., Regan, M., Wild, C., & Horton, N. J. (2010). Telling data stories: Essential dialogues for comparative reasoning. Journal of Statistics Education, 18(1) Article 10. https://doi.org/10.1080/10691898.2010.11889479
Philip, T. M., Olivares-Pasillas, M. C., & Rocha, J. (2016). Becoming racially literate about data and data-literate about race: Data visualizations in the classroom as a site of racial-ideological micro-contestations. Cognition & Instruction, 34(4), 361–388. https://doi.org/10.1080/07370008.2016.1210418
Pluta, W. J., Chinn, C. A., & Duncan, R. G. (2011). Learners' epistemic criteria for good scientific models. Journal of Research in Science Teaching, 48(5), 486–511. https://doi.org/10.1002/tea.20415
Ritchie, D., & Earnest, J. (1999). The Future of Instructional Design: Results of a Delphi Study. Educational Technology, 39(1), 35–42.
Roth, W. M., & Bowen, G. M. (2001). Professionals read graphs: A semiotic analysis. Journal for Research in Mathematics Education, 32(2), 159–194.
Sandoval, W. A., Greene, J. A., & Bråten, I. (2016). Understanding and promoting thinking about knowledge. Review of Research in Education, 40(1), 457–496.
Segel, E., & Heer, J. (2010). Narrative visualization: Telling stories with data. IEEE Transactions on Visualization and Computer Graphics, 16(6), 1139–1148. https://doi.org/10.1109/TVCG.2010.179
Serafini, F. (2012). Expanding the four resources model: Reading visual and multi-modal texts. Pedagogies: An International Journal, 7(2), 150–164.
Shah, P., & Hoeffner, J. (2002). Review of graph comprehension research: Implications for instruction. Educational Psychology Review, 14(1), 47–69. http://dx.doi.org/10.1023/A:1013180410169
Spradley, J. P. (1979). The ethnographic interview. Holt, Rinehart & Winston.
Takemura, A. (2018). New undergraduate departments and programs of data science in Japan. In M. A. Sorto & E. Paparistodemou (Eds.), Looking back, looking forward. Proceedings of the 10th International Conference on the Teaching of Statistics (ICOTS10), Kyoto, Japan, July 8–14. https://iase-web.org/icots/10/proceedings/pdfs/ICOTS10_1I2.pdf?1531364187
Thangaratinam, S., & Redman, C. W. (2005). The Delphi technique. The Obstetrician & Gynaecologist, 7(2), 120–125. https://doi.org/10.1576/toag.7.2.120.27071
Vahey, P., Rafanan, K., Patton, C., Swan, K., van ’t Hooft, M., Kratcoski, A., & Stanford, T. (2012). A cross-disciplinary approach to teaching data literacy and proportionality. Educational Studies in Mathematics, 81(2), 179–205. https://doi.org/10.1007/s10649-012-9392-z
VERBI Software. (2019). MAXQDA 2020 [computer software]. VERBI Software. www.maxqda.com
Wild, C. J., & Pfannkuch, M. (1999). Statistical thinking in empirical enquiry. International Statistical Review, 67(3), 223–248. https://doi.org/10.2307/1403699
