Perspective Article

Toward a Framework for Robust Design-Based Research

M. Shane Tutwiler 1 * , Denise M. Bressler 2, Joseph Reilly 3, Eileen McGivney 4, Tina A. Grotzer 5, Chris Dede 5
More Detail
1 College of Education, University of Rhode Island, Kingston, RI, 02881, USA2 College of Education, East Carolina University, Greenville, NC, 27858, USA3 College of Professional Studies, Northeastern University, Boston, MA, 02115, USA4 College of Art, Media & Design, Northeastern University, Boston, MA, 02115, USA5 Graduate School of Education, Harvard University, Cambridge, MA, 02138, USA* Corresponding Author
Educational Innovations and Emerging Technologies, 3(3), 2023, 1-7, https://doi.org/10.35745/eiet2023v03.03.0001
Published: 30 September 2023
OPEN ACCESS   211 Views   98 Downloads
Download Full Text (PDF)

ABSTRACT

Design-based research (DBR) is a popular approach for studying and maximizing the effectiveness of learning environments in the Learning Sciences. This approach has historically been approached from a mixed-methodological perspective. In this article, we argue that, with an ever-increasing focus on using the results of DBR to inform policy and practice, the design of DBR studies must be made more robust by addressing issues inherent to the quantitative methodologies employed to track gains in learning. We propose four key design principles (Measurement Matters, Learning is Longitudinal, Use Samples Smartly, and Invest in Fidelity), as well as an analytic framework within which to apply them. A brief case study is used to demonstrate some of these elements in practice.
Keywords: Design-based Research, DBR, Educational technology

CITATION (APA)

Tutwiler, M. S., Bressler, D. M., Reilly, J., McGivney, E., Grotzer, T. A., & Dede, C. (2023). Toward a Framework for Robust Design-Based Research. Educational Innovations and Emerging Technologies, 3(3), 1-7. https://doi.org/10.35745/eiet2023v03.03.0001

REFERENCES

  1. Bressler, D.M., & Bodzin, A.M. (2013). A mixed methods assessment of students’ flow experiences during a mobile augmented reality science game. Journal of Computer Assisted Learning, 29(6), 505–517.
  2. Bressler, D.M., & Bodzin, A.M. (2016). Investigating flow experience and scientific practices during a mobile serious educational game. Journal of Science Education and Technology, 25, 795–805.
  3. Bressler, D.M., Bodzin, A.M., & Tutwiler, M.S. (2019). Engaging middle school students in scientific practice with a collaborative mobile game. Journal of Computer Assisted Learning, 35(2), 197–207.
  4. Brown, A.L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141–178.
  5. Chen, J.A., Tutwiler, M.S., Metcalf, S.J., Kamarainen, A., Grotzer, T., & Dede, C. (2016). A multi-user virtual environment to support students' self-efficacy and interest in science: A latent growth model analysis. Learning and Instruction, 41, 11–22.
  6. Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. The Journal of the Learning Sciences, 13(1), 15–42.
  7. Courter, C.J., Grotzer, T.A., Derbiszewska, K.M., Powell, M., Tutwiler, M.S., Kamarainen, A., Metcalf, S.J., & Dede, C. (2014). A case study contrasting students’ exploration behaviors of a complex causal scenario in a virtual world: Mapping proximity and goal directed movement in EcoMUVE. In Proceedings of National Association of Research in Science Teaching (NARST) Annual International Conference. Pittsburgh, PA, USA, 30 March–3 April, 2014.
  8. Creswell, J.W. (2011). Controversies in mixed methods research. The Sage Handbook of Qualitative Research, 4, 269–284.
  9. Cronbach, L.J., & Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281.
  10. Cuzzolino, M.P., Grotzer, T.A., Tutwiler, M.S., & Torres, E. (2019). An agentive focus may limit learning about complex causality and systems dynamics: A study of seventh graders' explanations of ecosystems. Journal of Research in Science Teaching, 56 (8), 1083–1105.
  11. Dede, C. (2004). If design-based research is the answer, what is the question? A commentary on Collins, Joseph, and Bielaczyc; diSessa and Cobb; and Fishman, Marx, Blumenthal, Krajcik, and Soloway in the JLS special issue on design-based research. The Journal of the Learning Sciences, 13(1), 105–114.
  12. Dede, C., Grotzer, T. A., Kamarainen, A., & Metcalf, S. (2017). EcoXPT: Designing for deeper learning through experimentation in an immersive virtual ecosystem. Journal of Educational Technology & Society, 20(4), 166–178.
  13. Dede, C., Grotzer, T., Kamarainen, A., Metcalf, S., & Feldman, R. (2019). Designing immersive authentic simulations that enhance motivation and learning: EcoLearn. Learning science: Theory, research, practice, 229–259.
  14. Frias‐Navarro, D., Pascual‐Llobell, J., Pascual‐Soler, M., Perezgonzalez, J., & Berrios‐Riquelme, J. (2020). Replication crisis or an opportunity to improve scientific production?. European Journal of Education, 55(4), 618–631.
  15. Gelman, A. (2018). The failure of null hypothesis significance testing when studying incremental changes, and what to do about it. Personality and Social Psychology Bulletin, 44(1), 16–23.
  16. Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing type S (sign) and type M (magnitude) errors. Perspectives on Psychological Science, 9(6), 641–651.
  17. Gelman, A., Carlin, J.B., Rubin, D.B., Vehtari, A., Dunson, D.B., & Stern, H.S. (2013). Chapman and Hall/CRC. In Bayesian data analysis, New York, USA: Taylor & Francis.
  18. Gelman, A., & Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge, UK: Cambridge University Press.
  19. Gelman, A., Hill, J., & Vehtari, A. (2020). Regression and other stories. Cambridge, UK: Cambridge University Press.
  20. Grotzer, T.A., Kamarainen, A.M., Tutwiler, M.S., Metcalf, S., & Dede, C. (2013). Learning to reason about ecosystems dynamics over time: The challenges of an event-based causal focus. BioScience, 63(4), 288–296.
  21. Grotzer, T.A., Powell, M., Derbiszewska, K.M., Courter, C.J., Kamarainen, A.M., Metcalf, S.J., & Dede, C.J. (2015). Turning Transfer Inside Out: The Affordances of Virtual Worlds and Mobile Devices in Real World Contexts for Teaching About Causality Across Time and Distance in Ecosystems. Technology, Knowledge and Learning, 20(1), 43–69.
  22. Jackson, S., Eklund, R., & Martin, A. (2010). The FLOW manual. Queensland, Australia: Mind Garden, Inc.
  23. Kamarainen, A.M., Reilly, J.M., Metcalf, S.J., Grotzer, T.A., & Dede, C.J. (2018). Using Mobile Location-Based Augmented Reality to Support Outdoor Learning in Undergraduate Ecology and Environmental Science Courses. The Bulletin of the Ecological Society of America, 99 (2), 259–276.
  24. Light, R.J., Singer, J.D., & Willett, J.B. (1990). By design. Cambridge, MA: Harvard University.
  25. Maxwell, J.A. (2012). Qualitative research design: An interactive approach (Vol. 41). Los Angeles, CA, USA: Sage publications.
  26. McElreath, R. (2020). Statistical rethinking: A Bayesian course with examples in R and Stan. New York, NY, USA: Chapman and Hall/CRC Press.
  27. McGivney, E., Gonzalez, E., De Los Santos, S., Kamarainen, A. & Grotzer, T.A. (2019). Improving understanding of teaching practice for student learning: A holistic measure of fidelity of implementation. Presented at the National Association for Research in Science Teaching (NARST), Baltimore, MD, USA, 2 April 2019.
  28. Metcalf, S.J., Dickes, A.C., Reilly, J.M., Kamarainen, A.M., Brennan, K.A., Grotzer, T.A., & Dede, C.J. (2020). Impact of a Blended Immersive and Computational Modeling Tool on Elementary Ecosystems Science Learning. Available online: https://ecolearn.gse.harvard.edu/node/1477274 (accessed on 15 July 2023)
  29. Metcalf, S.J., Kamarainen, A.M., Torres, E., Grotzer, T.A., & Dede, C.J. (2018). Integrating Multi-User Virtual Environments in Modern Classrooms, In EcoMUVE: A Case Study on the Affordances of MUVEs in Ecosystem Science Education. Hershey,PA, USA: IGI Global.
  30. Metcalf, S.J., Reilly, J M., Jeon, S., Wang, A., Pyers, A., Brennan, K., & Dede, C. (2021). Assessing computational thinking through the lenses of functionality and computational fluency. Computer Science Education, 31(2), 199–223.
  31. Murnane, R.J., & Willett, J.B. (2010). Methods matter: Improving causal inference in educational and social science research. New York, NY: Oxford University Press.
  32. Puntambekar, S. (2018). Design-Based Research (DBR). In International Handbook of the Learning Sciences. New York, NY: Routledge Press; pp. 383–392.
  33. Renninger, K.A., Ren, Y., & Kern, H.M. (2018). Motivation, engagement, and interest. In The International Handbook of the Learning Sciences. New York, NY: Routledge Press; pp. 116–126.
  34. Singer, J.D., Willett, J.B., & Willett, J.B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York, NY: Oxford University press.
  35. Sommerhoff, D., Szameitat, A., Vogel, F., Chernikova, O., Loderer, K., & Fischer, F. (2018). What do we teach when we teach the learning sciences? A document analysis of 75 graduate programs. Journal of the Learning Sciences, 27(2), 319–351.
  36. Stahl, G., & Hesse, F. (2008). The many levels of CSCL. International Journal of Computer-Supported Collaborative Learning, 3(1), 1–4.
  37. Tutwiler, M.S. (2019). Exploring The relationship between attentional capture and prior knowledge in a science-based multi-user virtual environment: an individual growth model analysis. Journal of Science Education and Technology, 28, 299–309.
  38. Tutwiler, M.S., Grotzer, T.A., Thompson, M.M., Kamarainen, A.M., Metcalf, S.J., & Dede, C.J. (2016). Validation of an instrument measuring student complex causal assumptions. In Proceedings of National Association for Research in Science Teaching (NARST) Conference, Baltimore, MD, USA, 14–17 April, 2016.