An Investigation of the Relationship between Subjective and Objective Cognitive Load Measures of Language Item Difficulty

Document Type : Research article

Authors

1 MA Holder, English Department, Faculty of Foreign Languages and Literature, University of Tehran, Tehran, Iran

2 Assistant Professor, English Department, Faculty of Foreign Languages and Literature, University of Tehran, Tehran, Iran

3 Associate Professor, Department of English Language and Literature, Faculty of Persian Literature and Foreign Languages, Allameh Tabataba’i University, Tehran, Iran

Abstract

The current study strived to delve into the response behavior and perceptions of examinees while taking a test in light of cognitive load theory. The empirical data were collected from 60 MA English major graduates and students, with a high level of language proficiency. The participants were required to answer 60 multiple-choice language items (i.e., grammar and vocabulary questions), ‌‌taken from the high-stakes tests of the MA English majors of the Iranian university entrance examination (IUEE), as fast and as accurately as possible. After completing each test item, they rated their perceptions with regard to the difficulty of test items (Bratfisch et al., 1972) and the amount of mental effort (Paas, 1992). Their response time spent on each language item and their selected options were also stored by the Psychopy software (Peirce et al., 2019). Through running Pearson and Spearman rho correlations, the findings revealed that response time enjoyed a strong positive correlation with mental effort, meaning that both objective and subjective cognitive load measures matched in terms of sensitivity to cognitive load changes in language test items. Further, the subjective measures of perceived mental effort and perceived level of difficulty revealed to be the sound indicators of cognitive load changes. As predicted, response time also indicated that more difficult language test items imposed a greater amount of load. The implications of the study will be explained.

Keywords

References

  1.  Antonenko, P. D., & Niederhauser, D. S. (2010). The influence of leads on cognitive load and learning in a hypertext environment. Computers in Human Behavior, 26(2), 140-150. https://doi.org/10.1016/j.chb.2009.10.014
  2. Ary, D., Jacobs, L. C., Irvine, S., & Walker, D. (2019). Introduction to research in education (10th ed.). Wadsworth Cengage Learning.
  3. Ayres, P. (2006). Using subjective measures to detect variations of intrinsic cognitive load within problems. Learning and Instruction, 16(5), 389-400. https://doi.org/10.1016/j.learninstruc.2006.09.001
  4. Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch analysis in the human sciences. Springer.
  5. Bratfisch, O., Borg, G., & Dornic, S. (1972). Perceived item-difficulty in three tests of intellectual performance capacity (Report No. 29). Institute of Applied Psychology.
  6. Brünken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in multimedia learning. Educational Psychologist, 38(1), 53-61. https://doi.org/10.1207/S15326985EP3801_7
  7. Brünken, R., Seufert, T., & Paas, F. (2010). Measuring cognitive load. In J. Plass, R. Moreno, & R. Brünken (Eds.), Cognitive load theory (pp. 181-202). Cambridge University Press. https://doi.org/10.1017/CBO9780511844744.011
  8. Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8(4), 293-332. https://doi.org/10.1207/s1532690xci0804_2 
  9. Dave, A. (2004). Oxford placement test. Oxford University Press.                                
  10. de Jong, T. (2010). Cognitive load theory, educational research, and instructional design: Some food for thought. Instructional Science, 38(2), 105-134. https://doi.org/10.1007/s11251-009-9110-0
  11. DeLeeuw, K. E., & Mayer, R. E. (2008). A comparison of three measures of cognitive load: Evidence for separable measures of intrinsic, extraneous, and germane load. Journal of Educational Psychology, 100(1), 223-234. https://doi.org/10.1037/0022-0663.100.1.223
  12. Dindar, M., Yurdakul, I. K., & Dönmez, F. I. (2015). Measuring cognitive load in test items: Static graphics versus animated graphics. Journal of Computer Assisted Learning, 31(2), 148-161. https://doi.org/10.1111/jcal.12086
  13. Dornyei, Z. (2007). Research methods in applied linguistics. Oxford University Press. 
  14. Gass, S. M., Behney, J., & Plonsky, L. (2013). Second language acquisition: An introductory course (4th ed.). Routledge.
  15. Goldhammer, F., Naumann, J., Stelter, A., Tóth, K., Rölke, H., & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106(3), 608-626. https://doi.org/10.1037/a0034716
  16. Gvozdenko, E., & Chambers, D. (2007). Beyond test accuracy: Benefits of measuring response time in computerised testing. Australasian Journal of Educational Technology, 23(4), 542-558. https://doi.org/10.14742/ajet.1251
  17. Johannsen, G. (1979). Workload and workload measurement. In N. Moray (Ed). Mental workload: Its theory and measurement, (pp. 3-11). Springer.
  18. Lee, H. (2014). Measuring cognitive load with electroencephalography and self-report: Focus on the effect of English-medium learning for Korean students. Educational Psychology, 34(7), 838-848. https://doi.org/10.1080/01443410.2013.860217
  19. Lee, J. (2019). Task complexity, cognitive load, and L1 speech. Applied Linguistics, 40(3), 506-539. https://doi.org/10.1093/applin/amx054
  20. Leppink, J. (2017). Cognitive load theory: Practical implications and an important challenge. Journal of Taibah University Medical Sciences, 12(5), 385-391. https://doi.org/10.1016/j.jtumed.2017.05.003
  21. Martin, S. (2014). Measuring cognitive load and cognition: Metrics for technology enhanced learning. Educational Research and Evaluation: An International Journal on Theory and Practice, 20(8), 592-621. https://doi.org/10.1080/13803611.2014.997140
  22. Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81-97. https://doi.org/10.1037/h0043158
  23. Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84(4), 429-434. https://doi.org/10.1037/0022-0663.84.4.429
  24. Paas, F., Ayres, P., & Pachman, M. (2008). Assessment of cognitive load in multimedia learning environments: Theory, methods, and applications. In D. Robinson & G. Schraw (Eds.), Recent innovations in educational technology that facilitate student learning (pp. 11-35). Information Age Publishing.
  25. Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), 1-4. https://doi.org/10.1207/S15326985EP3801_1
  26. Paas, F., Tuovinen, J. E., Tabbers, H., & van Gerven, P. W. M. (2003). Cognitive load measurements as a means to advance cognitive load theory. Educational Psychologist, 38 (1), 63-71. https://doi.org/10.1207/S15326985EP3801_8
  27. Paas, F., & van Merriënboer, J. J. G. (1993). The efficiency of instructional conditions: An approach to combine mental effort and performance measures. Human Factors, 35(4), 737-743. https://doi.org/10.1177/001872089303500412
  28. Pallant, J. (2016). SPSS survival manual: A step by step guide to data analysis using IBM SPSS (6th ed.). George Allen & Unwin.
  29. Peirce, J., Gray, J. R., Simpson, S., MacAskill, M., Höchenberger, R., Sogo, H., Kastman E.,  & Lindeløv, J. K. (2019). Psychopy2: Experiments in behavior made easy. Behavior Research Methods, 51(1), 195-203. https://doi.org/10.3758/s13428-018-01193-y
  30. Peterson, L., & Peterson, M. J. (1959). Short-term retention of individual verbal items. Journal of Experimental Psychology, 58(3), 193-198. https://doi.org/10.1037/h0049234
  31. Pollock, E., Chandler, P., & Sweller, J. (2002). Assimilating complex information. Learning and Instruction, 12(1), 61-86. https://doi.org/10.1016/S0959-4752(01)00016-0
  32. Ponce, H. R., Mayer, R. E., Sitthiworachart, J., & Lopez, M. J. (2020). Effects on response time and accuracy of technology-enhanced cloze tests: An eye-tracking study. Educational Technology Research and Development, 68, 2033-2053. https://doi.org/10.1007/s11423-020-09740-1
  33. Pouw, W. T., Eielts, C., van Gog, T., Zwaan, R. A., & Paas, F. (2016). Does (non)meaningful sensori‐motor engagement promote learning with animated physical systems? Mind, Brain, and Education, 10, 91-104. https://doi.org/10.1111/mbe.12105
  34. Prisacari, A. A., & Danielson, J. (2017). Computer-based versus paper-based testing: Investigating testing mode with cognitive load and scratch paper use. Computers in Human Behavior, 77, 1-10. https://doi.org/10.1016/j.chb.2017.07.044
  35. Révész, A., Michel, M., & Gilabert, R. (2016). Measuring cognitive task demands using dual-task methodology, subjective self-ratings, and expert judgments: A validation study. Studies in Second Language Acquisition, 38(4), 703-737. https://doi.org/10.1017/S0272263115000339
  36. Révész, A., Sachs, R., & Hama., M. (2014). The effects of task complexity and input frequency on the acquisition of the past counterfactual construction through recasts. Language Learning, 64, 615-650. https://doi.org/10.1111/lang.12061
  37. Sasayama, S. (2016). Is a ‘complex’ task really complex? Validating the assumption of cognitive task complexity. The Modern Language Journal, 100, 231-254. https://doi.org/10.1111/modl.12313  
  38.  Scheiter, K., Ackerman, R., & Hoogerheide, V. (2020).  Looking at mental effort appraisals through a metacognitive lens: Are they biased? Educational Psychology Review, 32(4), 1003-1027.
  39. Schnotz, W., & Kürschner, C. (2007). A reconsideration of cognitive load theory. Educational Psychology Review, 19(4), 469-508. https://doi.org/10.1007/s10648-007-9053-4
  40. Shohamy, E., Donitsa-Schmidt, S., & Ferman, I. (1996). Test impact revisited washback effect  over time. Language Testing, 13(3), 298-317. https://doi.org/10.1177/026553229601300305
  41. Skulmowski, A., & Rey, G. D. (2017). Measuring cognitive load in embodied learning settings. Frontiers in Psychology, 8, 1-6. https://doi.org/10.3389/fpsyg.2017.01191
  42. Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4(4), 295-312. https://doi.org/10.1016/0959-4752(94)90003-5
  43. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257-285.  https://doi.org/10.1207/s15516709cog1202_4
  44. Sweller, J. (1999). Instructional design in technical areas. ACER.
  45. Sweller, J. (2010). Cognitive load theory: Recent theoretical advances. In J. L. Plass, R. Moreno, & R. Brünken (Eds.), Cognitive load theory (pp. 29-47). Cambridge University Press. https://doi.org/10.1017/CBO9780511844744.004
  46. Sweller, J. (2020). Cognitive load theory and educational technology. Educational Technology Research and Development, 68(1), 1-16. https://doi.org/10.1007/s11423-019-09701-3
  47. Sweller, J., Ayres, P., Kalyuga, S. (2011). Cognitive load theory. Springer.
  48. Sweller, J., van Merriënboer, J. J. G., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251-296. https://doi.org/10.1023/A:1022193728205
  49. Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019). Cognitive architecture and instructional design: 20 years later. Educational Psychology Review, 31, 261-292.
  50. van der Linden, W. J. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46, 247-272. https://doi.org/10.1111/j.1745-3984.2009.00080.x
  51. van Gerven, P. W. M., Paas, F., van Merriënboer, J. J. G., & Schmidt, H. G. (2006). Modality and variability as factors in training the elderly. Applied Cognitive Psychology, 20(3), 311-320. https://doi.org/10.1002/acp.1247
  52. van Gog, T., & Paas, F.  (2008). Instructional efficiency: Revisiting the original construct in educational research. Educational Psychologist, 43(1), 16-26. https://doi.org/10.1080/00461520701756248  
  53. Whelan, R. R. (2007). Neuroimaging of cognitive load in instructional multimedia. Educational Research Review, 2(1),1-12. https://doi.org/10.1016/j.edurev.2006.11.001
  54. Wiebe, E. N., Roberts, E., & Behrend, T. S. (2010). An examination of two mental workload measurement approaches to understanding multimedia learning. Computers in Human Behavior, 26(3), 474-481. https://doi.org/10.1016/j.chb.2009.12.006
  55. Young, M. S., Brookhuis, K. A., Wickens, C. D., & Hancock, P. A. (2015).  State of science: Mental workload in ergonomics. Ergonomics, 58(1), 1-17. https://doi.org/10.1080/00140139.2014.956151

Files

Share

How to cite

Statistics