Evidence-Based Design Approach for Assessing Universal Competencies in Higher Education: Advantages and Features

The article addresses the issue of assessing universal competencies, which are becoming increasingly important given modern demands on university graduates. It emphasizes the need for valid and reliable tools to measure skills such as critical thinking, creativity, communication, and complex problem-solving. The paper presents an analysis of an evidence-based design approach, which integrates cognitive theories, innovative educational technologies, and psychometric models to achieve objective assessments of these competencies.

Particular attention is given to the evidence-centered design methodology, which relies on scenario-based tasks closely reflecting real-life and professional situations. This method emphasizes constructing valid argument that link students’ observable behaviors to claims about their competence levels. The approach not only captures demonstrations of knowledge and skills in realistic contexts but also structures the instrument development process to ensure that empirical results are theoretically sound. A distinctive feature of this approach is the process of building arguments from collected evidence, differentiating it from traditional methods, which focus primarily on the statistical and psychometric properties of tests.

The principles of this method are illustrated using the CT Test for assessing university students’ critical thinking, a tool with established psychometric quality, which has been piloted with over 10,000 students in Russian universities. The article argues that implementing evidence-based tools will substantially enhance the validity and reliability of competency assessments among students, making this approach a promising candidate for integration into educational practice.

1. Razumova T.O., Telekhova I.G. (2023). Transformation of the System of Higher Professional Education: Challenges and Perspectives. Standard of Living of the Population of the Regions of Russia. Vol. 19, no. 3, pp. 338-349, doi: 10.52180/1999-98362023193333834 (In Russ., abstract in Eng.).

2. Care, E; Kim, H; Vista, A; Anderson, K. (2018). Education System Alignment for 21st Century Skills: Focus on Assessment. The Center for Universal Education at the Brookings Institution. Available at: https://www.researchgate.net/publication/330740772_Education_system_alignment_for_21st_century_skills_Focus_on_assessment (accessed: 25.10.2024).

3. Mislevy, R.J. (2018). Sociocognitive Foundations of Educational Measurement. Routledge, doi: 10.1111/jedm.12255

4. Shmigirilova I.B., Rvanova A.S. Grigorenko O.V. (2021). Assessment in education: Current trends, problems and contradictions (review of scientific publications). Education and Science Journal. Vol. 23, no. 6, pp. 43-83, doi: 10.17853/1994-5639-2021-6-43-83 (In Russ., abstract in Eng.).

5. Achtenhagen, F. (2012). The curriculum-instruction-assessment triad. Empirical Research in Vocational Education and Training. No. 4, pp. 5-25, doi: 10.1007/BF03546504

6. Biggs, J., Tang, C. (2011). Teaching for Quality Learning at University: What the Student Does. New York: McGraw-Hill Education. 480 р. ISBN 10: 0335242758.

7. Mislevy, R.J. (2013). Four Metaphors We Need to Understand Assessment. The Gordon Commission on the Future of Assessment in Education, 39 p. Available at: https://www.ets.org/Media/Research/pdf/mislevy_four_metaphors_understand_assessment.pdf (accessed: 25.10.2024).

8. Pellegrino, J.W., Chudowsky, N., Glaser, R. (2001). Knowing What Students Know: The science and Design of Educational Assessment. Washington, DC: National Academy Press. Available at: https://www.researchgate.net/publication/270584995_Knowing_What_Students_Know_The_Science_and_Design_of_Educational_Assessment (accessed: 25.10.2024).

9. Lamri, J., Lubart T. (2023) Reconciling Hard Skills and Soft Skills in a Common Framework: The Generic Skills Component Approach. Journal of Intelligence. No. 11: 107, doi: 10.3390/jintelligence11060107, doi: 10.1007/s11135-021-01149-z

10. Pellegrino, J.W., Hilton, M.L. (Eds.) (2012). Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century. Washington, DC: National Academies Press, 256 p., doi: 10.17226/13398

11. Shavelson, R.J., Zlatkin-Troitschanskaia, O., Mariño, J.P. (2018). International performance assessment of learning in higher education (iPAL): Research and development. Assessment of Learning Outcomes in Higher Education. P. 193-214, doi: 10.1007/978-3-319-74338-7_10

12. McClelland, D.C. (1973). Testing for competence rather than for “intelligence”. American Psychologist. American Psychological Association. Vol. 28, no. 1, pp. 1-14, doi: 10.1037/h0034092

13. Spencer, L.M., Spencer S.M. (1993). Competence at Work: Models for Superior Performance. New York: John Wiley & Sons. ISBN 0-471-54809-х. Available at: https://www.wiley.com/enus/Competence+at+Work%3A+Models+for+Superior+Performance-p-9780471548096 (accessed: 25.10.2024).

14. Ercikan, K., Oliveri, M.E. (2016). In search of validity evidence in support of the interpretation and use of assessments of complex constructs: Discussion of research on assessing 21st century skills. Applied Measurement in Education. Vol. 29, no. 4, pp. 310-318, doi: 10.1080/08957347.2016.1209210

15. Zlatkin-Troitschanskaia O., Jitomirski J., Happ R., Molerov D., Schlax J., Kühling-Thees C., Förster M., Brückner S. Validating a test for measuring knowledge and understanding of economics among university students // Zeitschrift für Pädagogische Psychologie. No. 33 (2), pp. 119-133, doi: 10.1024/1010-0652/a000239

16. Shavelson R.J., Zlatkin-Troitschanskaia O., Beck K., Schmidt S., Mariño J.P. (2019). Assessment of university students’ critical thinking: Next generation performance assessment. International Journal of Testing. Vol. 19, no. 4, pp. 337-362, doi: 10.1080/15305058.2018.1543309

17. Messick, S. (1992). The Interplay of Evidence and Consequences in the Validation of Performance Assessments. Educational Researcher. Vol. 23, no. 2, pp. 13-23, doi: 10.3102/0013189X023002013

18. Mislevy, R.J. (1994). Evidence and inference in educational assessment. Psychometrika. Vol. 59, no. 4, pp. 439-483. Available at: http://www.springerlink.com/content/l6116h6652714714 (accessed: 12.10.2024).

19. Toulmin, S.E. (2008). The Uses of Argument. Updated edition. Cambridge University Press, 247 p., doi: 10.1017/cbo9780511840005

20. Mislevy, R.J. (2003). Substance and structure in assessment arguments. Law, Probability and Risk. No. 2, pp. 237-258, doi: 10.1093/lpr/2.4.237

21. Messick, S. (1989). Validity. In R.L. Linn (Ed.). Educational Measurement: 3rd ed., pp. 13–103. New York: American Council on Education/Macmillan. Available at: https://psycnet.apa.org/record/1989-97348-002 (accessed: 25.10.2024).

22. Ferrara, S.; Lai, E.; Nichols, P. (2016). Principled Approaches to Assessment Design, Development, and Implementation. The Handbook of Cognition and Assessment: Frameworks, Methodologies, and Applications. Pp. 41-74, doi: 10.1002/9781118956588.ch3

23. Mislevy, R.J., Almond, R.G., Lukas, J.F. (2003). A Brief Introduction to Evidence-centered Design. Princeton: Educational Testing Service, 37 p. Available at: https://files.eric.ed.gov/fulltext/ED483399.pdf https://psycnet.apa.org/record/1989-97348-002 (accessed: 25.10.2024).

24. Uglanova, I.L., Brun, I.V., Vasin, G.M. (2018). Evidence-Centered Design method for measuring complex psychological constructs. Journal of Modern Foreign Psychology. Vol. 7, no. 3, pp. 1827, doi: 10.17759/jmfp.2018070302 (In Russ., abstract in Eng.).

25. Efremova, N.F. (2020). Techniques of Evidence-Based Argumentation for Competency Assessment. Innovatsionnaia nauka: Psikhologia, Pedagogika, Defektologia = Innovative Science: Psychology, Pedagogy, Defectology. Vol. 3, no. 2, pp. 112-124. Available at: https://elibrary.ru/download/elibrary_44268469_64255654.pdf (accessed: 25.10.2024) (In Russ., abstract in Eng.).

26. Uglanova, I., Orel, E., Gracheva, D., Tarasova, K. (2023). Computer-based performance approach for critical thinking assessment in children. British Journal of Educational Psychology. Vol. 93, no. 2, pp. 531-544, doi: 10.1111/bjep.12576

27. Avdeeva, S.M., Rudnev, M.G., Vasin, G.M., Tarasova, K.V., Panova, D.M. (2017). Assessing Information and Communication Technology Competence of Students: Approaches, Tools, Validity and Reliability of Results. Educational Studies Moscow. No. 4, pp. 104-132, doi: 10.17323/18149545-2017-4-104-132 (In Russ., abstract in Eng.).

28. Avdeeva, S.M., Tarasova, K.V. (2023). On measuring digital literacy: methodology, conceptual model and measurement tool. Educational Studies Moscow. No. 2, pp. 8-32, doi: 10.17323/18149545-2023-2-8-32 (In Russ., abstract in Eng.).

29. Mislevy, R.J., Behrens, J., DiCerbo, K.E., Levy, R. (2012). Design and discovery in educational assessment: Evidence centered design, psychometrics, and data mining. Journal of Educational Data Mining. No. 4, pp. 11-48. Available at: http://www.educationaldatamining.org/JEDM/images/articles/vol4/issue1/MislevyEtAlVol4Issue1P11_48.pdf (accessed: 12.10.2024).

30. DiBello, L.V., Roussos, L.A., Stout, W. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. Handbook of Statistics. No. 26, pp. 979-1030, doi: 10.1016/ S0169-7161(06)26031-0

31. Tjoe, H., & de la Torre, J. (2014). The identification and validation process of proportional reasoning attributes: An application of a cognitive diagnosis modeling framework. Mathematics Education Research Journal. No. 26, pp. 237-255, doi: 10.1007/s13394-013-0090-7

32. Tarasova, K.V., Orel, E.A. (2022). Measuring Students’ Critical Thinking in Online Environment: Methodology, Conceptual Framework and Tasks Typology. Educational Studies Moscow. No. 3, pp. 187-212, doi: 10.17323/1814-9545-2022-3-187-212 (In Russ., abstract in Eng.).

33. Lai, E.R. (2011). Critical Thinking: A Literature Review Research Report. London: Parsons Publishing, Available at: http://paluchja-pajecia.home.amu.edu.pl/seminarium_fakult/sem_f_krytyczne/Critical%20Thinking%20A%20Literature%20Review.pdf (acces- sed: 25.10.2024).

34. Liu, O.L., Frankel, L., Roohr, K.C. (2014). Assessing Сritical Thinking in Higher Education: Current State and Directions for Next-Generation Assessment. ETS Research Report Series. No. 1, pp. 1-23, doi: 10.1002/ets2.12009

35. Riconscente, M., Mislevy, R., Hamel, L. (2005). An introduction to PADI task templates. PADI Technical Report. Vol. 3. Available at: https://padi.sri.com/downloads/TR3_Templates.pdf (accessed: 25.10.2024).

36. Mislevy, R.J., Riconscente, M.M., & Rutstein, D.W. (2009). Design patterns for assessing model-based reasoning. Large Systems Technical Report 6. Menlo Park, CA: SRI International. Available at: http://ecd.sri.com/downloads/ECD_TR6_Model Based_Reasoning.pdf (accessed: 25.10.2024).

37. Zlatkin-Troitschanskaia, O., Shavelson, R.J. (2019). Advantages and Challenges of Performance Assessment of Student Learning in Higher Education. British Journal of Educational Psychology. Vol. 89, no. 3, pp. 413-415, doi: 10.1111/bjep.12314

38. Wang, W.C., Su, C.M., Qiu, X.L. (2014). Item response models for local dependence among multiple ratings. Journal of Educational Measurement. Vol. 51, no. 3, pp. 260-280, doi: 10.1111/jedm.12045

39. Tarasova, K.V., Gracheva, D.A. (2023). Computational Psychometrics: The Near Future orAlready a Reality. Review of the Book “Computational Psychometrics: New Methodologies for a New Generation of Digital Learning and Assessment“. Educational Studies Moscow. No. 3, pp. 221-230, doi: 10.17323/vo-2023-17938

40. Andrews-Todd, J., Forsyth, C.M. (2020). Exploring Social and Cognitive Dimensions of Collaborative Problem Solving in an Open Online Simulation-Based Task. Computers in Human Behavior. Vol. 104, article no. 105759, doi: 10.1016/j.chb.2018.10.025

41. Almond, R., Steinberg, L., Mislevy, R. (2002). Enhancing the design and delivery of assessment systems: A four-process architecture. Journal of Technology, Learning, and Assessment. Vol. 1, no. 5. Available at: https://ejournals.bc.edu/index.php/jtla/article/view/1671 (accessed: 25.10.2024).

42. Gracheva, D.A., Tarasova, K.V. (2022). Approaches to the development of scenario-based task forms within the framework of evidence-centered design. Otechestvennaia i zarubezhnaia pedagogika = Domestic and Foreign Pedagogy. Vol. 1, no. 3(84), pp. 83-97, doi: 10.24412/22240772-2022-84-83-97 (In Russ., abstract in Eng.).

43. Yan, D.; Rupp, A.; Foltz, P. (2020). The Handbook of Automated Scoring: Theory into Practice. CRC Press. ISBN: 9781351264808, DOI: 10.1201/9781351264808.

44. De Klerk, S., Eggen, T.J.H.M., Veldkamp, B.P. (2016). A Methodology for Applying Students’ Interactive Task Performance Scores from a Multimedia-based Performance Assessment in a Bayesian Network. Computers in Human Behavior. Vol. 60, i. C, pp. 264-279, doi: 10.1016/j.chb.2016.02.071

45. Mislevy, R. (2024). Sociocognitive and argumentation perspectives on psychometric modeling in educational assessment. Psychometrika. Vol. 89, no. 1, doi: 10.1007/s11336-024-09966-5