¿Puede ayudar la inteligencia artificial (IA) en la educación en ciencias de la computación? Un enfoque metaanalítico
DOI
10.22550/2174-0909.4172
Abstract
Several studies have investigated the effect of Artificial Intelligence (AI) on stu- dents’ learning achievement in education. However, limited research targeted Computer Science (CS) education, which is considered crucial regardless of the future profession. Consequently, scant information exists on how AI might impact students’ learning achievement in CS education. To address this research gap, this study conducts a systematic review and a meta-analysis to investigate how AI integration affects learning achievement in CS education and the potential moderating variables of this effect. Specifically, 28 studies (n = 2765 participants in total) were included and meta-analyzed, and the obtained effect size was very large (g = 1.36, p g=1.45; huge) as an AI technology. Additionally, the AI intervention duration and the geographical distribution of students are found to moderate the AI effect in CS education. The findings of this study can serve as a reference for various stakeholders (e. g., educators, computer scientists, instructional designers) on how to integrate AI and improve learning experiences and outcomes in CS education.
Referencias | References
References with an asterisk (*) indicate studies included in the analysis
AAAI. (2018). AAAI Launches “AI for K-12” Initiative in collaboration with the Computer Science Teachers Association (CSTA) and AI4All. https://m.aaai.org/Pressroom/Releases/ release-18-0515.pdf
*Abbasi, S., & Kazi, H. (2014). Measuring effectiveness of learning chatbot systems on student’s learning outcome and memory retention. Asian Journal of Applied Science and Engineering, 3 (2), 251-260.
*Abbasi, S., Kazi, H., & Hussaini, N. N. (2019). Effect of chatbot systems on student’s learning outcomes. Sylwan, 163 (10), 49-63.
*Ahn, Y.-H., y Oh, E.-Y. (2024). Effects of the international training program for enhancing intelligent capabilities through blended learning on computational thinking, artificial intelligence competencies, and core competencies for the future society in graduate students. Applied Sciences, 14 (3). https://doi.org/10.3390/app14030991
*Al-Abdullatif, A. M., Al-Dokhny, A. A., & Drwish, A. M. (2023). Implementing the Bashayer chatbot in Saudi higher education: Measuring the influence on students’ motivation and learning strategies. Frontiers in psychology, 14, 1129070. https://doi.org/10.3389/fpsyg.2023.1129070
*Arsovic, B., & Stefanovic, N. (2020). E-learning based on the adaptive learning model: Case study in Serbia. Sādhanā, 45 (1), 266. https://doi.org/10.1007/s12046-020-01499-8
Barnes, T., Boyer, K., Hsiao, S. I. H., Le, N. T., & Sosnovsky, S. (2017). Preface for the special issue on AI-supported education in computer science. International Journal of Artificial Intelligence in Education, 27, 1-4. https://doi.org/10.1007/s40593-016-0123-y
Borenstein, M. (2022). Comprehensive meta-analysis software. In M. Egger, J. P. T. Higgins, & G. D. Smith (Eds.), Systematic reviews in health research: Meta-analysis in context (pp. 535-548). BMJ Books. https://doi.org/10.1002/9781119099369.ch27
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research synthesis methods, 1 (2), 97-111. https://doi.org/10.1002/jrsm.12
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021). Introduction to meta-analysis (2nd ed.). John Wiley & Sons.
Çağataylı, M., & Çelebi, E. (2022). Estimating academic success in higher education using big five personality traits, a machine learning approach. Arab Journal Scientific Engineering, 47, 1289-1298. https://doi.org/10.1007/s13369-021-05873-4
*Chang, Y. H., Chen, Y. Y., Chen, N. S., Lu, Y. T., & Fang, R. J. (2016). Yet another adaptive learning management system based on Felder and Silverman’s learning styles and Mashup. Eurasia Journal of Mathematics, Science and Technology Education, 12 (5), 1273-1285. https://doi.org/10.12973/eura-sia.2016.1512a
Crow, T., Luxton-Reilly, A., & Wuensche, B. (2018). Intelligent tutoring systems for programming education: A systematic review. In Proceedings of the 20th Australasian Computing Education Conference (pp. 53-62). Association for Computing Machinery.
*Cuong, N. D. H., Arch-Int, N., & Arch-Int, S. (2018). FUSE: A fuzzy-semantic framework for personalizing learning recommendations. International Journal of Information Technology & Decision Making, 17 (4), 1173-1201. https://doi.org/10.1142/S0219622018500220
Dever, D. A., Azevedo, R., Cloude, E. B., & Wiedbusch, M. (2020). The impact of autonomy and types of informational text presentations in game-based environments on learning: Converging multi-channel processes data and learning outcomes. International Journal of Artificial Intelligence in Education, 30 (4), 581-615. https://doi.org/10.1007/s40593-020-00215-1
*Eryılmaz, M., & Adabashi, A. (2020). Development of an intelligent tutoring system using Bayesian networks and fuzzy logic for a higher student academic performance. Applied Sciences, 10 (19), 6638. https://doi.org/10.3390/app10196638
*Essel, H. B., Vlachopoulos, D., Tachie-Menson, A., Johnson, E. E., & Baah, P. K. (2022). The impact of a virtual teaching assistant (chatbot) on students’ learning in Ghanaian higher education. International Journal of Educational Technology in Higher Education, 19 (1), 1-19. https://doi.org/10.1186/s41239-022-00362-6
*Farah, J. C., Spaenlehauer, B., Sharma, V., Rodríguez-Triana, M. J., Ingram, S., & Gillet, D. (2022). Impersonating chatbots in a code review exercise to teach software engineering best practices. In 2022 IEEE Global Engineering Education Conference (EDUCON) (pp. 1634-1642). IEEE.
*Fidan, M., & Gencel, N. (2022). Supporting the instructional videos with chatbot and peer feedback mechanisms in online learning: The effects on learning performance and intrinsic motivation. Journal of Educational Computing Research, 60 (7), 1716-1741. https://doi.org/10.1177/07356331221077901
Francisco, R. E., & de Oliveira, F. (2022). Intelligent tutoring system for computer science education and the use of artificial intelligence: A literature review. CSEDU, (1), 338- 345.
García-Martínez, I., Fernández-Batanero, J., Fernández-Cerero, J., & León, S. (2023). Analysing the impact of artificial intelligence and computational sciences on student performance: Systematic review and meta-analysis. Journal of New Approaches in Educational Research, 12 (1), 171-197. https://doi.org/10.7821/naer.2023.1.1240
Gerdes, A., Heeren, B., Jeuring, J., & Van Binsbergen, L. T. (2017). Ask-Elle: An adaptable programming tutor for Haskell giving automated feedback. International Journal of Artificial Intelligence in Education, 27, 65-100. https://doi.org/10.1007/s40593-015-0080-x
Goel, A. K., & Joyner, D. A. (2017). Using AI to teach AI: Lessons from an online AI class. Ai Magazine, 38 (2), 48-59. https://doi.org/10.1609/aimag.v38i2.2732
*Grivokostopoulou, F., Perikos, I., & Hatzily- geroudis, I. (2017). An educational system for learning search algorithms and automatically assessing student performance. International Journal of Artificial Intelligence in Education, 27, 207-240. https://doi.org/10.1007/s40593-016-0116-x
*Hayashi, Y. (2019). Multiple pedagogical conversational agents to support learner-learner collaborative learning: Effects of splitting suggestion types. Cognitive Systems Research, 54, 246-257. https://doi.org/10.1016/j.cogsys.2018.04.005
Hedges, L. (1981). Distribution theory for glass’s estimator of effect size and related estimators. Journal of Educational Statistics, 6 (2), 107-128. https://doi.org/10.3102/10769986006002107
Hedges, L., & Olkin, I. (1985). Statistical methods for meta-analysis. Academic Press.
Hwang, S. (2022). Examining the effects of artificial intelligence on elementary students’ mathematics achievement: A meta-analysis. Sustainability, 14 (20), 1-20. https://doi.org/10.3390/su142013185
Jarota, M., (2023). Artificial intelligence in the work process. A reflection on the proposed European Union regulations on artificial intelligence from an occupational health and safety perspective. Computer Law & Security Review, 49, 1-14, https://doi.org/10.1016/j.clsr.2023.105825
Kim, J., & Lee, SS. (2023). Are two heads better than one? The effect of student-AI collaboration on students’ learning task performance. TechTrends. 67, 365-375. https://doi.org/10.1007/s11528-022-00788-9
Kitchenham, B. A., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Keele University and Durham University.
*Kumar, J. A. (2021). Educational chatbots for project-based learning: investigating learning outcomes for a team-based design course. International journal of educational technology in higher education, 18 (1), 1-28. https://doi.org/10.1080/10494820.2019.1580750
*Lai, Y. H., Chen, S. Y., Lai, C. F., Chang, Y. C., & Su, Y. S. (2021). Study on enhancing AIoT computational thinking skills by plot image-based VR. Interactive Learning Environments, 29 (3), 482-495. https://doi.org/10.1080/10494820.2019.1580750
*Lee, Y. F., Hwang, G. J., & Chen, P. Y. (2022). Impacts of an AI-based chatbot on college students’ after-class review, academic performance, self-efficacy, learning attitude, and motivation. Educational technology research and development, 70 (5), 1843- 1865. https://doi.org/10.1007/s11423-022-10142-8
Lin, L., Chu, H., Murad, M. H., Hong, C., Qu, Z., Cole, S. R., & Chen, Y. (2018). Empirical comparison of publication bias tests in meta-analysis. Journal of General Internal Medicine, 33 (8), 1260-1267. https://doi.org/10.1007/s11606-018-4425-7
Lin, R., Zhang, Q., Xi, L., & Chu, J. (2022). Exploring the effectiveness and moderators of artificial intelligence in the classroom: A meta-analysis. In J. Yang, D. Liu, Kinshuk, A. Tlili, M. Chang, E. Popescu, D. Burgos, & Z. Altınay (Eds.), Resilience and future of smart learning. Proceedings of 2022 International Conference on Smart Learning Environments. Springer. https://doi.org/10.1007/978-981-19-5967-7_7
Lin, C. C., Huang, A. Y., & Lu, O. H. (2023). Artificial intelligence in intelligent tutoring systems toward sustainable education: A systematic review. Smart Learning Environments, 10 (1), 41. https://doi.org/10.1186/s40561-023-00260-y
*Ma, Z. H., Hwang, W. Y., & Shih, T. K. (2020). Effects of a peer tutor recommender system (PTRS) with machine learning and automated assessment on vocational high school students’ computer application operating skills. Journal of Computers in Education, 7, 435-462. https://doi.org/10.1007/s40692-020-00162-9
Mihalic, S. (2004). The importance of implementation fidelity. Emotional and Behavioral Disorders in Youth, 4 (4), 83-105.
Merilampi, S., Sirkka, A., Leino, M., Koivisto, A. and Finn, E. (2014). Cognitive mobile games for memory impaired older adults. Journal of Assistive Technologies, 8 (4), 207-223. https://doi.org/10.1108/JAT-12-2013-0033
Nesbit, J. C., Adesope, O. O., Liu, Q., & Ma, W. (2014). How effective are intelligent tutoring systems in computer science education? In The 14th IEEE International Conference on Advanced Learning Technologies (ICALT 2014). Advanced Technologies for Supporting Open Access to Formal and Informal Learning (pp. 99-103). IEEE.
Nesbit, J., Liu, L., Liu, Q., and Adesope, O. O. (2015). Work in progress: Intelligent tutoring systems in computer science and software engineering education. In 2015 ASEE Annual Conference and Exposition Proceedings (pp. 26.1754.1-26.1754.12). American Society for Engineering Education.
Nye, B. D. (2015). Intelligent tutoring systems by and for the developing world: A review of trends and approaches for educational technology in a global context. International Journal of Artificial Intelligence in Education, 25, 177-203. https://doi.org/10.1007/s40593-014-0028-6
Onesi-Ozigagun, O., Ololade, Y. J., Eyo-Udo, N. L., & Ogundipe, D. O. (2024). Revolutionizing education through AI: A comprehensive review of enhancing learning experiences. International Journal of Applied Research in Social Sciences, 6 (4), 589-607. https://doi.org/10.51594/ijarss.v6i4.1011
*Ortega-Ochoa, E., Quiroga, J., Arguedas, M., Daradoumis, T., & Marquès, J. M. (2024). The effectiveness of empathic chatbot feedback for developing computer competencies, motivation, self-regulation, and metacognitive reasoning in online higher education. Internet of Things, 25, 101101. https://doi.org/10.1016/j.iot.2024.101101
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, M. A., Stewart, L. A., Thomas, J. Tricco, A. C., …, y Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. International journal of surgery, 88, 105906.
*Perikos, I., Grivokostopoulou, F., & Hatzily- geroudis, I. (2017). Assistance and feedback mechanism in an intelligent tutoring system for teaching conversion of natural language into logic. International Journal of Artificial Intelligence in Education, 27, 475-514. https://doi.org/10.1007/s40593-017-0139-y
Pirker, J., Riffnaller-Schiefer, M., & Gütl, C. (2014, June). Motivational active learning: Engaging university students in computer science education. In Proceedings of the 2014 Conference on Innovation y Technology in Computer Science Education (pp. 297-302). Association for Computing Machinery.
Pisapia, J., Schlesinger, J., & Parks, A. (1993). Learning technologies in the classroom: Review of the literature. Metropolitan Educational Research Consortium.
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86 (3), 638-641. https://doi.org/10.1037//0033-2909.86.3.638
Shaw, M. (1985). The nature of computer science. In M. Shaw, M (Ed.), The Carnegie-Mellon curriculum for undergraduate computer science. Springer.
Silva, D. B., Aguiar, R. d. L., Dvconlo, D. S., & Silla, C. N. (2019). Recent studies about teaching algorithms (CS1) and data structures (CS2) for computer science students. In 2019 IEEE Frontiers in Education Conference (FIE) (pp. 1-8). IEEE.
Smith, M. (2016, 30 January). Computer science for all [blog]. The White House - President Barack Obama. https://www.whitehouse.gov/blog/2016/01/30/computer-science-all
*Song, D., & Kim, D. (2021). Effects of self-regulation scaffolding on online participation and learning outcomes. Journal of Research on Technology in Education, 53 (3), 249-263. https://doi.org/10.1080/15391523.2020.1767525
*Stamper, J., Eagle, M., Barnes, T., & Croy, M. (2013). Experimental evaluation of automatic hint generation for a logic tutor. International Journal of Artificial Intelligence in Education, 22 (1-2), 3-17. https://doi.org/10.1007/978-3-642-21869-9_45
*Su, J. M. (2020). A rule-based self-regulated learning assistance scheme to facilitate personalized learning with adaptive scaffoldings: A case study for learning computer software. Computer Applications in Engineering Education, 28 (3), 536-555. https://doi.org/10.1002/cae.22222
*Sun, D., Boudouaia, A., Zhu, C., & Li, Y. (2024). Would ChatGPT-facilitated programming mode impact college students’ programming behaviors, performances, and perceptions? An empirical study. International Journal of Educational Technology in Higher Education, 21, 14. https://doi.org/10.1186/s41239-024-00446-5
Sung, Y. T., Chang, K. E., & Liu, T. C. (2016). The effects of integrating mobile devices with teaching and learning on students’ learning performance: A meta-analysis and research synthesis. Computers & Education, 94, 252-275. https://doi.org/10.1016/j.compedu.2015.11.008
Tangi, L., van Noordt, C., Combetto, M., Gattwinkel D., & Pignatelli F. (2022). AI watch. European landscape on the use of artificial intelligence by the public sector. Publications Office of the European Union. https://doi.org/10.2760/39336
*Tegos, S., & Demetriadis, S. (2017). Conversational agents improve peer learning through building on prior knowledge. Journal of Educational Technology & Society, 20 (1), 99-111.
*Tegos, S., Demetriadis, S., & Karakostas, A. (2015). Promoting academically productive talk with conversational agent interventions in collaborative learning settings. Computers & Education, 87, 309-325. https://doi.org/10.1016/j.compedu.2015.07.014
*Tegos, S., Demetriadis, S., Papadopoulos, P. M., & Weinberger, A. (2016). Conversational agents for academically productive talk: A comparison of directed and undirected agent interventions. International Journal of Computer-Supported Collaborative Learning, 11 (4), 417-440. https://doi.org/10.1007/s11412-016-9246-2
Tlili, A., Essalmi, F., & Jemni, M. (2015). An educational game for teaching computer architecture: Evaluation using learning analytics. In 2015 5th International Conference on Information & Communication Technology and Accessibility (ICTA) (pp. 1-6). IEEE.
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10 (1), 15. https://doi.org/10.1186/s40561-023-00237-x
Touretzky, D., Gardner-McCune, C., & Seehorn, D. (2023). Machine learning and the five big ideas in AI. International Journal of Artificial Intelligence in Education, 33 (2), 233-266. https://doi.org/10.1007/s40593-022-00314-1
Verdú, E., Regueras, L. M., Gal, E., De Castro, J. P., Verdú, M. J., & Kohen-Vacs, D. (2017). Integration of an intelligent tutoring system in a course of computer network design. Educational Technology Research and Development, 65, 653-677. https://doi.org/10.1007/s11423-016-9503-0
Waraich, A. (2004). Using narrative as a motivating device to teach binary arithmetic and logic gates. ACM SIGCSE Bulletin, 36 (3), 97-101. https://doi.org/10.1145/1026487.1008024
Wang, H., Tlili, A., Huang, R., Cai, Z., Li, M., Cheng, Z., Yang, D., Li, M., Zhu, X., & Fei, C. (2023). Examining the applications of intelligent tutoring systems in real educational contexts: A systematic literature review from the social experiment perspective. Education and information technologies, 28 (7), 9113-9148. https://doi.org/10.1007/s10639-022-11555-x
*Winkler, R., Hobert, S., Salovaara, A., Söllner, M., & Leimeister, J. M. (2020, April). Sara, the lecturer: Improving learning in online education with a scaffolding-based conversational agent. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-14). Association for Computing Machinery.
Wu, R., & Yu, Z. (2023). Do AI chatbots improve students learning outcomes? Evidence from a meta-analysis. British Journal of Educational Technology, 55 (1), 10-33. https://doi.org/10.1111/bjet.13334
*Yin, J., Goh, T. T., Yang, B., & Xiaobin, Y. (2021). Conversation technology with micro-learning: The impact of chatbot-based learning on students’ learning motivation and performance. Journal of Education- al Computing Research, 59 (1), 154-177. https://doi.org/10.1177/0735633120952067
Zhang, X., Tlili, A., Shubeck, K., Hu, X., Huang, R., & Zhu, L. (2021). Teachers’ adoption of an open and interactive e-book for teaching K-12 students artificial intelligence: A mixed methods inquiry. Smart Learning Environments, 8, 1-20. https://doi.org/10.1186/s40561-021-00176-5
Zheng, L., Niu, J., Zhong, L., & Gyasi, J. F. (2023). The effectiveness of artificial intelligence on learning achievement and learning perception: A meta-analysis. Interactive Learning Environments, 31 (9), 5650-5664. https://doi.org/10.1080/10494820.2021.2015693
Citación recomendada | Recommended citation
Tlili, A. (2024). Can artificial intelligence (AI) help in computer science education? A meta-analysis approach [¿Puede ayudar la inteligencia artificial (IA) en la educación en ciencias de la computación? Un enfoque metaanalítico]. Revista Española de Pedagogía, 82 (289), 469-490. https://doi.org/10.22550/2174-0909.4172
Licencia Creative Commons | Creative Commons License
Esta obra está bajo una licencia internacional Creative Commons Atribución-NoComercial 4.0.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License
Included in
Palabras clave | Keywords
Computer science, computing, artificial intelligence, education, learning, collaborative intelligence, meta-analysis, learning achievement.
