Validation of an instrument to evaluate art apps that enable the development of artistic skills in digital settings

Validación de un instrumento de evaluación de apps de arte que permiten desarrollar la competencia artística en entornos digitales

Feliciano F. Ordóñez Fernández, Martín Caeiro Rodríguez, Ana María Barbero Franco, María del Mar Martínez-Oña and Alberto Torres Pérez

DOI: https://doi.org/10.22550/REP79-3-2021-02

Apps that help art teachers work on areas relating directly to artistic competences are a valuable digital teaching resource. In this study we call them art apps. Aims: To validate a measurement instrument for digital applica­tions based on the results we obtained in the first design phase, and to modify the items and identify factors. Methodology: We carried out three processes in this second phase. An exploratory factorial analysis, a pre-confirm­atory analysis, and a confirmatory factorial analysis adjusted by the global or absolute fit indices for all of the scales of the test. This was based on an evaluation battery comprising 98 items distributed in three dimensions and four scales: artistic dimension (expressive and perceptive domain), technical dimension, and pedagogical dimension. Results: We obtained a definitive model of all the scales with optimal values in all the adjustment indicators. Dis­cussion/Conclusion: After completing this pro­cess, we designed the definitive test based on the results of the fitting. Taking into account the resulting number of items (74), the num­ber of factors (8), and the wording of each one, we consider it to be a useful and viable tool for evaluating art apps by connecting their ar­tistic, technical, and pedagogical dimensions according to the needs and possibilities of the educational activities.

 


 

Please, cite this article as follows: Ordoñez Fernández, F. F., Caeiro Rodríguez, M., Barbero Franco, A. M., Martínez Oña, M. M., & Torres Pérez, A. (2021). Validación de un instrumento de evaluación de apps de arte que permiten desarrollar la competencia artística en entornos digitales | Validation of an instrument to evaluate art apps that enable the development of artistic skills in digital settings. Revista Española de Pedagogía, 79 (280), 515-536. https://doi.org/10.22550/REP79-3-2021-02

 

Aiello, M., & Willem, C. (2004). El Blended Lear­ning como práctica transformadora [Blended Learning as a transformative practice]. Pixel­Bit. Revista de Medios y Educación, 23, 21-26. https://recyt.fecyt.es/index.php/pixel/article/ download/61238/37252

Aiken, L. (2003). Test psicológicos y evaluación [Psychological testing and assessment]. Pear­son Education.

Bajardi, A. (2015). B-Learning y arte contemporá­neo en educación artística: Construyendo iden­tidades personales y profesionales [B-Learand contemporary art in art education: Building personal and professional identities] [Docto­ral dissertation, Universidad de Granada]. DIGIBUG: Institutional repository of the Universidad de Granada. http://hdl.handle. net/10481/41757

Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of co­variance structures. Psychological Bulletin, 88, 588-606.

Bentrop, S. M. (2014). Creating an educational app rubric for teachers of students who are deaf and hard of hearing (Publication No. 680) [Docto­ral dissertation, Washington University School of Medicine in St. Louis]. Digital Commons@ Becker.

Bouck, E. C., Satsangi, R., & Flanagan, S. (2016). Focus on inclusive education: Evaluating apps for students with disabilities: Supporting aca­demic access and success. Childhood Educa­tion, 92 (4), 324- 328.

Burbules, N. (2014). El aprendizaje ubicuo: nuevos contextos, nuevos procesos [Ubiquitous learn-ing: New contexts, new processes]. Revista Entramados-Educación y sociedad, 1, 131-135. https://bit.ly/2lHdSE1

Caeiro Rodríguez, M. (2015). Tecnopatías de la educación artística. Reflexiones en torno a la enseñanza y aprendizaje digital del arte [Tech­nopathologies of art education. Reflections on digital art teaching and learning]. In J. A. Ibá­ñez Martín & J. L. Fuentes (Eds.), Aprendizaje Ético-Cívico en Entornos Virtuales: Simposio Internacional de Filosofía de la Educación (pp. 11-26). Biblioteca ONLINE.

Caeiro Rodríguez, M. (2020). Arts and Apps: apli­caciones digitales y educación artística. Tres instrumentos para trabajar con Apps de Arte [Arts and Apps: digital applications and arts education. Three tools for working with Arts Apps]. In E. Sánchez Rivas, E. Colomo Ma­gaña, J. Ruiz Palmero, & Sánchez Rodríguez (Eds.), Tecnologías educativas y estrategias di­dácticas (pp. 210-225). UMA Editorial. https:// hdl.handle.net/10630/20345

Caeiro Rodríguez, M., & Navarrete, C. (2020). Ex­periencias B-learning con Apps de Arte en la asignatura de Educación Plástica, Visual y Au­diovisual: logotipo y stopmotion como armas de construcción masiva [B-learning experiences with Art Apps in the subject of Plastic, Visual and Audiovisual Education: Logo and stopmo­tion as weapons of mass construction]. REI­DOCREA, 9, 107-120. https://www.ugr.es/~rei­docrea/9-9.pdf

Caeiro Rodríguez, M., Ordóñez, F. F., Callejón, M. D., & Castro-León, E. (2020). Diseño de un instrumento de evaluación de aplicaciones digitales (apps) que permiten desarrollar la competencia artística [Design of an evaluation instrument for digital applications (apps) that allow the development of artistic competence]. Pixel-Bit. Revista de Medios y Educación, 58, 7-25. https://doi.org/10.12795/pixelbit.74071

Camilleri, A., & Camilleri, M. (2019). Mobile Lear­ning via Educational Apps: An Interpretative Study. ICETT 2019, 88-92. https://dl.acm.org/ doi/pdf/10.1145/3337682.3337687

Campo-Arias, A., & Oviedo, H. C. (2008). Propieda­des psicométricas de una Escala: La Consisten­cia interna. Revista de Salud Pública, 10 (5), 831-839.

Cherner, T., Dix, J., & Lee, C. (2014). Cleaning up that mess: A framework for classifying educa­tional apps. Contemporary Issues in Techno­logy and Teacher Education, 14 (2). http://bit. ly/2VEiwQW

Del-Moral, M. E., Bellver, M. C., &. Guzmán-Du­que, A. P. (2019). Evaluación de la potenciali­dad creativa de aplicaciones móviles creadoras de relatos digitales para Educación Primaria [Evaluation of the creative potential of mobile applications for digital storytelling in Primary

 

School Education]. Ocnos, 18 (1), 7-20. https:// doi.org/10.18239/ocnos_2019.18.1.1866

DiStefano, C., Liu, J., Jiang, N., & Shi, D. (2018). Examination of the weighted root mean square residual: Evidence for trustworthiness? Struc­tural Equation Modeling, 25 (3), 453-466.

Escaño González, C. (2010). Hacia una educación artística 4.0. [Towards an Art Education 4.0.]. Revista Arte, Individuo y Sociedad, 2 (1), 135- 144. https://bit.ly/2kCVtb1

Escobar-Pérez, J., & Cuervo-Martínez, A. (2008). Validez de contenido y juicio de expertos: una aproximación a su utilización [Content validity and expert assessment: An approach to their use]. Avances en Medición, 6, 27-36.

Fernández-Cruz, F. J., Fernández-Díaz, M. J., & Rodríguez-Mantilla, J. M. (2018). Diseño y validación de un instrumento de medida del perfil de formación docente en tecnologías de la información y comunicación | Design and validation of an instrument to measure teacher training profiles in information and communi­cation technologies. revista española de pe­dagogía, 76 (270), 247-270.

Ferrando, P. J., & Lorenzo-Seva, U. (2016). A note on improving EAP trait estimation in oblique factor-analytic and item response theory mo­dels. Psicologica, 37, 235-247.

Fombona Cadavieco, J., Pascual Sevillano, M. A., & Vázquez-Cano, E. (2020). M-Learning en niveles iniciales, rasgos didácticos de las APPS educativas [M-Learning in initial levels, didactic features of educational APPS]. Cam­pus Virtuales, 9 (1), 17-27. http://hdl.handle. net/10272/17786

García de Rozas, A. M. (2017). La educación artísti­ca a través de las nuevas tecnologías: El uso de la tablet [Arts Education through new technolo­gies: The use of the tablet] [Final degree thesis, Universidad de Valladolid]. Document reposi­tory of the Universidad de Valladolid. http:// uvadoc.uva.es/handle/10324/26415

García-Umaña, A., & Córdoba, E. (2020). Vali­dación de la escala MPPUS-A sobre el uso problemático del smartphone [Validation of scale MPPUS-A on the problematic use of the smartphone]. Pixel-Bit. Revista de Me­dios y Educación, 57, 173-189. https://doi. org/10.12795/pixelbit.2020.i57.07

Giráldez, A. (2013). Enseñar artes en un mundo digital: diez propuestas para la formación del profesorado [Teaching the arts in a digital world: Ten proposals for teacher education]. Belo Horizonte, 3 (5), 85-104.

Hart, A. (2001). La educación artística y la educa­ción de los medios en la era digital [Arts edu­cation and media education in the digital age]. Revista Comunicar, 8 (16), 149-160. https://bit. ly/35s7iEY

Hernández Ramírez, J. (2014). Instrumento para la valoración preliminar del uso de las APPs en educación primaria [Instrument for the preliminary assessment of the use of apps in primary education]. In J. J. Maquilón Sánchez, A. Escarbajal Frutos, & R. Nortes Martínez-Ar­tero (Eds.), Vivencias innovadoras en las aulas de primaria (pp. 273-286). Universidad de Murcia.

IBM (2016). SPSS (v. 24) [Computer software].

Iglesias, L. (2012). Mobile ART: apps móviles en el contexto artístico [Mobile ART: Mobile apps in an artistic context]. Blog Mobile Art. https:// bit.ly/2lD3f4Z

Iglesias Antón, E. (2015). Nuevas tecnologías y medios en el proceso de creación fotográfica contemporánea. La fotografía móvil y las apps de autor [New technologies and media in the process of contemporary photographic creation. Mobile photography and author apps] [Final degree thesis, Universidad Pompeu Fabra]. Di­gital repository of Universidad Pompeu Fabra. http://hdl.handle.net/10230/25223

JASP Team (2019). JASP (Version. 0.11.1) [Compu­ter software]. https://jasp-stats.org/download/

Jöreskog, K. G., & Sörbon, D. (1986). LISREL: Análysis of Linear Structural Relationships by The Mettod of Maximun Likelihood. Mooresvi­lle, IN Scientific Software.

Kortabitarte, A., Ibáñez-Etxeberria, A., Luna, U., Vicent, N., Gillate, I., Molero, B., & Kintana, J. (2017). Dimensiones para la evaluación de aprendizajes en apps sobre patrimonio [Dimen­sions for learning assessment in heritage apps]. Pulso. Revista de educación, 40, 17-33. https:// bit.ly/2k83mVM

Lee, C. Y., & Cherner, T. (2015). A comprehensi­ve evaluation rubric for assessing instructio­nal apps. Journal of Information Technology Education: Research, 14, 21-53. http://bit. ly/2ph5xIS

Lee, J. S., & Kim, S. W. (2015). Validation of a tool evaluating educational apps for smart education. Journal of Educational Compu­ting Research, 52 (3), 435-450. https://doi. org/10.1177/0735633115571923

Li, C. (2016). Confirmatory factor analysis with or­dinal data: Comparing robust maximum likeli­hood and diagonally weighted least squares. Behavior Res, 48, 936-949.

Lloret, S., Ferreres, A., Hernández, A., & Tomás, I. (2017). The exploratory factor analysis of items: guided analysis based on empirical data and software. Anales de Psicología, 33 (2), 417- 432.

López Benito, V. (2013). ¿Educar en el arte median­te apps? Los museos de arte y el uso de aplica­ciones móviles en el contexto español [Art edu­cation through apps? Art museums and the use of mobile applications in the Spanish context]. Her&Mus: Heritage & Museography, 69-74.

Lorenzo-Seva, U., & Ferrando, P. J. (2013). FAC­TOR 9.2. A comprehensive program for fit­ting exploratory and semiconfirmatory factor analysis and IRT models. Applied Psychologi­cal Measurement, 37, 497-498.

Lorenzo-Seva, U., & Ferrando, P. J. (2018). FAC­TOR Unrestricted Factor Análysis, v. 10.9.02: A computer program to fit the exploratory factor analysis model. University Rovira y Virgili.

Lorenzo-Seva, U., & Van Ginkel, J. R. (2016). Mul­tiple imputation of missing values in explora­tory factor analysis of multidimensional scales: estimating latent trait scores. Anales de Psico­logía, 32, 596-608.

Luna, Ú., Etxeberria, Á. I., & Kortabitarte, A. (2019). Herramientas digitales innovadoras en espacios educativos. El uso de las apps para la enseñanza-aprendizaje del patrimonio en España [Innovative digital tools in educatio­nal spaces. The use of apps for heritage tea­ching-learning in Spain]. In J. Calvillo Barrios (Coord.), Escenarios, heterogeneidad y nuevas perspectivas para comprensión de la educación (pp. 90-106). Universidad Estatal de Oriente.

Malone, M., & Peterson, M. (2013). Is there an app for that? Developing an evaluation rubric for apps for use with adults with special needs. The Journal of BSN Honors Research, 6 (1), 38-56.

Marín García, T. (2011). Experiencia de innovación docente GOUMH: aprendizaje colaborativo en Bellas Artes con APPS de Google [Teaching innovation experience GOUMH: Collaborative learning in Fine Arts with Google APPS]. In Congreso Internacional de Innovación Docen­te (pp. 1845-1859). Universidad Politécnica de Cartagena. http://hdl.handle.net/10317/2270

Martín Ezama, P. (2016). El contenido digital y transmedia de los museos: webs, apps y disposi­tivos. Estudio descriptivo de la situación actual de las herramientas y plataformas de distribu­ción de contenido digital de los principales mu­seos de arte de titularidad pública de España [Digital and transmedia content in museums: Websites, apps and devices. Descriptive study of the current situation of the digital content dis­tribution tools and platforms of the main pu­blicly owned art museums in Spain] [Master’s thesis, Universidad Nacional de Educación a Distancia]. Repositorio Institucional de la Uni­versidad Nacional de Educación a Distancia. https://bit.ly/3pUZncA

Montaño Armendaríz, M. A. (2014). Modelo de de­sarrollo económico local para la diversificación de la estructura productiva y la articulación del tejido empresarial en baja California Sur [Local economic development model for the di­versification of the productive structure and the articulation of the business fabric in Baja Cali­fornia Sur] [Doctoral dissertation, Universidad Autónoma de Baja California]. Eumed.net. ht­tps://bit.ly/3xkG136

Mora Castro, D. M. (2018). Las aplicaciones móvi­les apps en el desarrollo de la expresión artís­tica de los niños [Mobile applications apps in the development of children’s artistic expresión] [Doctoral thesis, Universidad de Ambato]. Ins­titutional repository of the Universidad Téc­nica de Ambato. https://repositorio.uta.edu.ec/ jspui/handle/123456789/28344

Muñíz, J., & Fonsaca-Pedrero, E. (2019). Diez pasos para la construcción de un test [Ten steps in test construction]. Psicothema. 31 (1), 7-16.

Navarro Martínez, D. (2014). Cómic, curriculum en la escuela de secundaria y APPS [Comics, secondary school curriculum and APPS]. In R. Marín Viadel, J. Roldán, & X. Molinet Medina (Eds.), Fundamentación, criterios y contextos en Investigación basada en Artes e Investiga­ción Artística, vol. 4 (pp. 59-96). Universidad de Granada. http://hdl.handle.net/10481/34212

Norman, D. (1988). The psychology of everyday things. Basic Books.

Norman, D., & Drapper S. (1988). User centered system design. Lawrence Erlbaum.

Ondé Pérez, D. (2020). Revisión del concepto de causalidad en el marco del Análisis factorial Confirmatorio [Review of the concept of cau­sality in the framework of Confirmatory Factor Analysis]. Revista Iberoamericana de Diagnós­tico y Evaluación- e Avaliaçao Psicológica RI­DEP, 54 (1). 103-117.

Ordóñez, F. F., & González Fernández, J. (2021). Estadística descriptiva paso a paso. Manual Básico para Ciencias Sociales [Descriptive statistics step by step. Basic Manual for Social Sciences]. Pirámide.

Oviedo, C. H., & Campo-Arias, A. (2005). Aproxi­mación al uso del coeficiente alfa de Cronbach [Approach to the use of Cronbach’s alpha co­efficient]. Revista Colombiana de Psiquiatría, 34 (4), 572-580.

Papadakis, S., Kalogiannakis, M., & Zaranis, N. (2017). Designing and creating an educational app rubric for preschool teachers. Education and Information Technologies, 22 (6), 3147- 3165. http://bit.ly/2IQsjOD

Prensky, M. (2001). Digital natives, digital immi­grants. On the Horizon, 9 (5), 1-6.

Prieto, S. (2015, April 29). Criterios para la utili­zación y diseño de aplicaciones móviles educa­tivas [Criteria for the use and design of educa­tional mobile applications]. Educaweb. https:// bit.ly/3cMIyv3

Rico Rico, A. B. (2017). Evaluación del uso de las APPS que abordan los procesos creativos en la educación artística formal [Evaluating the use of APPs that address creative processes in formal arts education] [Doctoral dissertation, Universidad de Valladolid]. Document reposi­tory of the Universidad de Valladolid. http:// uvadoc.uva.es/handle/10324/22663

Rojas Torres, L. (2020). Robustez de los índices de ajuste del Análisis Factorial Confirmatorio a los valores extremos [Robustness of Confir­matory Factor Analysis fit indices to outliers]. Revista Matemática: Teoría y Aplicaciones, 27 (2), 383-404.

Sarrab, M. K., Alzahrani, A. A., Al Alwan, N. A., & Alfarraj, O. M. (2014). From tradicional learning into mobile learning in education at the university level: Undergraduate students perspective. International Journal of Mobile Learning and Organisation, 8 (3/4), 167-186. https://doi.org/10.1504/IJMLO.2014.067014

Sasiain Camarero-Núñez, A., & Aberasturi Apraiz, E. (2018). Pensar pedagógicamente desde el arte: el stop motion como herramienta de tra­bajo [Thinking pedagogically from art: Stop motion as a working tool]. In M. Caeiro Rodrí­guez (Ed.), Arte, Educación y Pensamiento di­gital: educar, crear y habitar en la quinta pared (pp. 127-146). Egregius Ediciones. https://bit. ly/3gtkrUv

Saura Pérez, Á. (2011). Innovación educativa con TIC en Educación artística, Plástica y Visual. Líneas de investigación y estudios de caso [Educational innovation with ICT in Art, Plas­tic and Visual Education. Lines of research and case studies]. MAD.

Schwab, K. (2016). La cuarta revolución industrial [The fourth industrial revolution]. Debate.

Seva, U., & Ferrando, P. J. (2015). POLYMAT-C: A comprehen-sive SPSS program for computing the polychoric correlation matrix. Behavioral Research Methods, 47 (3), 884-889.

Soriano, A. M. (2014). Diseño y validación de ins­trumentos de medición [Design and validation of measurement instruments]. Diálogos, 14, 19-40.

Stake, R. E. (2006). Evaluación comprensiva y eva­luación basada en estándares. Ariel.

Steiger, J. H., & Lind, J. C. (1980). Statistically-ba­sed tests for the number of common factors [Pa­per presentation]. Annual Spring Meeting of the Psychometric Society.

Vernet, M. (2014, May 10). El aprendizaje móvil. Algunas reflexiones sobre sus características y su puesta en práctica [Mobile learning. Some reflections on its characteristics and its imple­mentation]. Docentes en línea. https://bit.ly/3z­t5PMl3zt5PMl

Vicent, N. (2013). Evaluación de un programa de educación patrimonial basado en tecnología móvil [Evaluation of a heritage education pro­gramme based on mobile technology] [Doctoral dissertation, Universidad Autónoma de Ma­drid]. Institutional repository of the Univer­sidad Autónoma de Madrid. http://hdl.handle. net/10486/14321

Walker, H. C. (2011). Evaluating the effectiveness of apps for mobile devices. Journal of Special Education Technology, 26 (4), 59-66.

Walker, H. C. (2013). Establishing content validity of an evaluation rubric for mobile technology applications utilizing the Delphi method [Doc­toral dissertation, Johns Hopkins University]. Johns Hopkins Sheridan Libraries. http://jhir. library.jhu.edu/handle/1774.2/36935

Feliciano F. Ordóñez Fernández. Doctorate in Psychology from the Univer­sidad de Oviedo (1993). He is currently an Associate Professor in the Faculty of Edu­cation at the Universidad Internacional de la Rioja (UNIR). His research interests cen­tre on research methodology and statistics.

 https://orcid.org/0000-0001-9753-7872

Martín Caeiro Rodríguez. Doctorate in Fine Arts from the Universidad de Vigo (2008). He is currently an Assistant Professor in the Faculty of Education at the Universi­dad de Zaragoza affiliated to the Department of Didactics of Musical, Visual, and Corporeal Expression. His research interests centre on online art education, art and cognition, art education teaching and learning methodolo­gies, and art-based research.

 https://orcid.org/0000-0001-5616-3747

Ana María Barbero Franco. Doc­torate in Fine Arts from the Universidad de Salamanca (2007). She is currently an Associate Professor in the Faculty of Ed­ucation at the Universidad Internacional de la Rioja (UNIR). Her research interests focus on online art education, developing creativity, heritage education, and manag­ing historical heritage.

 https://orcid.org/0000-0002-3413-1602

María del Mar Martínez-Oña. Grad­uate in History of Art (1998) and doctorate in Women’s Studies and Gender Studies (2015) from the Universidad de Grana­da. She currently works as a Professor of Practice at the ESNE (University School of Design, Innovation, and Technology), a centre attached to the Universidad Camilo José Cela (Madrid). Her research interests focus on gender studies and innovation in teaching with digital technologies.

 https://orcid.org/0000-0001-9231-7264

Alberto Torres Pérez. Gradu­ate in Fine Arts from the Universidad Politécnica de Valencia (1997). He is an Associate Professor at the Universidad Internacional de La Rioja (UNIR) in the Faculty of Education. His research interests focus on online artistic educa­tion and teaching innovation with digital technologies.

 https://orcid.org/0000-0002-7991-1813