Quality Meta-Framework: MLOps, NLP and learning analytics
Article Sidebar
Main Article Content
Abstract
A theoretical proposal is presented for an integrated meta-framework for educational quality management based on Natural Language Processing (NLP) and Learning Analytics (LA), designed to enable the scalable, traceable, and auditable evaluation of teacher-student interactions in digital environments. The conceptual architecture articulates NLP and LA to transform academic discourse and behavioral metadata into key performance indicators (KPIs), aligned with the pedagogical, technical, social, and psychological dimensions of educational quality.
The model is structured under a cyclical architecture of three phases: Direction, Operation, and Optimization that integrates strategic design, technical deployment, and continuous calibration. It also incorporates a hybrid annotation strategy that combines the scalability of Large Language Models (LLMs) with periodic human validation, ensuring bias control, semantic consistency, and institutional oversight.
Its main contribution lies in the explicit incorporation of an Algorithmic Governance (AG) framework, based on MLOps practices, which operationalizes quality through auditable metrics, validation protocols, and mechanisms for monitoring data and model drift. In this way, the proposal transcends approaches based on isolated KPIs and establishes a second-order architecture that integrates pedagogical evaluation, technical control, and institutional responsibility, promoting transparency, accountability, and continuous improvement in AI-based educational systems.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Los autores conservan los derechos de autor y ceden a la revista el derecho de la primera publicación y pueda editarlo, reproducirlo, distribuirlo, exhibirlo y comunicarlo en el país y en el extranjero mediante medios impresos y electrónicos. Asimismo, asumen el compromiso sobre cualquier litigio o reclamación relacionada con derechos de propiedad intelectual, exonerando de responsabilidad a la Editorial Tecnológica de Costa Rica. Además, se establece que los autores pueden realizar otros acuerdos contractuales independientes y adicionales para la distribución no exclusiva de la versión del artículo publicado en esta revista (p. ej., incluirlo en un repositorio institucional o publicarlo en un libro) siempre que indiquen claramente que el trabajo se publicó por primera vez en esta revista.
References
[1] K. Alalawi, R. Athauda, y R. Chiong, “An extended learning analytics framework integrating machine learning and pedagogical approaches for student performance prediction and intervention,”International Journal of Artificial Intelligence in Education, vol. 35, no. 3, pp. 1239–1287, 2025.
[2] A. Anwar et al., “Emotions matter: A systematic review and meta analysis of the detection and classification of students’ emotions in STEM during online learning,” Education Sciences, vol. 13, no. 9, p. 914, 2023.
[3] Axelos, ITIL Foundation: ITIL 4 Edition. TSO (The Stationery Office), 2019.
[4] CMMI Institute, CMMI for Development, Version 2.0. CMMI Institute, 2018.
[5] D. Demszky, J. Liu, H. C. Hill, D. Jurafsky, y C. Piech, “Can automated feedback improve teachers’ uptake of student ideas? Evidence from a randomized controlled trial in a large-scale online course,” Educational Evaluation and Policy Analysis, vol. 46, no. 3, pp. 483–505, 2024.
[6] D. Herrero Pascual, Ciclo de Vida de los Datos y MLOps: Herramientas, metodologías, puesta en producción y mantenimiento de proyectos de Data Science, 2022.
[7] ISACA, COBIT 2019 Framework: Introduction and Methodology. ISACA, 2019.
[8] ISO/IEC,Information technology—Governance of IT for the organization, ISO/IEC 38500:2015, 2015.
[9] ISO, Quality management systems — Fundamentals and vocabulary, ISO 9000:2015, 2015.
[10] S. Issar y A. Aneesh, “What is algorithmic governance?,” Sociology Compass, vol. 16, no. 1, e12955, 2022.
[11] K. F. Hew, C. Qiao, y Y. Tang, “Understanding student engagement in large-scale open online courses: A machine learning facilitated analysis of student’s reflections in 18 highly rated MOOCs,” Int. Rev. Res. Open Distrib. Learn., vol. 19, no. 3, 2018.
[12] R. Kaliisa, B. Rienties, A. I. Mørch, y A. Kluge, “Social learning analytics in computer-supported collaborative learning environments:A systematic review of empirical studies,”Computers and Education Open,vol.3, p.100073, 2022.
[13] C. Katzenbach y L. Ulbricht, “Algorithmic governance,” Internet Policy Review, vol. 8, no. 4, pp. 1–18, 2019.
[14] S. Knight y S. Buckingham Shum,”Theory and learning analytics,”Handbook of Learning Analytics, vol. 1, pp. 17–22, 2017.
[15] S. Knight y K. Littleton, “Dialogue as data in learning analytics for productive educational dialogue,” Journal of Learning Analytics, vol. 2, no. 3, pp. 111–143, 2015.
[16] J. O. Krugmann y J. Hartmann, “Sentiment analysis in the age of generative AI,” Customer Needs and Solutions, vol. 11, no. 3, 2024.
[17] A. Kucharavy, O. Plancherel, V. Mulder, A. Mermoud, y V. Lenders, Eds., Large Language Models in Cybersecurity: Threats, Exposure and Mitigation. Springer, 2024.
[18] V. Lukarov, “Scaling up Learning Analytics in Blended Learning Scenarios,” Ph.D. dissertation, RWTH Aachen Univ., 2019.
[19] Q. Lu et al., “Sentiment analysis: Comprehensive reviews, recent advances, and open challenges,” IEEE Trans. Neural Netw. Learn. Syst., 2023.
[20] R. Martinez-Maldonado et al., “Interactive surfaces and learning analytics: Data, orchestration aspects, pedagogical uses and challenges,” in Proc. 6th Int. Conf. on Learning Analytics & Knowledge, 2016, pp. 124–133.
[21] X. Ochoa et al., “Multimodal learning analytics—Rationale, process, examples, and direction,” en The Handbook of Learning Analytics, vol. 2, pp. 54–65, 2022.
[22] D. Owusu-Mensah, P. A. Sarfo, y G. A. Kusi, “Exploring the Impact of Artificial Intelligence on Customer Experience Personalization and Marketing Strategy Optimization in Digital Marketing: An Empirical Analysis,” Journal of Management and Informatics, vol. 4, no. 2, pp. 822-843, 2025.
[23] S. Rüdian, J. Podelo, J. Kužílek, y N. Pinkwart, “Feedback on feedback: Student’s perceptions for feedback from teachers and few-shot LLMs,” in Proc. 15th Int. Learning Analytics and Knowledge Conf., Dublin, Ireland: ACM, Mar. 2025, pp. 82–92.
[24] M. Sokovic, D. Pavletic, y K. K. Pipan,”Quality improvement methodologies–PDCA cycle, RADAR matrix, DMAIC and DFSS,” Journal of achievements in materials and manufacturing engineering,vol. 43,no. 1, pp. 476-483, 2010.
[25] B. T. Wong y K. C. Li, “Learning analytics intervention: A review of case studies,” en 2018 International Symposium on Educational Technology (ISET), IEEE, 2018, pp. 178-182.
[26] B. Yang, “Learning analytics through machine learning and natural language processing,” 2023.
[27] L. Yan, L. Zhao, D. Gašević, y R. Martinez-Maldonado, “Scalability, sustainability, and ethicality of multimodal learning analytics,” in Proc. LAK’22: 12th Int. Learning Analytics and Knowledge Conf., 2022, pp. 13–23.
[28] L. Yan et al., “Practical and ethical challenges of large language models in education: A systematic scoping review,” British Journal of Educational Technology, vol. 55, no. 1, pp. 90–112, 2024.
[29] U. Hanson, C. A. Okonkwo, y C. U. Orakwe, “Implementing AI-enhanced learning analytics to improve educational outcomes using psychological insights,” Journal of Educational Technology Innovations, 2024.
[30] A. Alam y A. Mohanty, “Business models, business strategies, and innovations in EdTech companies: Integration of learning analytics and artificial intelligence in higher education,” in Proc. IEEE 6th Conf. on Information and Communication Technology (CICT), 2022, pp. 1–6.
[31] Dutta, D. (2022). Edtech with Perspective of Business Analysis. perception, 6(03).
[32] B. Williamson, “Meta-edtech,” Learning, Media and Technology, vol. 46, no. 1, pp. 1–5, 2021, doi: 10.1080/17439884.2021.1876089.
[33] M.A.Chatti et al.,“Learning analytics:Challenges and future research directions,”E-Learning and Education, no.10, 2014.
[34] R. Pelánek, “Learning analytics challenges: Trade-offs, methodology, scalability,” in Proc. 10th Int. Conf. on Learning Analytics & Knowledge, 2020, pp. 554–558.
[35] C. Herodotou et al., “The scalable implementation of predictive learning analytics at a distance learning university: Insights from a longitudinal case study,” The Internet and Higher Education, vol. 45, p. 100725, 2020.
[36] A. Wilson, C. Watson, T. L. Thompson, V. Drew, y S. Doyle, “Learning analytics: Challenges and limitations,” Teaching in Higher Education, vol. 22, no. 8, pp. 991–1007, 2017.
[37] S. Dyer, Feedback by Design: Exploring Interactions Between Novice Programmers and Automated Teaching Assistants, Ph.D. dissertation, Univ. of Auckland, 2025.
[38] J. C. Debuse, M. Lawley, y R. Shibl, “Educators’ perceptions of automated feedback systems,” Australasian Journal of Educational Technology, vol. 24, no. 4, 2008.
[39] UNESCO, A Global Conversation on the Ethics of AI for Children, UNESCO, 2024. [Online]. Available: https://media.unesco.org/sites/default/files/webform/ed3002/390204eng.pdf
[40] K. Vayadande, P. A. Bailke, L. S. Khedekar, R. Kumar, y V. R. Dange, “A review on text analysis using NLP,” in How Machine Learning is Innovating Today’s World: A Concise Technical Guide, pp. 13–23, 2024.
[41] R. Ranganath, A. K. Fetterman, A. N. Gesselman, y E. J. Johnson, “Using natural language processing to analyse text data in behavioural science,” Nature Human Behaviour, 2024.
[42] S. Rathje et al., “GPT is an effective tool for multilingual psychological text analysis,” Proc. Natl. Acad. Sci. USA, vol. 121, e2308950121, 2024.
[43] F. Gilardi, M. Alizadeh, y M. Kubli, “ChatGPT outperforms crowd workers for text-annotation tasks,” Proc. Natl. Acad. Sci. USA, vol. 120, e2305016120, 2023.
[44] A. S. Ziabari et al., “Reinforced multiple instance selection for speaker attribute prediction,” in Proc. Conf. North American Chapter of the Assoc. Computational Linguistics: Human Language Technologies, 2024, pp. 3307–3321.
[45] M. Kurday y G. Vladova, “Learning analytics and educational data mining applications: Bibliometric and ChatGPT-based analysis of research publications from 2014 to 2023,” Journal of Computers in Education, pp. 1–36, 2025.
[46] A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30, pp. 5998–6008, 2017.
[47] C. Romero y S. Ventura, “Educational data mining: A review of the state of the art,” IEEE Trans. Syst., Man, Cybern., Part C (Applications and Reviews), vol. 40, no. 6, pp. 601–618, 2010.
[48] V. Kagklis, A. Karatrantou, M. Tantoula, C. T. Panagiotakopoulos, y V. S. Verykios, «A learning analytics methodology for detecting sentiment in student fora: A Case Study in Distance Education.», European Journal of Open, Distance and E-learning, vol. 18, n.o 2, pp. 74-94, 2015.
[49] C. Lang, G. Siemens, A. Wise, y D. Gasevic, «Handbook of learning analytics», 2017.
[50] V. Kovanovic, S. Joksimovic, D. Gašević, y M. Hatala, “What is the source of social capital? The association between social network position and social presence in communities of inquiry,” in CEUR Workshop Proc., CEUR-WS, pp. 21–28, 2014.
[51] J. P. Bernius, S. Krusche, y B. Bruegge, “Machine learning based feedback on textual student answers in large courses,” Computers and Education: Artificial Intelligence, vol. 3, p. 100081, 2022.
[52] M. Kumar, L. Khan, y H.-T. Chang, “Evolving techniques in sentiment analysis: A comprehensive review,” PeerJ Comput. Sci., vol. 11, p. e2592, 2025.
[53] Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400.
[54] Long, P.,& Siemens, G.(2011).Penetrating the fog:Analytics in learning and education.EDUCAUSE Review,46(5), 31–40.
[55] Baker, R. S., & Inventado, P. S. (2014). Educational data mining and learning analytics. In Learning analytics (pp. 61–75). Springer.
[56] OECD (2019). OECD Principles on Artificial Intelligence.
[57] European Commission (2019). Ethics Guidelines for Trustworthy AI.
[58] NIST (2023). AI Risk Management Framework (AI RMF 1.0).
[59] Sculley, D. et al. (2015). Hidden technical debt in machine learning systems. NeurIPS.
[60] Breck, E. et al. (2017). The ML Test Score: A rubric for ML production readiness. Google Research.
[61] Amershi, S. et al. (2019). Software engineering for machine learning: A case study. ICSE.
[62] IBM (2020). AI Governance Framework.