Workload perception in educational resource recommendation supported by artificial intelligence: a controlled experiment with teachers

Abstract

Researchers are increasingly interested in enabling teachers to monitor and adapt gamification design in the context of intelligent tutoring systems (ITSs). These contributions rely on teachers’ needs and preferences to adjust the gamification design according to student performance. This work extends previous studies on teachers’ perception of their cognitive effort and dedication to creating and monitoring educational resource recommendations on a simulated gamified educational platform. This study compares teachers’ perceptions of workload using one of three scenarios— manual, automated, and semi-automated—to recommend educational resources through a randomized experiment. In this study, 151 participating teachers evaluated their perception of cognitive effort and time dedicated to creating recommendations for missions and monitoring students on the platform. The results indicate that the teachers’ perception that the automated scenario has a lower workload than the manual scenario significantly raises the hypotheses. Our results also suggest that teachers’ perception of the textbook scenario is different according to their level of knowledge about Information and Communication Technologies (ICT). For teachers with advanced ICT knowledge, the manual scenario is perceived as a scenario that indicates a more outstanding performance. According to the educational level of the teachers, the perception of mental demand for the automated scenario is significantly different. These significantly contribute to understanding teachers’ perceptions when using educational platforms in their classes.

Introduction

To increase student engagement and motivation when using intelligent tutoring systems (ITSs), researchers and practitioners are increasingly interested in adopting gamification (Dermeval, 2016). The application of gamification in ITSs implies using game elements (e.g., missions, points, ranking, levels) in instructional scenarios to enhance student engagement and participation (Hunter & Werbach, 2012). However, defining a gamification design is a challenging task, especially for teachers with no experience in, for instance, game design (Toda et al., 2019).

As explained by Dermeval et al. (2018), ITS authoring tools can simplify teachers’ active participation, helping them define ITS design, pedagogical resources, and interaction objectives. Moreover, teachers can develop their activities without having advanced technical skills on the educational platform in such a context. Using authoring tools to construct gamified ITSs can effectively increase teacher participation, allowing them to intervene in the gamification design at different times (pre, during, or post-interaction of students with ITS) (Dermeval et al., 2019).

The active participation of teachers in the design of gamified ITSs implies dealing with significant variability of data and different possibilities for decision-making (Baker, 2016). This variability may influence teachers’ perceptions of how complex it is to customize the design of gamified ITSs, assist in the instructional process in a complementary way to the ITS, and monitor how students interact with the system (Tenório et al., 2021a). The complexity of teachers’ decision-making in gamified ITSs and other educational scenarios has increasingly caught the attention of researchers. This interest has led to exploring combined artificial intelligence (AI) strategies. These strategies aim to enhance teachers’ human intelligence while reducing their workload. AI offers great capacity and speed in handling multidimensional data, which can complement the human aspects teachers bring to education. (Paiva & Bittencourt, 2020). This partnership between human intelligence and AI, where AI works with an assistance perspective and not just automation, is also known as Augmented Intelligence (i.e., the collaboration between human intelligence and artificial intelligence to enhance cognitive abilities, leveraging AI’s data processing capabilities to complement human intuition, creativity, and contextual understanding) Cerf (2013).

In the context of gamified ITSs, it is necessary to provide teachers with models that simplify the application of game elements in the design and customization of instructions for their students. This study advances educational technology research by addressing critical gaps in the literature concerning teachers’ roles in gamified intelligent tutoring systems (ITS). Traditional ITS designs often require advanced technical skills, limiting educators’ ability to engage deeply in the customization and monitoring of these platforms (Dermeval et al., 2018). Building on these insights, the study investigates semi-automated environments that integrate artificial intelligence (AI) to aid teachers in personalizing student learning experiences while reducing cognitive load and time commitment. By comparing manual, automated, and semi-automated scenarios, this research uniquely examines how varying levels of automation impact teachers’ cognitive workload and time investment, contributing novel insights into hybrid approaches that blend human and AI capabilities.

This research addresses specific scientific gaps, such as the challenge of reducing the complexity of designing gamified ITS and fostering teacher involvement through simplified tools (Paiva & Bittencourt, 2020; Dermeval et al., 2018a; Tenório et al., 2022). These findings align with the call for inclusive and accessible educational technologies (Tenório et al., 2020b), further emphasizing the transformative potential of AI to enhance learning environments. Teachers could use AI resources to expand their capacity in the decision-making process, stimulating an increase in motivation and engagement (positive effects) and avoiding a state of boredom (negative effect) in students (Tenório et al., 2021a).

Teachers could use AI resources to expand their capacity in the decision-making process, stimulating an increase in motivation and engagement (positive effects) and avoiding a state of boredom (negative effect) in students (Tenório et al., 2021a). This article investigates teachers’ perceptions of their workload using three simulated scenarios to recommend educational resources to students. It compares teachers’ insights using one of three scenarios—manual, automated, and semi-automated (see Sect. 4.6 for details).

Some researchers describe cognitive workload as the mental energy required to manage a certain amount of information Sweller (1988). The Cognitive Workload theory assumes that performance and learning decline when effort or mental load exceeds the memory’s capacity to process Sweller (1988). By decreasing the cognitive load, it is possible to achieve higher productivity or better learning outcomes Paas et al. (2003); Sweller et al. (2011). Recent studies of how mental workload can negatively impact the quality of teacher education lead to the need to implement effective program solutions to mitigate working hours and reduce teachers’ cognitive load to impact performance positively Malekpour et al. (2014).

Related works

Some studies in the literature are investigating the inclusion of teachers in the design of gamified educational applications. In a previous study, Paiva et al. (2016) investigated the active participation of teachers in the pedagogical decision-making process in an educational system, following the significant variability of data from interactions contained in the educational platforms. In Tenório et al. (2020), it was presented the Gamification Analytics Model for Teachers, a model that allows teachers to make mission-based interventions based on the visualization of student interactions with pedagogical and gamified resources in the educational system. In Tenório et al. (2021b), it evaluated teachers’ perceptions regarding learning and gamification dashboards to assist them in the pedagogical decision-making process in educational platforms.

The related articles made it possible to generate some functionalities for creation, release, and monitoring of simulated scenarios, such as in the work of Tenório et al. (2020b), by using specific scenarios, bringing approaches that can be applied by teachers. An attempt was made to reproduce the same intervention action, using automation by AI and the proposed combined use of these features in the semi-automated scenario. In the Table 1, we have a comparative table of the criteria used in each article for the creation of each scenario used in the study.

Table 1 Comparative Table of the Articles Related to this research

Table 1 highlights how this study differs from prior research. Furthermore, it was used as a reference for the construction of the prototyping of the screens from the simulated scenarios involved in the experiment. In that context, the following section presents the theoretical background supporting these considerations.

Theoretical background

This section presents background information on ITSs, Gamification, Authoring Tools, and Teachers’ Workload, as well as discusses related work.

Intelligent tutor systems

The term intelligent tutoring systems was developed by Auguste (1985) in a review of the state of the art in Computer Assisted Instruction (CAI) to differentiate it from ICAI (Intelligent Computer Assisted Instruction) systems, which are defined by instructional systems that use AI techniques to guide the teaching/learning process. ITS uses the theory of learning by doing, where instructions are personalized, and feedback is carried out according to actions in the system. According to what was established by Wenger (2014), the ITS has in its composition four essential components, which are: domain, tutor, user interface, and student. These components are known in the literature as “Traditional Intelligent Tutor Systems Architecture” Nkambou et al. (2010).

Gamification

According to Deterding et al. (2011), the term gamification was first used in 2008 in the digital media industry and widely adopted in 2011, and was defined as the “use of game design elements in non-game contexts.” According to Fardo (2013), gamification motivates and engages users in interactions with gamified environments with defined design elements, leading to a game experience (Ermi, 2005).

Gamification designs are divided into elements according to their levels (dynamics, mechanics, components), requiring them to be well structured and in line with users (Werbach, 2015), enabling an optimized game experience, leading to its maximum level of immersion, and reaching the flow state of Csikszentimihalyi Brown & Cairns (2004).

As explained by Herzig et al. (2015), the gamification analysis consists in analyzing the gamification process in its planning and generated documentation. This process can be summarized in four phases: Business and Requirements Modeling; Design; Implementation; Monitoring and Adaptation.

Authorship in ITS environments

Developing an ITS is complex because it involves many stakeholders, such as developers, instructors, and experts in a specific knowledge domain Escudero & Fuentes (2010). The need to implement simple authorship models and tools in ITS environments to provide management and customization of gamification design and its elements led authors such as Escudero & Fuentes (2010); Dermeval et al. (2018); Moundridou & Virvou (2002) to establish proposals for simplified authoring models and tools.

According to the systematic literature review conducted by Dermeval et al. (2018), there are about six types of authoring tools for ITSs, these being: Model tutor tracking; Cognitive example tutor; Content and problem tutor; Dialogue tutor; Constraint tutor; Machine and human tutor. Moreover, Dermeval et al. (2018) consider some essential steps for the ITS development process, such as: Identifying the tutors; Identifying the authors; Identify the target.

Gamification analytics model for teachers

Tenório et al. (2020b) proposed the gamification analytics model for teachers. According to the model, teachers can define, monitor, and adapt gamification learning systems. The model offers the possibility of adapting or readapting the design through the gamification element of missions in order to engage and motivate students who are not achieving the goals set by the teacher. The gamification analytics model for teachers is composed of the following structure: Definition of interaction goals; Monitoring of student interaction with learning resources; Monitoring of student interaction with gamification elements; Adaptation of gamification design through missions Tenório et al. (2020b), as presented in Fig. 1.

Fig. 1

Gamification Analytics Model for Teachers

Cognitive load theory

Models and tools used in educational settings should not only consider that teachers are able to use the technology itself but also that they find a purpose for making use of it, taking into consideration time, effort, and dedication to ensure a high level of competence, pedagogical understanding, and teaching effectiveness (Comas-Quinn, 2011).

Considering the factors cited as indicators of a cognitive mental load, authors such as Daniel (2020); Malekpour et al. (2014) understand that cognitive mental load is the amount of mental energy required to deal with a given amount of information. Where cognitive load theory assumes that performance and learning decrease when the amount of effort or mental load exceeds the memory’s ability to process (Sweller, 1988). Studies such as by Malekpour et al. (2014) investigate how mental load can negatively impact a teacher’s quality of teaching and how the treatment of this load, being reduced, can positively impact teacher performance in educational settings.

Summary

This section presents the theoretical background supporting this study. Particularly, it discussed the relevance of ITS, gamification, cognitive load, and how those concepts interact to background our proposal. Next, we detail the experimental study conducted to evaluate our proposal.

Experiment

This section details our experimental study following the framework introduced in Wohlin et al. (2012).

Proposal

The purpose of this paper is to investigate the perception of teachers regarding their workload and time in a simulated educational environment.

Scope

The study compared the results of teacher perception regarding their dedicated workload and time in a simulated educational environment, where teachers evaluated one of three proposed scenarios: CA (Automated Environment); CM (Manual Environment); CS (Semi-automated Environment). In each scenario, the study participant evaluated a set of screens, simulating the creation of missions in a gamified environment. By experiencing the set of screens in one of the three proposed scenarios, the teacher analyzed their perception of cognitive effort and dedication time to use the simulated environment.

Research questions and hypotheses

The conducted study analyzed the following research questions, which were then statistically tested based on the corresponding hypotheses.

QP1: What are teachers’ perceptions of workload and its six factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) in the combined use of recommendations by missions in automated, manual, and semi-automated scenarios?

H1.0: There is no significant difference in teachers’ “perceived workload” and its six factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) depending on the “scenario” that was presented to them in the questionnaire.

H1.1: There is a significant difference in teachers’ “perceived workload” and its six factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) depending on the “scenario” that was presented to them in the questionnaire.

QP2: What are teachers’ perceptions of workload and its six factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) in the combined use of recommendations, by missions in automated, manual, and semi-automated scenarios, according to their gender?

H2.0: There is no significant difference in teachers’ “perceived workload” and its six factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) depending on the “scenario” that was presented to them in the questionnaire and their “genders”.

H2.1: There is a significant difference in teachers’ “perceived workload” and its six factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) depending on the “scenario” that was presented to them in the questionnaire and their “genders”.

QP3: What are teachers’ perceptions of workload and its six factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) in the combined use of recommendations, by missions in automated, manual, and semi-automated scenarios, according to their level of knowledge regarding ICTs?

H3.0: There is no significant difference in teachers’ “perceived workload” and its six factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) depending on the “scenario” that was presented to them in the questionnaire and their “level of knowledge regarding ICTs”.

H3.1: There is a significant difference in teachers’ “perceived workload” and its six factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) depending on the “scenario” that was presented to them in the questionnaire and their “level of knowledge regarding ICTs”.

QP4: What are teachers’ perceptions of workload and its six factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) in the combined use of recommendations, by missions in automated, manual, and semi-automated scenarios, according to the educational level at which they teach?

H4.0: There is no significant difference in teachers’ “perceived workload” and its six factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) depending on the “scenario” that was presented to them in the questionnaire and the “educational level at which they teach”.

H4.1: There is a significant difference in teachers’ “perceived workload” and its six factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) depending on the “scenario” that was presented to them in the questionnaire and the “educational level at which they teach”.

Research method

The method used in the experiment was one factor, between-subject with three groups/levels with a desirable minimum grouping of 32 participating teachers in each proposed scenario. The selection of teachers was by convenience sampling, where each teacher invited to participate in the experiment already uses or has used an educational platform to support their subjects and monitor their students.

The goal of the scenarios was to lead the participating teacher to discuss the hypotheses raised and how they perceived the workload and time invested while using the simulation, bringing the evaluation of a set of recommendations by mission tracking screens and the different ways of intervention existing in one of the scenarios that they used: Manual Scenario, Automated Scenario, and Semi-Automated Scenario.

The participating teachers were randomly and equally allocated to the groups of each scenario adopted. The teachers evaluated one of the three simulated scenarios, which contains a set of screens representing the action of offering recommendations through missions as a form of pedagogical intervention with students who are not meeting the minimum requirements of interaction or pedagogical performance defined for the simulation of student interaction in the scenario.

Procedure

The phases performed in the experiment are represented in Fig. 2.

Fig. 2

Phases developed in the experiment

In Phase 1 (Participant Selection), e-mails were sent inviting teachers to participate in the study online, with a summary of the study, the ethics committee document (Informed Consent Form), and a link to the website with the simulation of the set of scenarios to be evaluated.

In Phase 2 (Preparation), teachers who accessed the website of the research experiment were oriented with more details about the object of study and information that by agreeing to participate, the e-mail address was collected to send them a copy of the Informed Consent Form, defining their role in the phases of the study.

In Phase 3 (Personal Data), data were collected such as gender, date of birth, level of education, level of activity as a teacher and the level of knowledge about technologies/systems/computers, whether the teacher uses or has used educational platforms for interaction or content recommendation to their students, the method of notification of these platforms, and whether they use or have used monitoring functions of these platforms such as reports, dashboards or indicators of student interactions with the available resources.

In Phase 4 (Basic Concepts), as a clarification to the teachers, some key terms were exposed and conceptualized for the elaboration of the simulated scenario and the object of study of the hypotheses that were presented.

In Phase 5 (Instructions), the teacher was instructed to consider each scenario of a situation in the educational environment, and these were evaluated by different teachers in the same context. The instructions to the teacher were according to the scenario they were allocated from Phase 2, where the participating teacher considered their performance on the simulated platform, having to analyze their perception of mental load and time commitment in interventions on the platform. For this, they observe a set of simulated screens for the action of creating and monitoring the suggested interventions.

In Phase 6 (Perception Questionnaire), the participating teacher answered a perception questionnaire using a scale from 1—(Strongly Disagree) to 5—(Strongly Agree) and open questions about the positive and negative points of the simulation of creating, releasing, and tracking the recommendations by missions in the simulation on the scenario they participated in.

The NASA Task Load IndeX questionnaire was also applied. It was answered by participating teachers to measure the teacher’s mental load and cognitive effort Daniel (2020) Hart & Staveland (1988), where each scenario was presented to the teacher again for him to reflect upon and answer the NASA questionnaire.

In Phase 7 (Data Collection), the data from the questionnaires were grouped by each scenario.

For Phase 8 (Data Analysis), statistical methods (see Sect. 4.8) were implemented with analysis tools such as R and JASP. For this, they were grouped into a single data file to be analyzed and discussed in the results section of this paper.

In Phase 9 (Presentation of Results), in the results section of this article, we will present the results of the collected data and how they relate to the hypotheses raised.

Proposed scenarios

Considering the objective of analyzing the perception of workload, time spent on interventions, and monitoring in an educational platform, a set of environment scenarios was developed, simulating the creation and monitoring of each environment. In the following, we detail some of the functionalities of the scenarios presented.

Recommending missions in the manual scenario (CM)

The participating teachers evaluated the set of screens of the manual scenario, where the teacher, in a simulated manner, performed the procedure of creating a mission for the listed students according to the criteria of minimum interaction with the pedagogical resources made available in a standard configuration for all scenarios. Figure 3 shows the initial view of the teacher participating in the manual scenario.

Fig. 3

Manual Recommendations Panel—Initial View

In the initial view of the manual scenario, the teacher has in the central part some data panels (dashboard) containing graphical information of student data such as Interaction with recommendations, performance without interaction, and performance with interventions. Another element is a quick view of students who have not interacted with the recommendations provided on the platform, giving the teacher a view of students who need close monitoring. On the right side, we can adjust some configuration parameters such as Activity Duration, Interaction Percentage, and Average Grade for Evaluations. On the left side, we have the possibility of creating manual interventions such as videos, reading, activities, and sites/links.

During the execution of the experiment, the participating teacher was directed to the set of screens used to create a video intervention for a set of students with simulated data. For this, they used the assignment creation screen, as shown in Fig. 4.

Fig. 4

Manual Recommendations Panel—Mission Creation

The participating teacher observed that the manual mission creation screen could require the following information for creating the mission intervention: Mission Start and End, which are the dates of availability of the resource, the topic of the subject studied, according to their teaching plan. However, the study focused on evaluating the teachers’ perception regarding the creation, release, and monitoring of the impact of missions manually.

It is possible to define extra rewards for students, such as: XP scoring (Experience Points) that are used in the gamification model and Grade Scoring, according to the evaluation criteria of the teaching plan. The teacher can define a name for the assignment, the assignment description and upload shortcuts to link the assignment video. Moreover, the teacher could assign the recommendation to all students or only to those he considers essential to release the intervention. Figure 5 shows the completed view of the assignment displayed for teacher evaluation.

Fig. 5

Manual Recommendations Panel—Mission Creation

By clicking on the “Create Assignment” button, confirmation of the data filled in the simulation was displayed for the participating teacher. From this point on, the teacher followed the other phases with the questionnaires to evaluate their perceptions of the exposed scenario.

Recommending missions in the automated scenario (CA)

In the Automated Scenario, the participating teacher viewed a simulation of the tracking of automatically generated missions using automation techniques or content recommendations. In the platform, when generating automated interventions, the participating teacher only has the role of monitoring the indicators and defining minimum interaction parameters, with the resources to be made available to students according to their lesson plan and the content to be taught. The teacher’s initial view of this scenario is shown in Fig. 6.

Fig. 6

Automated Recommendation Panel—Initial View

The automated scenario is very similar to the manual scenario. Its main difference is that the participating teacher evaluated his perception by only following the interactions of students and the recommendations made by the platform in an automated way. On the left side of Fig. 6, it is possible to follow the percentage of interventions performed by the platform according to the types presented above.

Assignment recommendation in the semi-automated scenario (SC)

The Semi-Automated scenario was created so that the teacher participating in the study could evaluate in a combined way the manual scenario with the automated one, combining features of monitoring and automation of the teacher’s activities. This allows the active participation of the teachers in the creation, release, and monitoring of the educational resources offered on the platform.

For the simulation of this scenario, the teacher was instructed to evaluate the proposed intervention situation. However, instead of just creating the mission, the automation of the platform suggested interventions for mission recommendations, leaving it to the teacher’s criteria to release or not the suggested resource. This allowed active participation of the teacher so that they were not passive in the recommendations suggested by the automation.

During the experiment, a set of screens was presented where the semi-automated scenario’s functioning was demonstrated. The participating teacher would receive an automated recommendation for a video intervention, similar to the previous scenarios. Figure 7 shows the initial view of the Semi-Automated scenario.

Fig. 7

Semi-Automated Recommendations Panel—Initial View Inicial

After evaluating the screens, the participating teacher was directed to answer the questionnaires about their perception of the activity developed in the Semi-Automated scenario.

Participants

This experiment was composed of 151 teacher participants. According to the strategy of balancing and randomization of the three scenarios, each group had the number of teachers as follows: automated scenario was composed of 46 (30,5%) teachers, and the manual scenario was composed of 50 (33,1%) teachers, and the semi-automated scenario was composed by 55 (36,4%) teachers. Table 2 details the participants’ demographic information.

Table 2 Demographics and ICT Knowledge Level of Participants

Data analysis

We applied the NASA Task Load IndeX questionnaire to measure the participating teacher’s mental load and cognitive effort, (Daniel, 2020; Hart & Staveland, 1988) concerning the scenario they evaluated. We applied the ANOVA test, considering the standard 95% confidence level, for the analyses performed to compare the results of the three scenarios developed.

During the analyses, the following dependent variables were used:

• Perception of workload;

• Perception of cognitive demand;

• Perception of physical demand;

• Perception of temporal demand;

• Perception of performance;

• Perception of effort;

• Perception of frustration.

During the application of the ANOVA test, all dependent variables presented non-normal distribution, so the transformation “sqrt(x + 1) - x)” was applied to all data collected before performing the parametric tests. Thus, when reporting the results of the descriptive statistics, the mean (M), standard deviation (SD), and confidence interval (CI) should be calculated by applying the following: $max(x+1) - y ~{2}$, where max(x + 1) = 11.

We used the following independent variables:

• Scenarios: With the manual recommendation, with an automated recommendation, and with semi-automated recommendation conditions.

• Gender: In the male and female conditions.

• Level of knowledge in reference to Information and Communication Technology (ICT).

• Educational level of the teacher.

Hypothesis testing

Null hypothesis testing was performed with the dependent and independent variables listed:

• H1(null): There is no significant difference in the “workload perception” and its six perception factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) of teachers depending on the “scenario” that was presented to them in the questionnaire.

• H2(null): There is no significant difference in the “workload perception” and its six perception factors (cognitive demand, physical demand, time demand, performance, effort, and frustration) of teachers depending on the “scenario” that was presented to them in the questionnaire and their “genders”.

• H3(null): There is no significant difference in “workload perception” and its six perception factors (cognitive demand, physical demand, demand, time demand, performance, effort, and frustration) of the teachers depending on the “scenario” that was presented to them in the questionnaire and their “level of knowledge about ICT”.

• H4(null): There is no significant difference in the “workload perception” and its six perception factors (cognitive demand, physical demand, demand, time demand, performance, effort, and frustration) of teachers depending on the “scenario” that was presented to them in the questionnaire and the “educational level at which they teach”.

Summary

This section detailed the experimental study presented in this article. Next, Sect. 5 presents the study’s findings.

Results

For H1, the ANOVA tests with independent variables between the subject “Scenario” (manual, automated, semi-automated) was performed to determine statistically significant differences in the dependent variables: “Work Load”; “Mental Demand”; “Physical Demand”; “Temporal Demand”; “Performance”; “Effort”; “Frustration”. For the dependent variable “Workload”, there was a statistically significant effect on the factor “Scenario” with F(2,136) = 3.842, p = 0.024 and ges = 0.053 (effect size). For the other dependent variables, there were no statistically significant effects.

Paired comparisons using Estimated Marginal Means (EMMs) were computed to find statistically significant differences between the groups defined by the independent variables and p-values adjusted by the Bonferroni method. For the dependent variable “Workload”, the mean in Scenario = “automated” (adj M = $-$2.066 and SD = 0.313) was significantly different from the mean in Scenario = “manual” (adj M = $-$1.821 and SD = 0.434) with p-adj = 0.03. Result for H1, the teachers’ perception is that the automated scenario has the lowest workload than the manual scenario, as shown in Fig. 8, while there was no significant differences for the other measures.

Fig. 8

Workload per Scenario

For H2, the ANOVA tests with independent variables between the subjects “Scenario” (manual, automated, semi-automated) and “Sex” (Female, Male) were performed to determine the statistically significant difference in the dependent variables: “Workload”; “Mental Demand"; "Physical Demand"; "Temporal Demand"; "Performance"; "Effort"; "Frustration". For the dependent variable "Frustration", there was a statistically significant effect on the factor "Scenario" with F(2,128) = 3.571, p = 0.031 and ges = 0.053 (effect size). For the other dependent variables, there were no statistically significant effects.

Paired comparisons using Estimated Marginal Means (EMMs) were computed to find statistically significant differences between the groups defined by the independent variables and with p-values adjusted by the Bonferroni method. For the dependent variable "Mental Demand", the mean in Sex = "Female" (adj M = $-$2.024 and SD = 0.447) was significantly different from the mean in Sex = "Male" (adj M = $-$1.705 and SD = 0.536) with p-adj = 0.045. Resulting for H2, the perception of mental demand of the automated scenario is significantly different according to gender. Female teachers have the perception that the automated scenario presents less mental demand, as shown in Fig. 9.

Fig. 9

Mental Demand of Scenarios by Gender

For H3, the ANOVA tests with independent variables between the subjects "Scenario" (manual, automated, semi-automated) and "ICT" (Advanced, Moderate) was performed to determine statistically significant differences in the dependent variables: "Workload"; "Mental Demand"; "Physical Demand"; "Temporal Demand"; "Performance"; "Effort"; "Frustration". For the dependent variable "Performance", there were statistically significant effects on the factor "ICT" with F(1,102) = 5.639, p = 0.019 and ges = 0.052 (effect size). For the other dependent variables, there were no statistically significant effects.

Paired comparisons using Estimated Marginal Means (EMMs) were computed to find statistically significant differences between the groups defined by the independent variables, with p-values adjusted by the Bonferroni method. For the dependent variable "Performance", the mean in TIC = "Advanced" (adj M = $-$1.453 and SD = 0.388) was significantly different from the mean in TIC = "Moderate" (adj M = $-$1.9 and SD = 0.298) with p-adj = 0.007. Therefore, the perception of performance in the manual scenario is different according to the level of ICT knowledge of the participating teachers. For teachers with advanced ICT knowledge, the manual scenario is perceived as a scenario that demands higher performance, as shown in Fig. 10.

Fig. 10

Performance by Level of ICT Knowledge

For H4, ANOVA tests with independent variables between the subjects "Scenario" (manual, automated, semi-automated) and "Teaching" (Secondary Vocational/Technical Education, Higher Education) were performed to determine statistically significant differences in the dependent variables: "Work Load"; "Mental Demand"; "Physical Demand"; "Temporal Demand"; "Performance"; "Effort"; "Frustration". For the dependent variable "Mental Demand", there were statistically significant effects in the interaction of the factors "Setting: Teaching" with F(2,158) = 3.378, p = 0.037 and g = 0.041 (effect size). For the other dependent variables, there were no statistically significant effects.

Paired comparisons using Estimated Marginal Means (EMMs) were computed to find statistically significant differences between the groups defined by the independent variables, with p-values adjusted by the Bonferroni method. For the dependent variable "Frustration," the mean in Education = " Secondary Vocational/Technical Education" (adj M = $-$1.598 and SD = 0.367) was significantly different from the mean in Education = "Higher Education" (adj M = $-$1.961 and SD = 0.549) with p-adj = 0.008; The mean in Teaching = " Secondary Vocational/Technical Education (adj M = $-$2.24 and SD = 0.561)" was significantly different from the mean in Teaching = "Higher Education" (adj M = $-$1.938 and SD = 0.533) with p-adj = 0.043. Resulting for H4, the perceived mental demand of the automated scenario is significantly different according to the educational level at which teachers teach. For vocational/technical high school teachers, the automated scenario is perceived as a scenario that needs more mental demand than for teachers in higher education, as shown in Fig. 11.

Fig. 11

Mental Demand by Teaching Area

In summary, this section demonstrates how varied recommendation approaches supported by AI differ in terms of teachers’ self-reported cognitive load. In that context, the following section interpret and discuss these insights.

Discussion

In the evaluation of the 151 teachers participating in the research, hypothesis testing H1 demonstrated that the participating teachers have the perception that the automated scenario presents the lowest workload than the manual scenario. However, the semi-automated scenario showed similarity in relation to the manual scenario. The Gamification Analytics Model for Teachers (Tenório et al., 2020b) demonstrates an intervention without the support of AI, and the evaluated scenarios bring the contribution to verifying the perception of teachers regarding the workload required to perform the intervention process.

Another contribution made by hypothesis H1 is that it demonstrates the impact of teachers’ perceived workload concerning an AI-automated environment. However, the importance of teachers’ active participation in the design of an AI-automated educational environment, such as an ITS (Dermeval et al., 2018), is necessary for approximate monitoring of the interventions to be performed by the platform.

In hypothesis H2, the result regarding the perception of mental demand of female participants is that the automated scenario presents the lowest mental workload demand, being that the number of participants of this gender may be a threat to the result since they were 37.7% of the participants in the study. However, the result of H2 may be a new opportunity to investigate the perception of teachers, according to gender, in a more balanced, realistic environment in future research.

Another interesting evaluation regarding hypothesis H2 is that male teacher have a similar perception of mental demand for the three scenarios. This may indicate the frequency of using the technologies in the presented scenarios, reinforcing the need to investigate further the perception of mental demand by gender in a realistic educational environment.

In Hypothesis H3, concerning the perception of teachers’ workload and their level of knowledge of Information and Communication Technology (ICT), for teachers with an advanced level of knowledge in ICT, the manual scenario demands higher performance. At the moderate level of ICT knowledge, the automated and semi-automated scenarios are similar with respect to the performance required to use the resources of the educational platform. This analysis allowed us to observe the need for the participating teachers to have adequate training on the objectives and resources available on the platform, where this information is a significant contribution to understanding how the training or the level of knowledge about ICT impacts the perception of teachers in relation to their performance in the work to be performed.

Hypothesis H3 also contributes to the understanding that even in an automated or semi-automated environment, the teacher needs to understand the purpose of the resources and how he/she will use and apply them in his/her work routine because, as seen, the level of ICT knowledge impacts on the teachers’ perception of performance.

In hypothesis H4, the results indicated that the perception of the teachers’ mental demand in relation to their level of activity reports that the automated scenario is significantly different according to the level at which they work. For the secondary vocational/technical teachers, the automated scenario is perceived as a scenario that needs more mental demand in comparison to the higher education teachers. This result may have been influenced by the fact that it is common for higher education teachers to apply educational tools as a way to complement their class activities. Alternatively, it may also have been because the study used a simulated environment to define the procedures for each scenario.

The result of hypothesis H4 contributes to understanding how teachers’ perception of mental demand can be influenced by their routine of applied classes, according to the educational level that teachers teach, and their ability to use online educational resources, such as educational teaching platforms. Especially the upper-level teachers, who perceive the automated scenario demands a lower mental demand than the other scenarios. Also, they perceive the manual and semi-automated scenario similarly, which may indicate that teachers tend to prefer to automate their work routine, whereas the use of the semi-automated scenario in a realistic environment can contribute to the pedagogical decision-making process, according to Paiva et al. (2016); Paiva & Bittencourt (2020).

Findings can be applied in educational settings to optimize teachers’ workloads and improve their experiences with technology. Understanding that teachers perceive reduced workloads in fully automated scenarios suggests that incorporating AI tools could alleviate burdens associated with manual processes, thus freeing teachers to focus more on student interactions and pedagogical strategies. However, the varied perceptions of mental demand based on gender and ICT competence highlight the need for tailored training programs that consider these differences, ensuring equitable and effective engagement with technology across diverse teaching populations. Moreover, identifying different workload perceptions across educational levels suggests that technology use needs customization based on the teachers’ context, such as secondary versus higher education. These insights emphasize the importance of involving teachers in designing and implementing educational technology to enhance the alignment of such tools with their actual needs, promoting efficient and user-friendly environments that facilitate rather than overwhelm educators.

Limitations

Importantly, this study has some limitations that must be acknowledged in interpreting its findings, which also guide future research. The first limitation is convenience sampling, which involves teachers already familiar with educational platforms. This sampling method was chosen because it ensures participants have a baseline understanding of technology, which is necessary to evaluate their perceptions accurately in a simulated digital environment. However, this choice may limit generalizability, as it may not reflect the broader range of experiences and technological comfort levels in the general teacher population.

Another limitation is the reliance on simulated environments to assess teacher perceptions. These controlled settings allow uniformity and control over variables that might complicate real-world studies. However, they might not capture the full complexity of actual classroom scenarios, potentially affecting the realism of the findings. Despite this, simulated environments are valuable for isolating specific variables and establishing initial understandings that can be explored in more varied settings in future research.

Lastly, the study emphasizes short-term perceptions of workload and uses self-reported data to gauge these perceptions. While such data can be subjective, the validated NASA Task Load Index adds reliability to the self-reporting process. Longitudinal studies could extend these findings by examining how perceptions and interventions’ effectiveness evolve. The immediate focus on short-term perceptions is justified as a necessary first step to understanding base reactions and responses that can inform longer-term studies. Examining workload through the six factors (cognitive, physical, temporal demands, performance, effort, and frustration) provides a comprehensively structured approach. However, future work could investigate how these factors interact across different contexts.

Conclusion

In this paper, we presented a study that demonstrated teachers’ perceptions regarding their workload and time commitment to create, release, and monitor educational interventions on teaching platforms by missions. From our findings, we highlight that:

• Teachers see the automated scenario as least burdensome, while the semi-automated scenario aligns more closely with the manual one in terms of workload. This underscores the need for teacher involvement in AI-driven educational environments.

• Female teachers perceive lower mental demand in automated scenarios, though sample distribution could affect this finding. Male teachers see similar mental demands across scenarios, indicating a need for further gender-based studies.

• Teachers with higher ICT knowledge find manual scenarios more demanding. Moderate ICT users have similar perceptions of automated and semi-automated scenarios, highlighting the importance of ICT training in shaping workload perceptions.

• Secondary teachers find automated scenarios more demanding than higher education teachers, who perceive less demand likely due to familiarization with educational tools, suggesting potential preferences for automation in educational settings.

These findings make significant contributions to the field of educational technology. By demonstrating how AI-driven systems can complement teachers’ decision-making processes, the research advances the development of adaptive learning environments with gamification features that reduce teachers’ workload. The study highlights the importance of tools integrating teacher-centric design principles, such as dashboards and mission-based recommendations, enabling educators to monitor and personalize student learning paths effectively. Furthermore, it shows a way to operationalize the concept of augmented intelligence, showcasing how AI can act as an assistive tool that enhances human decision-making rather than replacing it. The research also provides gender-specific insights, revealing that automated scenarios significantly reduce mental demand for female teachers, thus promoting the creation of more inclusive solutions. Finally, through its experimental design, this study bridges theoretical frameworks with practical applications, offering a roadmap for scaling ITS technologies across educational contexts while supporting teachers in using gamified ITS.

Accordingly, we recommend the following lines of future research:

• Improve the collection of data that demonstrates the perception of teachers in relation to their workload and time of dedication to educational platforms.

• Remodel the Analytic Model of Gamification for Teachers proposed by Tenório et al. (2020b), combining it with the model for the Pedagogical Decision Making Process proposed by Paiva et al. (2016), extending both models to be used in the semi-automated environment.

• Develop and implement the model of the semi-automated environment of recommendations by assignments in a pedagogical platform and reapply the study developed in this article in a realistic environment.

• Re-evaluate teachers’ perceptions regarding workload and time commitment on a realistic platform for an extended period.