An English course practice evaluation system based on multi-source mobile information and IoT technology

Research on intelligent text analysis methods

Some early textual entailment methods, such as those based on text similarity and those based on logical calculus, are too shallow to adapt to more complex semantics. The convolutional neural network (CNN)-based textual entailment model can automatically explore the N-gram features in the sentence, which has a good ability for local information mining, and the recurrent neural network (RNN)-based textual entailment model can analyze the long-term dependence and short-term dependence in the document sentence. After combining the two types of models with the attention mechanism, the discovery of the key content in the document sentence has been further improved.

Three types of models can be used to solve the problem of text understanding: attention-based models, pointer-wise models, and multi-hop mechanism models. An attention mechanism is characteristic of attention-based models. Hermann et al. (2015) proposed an attentive radar based on a connected attention mechanism, and Chen, Bolton & Manning (2016) proposed Stanford AR based on a bilinear attention mechanism. The model based on pointer-wise tries to decrease the model’s perplexity by narrowing the answer’s value range from the whole dictionary to a single discourse dictionary. The representatives of this model include AS Reader (Kadlec et al., 2016) and AOA Reader (Cui et al., 2016). The model simulates the multi-step reading of human beings and filters the essential parts layer by layer through the method of superimposed attention mechanism. The representatives of this type of model are AMRNN (Attention-based Multi-hop Recurrent Neural Network) (Tseng et al., 2016) and GAReader (Dhingra et al., 2017). 2018, Microsoft released the MS-MARCO dataset (Nguyen et al., 2016), a fragment synthesis dataset.

Meanwhile, the original version of Du Reader (He et al., 2017) is a complete fragment synthesis task consisting of multiple passages, an actual user's question in a Baidu search, and then a synthetic answer. This type of reading comprehension problem corresponds to certain heuristics. For example, it directly borrows the idea of extraction synthesis (S-Net (Tan et al., 2017)). There is also a novel read-then-verify model (Hu et al., 2018), which tries to construct a model containing two modules through read-verify.

Research on intelligent scoring methods for teaching

With the popularity of online intelligent scoring systems in modern teaching, more and more scholars have begun to join the ranks of related research. First of all, regarding the reliability and validity of the intelligent scoring system, the online intelligent scoring system has the advantages of immediacy, reliability, and interactivity, and the automatic scoring system can objectively improve the efficiency of feedback. Many scholars have a positive attitude toward the validity of the intelligent scoring system (Bull & McKenna, 2004). Wilson (2016) also conducted a study about the effective feedback of intelligent systems. He conducted a one-semester experiment on learners in lower grades and affirmed the effectiveness of intelligent feedback in assisting teachers in their feedback work.

Compared with teacher feedback, the feedback of an intelligent essay scoring system is more objective and efficient than teacher feedback, and it can better overcome the reliability and validity problems caused by some human factors, such as fatigue (Group, 2022). Reilly et al. (2014) and other researchers have also confirmed that there are differences between intelligent scoring systems and teacher feedback. In terms of the reliability and validity of the system, scholars generally think that it has the characteristics of timely feedback, is objective and effective, and has a positive attitude; they mainly believe that the system can be used as an auxiliary tool for teachers to make up for the lack of teacher feedback. However, the research on comparing teacher and machine feedback is not very comprehensive.

The online intelligent scoring system also has shortcomings in evaluating teachers’ emotions in courses. Firstly, some scholars believe that the system can only play a role in supervising course fluency, and whether it is conducive to supervising students’ teaching efficiency remains to be explored (Wang & Chen, 2017). Secondly, the feedback focuses on video and audio, which is not helpful for students’ psychological acceptance. However, such problems will be reduced with the update and improvement of the system. In the current situation of the online intelligent scoring system, the shortcomings in error recognition have restricted its wide application to a certain extent and reduced the quality of students’ modification feedback, so the system urgently needs improvement. Some scholars have proposed many ways to assist the course through Internet of Things technology. Zong, Jia & Zhang (2014) apply the IoT to managing sources in the school, such as students, the teaching environment, the library, and teaching instruments and equipment. Qi & Shen (2011) propose an application scheme based on the Internet of Things to solve the access defects and system security problems of RFID tags and improve the efficiency of traditional teaching management system methods. Chen & Huang (2021) designed an interactive online English teaching system based on Internet of Things technology and evaluated the teaching quality of the system using a gray correlation analysis algorithm.

Looking at the above content, there have been many relevant experimental studies on the intelligent teaching scoring system, and the system has also been widely used in schools at all stages. Most researchers have a positive attitude toward reliability and validity, and the research shows the system can improve the efficiency of students’ courses. Of course, some scholars think the system has limitations and defects in the teacher’s mental outlook.

Evaluation of English lesson preparation content based on intelligent text analysis method

To better understand the lesson plans compiled by teachers when preparing for lessons, we evaluate the English lesson preparation content by using the intelligent text analysis method. Through the image acquisition of the English lesson preparation content through the sensor device, the image of the lesson plan content for each class is obtained. Considering that the traditional recurrent neural network has a poor effect on long-distance sentence processing and cannot capture semantic information in all directions. Although the encoding information in all orders can be obtained by bidirectional encoding, it increases the computational complexity of the model. To solve the problems and fully understand the lesson preparation content, we use the improved Transformer Encoder as the basic feature extractor, as shown in Fig. 2, which has a great advantage in calculation speed. Thanks to the self-attention mechanism, it can better capture context semantics and solve problems such as long-distance dependence. In addition, the structure of the model is improved, and the attention alignment method is used to focus on some important word vectors, weaken the influence of irrelevant words, and generate reasoning vectors with more robust semantics. In the model ending, an interaction layer is added to share semantic information between sentence pairs to form a more semantically rich word vector, which is then input to the latter layer for processing. We set up an interaction layer to achieve the fused features and extract the same information from the sentence pairs.

Each improved Transformer module contains two consecutive multi-head self-attention modules. Each module comprises Layer Normalization (LN), a multi-head self-attention mechanism, residual connection, and Multi-Layer Perceptron (MLP). The overall calculation process is as follows:

(1) ${\hat{\mathrm{z}}}_{\mathrm{i}}=\mathrm{M}\mathrm{S}\mathrm{A}\left(\mathrm{L}\mathrm{N}\left({\mathrm{z}}_{\mathrm{i}-1}\right)\right)+{\mathrm{z}}_{\mathrm{i}+1}$

(2) ${\mathrm{z}}_{\mathrm{i}}=\mathrm{M}\mathrm{L}\mathrm{P}\left(\mathrm{L}\mathrm{N}\left({\hat{\mathrm{z}}}_{\mathrm{i}}\right)\right)+{\hat{\mathrm{z}}}_{\mathrm{i}}$

(3) ${\hat{\mathrm{z}}}_{\mathrm{i}+1}=\mathrm{M}\mathrm{S}\mathrm{A}\left(\mathrm{L}\mathrm{N}\left({\mathrm{z}}_{\mathrm{i}}\right)\right)+{\mathrm{z}}_{\mathrm{i}}$

(4) $${\mathrm{z}}_{\mathrm{i}+1}=\mathrm{M}\mathrm{L}\mathrm{P}\left(\mathrm{L}\mathrm{N}\left({\hat{\mathrm{z}}}_{\mathrm{i}+1}\right)\right)+{\hat{\mathrm{z}}}_{\mathrm{i}+1}$$

When calculating the multi-head self-attention mechanism, each head is calculated as follows:

(5) $$\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(\mathrm{Q},\mathrm{K},\mathrm{V}\right)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{\mathrm{Q}{\mathrm{K}}^{\mathrm{T}}}{\sqrt{\mathrm{D}}}+\mathrm{B}}\right)\mathrm{V}$$where $\mathrm{Q},\mathrm{K},\mathrm{V}\in {\mathrm{R}}^{{\mathrm{S}}^2\times \mathrm{D}}$ refer to Query, Key, and Value matrix, ${\mathrm{S}}^2$ is the number of image patches in a window, $\mathrm{D}$ means the number of feature dimensions of the sequence; The corresponding encoding matrix is $\mathrm{B}\in {\mathrm{R}}^{{\mathrm{S}}^2\times {\mathrm{S}}^2}$, whose value is obtained in the bias matrix ${\mathrm{R}}^{\left(2\mathrm{S}-1\right)\times \left(2\mathrm{S}+1\right)}$. Self-attention allows each sequential element (such as every word in a sentence or every pixel in an image) to interact. It calculates the relevance score (or attention weight) between elements to determine how important different elements are for a particular task. This global correlation modeling enables the model to consider the relationships between all aspects simultaneously to better understand the context information in the sequence. Self-attention does not depend on the location information of the element. Each element’s relevance weight to all other elements is obtained by calculating its similarity to the other elements, regardless of their actual position in the sequence. This makes the self-attention model better handle out-of-order and variation in the input sequence.

By fully extracting the visual information of the lesson plan image and establishing the semantic relationship through the encoder, we can obtain a set of semantic features. Then, the semantic features are used to design the decoder to obtain the final evaluation results of the English course lesson plan. The relevance weights of the word vectors’ relative semantic features ${\mathrm{z}}_{\mathrm{i}+1}$ in GT are calculated, and the calculation process is shown in Eq. (6). The weights and each word vector are used to obtain the aligned enhanced representation vector ${{\mathrm{p}}_{\mathrm{i}}}^{\mathrm{\prime }}$.

(6) $${\mathrm{p}}_{\mathrm{i}}^{\mathrm{\prime }}=\sum _{\mathrm{j}=1}^{\mathrm{n}}{\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{e}}_{\mathrm{i}\mathrm{j}}\right)}{{\sum }_{\mathrm{k}=1}^{\mathrm{n}}\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{e}}_{\mathrm{i}\mathrm{k}}\right)}{\mathrm{p}}_{\mathrm{i}}.}$$

After calculating the model’s output, the result vectors are collected and spliced together. Finally, the features for classification are extracted from the vectors to obtain the recognition results.

(7) $$\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\mathrm{F}\mathrm{C}\left({\mathrm{p}}_{\mathrm{i}}^{{}^{\prime }}\right)\right)$$where FC is a fully connected layer, the activation function uses the Tanh. Input ${\mathrm{p}}_{\mathrm{i}}^{\mathrm{\prime }}$ into a fully connected layer for linear transformation to reduce the dimension and then normalized by Softmax to obtain the predicted probability of each classification.

Teaching intelligent scoring methods

The convolutional neural network method analyzes the context semantics and feature learning of comments, and its anti-noise and classification degrees are high. Therefore, the CNN method is selected to extract the audio and video features of teaching quality in the class. A training set was established according to the audio and video information collected by the sensor, and labels marked all samples in the training set. After automatically inputting the test dataset by convolution and pooling, the trained neural network was applied to obtain the label of the new sample, and the extraction process is shown in Fig. 3.

The K-means is a clustering algorithm. Its principle is to make the objects of different clusters as different as possible and the objects of the same cluster as much as possible according to the corresponding similarity rules. The algorithm uses distance as the criterion for dividing clusters and usually uses the Euclidean distance formula to calculate the distance between data object samples, such as the formula

(8) $\mathrm{d}=\sqrt{\sum _{\mathrm{j}=1}^{\mathrm{n}}{\left({\mathrm{x}}_{\mathrm{i}}-{\mathrm{y}}_{\mathrm{i}}\right)}^2.}$

In the clustering process of the K-means clustering algorithm, the average value of all the samples in the cluster needs to be recalculated in each iteration, which is called the cluster center point. The updated calculation formula is: ${\mathrm{c}}_{\mathrm{i}}{\mathrm{c}}_{\mathrm{i}}$

(9) ${\mathrm{C}}_{\mathrm{i}}={\displaystyle \frac{1}{\left|{\mathrm{c}}_{\mathrm{i}}\right|}\sum {\mathrm{x}}_{\mathrm{i}}}$where ${\mathrm{C}}_{\mathrm{i}}$ denotes a cluster.

The K-means clustering algorithm must iteratively update the divided categories and cluster centers until the termination condition is met. The default termination condition is that the model reaches the maximum or the algorithm’s objective function is less than the threshold. As seen from the above description, the K-means core is the squares criterion’s minimum error sum. The basic idea is to divide the given data object into the same cluster through some iterations and then recalculate the cluster center by the cyclic iterations when the criterion function converges. And obtain the output result. The k-criterion function E is defined as follows.

(10) $\mathrm{E}=\sum _{\mathrm{i}=1}^{\mathrm{k}}\sum _{{\mathrm{x}}_{\mathrm{i}}\in {\mathrm{C}}_{\mathrm{i}}}{\left({\mathrm{x}}_{\mathrm{i}}-\overline{{\mathrm{x}}_{\mathrm{i}}}\right)}^2.$

The more significant E is, the lower the intra-cluster similarity is. On the contrary, the smaller E is, the higher the intra-cluster similarity is.

In the teaching intelligent scoring method, the k-means algorithm can cluster students’ performance or learning behavior to classify and evaluate students. K-means is an unsupervised machine learning algorithm that divides data points into K clusters so that the data points within the same cluster are similar, and the data points between different groups are quite different. By collecting students’ data in the learning process, such as answering questions, learning time, interactive behavior, etc., students can be divided into several groups (clusters), each representing a kind of teaching quality. By inputting students’ learning performance data into the K-means algorithm, students can be divided into different groups to assess their learning level and teaching quality according to their group. By inputting the extracted audio and video features of English teaching into the k-means algorithm model, the evaluation grade in the process of English teaching practice can be directly obtained. Among them, audio and video feature fusion refers to integrating and combining various features extracted from teaching video and audio to receive more comprehensive and accurate information to better understand the learner’s behavior, emotions, and learning process. Integrating audio and video features can improve the accuracy of teaching quality recognition. Learners who watch teaching videos may have different learning behaviors, such as concentration, inattention, confusion, and so on. By integrating visual information in video and sound features in audio, students’ current learning behavior can be more accurately identified, reflecting the quality of teaching from the side.

Dataset and implement details

We use the Teaching and Learning dataset to verify the effect of our evaluation system. The lesson plan is shown in Fig. 4.

We conduct experiments on a device with an i7-12700 CPU and Rtx 3090 GPU. The operating system is Unbuntu, and the network model is implemented under the PyTorch framework. The total number of rounds of the experiment was set to 1,000, the batch size was set to 16, and the initial learning rate was set to 0.1. SGD is used as the optimizer of the model, the momentum is 0.9, and the weight decay term is set to. $1\times {10}^{-4}$. We use commonly used word segmentation tools, including NLTK, spaCy, jieba, etc., to break down text in English teaching into sequences of words or sub-words that computers can understand and process. Since the length of the sentences may not be consistent, it is necessary to fill or truncate them so that all the sentences are the same length for batch processing. Finally, Word2Vec, GloVe, FastText and other methods map words to low-dimensional vector space so the model can learn word meaning and semantic information.

To evaluate the performance of our method, we use accuracy as the evaluation criterion, which is calculated as follows:

(11) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}}$$where TP represents the number of correctly predicted samples, FP is the number of incorrectly predicted samples.

Results and discussion

We conducted experiments on the evaluation methods of pre-class lesson plans for English courses on the Teaching and Learning dataset. At the same time, three evaluation grades are set up: excellent, good, and medium. Compared with three advanced technologies: Transformer (Xiao et al., 2023), Swin-Transformer (Liu et al., 2022), and Visual Transformer (Li et al., 2023), the results are in Fig. 5. Our model achieves the best teaching plan content score classification performance, with an accuracy of 80.23%. Compared with the Visual Transformer, the accuracy evaluation index is improved by about 3.01%. As shown in Fig. 6, after comparing several mainstream networks with other networks that introduce attention mechanisms, such as Swin-Transformer and Visual Transformer, Our model has fewer parameters, a faster inference time, lower computational complexity, and is more lightweight. Transformer-based methods are prone to losing some features and suffer from inaccurate classification. Our model not only ensures sensitivity but also prevents the loss of features. Finally, we show the ratings made by our method for some lesson plans, as shown in Fig. 7. It can be found that our pre-class lesson plan evaluation method can accurately grade the lesson plans made by teachers. As for the content of lesson preparation, we can see that the perfect content should be shown in the leftmost sample in Fig. 7, which has a complete outline and teaching content and contains certain expansibility. In contrast, the middle teaching plan lacks these necessary contents and is vague and unclear, which is a poor teaching plan. The sample on the far right is a good-quality lesson plan that is complete in content but lacks some extension content.

In addition, the results of our methods about pre-lessons in the rating system were validated on the Coin video dataset. We select Transformer, Deit (Touvron et al., 2021), CLIP (Radford et al., 2021), and Vit to conduct comparative experiments with our model.

As shown in Table 1, the experimental results show that the CNN features obtained by FPN enhance the global learning ability of audio and video features, extract multi-level feature information, improve feature interaction ability by information fusion, aggregate all global features for deep and shallow feature deep fusion, and finally improve the accuracy of teaching quality in model recognition courses in Fig. 7. Compared to Transformer, our method improves accuracy by 4.02%. In contrast, the number of model parameters decreases by 99.22 M. Compared with Deit and Vit, our approach enhances accuracy by more than 5%, and the number of model parameters is much smaller. Compared with the more effective Vit, our method is significantly ahead in the number of model parameters, and the model accuracy is also better than CLIP. In addition, we show the training process of our classroom quality assessment model in Fig. 8. Although the k-means algorithm can help us improve the accuracy to 97.44%, there are still some drawbacks. The K-means algorithm is susceptible to selecting initial clustering centers, and different initial centers may lead to different clustering results. Each cluster is assumed to be convex; all points within the cluster are located in a circle centered around the cluster’s center. This limits the effectiveness of K-means when dealing with clusters with non-convex shapes. The performance of K-means can deteriorate on high-dimensional data sets, known as the “dimensional disaster” problem. The distance between data points in high-dimensional space becomes more similar, and the clustering effect may become poor.

Discuss and application

Through the above experiments, it can be found that our proposed English curriculum practice evaluation system, including the text intelligent analysis method of English lesson preparation content evaluation method and the teaching intelligent scoring method, can be applied to English curriculum practice to help teachers better manage the classroom and improve the quality of teaching. Our approach utilizes natural language processing and computer vision techniques to deeply analyze English text data for a more comprehensive, objective, and accurate assessment and guidance.

The text intelligence analysis method can automatically evaluate the English teaching content of teachers’ lesson preparation, which helps teachers understand the textbook’s coverage degree and difficulty more quickly and saves lesson preparation time. Text intelligence analysis can be used to evaluate multiple dimensions of English lesson preparation content, such as grammar, vocabulary, semantics, difficulty, etc., to find possible problems in teaching. According to the results of text intelligence analysis, personalized lesson preparation suggestions can be provided to teachers, helping them to better design teaching content according to the needs and levels of students. The teaching intelligent scoring method can avoid subjective bias and improve the accuracy and fairness of the evaluation results by objectively evaluating the text data of students. Teaching intelligent scoring methods can also analyze students’ English writing or oral expression in real-time and provide timely feedback and suggestions to help them improve their language expression ability. With the use of instructional intelligence scoring, teachers and schools can monitor and track students’ learning processes, find learning problems, and take timely teaching intervention measures. At the same time, the intelligent scoring method can provide personalized learning resources and suggestions according to the performance and level of students and help them make learning plans according to their own needs and abilities.

To give full play to the role of our English curriculum practice evaluation system, we rely on the school intranet platform and some sensors to build an English curriculum practice evaluation system integrating a multi-source mobile Internet of Things, as shown in Fig. 9.

By integrating the system into the school intranet platform, we conduct a comprehensive evaluation of English course practice according to the audio and video sensors in each classroom and the lesson preparation content of each English teacher. Firstly, the system uses sensors to collect the audio and video information of the teacher in class and sends it to the Internet of Things platform for intelligent evaluation to obtain the evaluation score in the class. Then, through the uploaded lesson preparation content, the intelligent model of the Internet of Things platform was used to rate it, and the specific score of the lesson preparation content was obtained. The score of lesson preparation content and the evaluation score in the class were integrated to complete the evaluation of English course practice.