LPITutor: an LLM based personalized intelligent tutoring system using RAG and prompt engineering

Motivation

The limitation of traditional tutoring methods and the growing demand for more personalized, scalable, and flexible learning solutions motivate the development of LLM-based ITS. Its been difficult for educators in a conventional classroom environment to provide individualized attention to every student in large groups. This leads to the gaps in understanding because every student learns at different paces and requires different levels of clarification. ITS have solved this problem, but their effectiveness is limited by their static content and their lack of adaptability to a wide range of students and learning contexts (Kabir & Lin, 2023).

The integration of LLMs into tutoring systems with support of RAG and prompt engineering solves these challenges by enabling dynamic content generation and real-time adaptation to each learner’s progress. LLMs not only provide immediate and contextually relevant feedback, but also customize their responses to previous interactions of the learner, creating a more personalized and effective learning experience (Modran et al., 2024). In addition, LLMs have access to a vast repository of knowledge that enables them to present subject matter in diverse disciplines without the constant manual updates of the educator (Stamper, Xiao & Hou, 2024).

Another significant motivation is the growing need for scalable education technologies. Global access to education increases the demand for tools that reach large numbers of students and offer personalized instruction. The LLM-based tutoring system offers the scalability that enables universal access to high-quality education on a global scale (Nye, Mee & Core, 2023).

Challenges

Although LLM based ITS holds significant potential, both conventional and LLM based tutoring approaches still face several challenges. In traditional teaching environments, an educator faces the primary challenge of providing personalized real-time feedback to students. Educators often struggle to adapt their teaching strategies to meet the needs of the individual learner in a large group of students due to time and resource constraints (Meyer et al., 2024). As a result, learners may not receive the targeted support to overcome learning difficulties they require, and thus it creates gaps in their knowledge and understanding.

Conventional ITS offer some level of personalization, but also limit users to their predefined content and rule-based frameworks. These systems often lack the flexibility to cover different courses and to adapt dynamically to a learner’s needs. Educators put in significant manual effort to update these systems to include new topics or learning scenarios (Fu et al., 2023). This lack of customization and flexibility creates a significant barrier to wider acceptance and limits their effectiveness in supporting a variety of learners across various educational domains.

Although LLMs-based tutoring system addresses these challenges, it also introduces its own set of complexities. One major challenge is to ensure the accuracy and relevancy of LLM-generated content in case of complex educational topics. Sometimes, LLMs generate incorrect responses that could mislead students (Stamper, Xiao & Hou, 2024). Other important concerns are related to the ethical use of AI in education, such as data privacy, potential bias in the content created, and the need to ensure fair access to these advanced technologies (Ilagan, 2023).

Despite these challenges, LLM-enabled systems with RAG and prompt engineering have the potential to overcome many constraints of traditional tutoring approaches. Educational environments are increasingly interested in implementing LLM based systems because they offer a more scalable, flexible, and personalized approach to learning.

LPITutor: personalized intelligent tutoring system

The proposed LLM based Personalized Intelligent Tutoring System (LPITutor) model creates a highly personalized ITS. It combines RAG and prompt engineering to offer a unique learning experience that responds to each learner based on the requirements. Unlike conventional tutoring systems, LPITutor does not rely on predefined content; rather, it dynamically generates personalized explanations and learning resources in real time based on the student’s progress and interactions. The main contribution of LPITutor is to use RAG to retrieve relevant information from a wide range of external sources and prompt engineering to provide accurate and contextually appropriate responses. The hypothesis for this research is considered as “When RAG, prompt engineering and LLM are personalized using contextual data specific to the learner, it can generate more accurate, relevant and pedagogically effective responses than traditional LLM-based tutoring systems”. To investigate this hypothesis, we formulate the following research questions:

1. How effectively does the LPITutor framework integrate RAG, LLMs, and prompt engineering to generate context-aware and content-based tutoring responses?

2. In what ways does LPITutor personalize instructional content based on learner profiles and how does this personalization impact the quality of learning interactions?

3. How do domain experts and learners perceive the accuracy, relevance, and usefulness of LPITutor-generated responses compared to baseline LLM outputs?

4. What are the ethical, practical, and pedagogical considerations involved in deploying an LLM-based intelligent tutoring system in real-world educational settings?

Large language models

LPITutor exploits a pre-trained LLM (based on the GPT-3.5 architecture) as the central generative engine responsible for producing natural language explanations. The LLM is not fine-tuned on our dataset; instead, we accessed it using API and explored its output through strategic input modulation using prompt engineering and document grounding. The proposed model exhibits strong capabilities in knowledge synthesis, language fluency, and pedagogical expression to make it relevant to generate human-like instructional content.

LLM and prompt engineering

Prompt engineering guides LLMs in delivering more accurate and targeted responses. It carefully designs LLM queries in a way that LLM responds to specific educational goals (Sonkar et al., 2023). It is challenging to customize the tutoring experience to each student’s level of understanding and learning style. Effective prompt engineering combined with RAG ensures that the system dynamically refines the tutoring session based on a learner’s responses.

Pre-trained LLMs are able to understand, generate, and adapt to natural language that enables them to deliver personalized learning experiences. Recent advances in GPT-4 and similar architectures offer a rich dialogue interface between the learner and the tutor system (Fu et al., 2023). LLMs are the foundation of ITS that performs three major tasks: (i) understand complex queries, (ii) provide contextually relevant responses, and (iii) produce explanations based on a vast corpus of information (Nye, Mee & Core, 2023). LLMs in educational applications offer significant benefits because they can be easily adapted to various subjects and generate feedback on various learning tasks, ranging from theoretical explanations to real-time problem solving (Jiang & Jiang, 2024). This broad flexibility increases the tutor’s ability to meet a variety of learning methods and levels of student comprehension.

The proposed LPITutor employs a dual-layer prompt engineering strategy: (i) a static layer that contains pedagogically aligned templates (e.g., “Explain the concept of ¡topic¿ using examples relevant to ¡student profile¿”), and (ii) a dynamic layer that injects retrieved content and learner-specific metadata into the prompt structure. This enables the system to adjust the style, depth, and complexity of its responses based on the learner’s level of knowledge, query history, and preferred learning style. Prompt variations are automatically selected using an intent classification module and a personalized engine, which ensure that the LLM is oriented towards generating responses with educational relevance and clarity.

RAG

Apart from harnessing the higher capabilities of LLM, ITS also incorporates RAG to improve its knowledge base and provide more accurate and contextual responses. RAG enables the system to capture critical information from external sources and merge that information into its output to make sure that the response is current and contextual (Modran et al., 2024). The method is practical in academic scenarios where accuracy and context are imperative.

Figure 1 illustrates the RAG method that facilitates dynamic content generation and proactively looks for the most pertinent data in real time to provide a high level of relevance and accuracy. In RAG architecture, the system operates in five main steps. (i) The user inputs a query and passes it to the retrieval model. (ii) In retrieval model, retrieval engine searches for relevant information from external data sources and obtains relevant results in step (iii). Afterwards in step (iv) the retrieved results combined with the appropriate prompt are passed to a generative model (LLM). In step (v), the generative model processes the prompt and generates a final response and returns it to the user as a final result. This hybrid approach efficiently integrates the knowledge from external databases with the generative power of LLMs to deliver precise and context-sensitive responses. This process improves learning outcomes by providing the most relevant and personalized topic-specific explanation (Chenxi, 2023).

To address the challenge of hallucination and ensure that the generated content aligns with verified instructional material, LPITutor integrates a RAG mechanism. The system embeds course documents into a vector space using sentence-transformer-based encoders and stores them in a vector database. When a learner submits a query, it is similarly embedded and matched with this vector database using cosine similarity. The retrieved passages on top k form the context window that is fed into the LLM, thus anchoring the generation of authoritative course content and allowing for explainable answers.

Collectively, these core technologies enable LPITutor to act as a context-sensitive, learner-sensitive, and content-based intelligent tutoring system. Their integration ensures a balance between linguistic fluency, factual accuracy, and instructional personalization, making the system suitable for deployment in formal educational settings.

User view

In the user-view phase, the system receives a query from the user. The query can be a question or a request related to the course materials that the user wants to learn. This input query is processed through a query processing system that interprets the user’s intent and prepares it for further action. The primary goal of this phase is to understand the requirement of a user using NLP operations. NLP operations are performed for extraction of keywords and other relevant information from the ambiguous and complex queries which are useful for further processing. This phase involves breaking down the query into components that can be more easily mapped to document search or knowledge retrieval tasks.

Document processing

This phase reads the course materials or relevant documents and is stored in the knowledge base in text vector format. Initially, the relevant documents are collected in the course documents step. These documents will be considered as an external source for this architecture. For further processing, this document is converted into embeddings, i.e., a vectorized form of text that allows us to understand the semantic meaning of the content within the document. These embedding are generated by transformers because it helps to capture the contextual information from the text and makes it easier to match relevant documents according to the user’s query. Afterwards, all the generated embeddings are stored in the vector database. This phase is responsible for converting the relevant documents into vectors for an efficient and accurate result.

Response generator

This phase reads the query from user view and vector database form document processing phase and generates a final response using LLM and appropriate prompt. The vector generated from the query processing step is applied to the knowledge base, and then the appropriate text is retrieved. The extracted text is passed to the LLM with an appropriate prompt for a contextually appropriate response. The response generation process generates the personalized response based on user requirements. This response involves summarizing the text, answering a specific question, or providing an explanation on a specific topic.

LPITutor architecture leverages the vast knowledge and understanding of both general language patterns and domain-specific content to provide precise responses.

Experimental design

The effectiveness of a customized approach in real world educational settings is validated through tests that evaluate its retrieval and relevance to actual information (Verma, 2024). A prototype was developed that helps generate questions for each topic present in the vector database. Domain experts evaluated the responses and feedback, and the system intentionally provided the wrong answer to determine its functionality. These tests help establish credibility in real-world educational settings. The prototype was tested with domain experts.

The proposed model examines the impact of a structured curriculum on the performance of LLMs in generating educational content. The research uses GPT-4 as a baseline to compare its output under two conditions: (i) a pre-defined syllabus, and (ii) without syllabus. LLMs process large amounts of information and generate coherent content across multiple domains. The first condition involves GPT-4 generating content based on an established syllabus, providing clear objectives, topics, and timelines for learning. This ensures that the model generates relevant, aligned content, and maintains logical flow. The second condition involves GPT-4 generating content without a syllabus. This leaves the model independent and rely on its internal knowledge base and contextual understanding of educational content. This model evaluates the effect of a structured syllabus on the caliber, coherence, and applicability of the model’s educational content. If a structured syllabus improves performance, it suggests that LLMs can create informative and pedagogically sound educational material. In contrast, if the absence of a syllabus leads to more scattered or less useful content, it needs further refinement or human intervention to ensure alignment with instructional goals.

Dataset design

The suggested research assesses the model’s ability to create learning content and to answer varying levels of complexity. For every subject, the user inputs the questions based on their level of expertise. The questions are grouped as beginner, intermediate and advanced levels of difficulty. The set of answers reflects a progression of learning, starting from fundamental concepts at the beginner level, building upon them at the intermediate level, and culminating in more complex topics at the advanced level. A guided condition is enabled to explore and examine how external guidance influences the model’s performance. This state offers a clear syllabus equivalent to the subject, acting as a blueprint to create responses as educational material. The syllabus defines the main elements of the subject, consisting of core principles, key terminologies, and precise learning goals. This mechanism is intended to mimic a truer educational experience where content dissemination is synchronized with pedagogic objectives. The model subsequently creates responses which are directly synchronized with the defined syllabus. It guarantees that the essential ideas are addressed methodically and learning outcomes are achieved. It is a guided process with the aim of utilizing the strength of LLMs like GPT-4, tapping their immense knowledge into organized, targeted answers which are informative and aligned with learning objectives.

The dataset used for evaluation consists of 300 queries distributed evenly across difficulty levels. These questions were either taken from open learning sources or created by subject matter specialists. Every question was manually examined for readability, aptness, and instructional utility. There was a preprocessing step carried out to normalize input form and to ensure linguistic cohesion. For the controlled condition, every question was matched with a syllabus segment related to the subject. It offers a realistic simulation of organized learning environments that is appropriate for assessing both spontaneous and guided generation situations.

By providing this organized dataset and explicit experimental settings, research work guarantees that others can reproduce the setup, baseline performance, and build upon the work using related methods. The dataset contains difficulty label sets of queries and corresponding syllabus content that can be made available for request or released under an open-access repository for further research into educational AI and language model testing.

Evaluation metrics

The proposed research used specific metrics to assess the quality and difficulty of GPT-4 answers to various queries. These metrics provided a multidimensional assessment that ensured the accuracy and appropriateness of the content. They allowed a comprehensive evaluation of the model’s performance across different difficulty levels and its ability to meet educational standards for different subjects.

Accuracy was assessed by comparing model output with authoritative sources and standard textbooks in the relevant subject to ensure that key facts, definitions, and explanations were aligned with well-established knowledge. This metric was applied to all difficulty levels. The degree to which the model answered the query taking into account the need for in-depth explanations, numerous steps, or the inclusion of several concepts was what determined its completeness. The completeness was used to determine whether the model was capable of appropriately scaling its responses in terms of depth and detail as the difficulty of the questions increased.

The clarity of the model was a determining factor in an educational environment, as it guaranteed that the information was not only accurate but also clearly comprehensible to the learner. The success of the model lay in the efficiency of its capacity to communicate intricate concepts in a format the target group would comprehend. For low-level beginner questions, plain language was adopted, shying away from jargon and rendering straightforward examples. For intermediate-level and challenging questions, more technical terms and detailed explanations were suitable, but the model had to explain concepts logically and coherently. Clarity made the responses correct and easily understandable, enabling students with varying levels of expertise to understand the content. The difficulty alignment measure was utilized to evaluate the model’s capacity to deal with questions of different difficulty levels, such that the model’s answer was suited to the student’s learning stage as intended. GPT-4 could adapt its tone, depth and detail depending on the question’s complexity, providing suitable pedagogic answers.

The study tested the performance of the model in answering various questions of various difficulty levels based on different metrics. Coherence was utilized to measure logical information presentation, especially in intermediate and advanced questions where there were several concepts combined. Relevance was utilized to keep the model on topic and directly answer the question without deviating into other information. For higher-level questions, the model produced an answer that was closely related to the question’s purpose. By aggregating these measures, the study shed light on the model’s strengths and areas of improvement, guaranteeing that the content was of high quality and pedagogical value. The findings indicated that the model performance was effective in leading learners through intricate ideas with a structured and clear explanation.

Experimental results

While this research demonstrates the potential of the model to generate structured educational content at varying difficulty levels, validating its practical usability requires testing in real-world learning environments. User testing was performed with students to validate the results and assess how effectively AI-generated content supports actual learning outcomes and engagement.

Advanced level results

The model accuracy at the advanced level remained strong but slightly decreased to 89%, with problems in specialized and complex topics. Completeness became a significant challenge at this level, with a rating of 75%. The model could generate substantial responses, but sometimes failed to delve into complex topics or miss out on advanced nuances. The clarity at the advanced level decreased to around 80%, with the explanations becoming too dense or technical, making it harder for the learners to grasp the information without additional clarification. Balancing advanced insights with making content understandable was a challenge.

The model scored 85% for alignment of difficulty, and the answers are often too simplistic or complex without an adequate explanation, leading to mismatches between the difficulty of the question and the response. Coherence was also a concern at the advanced level, with complex answers 78% disrupting the logical flow or omitting crucial transitions. The relevance dropped to 82% at the advanced level, and the answers sometimes contained tangential information that was not necessary for the specific question, resulting in unfocused or superfluous details.

Figure 3 showing the real-world behavior of LPITutor using sample question and answer session. This image represents Q&A interactions comparing outputs from the LLM based approach and the LPITutor.