The aim of this study is to design and propose a governance-aware retrieval-augmented chatbot framework for supporting student access to academic regulations within a university environment. The system is intended to function as a first-line informational support tool, enabling students to query institutional policies while ensuring that responses remain grounded in authoritative regulatory documents and aligned with institutional governance requirements.

More specifically, the study aims to develop a conceptual system architecture and evaluation framework that integrates:

The overall objective is to propose how conversational AI systems can be designed to support transparent and accountable access to institutional regulations while respecting the governance boundaries of higher education institutions.

Recent advances in LLMs and conversational AI have led to the rapid adoption of chatbot systems in higher education for student support services. These systems are often used to answer questions related to admissions, course information, and administrative procedures (Kim & Baylor, 2016). However, existing implementations typically focus on general informational assistance and often rely on generative models that may produce responses without explicit grounding in institutional policy documents.

This raises several challenges when applying chatbot systems to regulatory guidance, where incorrect or unsupported responses may have academic, administrative, or legal consequences for students and institutions (Sharda et al., 2021).

Despite the growing use of conversational AI in education, several gaps remain in the literature:

These gaps highlight the need for system designs that integrate information retrieval, governance constraints, and responsible AI practices when deploying conversational interfaces for institutional regulations (Holmes et al., 2019).

This paper introduces the conceptual framework for an AI-assisted Regulations & Student Support Chatbot for the FoE. It discusses the motivations, system architecture, expected benefits, ethical considerations, and implementation challenges. By conceptualising a tailored, context-aware and regulation-grounded AI system, this work can contribute to emerging scholarly discussions on AI-driven student support in the context of higher education in developing countries and offers a practical roadmap for enhancing student experience through responsible technological innovation.

The advent of AI has transformed educational technology, enabling personalised learning experiences that cater to individual student needs. Luckin et al. (2016) emphasise the potential of AI to enhance learning by providing adaptive feedback and fostering critical thinking skills. Moreover, AI technologies facilitate the development of intelligent tutoring systems, which can offer real-time support and resources to students.

AI chatbots have emerged as a viable solution for student support across various academic disciplines. Recent studies indicate that chatbots can effectively address common student inquiries, thus reducing the workload on administrative staff (De La Roca et al., 2024). For instance, the use of chatbots in universities has been shown to streamline processes such as course registration, scheduling, and answering frequently asked questions. A notable case study at Georgia State University demonstrated a significant increase in student retention rates after the implementation of a chatbot system (Nurshatayeva et al., 2021).

Regulatory compliance is an essential aspect of higher education, ensuring that institutions meet established academic standards and practices. AI-driven solutions can assist in monitoring compliance by automatically tracking changes in regulations and facilitating timely updates to institutional policies. Duarte & Vardasca (2023) highlight the importance of integrating compliance systems within educational technology to maintain institutional integrity and support accreditation processes.

The effectiveness of a chatbot is greatly influenced by its design, including user experience and interaction capabilities. A user-centric approach that prioritises intuitive interfaces and NLP enhances user engagement with chatbots. Furthermore, studies indicate that incorporating empathy and emotional intelligence within chatbot interactions can foster positive user experiences, leading to higher satisfaction among students (Jiang et al., 2022). The design of the chatbot for the FoE will prioritise these aspects to ensure effective communication and support.

When developing AI chatbots for educational contexts, it is crucial to consider cultural and contextual factors. Luo & Hsiao-Chin (2023) suggest that chatbots must be tailored to align with the linguistic and cultural nuances of their user base to enhance effectiveness. In the context of the UoM, understanding the local culture and communication styles will be pivotal in designing the chatbot that will resonate with students and staff alike.

Despite the potential benefits of AI chatbots in education, challenges exist that must be addressed for successful implementation. Common barriers include resistance to change from staff and students, concerns over data privacy, and the need for ongoing maintenance and updates (Yin, 2021). It is essential for the UoM to develop a comprehensive strategy that includes training for users and mechanisms for continuous improvement of the chatbot.

Looking forward, the evolution of AI in education suggests a shift towards more integrated systems that combine chatbots with other technological innovations, such as LMS and data analytics tools. By enabling seamless interactions among various platforms, higher educational institutions can create a comprehensive support ecosystem that enhances both academic and administrative functions (Ashok et al., 2021).

The literature underscores the transformative potential of AI-assisted chatbots in enhancing student support and regulatory compliance within higher educational institutions. By leveraging insights from existing research, the proposed chatbot for the FoE aims to foster a supportive academic environment that aligns with current technological advancements and meets the needs of its diverse student body. Future research should continue to explore the impacts of such technologies and iterate on their designs to maximise effectiveness and user engagement.

This research adopts a design-oriented conceptual study that proposes a system architecture and evaluation framework for a regulatory-support chatbot.

The study does not present a deployed system or empirical performance evaluation. Instead, it focuses on:

The work can therefore be characterised as a design science research contribution, where the primary output is a conceptual artefact consisting of:

By articulating the design principles, implementation considerations and evaluation criteria, the study aims to provide a foundation for future empirical implementations and experimental evaluations of AI-assisted regulatory support tools in higher education.

The contribution of this study lies in the conceptualisation of a governance-aware chatbot framework tailored to institutional regulations in higher education. Specifically, the study contributes:

Together, these elements provide a methodological foundation that can support future research and practical deployments of AI systems for regulatory guidance in higher education institutions.

The proposed AI-assisted Regulations and Student Support Chatbot is designed to provide reliable, regulation-grounded guidance to students of the FoE. The system will prioritise accuracy, traceability, and accountability by grounding all responses in official institutional documents and embedding governance mechanisms that enable human oversight.

To achieve these objectives, the system adopts a Retrieval-Augmented Conversational Architecture, where semantic retrieval from a curated regulatory knowledge base precedes controlled response generation.

The block diagram of the proposed system is shown in Figure 1.

The interaction between the functional blocks will ensure that:

By embedding governance and escalation mechanisms directly into the system architecture, the proposed model will be designed to align with best practices for deploying AI systems in regulated educational environments.

The proposed system architecture will adopt a risk classifier coupled with a human-in-the-loop (HITL) decision layer, grounded in established principles from decision support systems (DSS) and AI ethics. In DSS literature, AI systems are explicitly designed not as autonomous decision-makers but as tools that support human judgment, providing recommendations that must be validated or overridden by a human expert. For example, Power (2002) emphasises that DSS are intended to augment, rather than replace, human decision-making, particularly in semi-structured and unstructured problem domains.

The inclusion of a risk classifier is motivated by the need to operationalise uncertainty and decision criticality. By estimating the likelihood that a response may be incorrect, ambiguous, or context-dependent, the system can distinguish between low-risk queries (e.g., factual FAQs) and high-risk cases (e.g., policy interpretation or academic decisions). This aligns with research in human-AI systems design, where Horvitz (1999) introduced principles of mixed-initiative interaction, advocating for systems that dynamically determine when to act autonomously and when to defer to human control. Hypothetically, effective deployment would result in high automation accuracy for routine queries while selectively escalating complex cases, thereby balancing efficiency and reliability.

From an AI ethics perspective, the HITL approach is essential to ensure accountability, fairness, and oversight. The European Commission’s High-Level Expert Group on Artificial Intelligence (European Commission, 2019) explicitly identifies human agency and oversight as a core requirement for trustworthy AI systems. Similarly, Dignum (2019) argues that responsible AI must incorporate mechanisms for human control, particularly in high-stakes or socially sensitive contexts such as education. In practice, this means that the system should not only generate answers but also provide confidence estimates and trigger escalation mechanisms when thresholds are not met. A desirable outcome would be a system where human oversight is concentrated on genuinely complex or high-stakes cases, while routine interactions are safely automated.

Finally, the integration of explainability within this architecture reflects broader regulatory and ethical trends. Explainable AI is increasingly viewed as a prerequisite for trustworthy DSS, particularly in domains where users must justify or rely on AI-supported decisions. Doshi-Velez & Kim, (2017) highlight the importance of interpretability for ensuring transparency, user trust, and effective human-AI collaboration. In this context, the combination of retrieval grounding, risk classification, and human oversight forms a coherent socio-technical design: the system provides evidence-based responses, evaluates its own uncertainty, and defers to human judgment when necessary. This layered approach is therefore well-justified both theoretically and practically for an AI system intended to support student success and institutional decision-making.

The block diagram and accompanying system model illustrate a modular, governance-aware AI chatbot architecture tailored to the regulatory and operational context of the FoE. The design balances automation with institutional oversight, ensuring that the system enhances student support while preserving academic integrity and regulatory compliance.

The proposed system model proposes how an AI-assisted chatbot can be safely deployed in a highly regulated academic environment such as the FoE. Unlike general-purpose conversational agents, this system will intentionally be constrained by design choices that prioritise regulatory correctness, traceability, and institutional accountability.

The layered architecture—comprising query processing, semantic retrieval, retrieval-augmented response generation, validation, and human escalation—ensures that automation is applied selectively. This design directly addresses concerns identified in the literature regarding hallucinations, liability, and loss of trust in AI-driven student services. The discussion that follows critically examines how each component of the system model influences dataset design and metric selection as reflected in the evaluation tables.

This section describes the implementation of the regulatory-support chatbot system, including the architecture, major design decisions, conflict-resolution mechanisms, and potential failure modes. The objective of this description is to enable reproducibility of the system design and facilitate replication in similar institutional environments.

Several failure modes were identified during system design.

Retrieval failure

Relevant clauses may not be retrieved due to:

Mitigation strategies include expanding the document corpus and tuning the retrieval model.

Hallucinated responses

The language model may generate unsupported statements not present in the retrieved evidence.

To reduce this risk, the system restricts responses to retrieved clauses and requires explicit citation.

Metadata errors

Incorrect metadata annotation (e.g., wrong validity period) may cause outdated regulations to appear in responses.

Periodic dataset validation is required to maintain metadata accuracy.

Risk misclassification

A query may be incorrectly classified as low-risk when it requires administrative oversight.

To mitigate this, the system adopts a conservative strategy where ambiguous queries are escalated.

To replicate the implementation described in this study, the following components are required:

Together, these components form a reproducible framework for building regulatory-aware chatbot systems in higher education environments.

A central strength of the system model lies in the tight coupling between architecture and dataset construction. The regulatory document corpus, described in Table 1 (dataset composition summary), will not be treated as unstructured text but as a curated, version-controlled institutional knowledge base.

Table 1 categorises the dataset into document types (e.g., general regulations, faculty handbooks, programme specifications, assessment regulations) and reports attributes such as:

Table 1 highlights that document granularity directly affects retrieval performance. Clause-level segmentation enables the Information Retrieval Module to achieve higher Recall scores compared to paragraph-level segmentation, as it reduces ambiguity and improves semantic matching. However, finer segmentation also increases index size and retrieval complexity, which must be managed through efficient indexing strategies.

The inclusion of metadata fields such as validity period and faculty applicability—outlined in Table 1—supports the system’s governance objectives and prevents outdated or cross-faculty regulations from being surfaced to students.

Table 2 (query dataset characteristics) summarises the evaluation query set used to test the system. This table typically includes:

Retrieval performance metrics presented in Table 3 (information retrieval performance) would be useful in evaluating how effectively the system identifies relevant regulatory clauses. The table is typically expected to report:

Table 3 is designed to help assess the semantic retrieval module performance. The findings can thus help inform future regulatory drafting and consolidation efforts at the institutional level.

Table 4 (response quality and grounding metrics) presents expert-evaluated measures such as:

Hallucination rate measures the extent to which the system generates information that is not supported by any source document. A low hallucination rate would indicate that the RAG approach is effectively grounding responses in verified content, whereas a high rate would suggest that the model is still relying too heavily on its generative component, increasing the risk of misinformation.

Citation correctness evaluates whether the references provided by the system accurately point to the relevant sections of the source documents. High citation correctness would imply that the system not only retrieves appropriate information but also links it transparently to its origin, thereby enhancing user trust. Conversely, low citation correctness would indicate mismatches between answers and their cited sources, which could undermine confidence even if the answer itself appears plausible.

Answer accuracy assesses the factual correctness and completeness of the generated response relative to the query. Strong performance on this metric would suggest that the system can correctly interpret and synthesise information from retrieved documents, while weaker performance would highlight limitations in reasoning or comprehension, particularly for complex or multi-part queries.

Grounding completeness measures how fully the response is supported by the retrieved evidence, especially in cases where multiple clauses or documents must be combined. High grounding completeness would indicate that the system successfully integrates all relevant pieces of information, whereas partial scores would suggest that some necessary context is missing, potentially leading to incomplete or oversimplified answers.

Finally, escalation or confidence calibration metrics capture the system’s ability to recognise its own uncertainty and defer to human intervention when necessary. Effective calibration would be reflected in the system flagging ambiguous or low-confidence cases for review, thereby reducing the risk of misleading outputs. Poor calibration, on the other hand, would result in overconfident responses in situations that require nuanced interpretation, particularly when regulations span multiple clauses or contain inherent ambiguities.

Table 5 (risk handling and escalation metrics) evaluates the system’s governance behaviour through:

Correct escalation rate measures the proportion of cases where the system appropriately identifies that it cannot confidently or safely provide a fully reliable answer and escalates the query to a human. A high correct escalation rate would suggest that the system is effectively recognising its limitations, particularly in complex, ambiguous, or high-stakes scenarios. A low rate, on the other hand, would indicate that the system is failing to flag cases that genuinely require human judgment, increasing the risk of incorrect or misleading responses being delivered automatically.

False automation rate captures the proportion of cases where the system chooses to provide an automated response when it should have escalated instead. This is a critical risk metric, as high false automation implies overconfidence: the system is answering queries autonomously despite insufficient evidence, ambiguity, or low certainty. A low false automation rate would indicate safer behaviour, where the system avoids making unsupported decisions and defers appropriately when needed.

Unnecessary escalation rate measures how often the system escalates queries that it could have handled correctly on its own. A high unnecessary escalation rate would suggest that the system is overly conservative, potentially leading to inefficiencies, increased workload for staff, and reduced user satisfaction due to delays. Conversely, a low unnecessary escalation rate would indicate that the system is making efficient use of automation, handling straightforward queries independently while reserving escalation for genuinely difficult or uncertain cases.

Together, these three metrics provide a balanced view of system behaviour: correct escalation reflects appropriate caution, false automation highlights risky overconfidence, and unnecessary escalation captures inefficiency due to excessive caution. An effective system should aim for high correct escalation, low false automation, and low unnecessary escalation.

Table 6 (user experience and trust metrics) gives visibility on survey-based measures including:

The discussion of Table 6 will help reveal how trustworthiness is correlated with citation transparency and response verbosity or conversational fluency. This finding will validate the system model’s prioritisation of traceability over purely natural dialogue.

The discussion will also critically examine limitations exposed by the metrics and datasets:

These limitations are architectural rather than algorithmic and reinforce the need for continuous governance and institutional oversight.

From an institutional perspective, the discussion aims to illustrate that the proposed system model is not merely a technical artefact but an operational support mechanism. The evaluation tables collectively will provide indications on how the chatbot will:

The modular design will also enable future extensions, such as integration with LMS or analytics-driven policy refinement. The effectiveness is a theoretical expectation until the pilot phase is complete.

The discussion confirms that the proposed system model, when evaluated using carefully designed datasets and metrics, will potentially achieve a balanced integration of automation and governance. The tables on datasets and metrics are not only evaluative tools but also diagnostic instruments that reveal strengths, limitations, and opportunities for institutional improvement.

Overall, the system model is shown to be technically sound, institutionally aligned, and ethically grounded, making it suitable for deployment within the FoE and adaptable to similar higher education contexts.

This paper presented the conceptual design, system model, and evaluation framework for an AI-assisted Regulations and Student Support Chatbot tailored to the FoE at the UoM. Unlike generic conversational systems, the proposed chatbot will explicitly be designed for high-stakes regulatory guidance, where accuracy, traceability, and institutional accountability are paramount.

The system model integrates semantic information retrieval, retrieval-augmented response generation, confidence-based validation, and human escalation mechanisms into a cohesive, governance-aware architecture. By grounding all responses in official University and Faculty regulations, the chatbot mitigates the risk of hallucinated or misleading information—an issue frequently cited in the literature on LLMs in education.

The implementation strategy emphasises modularity and sustainability, enabling the system to be maintained and updated in line with evolving academic regulations. The dataset construction process, based on authoritative regulatory documents and representative student queries, ensures contextual relevance and institutional alignment.

Tables 1, 2, 3, 4, 5, 6 present the proposed evaluation framework designed to assess the performance and governance compliance of the system as a first-line student support tool. The framework includes metrics related to retrieval accuracy, regulatory correctness, hallucination rate, and citation correctness, which will be intended to evaluate how effectively the system identifies, interprets, and references relevant institutional regulations. These indicators are included to ensure that responses remain grounded in authoritative sources and aligned with institutional policies.

In addition, the framework incorporates measures for evaluating the behaviour of the risk-aware escalation strategy. Metrics such as correct escalation rate and false automation rate are defined to assess whether high-risk or policy-sensitive queries will be appropriately redirected to human administrators. These measures aim to support the monitoring of institutional boundaries and ensure that automated responses will be limited to suitable contexts.

User-centred evaluation metrics are also included to capture student perceptions of the system. Indicators such as satisfaction, perceived trustworthiness, and intention to reuse are proposed to assess how students experience and interact with the chatbot interface. The inclusion of these measures reflects the design emphasis on transparency, traceability, and the provision of clearly cited regulatory information rather than purely conversational fluency.

Overall, the evaluation framework is intended to provide a structured approach for assessing how a regulatory-support chatbot could operate within the FoE. The framework aims to support future empirical studies that examine regulatory clarity, administrative workload, and the student information experience while maintaining alignment with institutional governance and academic integrity requirements.

Despite its strengths, the proposed system has several limitations. First, regulatory language is inherently complex and, in some cases, open to interpretation, which constrains full automation. Second, the evaluation dataset, while representative, may not capture rare or exceptional edge cases. Third, system performance is dependent on the timeliness and completeness of regulatory updates; delays in document integration could affect response accuracy.

These limitations are primarily institutional and procedural rather than algorithmic, underscoring the importance of sustained governance and human oversight.

Several avenues for future work emerge from this conceptual framework:

This work proposes that AI-assisted chatbots, when carefully designed with retrieval grounding, risk awareness, and governance mechanisms, can serve as trusted institutional support tools rather than mere conversational agents. The proposed system offers a conceptual approach to improving regulatory accessibility in higher education, providing a foundation for responsible AI deployment within the UoM and beyond.