Metacognition And Machine Cognition: Secondary Thinking As A Mediator Of Human–AI Reasoning
- 6 days ago
- 26 min read
Read the full journal, including this article, by downloading the PDF below.
Image credit: AI-generated illustration produced with OpenAI (DALL·E), 2026.
by Prof Wynand Goosen
Abstract
The rapid diffusion of generative artificial intelligence (AI) across education, professional practice, and decision-making has transformed human reasoning into a hybrid process in which humans and machines jointly generate interpretations, judgments, and knowledge claims. While critical thinking is widely promoted as a safeguard against automation bias and uncritical reliance on AI outputs, growing empirical and theoretical evidence suggests that critical thinking alone is insufficient in AI-mediated environments. Instead, effective reasoning depends on metacognition, the monitoring, regulation, and evaluation of one’s own cognitive processes, extended to include reflection on machine-generated contributions (Flavell, 1979; Dunlosky & Metcalfe, 2009). Metacognition implies thinking on an extended basis. This extended basis identifies “consequence” as the missing link. Understanding the real impact of any form of thinking, whether originating from natural humans or artificial intelligence.
This article reconceptualises metacognition as secondary thinking: a supervisory cognitive layer that governs how humans engage with, evaluate, and rely on AI systems. Drawing on cognitive psychology, education, human–computer interaction, and decision sciences, the paper argues that secondary thinking is the central mechanism through which epistemic agency, trust calibration, and responsibility are maintained in human–AI reasoning (Parasuraman & Riley, 1997; Liao & Sundar, 2022).
Three core claims are advanced. First, generative AI shifts cognition from an individual activity to a distributed human–machine system, intensifying the need for metacognitive regulation (Hutchins, 1995; Hollan et al., 2000). Second, critical thinking in AI-supported contexts is contingent on secondary thinking, which determines when and how analytical evaluation is activated (Bjork et al., 2013; Halpern, 2014). Third, failures of secondary thinking explain phenomena such as automation bias, cognitive offloading, and over-trust in fluent AI outputs (Skitka et al., 2000; Kasneci et al., 2023).
The article establishes a conceptual foundation for understanding secondary thinking as a mediator between machine cognition and human judgment, preparing the ground for a structured literature survey and the development of an integrative Human–AI Metacognitive Mediation (HAMM) model in subsequent sections.
Keywords: Metacognition; Secondary Thinking; Critical Thinking; Human–AI Collaboration; Generative Artificial Intelligence; Distributed Cognition; Automation Bias; AI Literacy
1. Introduction: From Individual Cognition to Hybrid Human–AI Reasoning
The integration of generative artificial intelligence (AI) into everyday cognitive work has fundamentally altered how reasoning is performed across educational, professional, and organisational contexts. Tasks such as writing, analysing information, planning, problem-solving, and decision-making now increasingly involve sustained interaction with AI systems capable of producing fluent, context-sensitive, and seemingly reasoned outputs. These systems do not merely support cognition at the margins or automate routine functions; rather, they participate directly in the generation, structuring, and evaluation of ideas. In many contemporary settings, AI-generated outputs function as primary cognitive inputs, the starting point for reasoning rather than its endpoint, thereby reshaping how judgments are formed.
As a consequence, cognition can no longer be adequately understood as a purely internal process confined to the individual mind. Instead, reasoning unfolds within hybrid human–machine systems, in which cognitive labour is distributed across human intuitions, analytical processes, and machine-generated contributions (Salomon, 1993; Hutchins, 1995; Kirsh, 2010). Within such systems, reasoning outcomes (consequences) emerge not from isolated human deliberation or automated computation alone, but from the interaction between fast human judgments, AI-generated outputs, and the regulatory processes that govern their integration. The consequences, or impact, of the integrated human and artificial thinking suggest a level of “metacognition” —thinking in such a way that we understand the real consequences.
This reconceptualisation is consistent with foundational work in distributed cognition, which demonstrates that reasoning routinely extends beyond the individual to include artefacts, representations, technologies, and social structures that shape how problems are framed and solved (Salomon, 1993; Hutchins, 1995). Generative AI intensifies this dynamic by functioning not only as an external resource but as an active contributor to reasoning, proposing arguments, synthesising evidence, generating counterfactuals, and simulating inferential pathways. In practice, humans increasingly co-think with AI systems, engaging them as interlocutors within cognitive activity rather than as neutral tools executing predefined instructions.
This shift fundamentally alters the epistemic conditions under which reasoning occurs. When AI systems contribute directly to the content and structure of reasoning, questions of agency, responsibility, authorship, and justification become distributed rather than singular. The apparent coherence, confidence, and completeness of AI-generated outputs can obscure their probabilistic and non-semantic foundations, while their speed and scale can reduce opportunities for reflective human oversight. As a result, epistemic risk no longer arises solely from human bias or error, but from mismanaged interaction between human judgment and machine-generated contributions.
Against this backdrop, a foundational epistemic question emerges: What forms of thinking enable humans to remain responsible epistemic agents when reasoning with and through AI? In educational and professional discourse, critical thinking is commonly positioned as the primary response to this challenge (Facione, 2015; Ennis, 2016). Learners and practitioners are urged to evaluate AI outputs, interrogate assumptions, verify sources, and resist uncritical acceptance. While normatively compelling, this response implicitly assumes that critical thinking is continuously available and readily activated.
However, extensive research in cognitive psychology demonstrates that critical thinking is effortful, situational, and conditionally deployed, rather than a default mode of reasoning (Kuhn, 1999; Halpern, 2014). Analytical evaluation requires sustained attention and cognitive control, and it is frequently not activated in environments characterised by time pressure, informational overload, or reduced perceived effort. When tasks appear fluent, complete, or cognitively efficient, as is often the case with generative AI outputs, individuals are more likely to rely on surface plausibility than on sustained analytical scrutiny.
Empirical evidence further indicates that analytical reasoning is governed by higher-order regulatory processes rather than operating autonomously. Individuals do not engage in continuous critical evaluation; instead, such evaluation is triggered by signals of uncertainty, contradiction, or epistemic risk. In the absence of these signals, even highly trained individuals may accept outputs unreflectively, particularly when those outputs align with prior beliefs or expectations. AI-mediated environments systematically attenuate many of the cues that typically prompt scrutiny—such as visible uncertainty, hesitation, or partial explanations—thereby increasing the likelihood that critical thinking remains dormant.
These regulatory processes are captured by metacognition, broadly defined as awareness and control of one’s own thinking (Flavell, 1979; Nelson & Narens, 1990). Metacognition enables individuals to monitor uncertainty, evaluate progress, and adjust cognitive strategies. In AI-mediated reasoning, however, metacognition must operate at an expanded scope. It must regulate not only human thought processes, but also reliance on machine-generated outputs, including decisions about when to accept, interrogate, override, or suspend AI assistance (Liao & Sundar, 2022).
Despite this necessity, research on metacognition, critical thinking, and human–AI interaction remains fragmented. Educational psychology has developed robust models of self-regulation yet typically treats cognition as individual (Schraw & Dennison, 1994). Human–AI interaction research has focused extensively on usability, trust, and performance, but with limited integration of metacognitive theory (Parasuraman et al., 2000). Meanwhile, educational discussions of generative AI often frame critical thinking as a static skill to be preserved, rather than as a conditionally activated process governed by metacognitive mediation (Holmes et al., 2023).
This fragmentation obscures a central insight: failures in human–AI reasoning are frequently not failures of critical thinking capacity, but failures of metacognitive regulation within hybrid systems. When regulatory oversight is weak, critical thinking may never be activated, regardless of an individual’s expertise or intent. This article addresses that gap by advancing secondary thinking as the supervisory mechanism that mediates between human cognition and machine-generated contributions.
Building on this insight, the article proposes the Human–AI Metacognitive Mediation (HAMM) model, which conceptualises reasoning in AI-mediated environments as a multi-layered, recursively regulated system. Within this model, secondary thinking governs the activation of critical thinking, the calibration of trust in AI systems, and the integration of machine outputs into human judgment. By foregrounding metacognitive mediation, the HAMM model provides a coherent theoretical foundation for understanding epistemic agency, accountability, and responsible reasoning in the age of generative AI.
2. Metacognition as Secondary Thinking
Metacognition has long been defined as “knowledge and cognition about cognitive phenomena”, encompassing the processes by which individuals plan, monitor, and regulate their thinking (Flavell, 1979). This foundational formulation established that cognition is not limited to first-order processes such as perception, inference, or memory retrieval, but also includes higher-order awareness of how those processes unfold and how effectively they serve one’s goals. Subsequent theoretical models refined this view by distinguishing between metacognitive knowledge, metacognitive monitoring, and metacognitive control, emphasising that effective reasoning depends not only on the content of thought, but on the continuous oversight and regulation of cognitive activity (Nelson & Narens, 1990; Schraw & Dennison, 1994; Efklides, 2008).
Across decades of empirical research, metacognition has been shown to predict learning quality, problem-solving success, and error detection across a wide range of domains and age groups. Importantly, these effects are most pronounced in complex, uncertain, or ill-structured task contexts in which individuals must decide not only how to think, but whether their current understanding is adequate and when to revise their approach (Zimmerman, 2002; Dunlosky & Metcalfe, 2009). These characteristics closely resemble the cognitive demands imposed by AI-mediated reasoning environments.
In this article, metacognition is conceptualised functionally as secondary thinking—a supervisory layer of cognition that operates on thinking rather than as thinking. This distinction is analytically important. Secondary thinking does not directly generate ideas, arguments, or solutions; instead, it governs the conditions under which such cognitive products are produced, evaluated, and revised. Framed in this way, secondary thinking functions as a form of cognitive governance, shaping the reliability, depth, and epistemic quality of reasoning rather than its immediate outputs.
More specifically, secondary thinking regulates cognition through several interrelated functions. It involves monitoring the coherence, plausibility, and uncertainty of one’s own reasoning, allowing individuals to recognise gaps in understanding or mismatches between confidence and justification (Schraw, 1998; Dunlosky & Metcalfe, 2009). It also includes evaluating the reliability, scope, and limitations of AI-generated outputs, particularly in light of the probabilistic and opaque nature of generative models (Van den Bosch & Bronkhorst, 2018; Liao & Sundar, 2022). A further function of secondary thinking is detecting cognitive conflict, such as discrepancies between intuitive judgments and analytical considerations, or between human reasoning and machine suggestions (Kahneman, 2011; Stanovich et al., 2016). Finally, secondary thinking supports strategic control, enabling individuals to decide when to rely on AI, when to override it, and when to seek alternative sources or verification (Parasuraman et al., 2000; Endsley, 2017).
This framing clarifies the role of metacognition in AI-mediated reasoning. Rather than treating metacognition as a generic learning skill or background trait, secondary thinking is understood as a dynamic regulatory mechanism that governs the interaction between human cognition and machine cognition. It operates above both intuitive, fast processes and deliberate analytical processes, coordinating their deployment in response to contextual demands and epistemic risk (Bjork et al., 2013).
Crucially, secondary thinking is not automatically triggered. Its activation depends on task design, perceived stakes, time pressure, and the presence of explicit reflective cues. In environments where AI systems produce fluent, confident, and seemingly authoritative outputs, the absence of such cues can suppress secondary thinking altogether. Under these conditions, even highly educated individuals may accept outputs uncritically, mistaking coherence, confidence, or stylistic sophistication for epistemic correctness (Williams et al., 2022). This vulnerability underscores the central importance of secondary thinking in hybrid human–AI reasoning systems.
3. Critical Thinking in AI-Mediated Contexts
Critical thinking is typically defined as purposeful, reflective judgment involving the analysis and evaluation of arguments, evidence, and assumptions (Facione, 1990; Ennis, 2016). Within higher education and professional practice, it is widely regarded as a cornerstone competence, underpinning informed decision-making, complex problem-solving, ethical judgment, and responsible participation in knowledge-based societies. Critical thinking is often positioned as a safeguard against error, bias, and misinformation, particularly in environments characterised by informational abundance, competing claims, and epistemic uncertainty. In policy discourse and educational reform, it is frequently presented as the primary defence against the epistemic risks associated with digital technologies and automated systems.
However, a substantial body of research demonstrates that critical thinking is neither context-free nor continuously active. Rather than operating as a default cognitive mode, critical thinking is situational, effortful, and strongly shaped by higher-order regulatory processes that govern attention, effort allocation, and strategy selection (Halpern, 2014; Kuhn, 2015). Individuals do not constantly evaluate the quality of information they encounter. Instead, analytical reasoning is selectively engaged in response to cues such as uncertainty, perceived risk, contradiction, or explicit demands for justification. When these cues are absent, even well-trained individuals may rely on surface plausibility rather than deliberate evaluation.
From a cognitive perspective, critical thinking requires sustained attention, effortful processing, and the inhibition of intuitive or habitual responses. These demands make it cognitively costly and therefore selectively deployed. Individuals are more likely to engage in critical scrutiny when they perceive high stakes or epistemic danger, and less likely to do so when tasks appear familiar, efficient, or well-structured. Consequently, critical thinking is highly sensitive to features of the surrounding cognitive environment, including time pressure, perceived task difficulty, motivational factors, and the availability of external cognitive supports. Environments that reduce effort or signal completion can unintentionally suppress analytical engagement.
In AI-supported environments, this conditionality becomes especially pronounced. Generative AI systems can, under appropriate conditions, scaffold critical thinking by exposing users to alternative perspectives, counterarguments, illustrative examples, and structured explanations that prompt comparison, evaluation, and synthesis (Chi, 2009; Kerrigan & Azevedo, 2022). When AI is used dialogically and reflectively—through iterative prompting, critique, explanation requests, or justification cycles—it can function as a cognitive partner that extends rather than replaces human reasoning capacity. Empirical work suggests that such reflective use of AI can enhance argument quality, conceptual integration, and depth of understanding, particularly when users remain actively engaged in evaluating the AI’s contributions (Xie et al., 2024).
At the same time, generative AI can suppress critical thinking, particularly when its outputs are fluent, confident, and presented as complete or authoritative solutions. In these cases, AI reduces perceived task difficulty and creates an impression that cognitive work has already been done, encouraging cognitive offloading and premature closure (Kasneci et al., 2023). This suppression is amplified by well-established cognitive biases, most notably the tendency to equate linguistic fluency with epistemic reliability (Kahneman, 2011). As Bender and Koller (2020) argue, the surface coherence of language-model outputs can obscure underlying inaccuracies, fabrications, or unsupported inferences, creating a powerful illusion of understanding that discourages further scrutiny.
Importantly, in such cases, critical thinking is not absent because users lack the relevant skills, dispositions, or educational preparation. Rather, it is absent because it is never activated. The cognitive environment created by generative AI often removes or attenuates the signals—such as visible uncertainty, hesitation, partial reasoning, or error—that typically prompt analytical evaluation. Without these epistemic triggers, individuals may default to acceptance, particularly when AI outputs align with prior beliefs, expectations, or goals. As a result, critical thinking remains latent, even among individuals with substantial training and experience in analytical reasoning and professional judgment.
These observations indicate that critical thinking alone cannot adequately explain reasoning quality in AI-mediated contexts. Instead, critical thinking is governed by secondary thinking, which determines when analytical evaluation is warranted, how intensively it should be applied, and whether AI outputs should be accepted, questioned, or rejected (Dunlosky & Metcalfe, 2009; Bjork et al., 2013). Secondary thinking monitors epistemic risk, detects overconfidence, and initiates reflective scrutiny when conditions demand it. Without such metacognitive oversight, critical thinking cannot reliably fulfil its epistemic function within hybrid cognitive systems, regardless of an individual’s formal competence, experience, or intentions.
4. Machine Cognition, Automation Bias, and Distributed Responsibility
Although contemporary AI systems do not possess cognition in the human sense, they increasingly perform functions that simulate the outward characteristics of reasoning. These include generating explanations, synthesising large bodies of information, identifying patterns across datasets, and proposing decisions or action pathways that resemble deliberative judgment (Miller, 2019; Bender et al., 2021). In interactive contexts, these outputs are linguistically fluent, context-sensitive, and often structurally indistinguishable from human-authored reasoning. They follow the conventions of argumentation, explanation, and narrative coherence that humans associate with understanding and rational agency. As a result, users frequently respond to AI systems as if they were epistemic agents—entities capable of knowing, reasoning, and justifying—rather than as computational artefacts executing statistical inference.
This functional resemblance has profound cognitive consequences. Humans are evolutionarily and socially predisposed to attribute understanding, intention, and authority to entities that produce coherent language and plausible explanations. Language fluency, in particular, acts as a powerful cue for competence and trustworthiness. In the case of generative AI, this attribution occurs despite extensive evidence that such systems operate through probabilistic pattern completion rather than semantic comprehension, intentional reasoning, or truth-directed inference (Bender & Koller, 2020; Ji et al., 2023). The system does not “know” what it states, nor does it possess commitments to truth, evidence, or justification. The growing disjunction between epistemic appearance and epistemic reality therefore becomes a central source of risk in AI-mediated reasoning.
Crucially, the problem is not simply that users misunderstand how AI systems work at a technical level. Rather, AI systems actively reshape the epistemic environment in which judgments are formed. Generative AI reduces informational friction by presenting outputs that are immediate, confident, syntactically complete, and often framed as solutions rather than as provisional suggestions. Traditional cues that signal uncertainty—hesitation, partial explanations, visible effort, acknowledged gaps, or expressions of doubt—are largely absent. This absence alters the cognitive ecology of reasoning, shifting users toward acceptance rather than interrogation. Even when errors or inconsistencies are present, the smoothness of presentation can suppress the impulse to scrutinise.
One of the most robustly documented consequences of this altered epistemic environment is automation bias, defined as the systematic tendency to over-rely on automated systems, particularly when they appear authoritative, consistent, or cognitively efficient (Parasuraman & Riley, 1997; Skitka et al., 2000). Research across aviation, healthcare, finance, and military decision-making consistently demonstrates that individuals frequently defer to automated recommendations even when contradictory evidence is available or when the system is demonstrably fallible. Importantly, automation bias persists among experts and does not reliably diminish with experience alone, especially under conditions of cognitive load, time pressure, or task complexity (Mosier & Skitka, 2018).
In AI-mediated reasoning, automation bias manifests in distinct but related forms. Users may accept AI-generated explanations without verification, fail to notice hallucinated citations or fabricated claims, or defer judgment to the system when outputs align with prior beliefs or goals. Generative AI intensifies these effects by masking epistemic uncertainty behind linguistic fluency, thereby exploiting well-known cognitive heuristics that equate coherence with correctness (Kahneman, 2011). The result is not blind trust in technology per se, but a subtle recalibration of epistemic vigilance in which scrutiny is deferred unless explicitly triggered.
A key construct for understanding these dynamics is trust calibration, defined as the alignment between a user’s confidence in an AI system and the system’s actual reliability in a given context (Van den Bosch & Bronkhorst, 2018; Liao & Sundar, 2022). Miscalibrated trust produces two symmetrical failure modes: over-reliance, in which erroneous or inappropriate outputs are accepted, and under-reliance, in which valuable assistance is dismissed or ignored. Importantly, trust calibration is not achieved solely through improvements in system accuracy, transparency, or explainability. It is fundamentally a metacognitive accomplishment, requiring users to recognise uncertainty, reflect on system limitations, assess contextual appropriateness, and dynamically adjust reliance strategies (Gigerenzer, 2020).
From the perspective of distributed cognition, reasoning outcomes remain human responsibilities, even when machines contribute substantially to the cognitive process (Hutchins, 1995). Cognitive artefacts can extend, scaffold, and transform reasoning by externalising memory, generating representations, or accelerating comparison. However, they do not assume epistemic agency. The delegation of cognitive labour—such as information retrieval or synthesis—does not entail the delegation of epistemic accountability. Decisions, interpretations, and justifications ultimately remain attributable to human agents. Ethical and epistemic responsibility therefore cannot be offloaded to machines, regardless of their apparent sophistication or autonomy.
Within this distributed cognitive system, secondary thinking emerges as the mechanism through which responsibility is exercised and maintained. Secondary thinking governs how AI outputs are interpreted, how trust is dynamically calibrated, and how final judgments are justified and defended. It enables users to step back from fluent outputs, interrogate their reliability, and reflect on how machine-generated content has shaped their reasoning process. By regulating the interaction between human judgment and machine-generated contributions, secondary thinking supports epistemic vigilance, accountability, and adaptive reliance within hybrid human–AI reasoning systems (Fricker, 2007; Medina, 2013).
5. Synthesis of the Literature: Secondary Thinking as a Mediator in Human–AI Reasoning
The preceding sections establish three interlocking premises that together reframe how reasoning must be understood in AI-mediated contexts. First, cognition in AI-rich environments is increasingly distributed across human and machine agents, rather than confined to the individual mind. Second, critical thinking—while indispensable to sound judgment—does not operate continuously or autonomously in such environments; its activation is conditional and effortful. Third, metacognitive regulation, conceptualised in this article as secondary thinking, is required to govern reasoning processes within hybrid cognitive systems. This section synthesises findings across educational psychology, cognitive science, and human–AI interaction research to demonstrate that secondary thinking operates not merely as an auxiliary skill, but as a mediating mechanism that determines how machine-generated contributions influence human judgment.
Within educational psychology, decades of empirical research consistently demonstrate that metacognition is a stronger predictor of learning quality, error detection, transfer, and long-term retention than cognitive ability or domain knowledge alone (Schraw & Dennison, 1994; Zimmerman, 2002; Dunlosky & Metcalfe, 2009). Learners with strong metacognitive skills are better able to monitor their understanding, detect misconceptions, allocate effort strategically, and adapt their approaches when faced with difficulty. These advantages are especially pronounced in ill-structured problem-solving contexts, where there is no single correct solution path and where success depends on evaluating the adequacy and reliability of one’s reasoning strategies (Efklides, 2008). In such contexts, progress hinges less on executing procedures and more on regulating cognition itself.
Crucially, these metacognitive advantages are not limited to formal learning settings. They generalise to any environment characterised by uncertainty, complexity, and incomplete information—conditions that closely resemble contemporary AI-mediated reasoning contexts. Generative AI systems introduce precisely the circumstances under which metacognitive regulation becomes essential: informational abundance, reduced cognitive effort, and asymmetric epistemic authority between human users and machine-generated outputs. When AI systems rapidly generate fluent explanations, summaries, or recommendations, users must decide not only what to accept, but whether acceptance is warranted at all. These decisions are inherently metacognitive.
Parallel insights emerge from the literature on human–automation interaction and decision sciences. Decades of research on automation bias demonstrate that individuals frequently over-rely on automated systems, even when those systems are demonstrably fallible or when contradictory evidence is available (Parasuraman & Riley, 1997; Skitka et al., 2000). This over-reliance is exacerbated under conditions of cognitive load, time pressure, or perceived system authority, and persists even among highly trained professionals (Mosier & Skitka, 2018). Importantly, these failures cannot be adequately explained by lack of expertise, motivation, or training. Instead, they reflect breakdowns in metacognitive monitoring, particularly failures to recognise uncertainty, question reliability, or recalibrate trust in light of contextual cues.
Research on generative AI in educational and professional settings further reinforces this interpretation. Studies consistently show that passive engagement with AI outputs—such as copying, accepting, or minimally editing generated content—is associated with shallow processing, reduced cognitive effort, and uncritical acceptance of plausible but flawed information (Ji et al., 2023; Kasneci et al., 2023). In contrast, when AI use is embedded within reflective and metacognitively demanding practices—including self-explanation, justification prompts, comparison of alternatives, or iterative critique—reasoning quality improves substantially (Kerrigan & Azevedo, 2022; Xie et al., 2024). Learners and practitioners in these conditions demonstrate greater awareness of uncertainty, stronger argumentation, and improved ability to detect errors or inconsistencies.
Taken together, these findings indicate that AI itself is neither inherently beneficial nor inherently detrimental to human cognition. Rather, its cognitive impact is conditional, shaped by how individuals regulate their interaction with machine-generated outputs. AI can function as a powerful cognitive scaffold when it is used within a metacognitively regulated framework that encourages reflection, evaluation, and strategic control. Conversely, AI can function as a cognitive substitute when such regulation is absent, leading to cognitive offloading, automation bias, and diminished epistemic vigilance.
Synthesised across these literatures, a central explanatory insight emerges: secondary thinking governs whether AI functions as a cognitive scaffold or as a cognitive substitute. When secondary thinking is active, individuals monitor both their own reasoning and the AI’s contributions, calibrate trust appropriately, and engage critical thinking when warranted. Under these conditions, AI supports exploration, comparison, and epistemic reflection. When secondary thinking is inactive or suppressed, AI reduces epistemic friction, obscures uncertainty, and encourages uncritical acceptance, thereby suppressing reasoning rather than enhancing it.
This mediating role of secondary thinking provides the missing theoretical bridge linking metacognition research, critical thinking theory, and human–AI interaction studies. It explains why similar AI tools can produce radically different cognitive outcomes across users and contexts, and why interventions focused solely on improving AI accuracy or transparency are insufficient. Without attention to the metacognitive processes that regulate human–AI interaction, critical thinking cannot reliably fulfil its epistemic function within hybrid cognitive systems.
By positioning secondary thinking as a mediator rather than a peripheral skill, this synthesis reframes human–AI reasoning as a metacognitively governed system. This reframing sets the foundation for the Human–AI Metacognitive Mediation (HAMM) model developed in the following section, which formalises these relationships and articulates their implications for education, leadership, and AI literacy.
6. The Human–AI Metacognitive Mediation (HAMM) Model
6.1 Conceptual Architecture
Building on the preceding synthesis, the Human–AI Metacognitive Mediation (HAMM) model is proposed as an integrative theoretical framework for understanding reasoning in AI-mediated environments. The model is designed to explain how and why interactions between human cognition and machine-generated outputs produce variable epistemic outcomes, ranging from enhanced reasoning and learning to automation bias and cognitive offloading. Rather than conceptualising reasoning as a linear sequence in which human input is followed by machine output and then acceptance or rejection, the HAMM model frames human–AI reasoning as a dynamic, recursive system governed by metacognitive oversight.
At the core of the HAMM model is the claim that reasoning quality in AI-mediated contexts is not determined solely by the accuracy of AI systems or the critical thinking ability of users. Instead, it is shaped by the regulatory processes that govern how human cognition and machine-generated contributions are integrated, monitored, and revised over time. Secondary thinking functions as the supervisory mechanism that coordinates this integration, ensuring that reasoning remains adaptive, context-sensitive, and epistemically responsible.
The model comprises three interacting layers, each of which plays a distinct but interdependent role in hybrid reasoning:
Layer 1: Primary Cognitive Inputs
The first layer consists of primary cognitive inputs, which include fast, automatic human processes—such as heuristics, intuitions, emotional responses, and prior beliefs—as well as machine-generated outputs produced through probabilistic inference and pattern recognition. These inputs are characterised by speed and efficiency, allowing rapid generation of ideas, interpretations, and responses. However, they are also epistemically opaque, offering little intrinsic assurance of reliability or justification (Kahneman, 2011; Bender & Koller, 2020).
In human cognition, primary processes are adaptive but vulnerable to bias and overconfidence. In machine cognition, outputs are fluent and scalable but indifferent to truth, meaning, or epistemic commitment. The HAMM model treats both sources symmetrically at this level: they provide raw material for reasoning, not warranted conclusions. Importantly, the model highlights that AI-generated content enters the reasoning process at the same level as intuitive human judgments, making regulatory oversight essential.
Layer 2: Critical Thinking Processes
The second layer comprises critical thinking processes, involving deliberate evaluation of claims, evidence, assumptions, and alternatives (Facione, 1990; Ennis, 2016). At this level, individuals assess coherence, consistency, relevance, and justification, including those associated with AI-generated outputs. Critical thinking enables comparison of alternatives, detection of inconsistencies, and evaluation of evidentiary support.
Crucially, the HAMM model explicitly rejects the assumption that critical thinking is continuously active. Instead, it is treated as a conditionally deployed cognitive resource. Activation depends on perceived uncertainty, task demands, epistemic risk, and—most importantly—metacognitive signals originating from the supervisory layer. Without such activation, primary inputs may pass directly to judgment without scrutiny, regardless of their origin.
Layer 3: Secondary Thinking (Metacognitive Mediation)
The third layer consists of secondary thinking, conceptualised as metacognitive mediation. This supervisory layer performs three central regulatory functions: monitoring, control, and reflection. It monitors uncertainty, coherence, and conflict between human judgments and AI-generated suggestions; it controls reliance strategies by deciding when to defer to AI, when to override it, and when to seek alternative sources; and it supports reflection on epistemic assumptions, biases, and prior outcomes (Flavell, 1979; Efklides, 2008).
Secondary thinking determines when critical thinking is activated, how intensively it is applied, and whether reliance on AI should be increased, reduced, or suspended. It thus functions as the gatekeeper of analytical engagement, ensuring that critical thinking is deployed where it is epistemically warranted rather than indiscriminately or not at all.
A defining feature of the HAMM model is the recursive interaction among these layers. Outputs from critical thinking feed back into secondary thinking, updating trust calibration, confidence judgments, and future reliance strategies. Over time, this recursive process supports learning, adaptation, and the development of AI-specific metacognitive skills, allowing users to become more discerning and reflective participants in hybrid reasoning systems.
6.2 Theoretical Propositions
From the conceptual architecture of the HAMM model, several theoretically grounded propositions follow. These propositions articulate testable relationships that link metacognitive regulation, AI use, and epistemic outcomes:
a. Secondary thinking positively predicts calibrated trust in AI systems.Individuals with stronger metacognitive monitoring and control are better able to align their confidence in AI outputs with the system's actual reliability, thereby avoiding both over- and under-reliance.
b. Trust calibration mediates the relationship between AI use and reasoning quality. AI use enhances reasoning outcomes only when trust is appropriately calibrated; miscalibration leads either to automation bias or to the rejection of valuable cognitive support.
c. High AI fluency combined with weak secondary thinking increases susceptibility to automation bias.Fluent, confident AI outputs suppress analytical engagement when metacognitive oversight is weak, particularly under conditions of cognitive load or time pressure.
d. Metacognitive scaffolds enhance epistemic outcomes in generative-AI tasks. Design features such as reflective prompts, uncertainty visualisations, self-explanation routines, and justification requirements activate secondary thinking and improve reasoning quality, transfer, and epistemic accountability (Azevedo et al., 2010; Xie et al., 2024).
Together, these propositions position secondary thinking as the central explanatory variable linking machine cognition to human judgment and epistemic outcomes. They provide a theoretically coherent basis for empirical testing and for the design of educational, professional, and organisational interventions aimed at improving human–AI reasoning.
7. Implications for Education, Leadership, and AI Literacy
7.1 Doctoral and Higher Education
Doctoral and advanced higher education constitute high-stakes epistemic environments in which originality, rigour, transparency, and accountability are central to scholarly legitimacy. Doctoral candidates are expected not only to produce novel contributions to knowledge, but also to demonstrate mastery over reasoning processes, evidentiary standards, and epistemic justification. In this context, the increasing reliance on generative AI for activities such as literature exploration, conceptual framing, methodological planning, data analysis, and academic writing introduces profound opportunities alongside equally significant risks.
Empirical studies indicate that unregulated use of generative AI can result in hallucinated or fabricated citations, superficial synthesis of literature, erosion of argumentative coherence, and diminished engagement with primary sources (Ji et al., 2023; Kasneci et al., 2023). These risks are not merely technical or procedural; they are fundamentally epistemic. When AI-generated content is incorporated without reflective scrutiny, the integrity of scholarly reasoning is compromised, even when surface-level outputs appear coherent or sophisticated.
The HAMM model reframes AI use in doctoral education not as a productivity shortcut to be managed through prohibition or detection, but as an object of metacognitive scrutiny. From this perspective, the core educational challenge is not whether doctoral candidates use AI, but how they regulate its influence on their reasoning processes. Secondary thinking becomes a central scholarly competence, enabling candidates to monitor how AI-generated suggestions shape their conceptual decisions, argumentative structure, and interpretation of evidence.
Practically, this implies that doctoral education should incorporate explicit metacognitive practices related to AI use. These may include reflective AI-use logs, structured justification of AI-assisted decisions, and critical annotation of AI-generated outputs. Supervisory practices can also be adapted to foreground reasoning transparency, asking candidates not only what conclusions they reached, but how AI tools influenced the path to those conclusions. In this way, AI becomes a catalyst for deeper epistemic reflection rather than a threat to scholarly rigour.
7.2 Leadership and Strategic Decision-Making
In leadership and executive contexts, AI systems are increasingly deployed for forecasting, risk modelling, scenario planning, policy analysis, and strategic decision-making. These applications place AI at the core of organisational judgment, often under conditions of uncertainty, time pressure, and high consequence. While AI promises enhanced analytical capacity and efficiency, it also amplifies well-documented cognitive vulnerabilities, particularly automation bias, when outputs align with intuitive judgments or reduce perceived cognitive effort (Green & Chen, 2019; Raisch & Krakowski, 2021).
Leaders are especially susceptible to these dynamics because strategic decisions often involve ambiguous data, competing priorities, and limited feedback loops. Fluent AI-generated recommendations can create an illusion of certainty or objectivity, leading leaders to overestimate the reliability of algorithmic outputs or to defer judgment prematurely. In such contexts, failures are rarely attributable to lack of intelligence or experience. Instead, they reflect insufficient regulation of how machine-generated insights are interpreted and integrated into human judgment.
The HAMM model positions secondary thinking as a core leadership capability. Effective leaders must be able to step back from AI outputs, interrogate underlying assumptions, recognise model limitations, and reflect on how cognitive biases—both human and algorithmic—shape decision outcomes. Secondary thinking supports epistemic humility by reminding leaders that AI systems are probabilistic, context-dependent, and value-laden rather than neutral arbiters of truth (Floridi, 2019; Gigerenzer, 2020).
Leadership development programmes should therefore move beyond instrumental AI training and incorporate metacognitive components. AI-augmented decision simulations, followed by structured debriefs focused on reasoning processes rather than outcomes alone, can help leaders develop awareness of trust calibration, over-reliance, and cognitive offloading. Such practices align strategic competence with epistemic responsibility, ensuring that AI enhances rather than displaces human judgment.
7.3 AI Literacy Frameworks
Current AI literacy frameworks typically emphasise technical understanding, ethical awareness, and the ability to critically evaluate AI outputs (Long & Magerko, 2020). While these dimensions are essential, they often underemphasise the metacognitive regulation of human–AI interaction. As a result, AI literacy is frequently framed as knowledge about AI rather than competence in managing how AI reshapes one’s own thinking.
The HAMM model extends existing AI literacy frameworks by positioning secondary thinking as a foundational literacy component. From this perspective, AI literacy involves not only understanding what AI systems can and cannot do, but also recognising how interaction with AI alters cognitive effort, attention allocation, confidence judgments, and epistemic vigilance. Literate users are those who can monitor their reliance on AI, detect when critical thinking is being suppressed, and deliberately re-engage analytical scrutiny when warranted.
Embedding secondary thinking into AI literacy initiatives has implications across educational and professional domains. Curricula should incorporate reflective prompts, uncertainty interrogation, and explicit discussion of cognitive offloading and automation bias. Rather than focusing solely on correct use of tools, AI literacy should cultivate awareness of how reasoning itself changes in AI-mediated environments.
By foregrounding metacognitive regulation, the HAMM model reframes AI literacy as a form of cognitive self-governance. This reframing aligns AI education with broader goals of epistemic agency, responsible judgment, and human flourishing in technologically mediated societies.
8. Conclusion
This article demonstrates that the central cognitive challenge of the AI era is not a deficit of critical thinking, but a deficit of effective metacognitive regulation in hybrid human–AI reasoning systems. By integrating metacognition theory, critical thinking research, and human–AI interaction studies, it shows that secondary thinking mediates the epistemic impact of machine cognition.
The HAMM model provides a theoretically grounded explanation for automation bias, cognitive offloading, and trust miscalibration, while offering actionable implications for education, leadership, and AI literacy. As AI systems become increasingly embedded in cognitive and societal processes, human agency will depend not only on what we think, but on our capacity to think about how we think with machines.
References
Azevedo, R., Moos, D.C., Johnson, A.M. and Chauncey, A.D. 2010. Measuring cognitive and metacognitive regulatory processes during hypermedia learning, Educational Psychologist, 45(4), pp. 210–223.
Bandura, A. 1997. Self-efficacy: The exercise of control. New York: Freeman.
Bender, E.M. and Koller, A. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data, Proceedings of the ACL, pp. 5185–5198.
Bender, E.M., Gebru, T., McMillan-Major, A. and Shmitchell, S. 2021. On the dangers of stochastic parrots, Proceedings of the ACM FAccT Conference, pp. 610–623.
Bjork, R.A., Dunlosky, J. and Kornell, N. 2013. Self-regulated learning, Annual Review of Psychology, 64, pp. 417–444.
Bransford, J.D., Brown, A.L. and Cocking, R.R. 2000. How people learn. Washington, DC: National Academy Press.
Chi, M.T.H. 2009. Active-constructive-interactive learning, Topics in Cognitive Science, 1(1), pp. 73–105.
Dede, C. 2014. Digital tools for deeper learning, Harvard Educational Review, 84(4), pp. 545–568.
Dennett, D.C. 2017. From bacteria to Bach and back. New York: Norton.
Dunlosky, J. and Metcalfe, J. 2009. Metacognition. Thousand Oaks, CA: Sage.
Efklides, A. 2008. Metacognition and self-regulation, European Psychologist, 13(4), pp. 277–287.
Endsley, M.R. 2017. From here to autonomy, Human Factors, 59(1), pp. 5–27.
Ennis, R.H. 2016. Critical thinking across the curriculum. Lanham, MD: Rowman & Littlefield.
Facione, P.A. 1990. Critical thinking: A statement of expert consensus. Millbrae, CA: APA Delphi Report.
Facione, P.A. 2015. Critical thinking: What it is and why it counts. Insight Assessment.
Flavell, J.H. 1979. Metacognition and cognitive monitoring, American Psychologist, 34(10), pp. 906–911.
Floridi, L. 2013. The ethics of information. Oxford: Oxford University Press.
Floridi, L. 2019. Establishing the rules for trustworthy AI, Nature Machine Intelligence, 1, pp. 261–262.
Fricker, M. 2007. Epistemic injustice. Oxford: Oxford University Press.
Gigerenzer, G. 2020. How to stay smart in a smart world. Cambridge, MA: MIT Press.
Green, B. and Chen, Y. 2019. Algorithmic bias mitigation, Proceedings of FAT ’19*, pp. 1–10.
Halpern, D.F. 2014. Thought and knowledge. 5th ed. New York: Psychology Press.
Hollan, J., Hutchins, E. and Kirsh, D. 2000. Distributed cognition, ACM Transactions on Computer–Human Interaction, 7(2), pp. 174–196.
Holmes, W., Bialik, M. and Fadel, C. 2023. Artificial intelligence in education. Boston: Center for Curriculum Redesign.
Hutchins, E. 1995. Cognition in the wild. Cambridge, MA: MIT Press.
Ji, Z., Lee, N., Frieske, R. et al. 2023. Survey of hallucination in natural language generation, ACM Computing Surveys, 55(12), pp. 1–38.
Kahneman, D. 2011. Thinking, fast and slow. New York: Farrar, Straus and Giroux.
Kasneci, E. et al. 2023. ChatGPT for education, Computers & Education: Artificial Intelligence, 5, 100135.
Kirsh, D. 2010. Thinking with external representations, AI & Society, 25(4), pp. 441–454.
Kuhn, D. 1999. A developmental model of critical thinking, Educational Researcher, 28(2), pp. 16–25.
Kuhn, D. 2015. Thinking together and alone, Educational Researcher, 44(1), pp. 46–53.
Liao, Q.V. and Sundar, S.S. 2022. Designing for calibrated trust in AI, Human–Computer Interaction, 37(4), pp. 1–45.
Long, D. and Magerko, B. 2020. What is AI literacy? Proceedings of CHI 2020, pp. 1–16.
Medina, J. 2013. The epistemology of resistance. Oxford: Oxford University Press.
Miller, T. 2019. Explanation in artificial intelligence, Artificial Intelligence, 267, pp. 1–38.
Mosier, K.L. and Skitka, L.J. 2018. Human decision makers and automation, in Routledge Handbook of Human–Automation Interaction. New York: Routledge.
Nelson, T.O. and Narens, L. 1990. Metamemory, Psychology of Learning and Motivation, 26, pp. 125–173.
Noble, S.U. 2018. Algorithms of oppression. New York: NYU Press.
Norman, D.A. 2013. The design of everyday things. Revised ed. New York: Basic Books.
O’Neil, C. 2016. Weapons of math destruction. New York: Crown.
Parasuraman, R. and Riley, V. 1997. Humans and automation, Human Factors, 39(2), pp. 230–253.
Parasuraman, R., Sheridan, T.B. and Wickens, C.D. 2000. Levels of automation, IEEE Transactions on Systems, Man, and Cybernetics, 30(3), pp. 286–297.
Raisch, S. and Krakowski, S. 2021. Artificial intelligence and management, Academy of Management Review, 46(1), pp. 192–214.
Salomon, G. 1993. Distributed cognitions. Cambridge: Cambridge University Press.
Schraw, G. 1998. Promoting general metacognitive awareness, Instructional Science, 26, pp. 113–125.
Schraw, G. and Dennison, R.S. 1994. Assessing metacognitive awareness, Contemporary Educational Psychology, 19(4), pp. 460–475.
Skitka, L.J., Mosier, K. and Burdick, M. 2000. Automation bias, International Journal of Human–Computer Studies, 52(4), pp. 701–717.
Stanovich, K.E., West, R.F. and Toplak, M.E. 2016. The rationality quotient. Cambridge, MA: MIT Press.
Suchman, L. 2007. Human–machine reconfigurations. Cambridge: Cambridge University Press.
Van den Bosch, K. and Bronkhorst, A. 2018. Trust calibration, Cognitive Systems Research, 50, pp. 92–107.
Veenman, M.V.J. 2016. Metacognition in science education, International Journal of Science Education, 38(4), pp. 1–17.
Williams, J.J., Lombrozo, T. and Rehder, B. 2022. The illusion of explanatory depth, Cognitive Science, 46(4), e13125.
Xie, Y., Liu, X. and Bradshaw, A. 2024. Generative AI and argument quality, Educational Technology Research and Development, 72, pp. 1–24.
Zimmerman, B.J. 2002. Becoming a self-regulated learner, Theory Into Practice, 41(2), pp. 64–70.
Zuboff, S. 2019. The age of surveillance capitalism. London: Profile Books.
This report has been published by the Inclusive Society Institute
The Inclusive Society Institute (ISI) is an autonomous and independent institution that functions independently from any other entity. It is founded for the purpose of supporting and further deepening multi-party democracy. The ISI’s work is motivated by its desire to achieve non-racialism, non-sexism, social justice and cohesion, economic development and equality in South Africa, through a value system that embodies the social and national democratic principles associated with a developmental state. It recognises that a well-functioning democracy requires well-functioning political formations that are suitably equipped and capacitated. It further acknowledges that South Africa is inextricably linked to the ever transforming and interdependent global world, which necessitates international and multilateral cooperation. As such, the ISI also seeks to achieve its ideals at a global level through cooperation with like-minded parties and organs of civil society who share its basic values. In South Africa, ISI’s ideological positioning is aligned with that of the current ruling party and others in broader society with similar ideals.
Email: info@inclusivesociety.org.za
Phone: +27 (0) 21 201 1589
Comments