Quantifying metacognitive performance with the hmetad package in R

Kevin O’Neill

Department of Experimental Psychology
University College London

https://metacoglab.github.io/hmetad_workshop

What is metacognition?

Thinking about thinking

Nelson (1990)

Hierarchies of metacognition

Seow et al. (2021)

Type 1 and type 2 decisions

Type 1 and type 2 decisions


Type 1

Decision about the world

  • Is there a stimulus present?
  • Are the dots moving left or right?
  • Have I seen this image before?

Type 2

Decision about yourself

  • Was my decision correct?
  • How good am I at this task?
  • Have I been paying attention?

Clarke et al. (1959)

Quantifying type 1 decisions

To understand how well someone performs a task, why not just calculate:

  • % correct?
  • correlation(stimulus, response)?

Need to control for response bias!

Quantifying type 1 decisions

Sensitivity (\(d'\))

How much do responses distinguish between stimuli?

Response Bias (\(c\))

Is one response more likely overall?


Green and Swets (1966)

Quantifying type 2 decisions

Metacognitive sensitivity

How much does distinguish between correct and incorrect decisions?

Metacognitive Bias

How much evidence is required to respond with high confidence?


We would like the same dissociation as for type 1 decisions

But now we need to control for type 1 sensitivity and bias!

Fleming and Lau (2014)

Measures of
metacognitive
sensitivity

General approach


Stimulus = 0 Stimulus = 1
Response = 0 Correct Rejection Miss
Response = 1 False Alarm Hit
Accuracy = 0 Accuracy = 1
Confidence = 0 Correct Rejection 2 Miss 2
Confidence = 1 False Alarm 2 Hit 2


Each approach determines metacognitive sensitivity using these probabilities

Correlational approaches


accuracy = [1, 0, 1, 1, 0, 1, 0, ...]
confidence = [0, 0, 1, 1, 0, 1, 0, ...]

phi = cor(accuracy, confidence)

Fleming and Lau (2014)

Should I use correlational measures?

Pros

  • Very simple, can be used with any design
  • Can establish presence (vs. absence) of metacognition

Cons

  • Confounded by type 1 performance
  • Confounded by metacognitive bias
  • Requires careful controls to compare between conditions

Area under type 2 ROC (AUROC2)

Clarke et al. (1959)

Issues with AUROC2

Does not control for type 1 performance!

Should I use AUROC2?

Pros

  • Non-parametric, can be used with any design
  • Usually independent of metacognitive bias

Cons

  • Confounded by type 1 performance
  • Confounded by type 1 response bias
  • Requires controlling for performance (experimentally or statistically)

meta-\(d'\)

The meta-\(d'\) model

If we make assumptions about the distribution of evidence,
we can model the type 2 ROC!

Taxonomy of metacognitive measures

  • Metacognitive sensitivity: how much information one has about the stimulus when making confidence ratings (i.e., \(\textrm{meta-}d'\))
  • Metacognitive efficiency: metacognitive capacity, relative to type 1 performance (i.e., \(M = \frac{\textrm{meta-}d'}{d'}\))
  • Metacognitive bias: overall level of confidence (i.e., \(\textrm{meta-}\Delta\), based on \(\textrm{meta-}c_2\))

Model-fitting approach

Bayesian inference

Determine the model parameters (e.g., \(\textrm{meta-}d'\), \(\textrm{meta-}c_2\)) are most likely given the participant’s observed type 1 and type 2 responses

\[ \begin{align*} p(\theta \;\vert\; \mathcal{D}) \propto p(\mathcal{D} \;\vert\; \theta) p(\theta) \end{align*} \]

Posterior Likelihood Prior


In the hmetad package, we approximate this calculation using the probabilistic programming language Stan

Fleming (2017)

Should I use meta-\(d'\)/\(M\)-ratio?

Pros

  • Provides principled metric for metacognitive sensitivity
  • Takes into account both type 1 and type 2 response biases
  • Easy to control for performance (e.g. using \(M = \frac{\textrm{meta-}d'}{d'}\))
  • Allows hierarchical modeling of participants/stimuli

Cons

  • Currently only developed for 2-choice discrimination tasks
  • Equal-variance Gaussian assumptions may not hold for some tasks
  • Claims of invariance to type 1 criterion and type 2 bias have been questioned (Rausch et al. 2023; Xue et al. 2021)

Alternative approaches

There are a wide range of other kinds of models of confidence:

Helpful reviews (Rausch et al. 2023; Rahnev 2025)

Entry points into the literature

Over to you!

Open https://metacoglab.github.io/hmetad_workshop in your browser and follow along the instructions.

If you can’t get it installed/running, let me know! Otherwise, you can look at the pre-generated outputs.

Bonus slides

The meta-\(d'\) model

\[ \begin{align*} \mathbf{y}_s &\sim \text{Multinomial}(\pi_s) \\ \pi_s &= \textrm{metad_pmf}_s(d', c, \textrm{meta-}d', \textrm{meta-}c, \mathbf{\textbf{meta-}c_{2}^0}, \mathbf{\textbf{meta-}c_{2}^{1}}) \\ \textrm{meta-}d' &= \textrm{exp}(\textrm{log-}M) \; d' \\ \textrm{meta-}c &= \begin{cases} c & \textrm{ABSOLUTE} \\ \textrm{exp}(\textrm{log-}M) \; c & \textrm{RELATIVE} \\ \end{cases} \\ \mathbf{\textbf{meta-}c_{2}^0} &= \textrm{meta-}c - \text{cumulative_sum}(\text{exp}(\mathbf{\textbf{log-dmeta-}c_2^0})) \\ \mathbf{\textbf{meta-}c_{2}^1} &= \textrm{meta-}c + \text{cumulative_sum}(\text{exp}(\mathbf{\textbf{log-dmeta-}c_2^1})) \end{align*} \]

Advantages of hierarchical Bayesian modeling

  • Point estimates of meta-\(d'\) are noisy; frequentist estimates of hit/false alarm rates fail to account for uncertainty
  • Optimally combines information about within-participant and between-participant variability
  • Avoids padding (edge-correction) for zero-count cells
  • Straightforward testing of group-level and participant-level hypotheses
  • Can also model stimulus variability (i.e., crossed random effects)

Advantages of hierarchical Bayesian modeling

Fleming (2017)

Differences between hmetad and the Hmeta-d toolbox

hmetad

  • implemented in Stan
  • \(d'\) and \(c\) modeled
  • meta-\(c_2\) modeled using ordered transform
  • uses R formula syntax for arbitrary model specifications

Hmeta-d toolbox

  • implemented in JAGS
  • \(d'\) and \(c\) point estimates taken as data
  • meta-\(c_2\) modeled on identity scale
  • has separate files for common model specifications

References

Boundy-Singer, Zoe M, Corey M Ziemba, and Robbe LT Goris. 2023. “Confidence Reflects a Noisy Decision Reliability Estimate.” Nature Human Behaviour 7 (1): 142–54.
Clarke, Frank R, Theodore G Birdsall, and Wilson P Tanner Jr. 1959. “Two Types of ROC Curves and Definitions of Parameters.” The Journal of the Acoustical Society of America 31 (5): 629–30.
Desender, Kobe, Luc Vermeylen, and Tom Verguts. 2022. “Dynamic Influences on Static Measures of Metacognition.” Nature Communications 13 (1): 4208.
Fleming, Stephen M. 2017. “HMeta-d: Hierarchical Bayesian Estimation of Metacognitive Efficiency from Confidence Ratings.” Neuroscience of Consciousness 2017 (1): nix007.
Fleming, Stephen M. 2024. “Metacognition and Confidence: A Review and Synthesis.” Annual Review of Psychology 75 (1): 241–68.
Fleming, Stephen M, and Nathaniel D Daw. 2017. “Self-Evaluation of Decision-Making: A General Bayesian Framework for Metacognitive Computation.” Psychological Review 124 (1): 91.
Fleming, Stephen M, and Hakwan C Lau. 2014. “How to Measure Metacognition.” Frontiers in Human Neuroscience 8: 443.
Green, David Marvin., and John A. Swets. 1966. Signal Detection Theory and Psychophysics. Wiley.
Hellmann, Sebastian, Michael Zehetleitner, and Manuel Rausch. 2024. “Confidence Is Influenced by Evidence Accumulation Time in Dynamical Decision Models.” Computational Brain & Behavior 7 (3): 287–313.
Katyal, Sucharit, Quentin JM Huys, Raymond J Dolan, and Stephen M Fleming. 2025. “Distorted Learning from Local Metacognition Supports Transdiagnostic Underconfidence.” Nature Communications 16 (1): 1854.
Kozyra, Wiktoria, Kevin O’Neill, and Stephen M. Fleming. 2026. “Confidence Is Detection-Like in High-Dimensional Spaces.” Open Mind.
Mamassian, Pascal. 2016. “Visual Confidence.” Annual Review of Vision Science 2 (1): 459–81.
Mamassian, Pascal, and Vincent de Gardelle. 2025. “The Confidence-Noise Confidence-Boost (CNCB) Model of Confidence Rating Data.” PLOS Computational Biology 21 (4): e1012451.
Maniscalco, Brian, Brian Odegaard, Piercesare Grimaldi, et al. 2021. “Tuned Inhibition in Perceptual Decision-Making Circuits Can Explain Seemingly Suboptimal Confidence Behavior.” PLoS Computational Biology 17 (3): e1008779.
Navajas, Joaquin, Bahador Bahrami, and Peter E Latham. 2016. “Post-Decisional Accounts of Biases in Confidence.” Current Opinion in Behavioral Sciences 11: 55–60.
Nelson, Thomas O. 1990. Metamemory: A Theoretical Framework and New Findings. Edited by Gordon H. Bower. Vol. 26. Psychology of Learning and Motivation. Academic Press. https://doi.org/10.1016/S0079-7421(08)60053-5.
Peters, Megan AK. 2022. “Towards Characterizing the Canonical Computations Generating Phenomenal Experience.” Neuroscience & Biobehavioral Reviews 142: 104903.
Rahnev, Dobromir. 2021. “Visual Metacognition: Measures, Models, and Neural Correlates.” American Psychologist 76 (9): 1445.
Rahnev, Dobromir. 2025. “A Comprehensive Assessment of Current Methods for Measuring Metacognition.” Nature Communications 16 (1): 701.
Rahnev, Dobromir, Tarryn Balsdon, Lucie Charles, et al. 2022. “Consensus Goals in the Field of Visual Metacognition.” Perspectives on Psychological Science 17 (6): 1746–65.
Rausch, Manuel, Sebastian Hellmann, and Michael Zehetleitner. 2023. “Measures of Metacognitive Efficiency Across Cognitive Models of Decision Confidence.” Psychological Methods.
Rouault, Marion, Peter Dayan, and Stephen M Fleming. 2019. “Forming Global Estimates of Self-Performance from Local Confidence.” Nature Communications 10 (1): 1141.
Seow, Tricia XF, Marion Rouault, Claire M Gillan, and Stephen M Fleming. 2021. “How Local and Global Metacognition Shape Mental Health.” Biological Psychiatry 90 (7): 436–46.
Xue, Kai, Medha Shekhar, and Dobromir Rahnev. 2021. “Examining the Robustness of the Relationship Between Metacognitive Efficiency and Metacognitive Bias.” Consciousness and Cognition 95: 103196.