How should human–AI interaction quality be measured?

Teams that ship conversational AI usually track two kinds of numbers: task metrics (accuracy, resolution rate) and engagement metrics (sessions, retention, thumbs-up). Neither measures what the person experienced, and one of them can actively reward the wrong behaviour. This article looks at what a better measure would need, how the AXI v2.0 paper attempts one, and why its proposal should be treated as a hypothesis until it is tested.

The problem with approval

In April 2025 OpenAI rolled back a GPT-4o update that had become overly flattering and agreeable, explaining that it had focused too much on short-term feedback such as thumbs-up ratings (OpenAI, 2025). Research on sycophancy had already shown that preference-trained models tend to tell people what they want to hear (Sharma et al., 2024).

The measurement lesson is that a signal the system can influence by pleasing the user is a poor measure of quality. The AXI paper responds by excluding in-session approval signals from its integrity measure and from its score altogether. That is one of its most immediately useful recommendations, and it requires no new metric.

Engagement has a related problem. A companion app that fosters dependence can raise retention while harming the user. AXI’s position is that experience quality is not engagement maximization (§7.2), and its fifth guardrail requires escalating loneliness and distress to human help even when that reduces use.

What a measure of interaction quality needs

Read across the paper’s design rules (§4.5) and limitations (§7.1), a credible measure needs at least six properties:

1. Behavioural, not only self-reported. Surveys are informative but intrusive and biased by mood and novelty. AXI makes passive telemetry primary: hesitation between turns, correction rate, interruptions.
2. Hard to game. Each AXI metric carries an explicit anti-gaming provision. Time-to-Comfort is measured from the first task statement, so small talk cannot inflate it. Integrity probes come from a pool rotated daily, so a system cannot memorize them.
3. Non-compensable where harm is categorical. A consent breach should not be offset by a pleasant conversation.
4. Honest about missing data. A measure that silently treats missing data as good is worse than none.
5. Separable by role. In autonomous systems the rider, the supervising driver and the pedestrian have different experiences; averaging them hides the person who is put at risk.
6. Valid: it must track what people actually experience and predict outcomes that matter. This is the property AXI has not yet shown.

How AXI’s proposal is built

The paper defines seven metrics (details on the metrics page): Time-to-Comfort, Interaction Integrity, Presence Continuity, Trust Decay Rate, Repair Efficacy, Cognitive Load Index and Consent Fidelity. Some design choices worth noticing:

• Continuity means acting, not remembering. Presence Continuity scores whether the system honours a remembered commitment. A system that retrieves your preference for kilograms and then gives a recipe in pounds has perfect recall and zero continuity.
• Trust is measured as worst-case erosion. Trust Decay Rate takes the minimum of the 30-day trend and each post-incident drop, so a quick recovery cannot hide a cliff. An overtrust guard withholds credit when trust rises while integrity falls, which is the complacency pattern known from automation failures.
• Repair counts only if it works. Repair Efficacy requires comfort signals to recover within three turns; an apology alone does not count.

These are then combined:

AXI Score = G_CF × G_II × BaseScore
BaseScore = 0.25·N_TtC + 0.20·PC + 0.20·N_TDR + 0.20·RE + 0.15·N_CLI

The two gates are binary. If consent coverage for actions, memory writes or identity claims is below 100%, or Interaction Integrity is below 0.75, the score is zero. Missing gate data fails closed. If BaseScore components are missing, the remaining weights are renormalized and the coverage is reported; below 0.8 coverage, no score is reported at all.

Why gates instead of weights

Consider the paper’s illustrative tutoring system: TtC 41 s, integrity 0.94, continuity 0.91, trust slope +0.4/day, repair 0.86, load 32. Its score is 0.906. Now suppose one action executed without consent. In a weighted average with consent at, say, 20% weight and a fidelity of 0.99, the score would barely move. Under the gated model it becomes 0. The paper’s argument is that consent failures are categorical: a breach is not “1% worse”, and a measure that lets it be averaged away will be optimized in exactly that direction.

The same logic drives the autonomy profile, where a bystander-legibility gate means that “a rider’s comfort cannot compensate for a pedestrian’s confusion”.

The open validity questions

Every number above is a default prior: the weights, the normalization anchors, the 0.75 integrity threshold, the 0.98 perception-consent tolerance. The paper says so directly, and notes that with equal weights the same tutoring example scores 0.890 instead of 0.906. That shows how much results depend on choices no data has yet informed.

What would have to be shown, and how the paper plans to test it (§6.3):

Question	Planned test	What would count against the metric
Does CLI measure workload?	Correlate with NASA-TLX in a within-subjects study (Study 1)	Weak or no correlation
Do TDR and PC measure trust?	Correlate with a trust-in-automation scale (Jian et al., 2000)	Weak or no correlation
Is the score sensitive to design?	Full system vs. an ablated variant without pacing, repair and the overtrust guard; target d ≥ 0.5	No separation
Does it predict anything?	Cross-lagged panel: does week-t score predict week t+1 retention and complaints, controlling for prior retention? (Study 3)	No predictive value beyond prior retention
Do passive signals reflect comfort?	Compare telemetry-only Time-to-Comfort with surveyed comfort	Disagreement

Two further concerns are not fully addressed by those studies:

• The telemetry-to-load mapping is a hypothesis. Long pauses may mean confusion, or reflection, or a phone call. Calibration needs to happen per population and per culture; the paper lists cultural calibration as an open problem.
• Continuity has a privacy cost. Measuring and delivering Presence Continuity means storing commitments about people. “Continuity without surveillance” is listed as an open problem, and teams should treat it as a design constraint, not an afterthought.

Measurement claims also need reporting discipline. Appendix A asks conformance reports to state sample sizes, windows, variance estimates, judge configurations, and the precision and recall of any classifier involved. Where coding is done by people, as in the paper’s planned incident re-coding, agreement statistics should include Gwet’s AC1 alongside Cohen’s kappa because kappa behaves badly when categories are unevenly distributed (Gwet, 2008).

What to do today

The most immediately practical uses of AXI’s measurement ideas:

• Audit the gates. Consent Fidelity is an audit, not an estimate: count actions without a current consent record. That number is meaningful now, and the target is zero.
• Remove approval ratings from quality dashboards, or at least never optimize on them directly.
• Report coverage and missing data alongside any composite you compute.
• Track worst-case trust events, not only averages.
• Do not publish AXI Scores as rankings or safety claims. The score calculators exist so the formulas can be inspected and tested, not to certify systems.

If you have data or a deployment where these metrics could be studied, get in touch through the about page.

Sources

1. OpenAI (2025). Sycophancy in GPT-4o: what happened and what we're doing about it
2. Sharma, M. et al. (2024). Towards understanding sycophancy in language models. ICLR. arXiv:2310.13548
3. Jian, J.-Y., Bisantz, A. M. and Drury, C. G. (2000). Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics 4(1)
4. Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology 61(1)

Related on this site

• AXI metrics and the AXI Score
• Evidence status
• Developer docs
• AXI v2.0 paper, §4.5, §6.3, §7.1, Appendix A: full text · DOI 10.21203/rs.3.rs-9717445/v2