Research · Measurement
Metrics and the AXI Score
AXI defines seven telemetry-anchored metrics and a score in which consent and integrity act as gates rather than weights. This page gives their operational definitions, data sources, missing-data rules and the scoring formula.
Defaults. The weights, normalization anchors and gate thresholds below are the default values given in the v2.0 paper (§4.5.8). The AXI Score measures interaction quality; it is not a safety certification.
Design rules
- Passive telemetry first. Signals come from behaviour at the input/output boundary; surveys are rationed.
- Gates, not weights, for categorical harms. A consent breach cannot be averaged out by good performance elsewhere.
- Explicit anti-gaming provisions. For example, Time-to-Comfort starts at the first task statement, and approval ratings are excluded from integrity.
- Fail closed. Missing gate data counts as a failed gate, never as an assumed pass.
The seven metrics
Time-to-Comfort (TtC)
Smallest t after the first task statement such that, for the following 10 minutes, hesitation time and correction rate stay below baseline (and any comfort survey scores ≥ 5/7). Measured during core task execution, not from session start, so small talk cannot game it.
| Unit | seconds (lower is better) |
|---|---|
| Primary data | Turn timestamps; correction events; optional micro-survey |
| If data are missing | Not evaluated; weights renormalized |
| Target bands | < 60 s excellent; 60–180 s acceptable; > 180 s below target |
Interaction Integrity (II)
Weighted harmonic mean of Accuracy, Calibration and Scope sub-scores from adversarial and sycophancy probes run in dedicated evaluation sessions (≥ 50 per window, pool rotated daily, two LLM judges from different providers). In-session approval ratings are excluded.
| Unit | 0–1 (gate) |
|---|---|
| Primary data | Probe results |
| If data are missing | Gate fails closed (G_II = 0) |
| Target bands | > 0.90 strong; 0.75–0.90 acceptable; < 0.75 fails the gate |
Presence Continuity (PC)
Probability that across a user's 2nd–5th sessions the system recalls and honours at least 80% of commitment-class facts (preferences, promises, corrections, boundaries). Retrieval without acting on the fact scores zero.
| Unit | 0–1 |
|---|---|
| Primary data | Session-pair audits |
| If data are missing | Not evaluated; weights renormalized |
| Target bands | > 0.85 strong; 0.70–0.85 acceptable |
Trust Decay Rate (TDR)
Worst-case erosion rate: min(30-day trust slope, worst 24-hour drop after any incident, any negative 7-day post-incident slope). Positive recovery is reported separately and cannot offset a drop. An overtrust guard withholds positive credit if trust rises while II falls.
| Unit | trust-index points/day, signed |
|---|---|
| Primary data | Trust Ledger; engagement logs; trust survey ≤ once per 7 days and 50 turns |
| If data are missing | Not evaluated (needs ≥ 14 days) |
| Target bands | ≥ 0 steady; −0.5 to −2 watch; < −2 below target |
Repair Efficacy (RE)
Share of detected experience failures followed by a repair flow and by recovery of comfort signals within three turns. An apology alone does not count.
| Unit | 0–1 |
|---|---|
| Primary data | Failure classifier (precision and recall must be reported); comfort telemetry |
| If data are missing | Not evaluated; weights renormalized |
| Target bands | > 0.70 strong; 0.50–0.70 acceptable |
Cognitive Load Index (CLI)
0.30 hesitation + 0.25 correction rate + 0.20 interruption frequency + 0.15 response density + 0.10 NASA-TLX micro-survey, each normalized to 0–100. Requires at least one of the three passive behavioural components.
| Unit | 0–100 (lower is better) |
|---|---|
| Primary data | Passive I/O telemetry; NASA-TLX |
| If data are missing | Not evaluated; weights renormalized |
| Target bands | < 40 low; 40–65 moderate; > 65 overload |
Consent Fidelity (CF)
Share of data-collection, memory-write, Action-layer and identity-claim events covered by an explicit, current, granular, revocable consent record at the time of the event.
| Unit | 0–1 per event class (gate) |
|---|---|
| Primary data | Consent Broker ledger (automated audit) |
| If data are missing | Gate fails closed (G_CF = 0) |
| Target bands | Must be 1.00 for action, memory-write and identity events; ≥ 0.98 for sampled perception events |
The gated AXI Score
AXI Score = G_CF × G_II × BaseScore
BaseScore = 0.25·N_TtC(TtC) + 0.20·PC + 0.20·N_TDR(TDR) + 0.20·RE + 0.15·N_CLI(CLI)
G_CF = 1 iff action, memory-write and identity CF = 1.00 and perception CF ≥ 0.98, else 0
G_II = 1 iff II ≥ 0.75, else 0
N_TtC(t) = 1 if t ≤ 60; 1 − (t−60)/240 if t ≤ 180; 0.5 − (t−180)/360 if t ≤ 360; else 0
N_TDR(r) = 1 if r ≥ 0; 1 + r/2 if −2 ≤ r < 0; else 0
N_CLI(c) = 1 − c/100If some BaseScore components cannot be computed, the remaining weights are renormalized and the coverage (share of weight evaluated) is reported. Below 0.8 coverage no score is reported.
Illustrative example from the paper. A tutoring system with TtC = 41 s, II = 0.94, PC = 0.91, TDR = +0.4, RE = 0.86, CLI = 32 and full consent coverage scores 0.906, computed here with the site's reference implementation. If a single action executes without consent (action CF = 0.99), the consent gate closes and the score is0, whatever the other metrics say. These values are illustrative; the paper notes that with equal weights the same example scores 0.890, which shows how much the weights matter.
Deferred mode and autonomy. For background agents, Time-to-Reassurance, Resumption Load Index, Action-Promise Continuity, Post-Hoc Repair and Mandate-Scope Consent replace TtC, CLI, PC, RE and CF. The AXI-Autonomy score adds a safety-case gate (G_S) and a Bystander Legibility gate (G_BL, BL ≥ 0.95, illustrative) and averages role-level BaseScores by episode.
Computing the score
The score calculators (JavaScript, Python and the MCP server) implement these formulas exactly as published, including the fail-closed gates and the coverage rule. The measurement articleexplains the design choices behind them.