Benchmarks

Why we publish benchmarks

Panora Health AI is a wellness platform. Wellness products that touch health data should show their work. This page is the public record of how we measure Panora's output against published reference standards, what we pre-register before each run, and what we deliberately defer to a future external evaluation.

Panora is a General Wellness Product as defined by FDA guidance. It is not a medical device, has not been independently certified by a medical board, and does not claim physician-level accuracy. The numbers below describe how Panora's outputs compare to physician-annotated reference labels in third-party datasets — not whether Panora is safe or effective for any specific clinical use.

Headline result

Our internal benchmark on HealthBench Consensus (a 34-dimensional conversational-rubric set authored by 262 physicians across 60 countries and released by OpenAI, Arora et al. 2025) scored 92.3% mean (Wilson 95% confidence interval 88.2–95.8%) on n=100 dev-split cases. This is an internal development benchmark — not a blind, independent, or production-performance result.

Panora's outputs were evaluated by an LLM judge against the Consensus rubric labels — the reference labels were produced by the dataset's physician panel and cross-validated among them; reported physician-physician agreement is 55–75% (an explicit Cohen’s κ was not published by the authors). Panora's outputs were not independently re-annotated by physicians. Evaluation-side physician adjudication is planned for Q3 2026 pending medical advisory board formation.

Source citation

• Arora, R. K. et al. HealthBench: Evaluating Large Language Models Toward Improved Human Health. arXiv preprint arXiv:2505.08775 (2025).
• License: MIT. Dataset published by OpenAI.
• Subset evaluated: HealthBench Consensus (n=100 cases drawn from the cross-validated Consensus subset).
• Reliability statistic cited from authors: 55–75% physician-physician agreement (Section 4.2 of the source paper).

Methodology

Three-way split

Every benchmark dataset Panora uses is partitioned into three splits, enforced at the database layer:

• Dev(1,000–5,000 cases) — used for iterating on rules, prompts, and agents. Per-case inspection allowed.
• Validation(200 cases) — aggregate accuracy only. Per-case inspection is disabled in the runner so we cannot teach the system to specific failures.
• Blind(500 cases) — pre-registered, sealed, and read once per calendar quarter. Database trigger enforces a unique constraint per quarter so a blind run cannot be repeated within the quarter.

Pre-registration

Before any blind run, a pre-registration file is committed to the repository specifying the dataset, exact n, the SHA-256 of the sorted case identifiers, the scoring rubric, the model name and version, the grader version SHA, the expected confidence-interval width at the anticipated accuracy, the budget cap, and the ablation pipelines that will be run on the same cases. The result file references the pre-registration SHA so any post-hoc edit is detectable in git history.

The most recent run's full results, methodology, and integrity disclosures are documented internally and summarized on this page.

Annotation κ vs evaluation κ

Two reliability statistics often get conflated in medical-AI marketing. Panora separates them strictly:

• Annotation κ measures how well the reference-label producers agreed among themselves on the ground-truth answers. We citethe annotation reliability that source datasets publish (or note when authors did not publish an explicit Cohen’s κ).
• Evaluation κwould measure how well credentialed physicians agree that Panora's outputs match the reference labels. Panora has not produced this yet. Until a credentialed medical advisory board is seated (target Q3 2026), Panora reports evaluation_reliability_method: llm_judge_only.

Ablation reporting

Each blind run reports the same cases through multiple pipelines (rules-only baseline, single-agent, multi-agent, +memory, +consensus). Components that degrade accuracy versus the prior baseline are not shipped to production, regardless of in-development performance.

Multi-dataset sanity check

Any advertised number is paired with a second number from a different dataset on the same pipeline in the same quarter. If the gap exceeds 10 percentage points, the discrepancy is resolved before publication.

Required disclosures with every published number

For each benchmark run, Panora records the following metadata. The headline result above lists the key figures (n, confidence interval, dataset, split, annotation authority, and evaluation method); the complete set is documented internally with the run and available on request:

• Exact n
• Wilson 95% confidence interval (lower and upper)
• Dataset name, split, and cycle
• Model name, version, and release date
• Benchmark date and pre-registration file SHA
• Grader and clinical-rules git SHAs
• Ablation breakdown
• Annotation authority (third-party physician panel, government curation, single-editor curation, internal, or not applicable)
• Annotation physician count (when applicable) and the source-published reliability statistic verbatim
• Evaluation reliability method ( llm_judge_only, llm_judge_with_physician_kappa, physician_only, or rules_only)

Open questions and what comes next

A board-certified medical advisory board is the prerequisite for evaluation-side adjudication. Once seated (target Q3 2026), the same n=100 HealthBench Consensus subset will be re-graded by physicians independent of Panora and a published Cohen’s κ will be reported.

Panora is also preparing a separate public benchmark repository (MIT license) with the harness, configurations, and result artifacts so the methodology is reproducible outside Panora's codebase.

Questions or peer-review feedback are welcome at admin@panorahealth.ai.