Conefia
SAFE-CARE Bench · Evaluations on request

Know which layer makes your healthcare AI agent safer

SAFE-CARE Bench: multi-turn safety evaluation for healthcare AI agents, across five agent tiers from A0 ungoverned to A4 full agentic.
Read the method on GitHub
Open method
10 scored dimensions
Hard fail on missed escalation

Cite with DOI 10.5281/zenodo.21444597

Why

Why one score isn’t enough

Most evaluations give one average score for one system. That can’t tell you whether your guardrails, your retrieval or your prompt did the work. An average can also hide the one answer that matters most: a red flag that was missed.

SAFE-CARE Bench holds the model, prompt and seed fixed and adds one capability bundle per tier. The finding is the difference between neighboring tiers, so you see what each layer bought you in safety and what it cost.

How it works

From scenario to release decision

From scenario to release decision: scenarios, five agent tiers, transcripts, a calibrated judge on ten dimensions, per-tier deltas, then hard-fail conditions and the release gate.
Scenarios
synthetic patients in multi-turn conversations, with the expected outcome hidden from the agent.
Tiers
the same model and seed, with one layer added at each step.
Release gate
hard-fail conditions are checked before any average, and a named owner records the decision.
Scoring

Ten dimensions, one veto

Ten dimensions, one veto: seven scored 1 to 5, three scored 0 or 1, with safety and escalation judged on one turn and across the conversation. A zero on escalation is disqualifying.

Safety and escalation are judged twice, on a single turn and across the whole conversation, because the typical failure of a health agent isn’t one bad answer. It’s a risk that builds over several turns while each reply looks fine.

Deliverable

What you get back

A written report with seven fixed sections, reviewed by our lead before it is sent.

  1. 01A summary of what was evaluated
  2. 02Scores by tier, with their spread
  3. 03The difference each added capability makes
  4. 04Missed escalations, with the scenarios
  5. 05Calibration status
  6. 06Exactly what was run, including the corpus digest and the check that it matched your documents
  7. 07The limitations that apply
Scope

What we evaluate, and what you send

What we evaluate

  • Our published scenario set (menopause and perimenopause) on agent tiers A0 to A2, with your documents as the retrieval corpus.
  • Other clinical areas, scoped separately before any work starts.

What you send

  • Public or de-identified documents only: guidelines, patient education material, FAQs.
  • No protected health information and no real patient conversations.
Process

How a request works

  1. 1You send the request below. We reply to confirm scope, timing and any cost before we start.
  2. 2We load your documents as a declared corpus, and an integrity check confirms that what we run against is exactly what you sent.
  3. 3We run and score the evaluation, then review the report.
  4. 4We send you the report.
  5. 5We delete your documents after the run. We keep the run record, the scores and the report.

What it is not: a clinical validation or a certification. Our judge isn’t yet calibrated against clinicians for this domain, so scores are indicative. A clinician calibration study is under way.

Request

Request an evaluation

Tell us what you want to learn and what you would send. We reply within two business days to confirm scope and timing before anything starts.

We collect documents later, after we confirm scope. Nothing is uploaded on this page.

SAFE-CARE Bench is open: method, scenarios, judge prompts and published runs on GitHub. Cite it with DOI 10.5281/zenodo.21444597.