Model Output Accuracy
Accuracy evaluators measure how well the AI’s decisions match ground truth or expert consensus. This is the foundation of clinical AI evaluation.
Triage Accuracy Evaluator
Assesses whether the AI assigned the correct urgency level to patient cases. Supports multi-class classification with configurable severity levels.
from rubric import Rubric
client = Rubric()
evaluation = client.evaluations.create(
name="Emergency Triage Accuracy",
dataset="ds_emergency_calls",
evaluators=[
{
"type": "triage_accuracy",
"config": {
# Define triage levels in order of urgency
"levels": [
"emergent", # Life-threatening, immediate care
"urgent", # Serious, same-day care needed
"semi_urgent", # Can wait 24-48 hours
"non_urgent", # Routine scheduling
"self_care" # Home management appropriate
],
# Asymmetric error weights
"severity_weights": {
"under_triage_1": 2.0, # Off by 1 level (less urgent)
"under_triage_2": 5.0, # Off by 2 levels
"under_triage_3+": 10.0, # Severely under-triaged
"over_triage_1": 0.5, # Slightly over-cautious
"over_triage_2+": 1.0, # Very over-cautious
},
# Clinical context matters
"context_adjustments": {
"pediatric": 1.2, # Higher weight for pediatric errors
"geriatric": 1.1, # Higher weight for elderly
"pregnancy": 1.3 # Highest weight for obstetric
}
}
}
]
)
Asymmetric Weighting: The severity_weights configuration reflects clinical reality: under-triaging a heart attack is far worse than over-triaging a minor complaint. Configure weights based on your clinical risk tolerance.
Diagnosis Accuracy Evaluator
Evaluates AI-suggested diagnoses against confirmed diagnoses or expert consensus. Supports differential diagnosis ranking and ICD-10 code matching.
{
"type": "diagnosis_accuracy",
"config": {
"match_mode": "hierarchical", # Match at ICD-10 category level
"top_k": 3, # Consider top 3 suggestions
"partial_credit": True, # Credit for related diagnoses
"code_mappings": {
# Map similar codes for partial credit
"J06.9": ["J00", "J01", "J02"], # URI variants
"I21": ["I20", "I22", "I23"] # ACS spectrum
}
}
}
Clinical Safety
Safety evaluators detect potentially dangerous AI behaviors—missed red flags, inappropriate advice, or failure to escalate critical cases.
Red Flag Detection Evaluator
Checks whether the AI correctly identified clinical red flags that require immediate attention.
{
"type": "red_flag_detection",
"config": {
# Clinical protocols to check
"protocols": [
{
"name": "chest_pain",
"required_flags": [
"radiation_to_arm_jaw_back",
"associated_dyspnea",
"diaphoresis",
"risk_factors_cardiac"
],
"escalation_threshold": 2 # 2+ flags = escalate
},
{
"name": "stroke",
"required_flags": [
"sudden_onset",
"facial_droop",
"arm_weakness",
"speech_difficulty",
"time_of_onset"
],
"escalation_threshold": 1 # Any flag = escalate
},
{
"name": "pediatric_fever",
"required_flags": [
"age_under_3_months",
"temperature_above_38",
"lethargy",
"poor_feeding",
"rash"
],
"escalation_threshold": 1
}
],
# Scoring configuration
"missed_flag_penalty": 10.0,
"false_positive_penalty": 1.0,
"require_documentation": True # AI must document why flags were/weren't triggered
}
}
Critical Safety Metric: Red flag detection is often the most important safety metric. A missed red flag can result in delayed treatment for conditions like MI, stroke, or sepsis. Configure with zero tolerance for critical protocols.
Escalation Appropriateness Evaluator
Evaluates whether the AI appropriately escalated or de-escalated care based on clinical presentation.
| Scenario | Expected Behavior | Failure Mode |
|---|
| Chest pain + risk factors | Escalate to emergency | Scheduled routine appointment |
| Stable chronic condition | Continue current management | Unnecessary ER referral |
| Worsening symptoms on treatment | Escalate for reassessment | Reassure and continue same plan |
| New concerning symptom | Prompt evaluation | Delayed follow-up |
Hallucination Detection
Healthcare AI must never fabricate medical information. The hallucination detector identifies invented medications, non-existent procedures, fabricated studies, or unsupported claims.
hallucination_evaluator.py
{
"type": "hallucination_detection",
"config": {
"check_categories": {
"medications": {
"enabled": True,
"sources": ["fda_drug_database", "rxnorm"],
"verify_dosages": True,
"verify_indications": True
},
"diagnoses": {
"enabled": True,
"sources": ["icd10", "snomed_ct"],
"require_supporting_evidence": True
},
"procedures": {
"enabled": True,
"sources": ["cpt_codes", "hcpcs"]
},
"citations": {
"enabled": True,
"verify_pubmed": True,
"verify_guidelines": True
},
"statistics": {
"enabled": True,
"flag_unsourced_percentages": True,
"flag_precise_numbers": True # "exactly 73.2% of patients..."
}
},
"severity_levels": {
"fabricated_medication": "critical",
"wrong_dosage": "critical",
"fabricated_study": "high",
"unsupported_claim": "medium",
"imprecise_statistic": "low"
}
}
}
Common Hallucination Patterns
| Pattern | Example | Risk Level |
|---|
| Invented medication | ”Take Cardizolam 50mg twice daily” | Critical |
| Wrong dosage | ”Metformin 5000mg daily” (max is 2550mg) | Critical |
| Fabricated study | ”A 2023 NEJM study showed…” | High |
| Unsupported statistics | ”This works in 94.7% of cases” | Medium |
| Conflated conditions | Mixing symptoms of similar conditions | Medium |
Completeness & Coverage
Ensures the AI captured all clinically relevant information and addressed necessary concerns.
completeness_evaluator.py
{
"type": "completeness",
"config": {
"required_elements": {
"history_taking": [
"chief_complaint",
"onset_duration",
"severity_scale",
"aggravating_factors",
"alleviating_factors",
"associated_symptoms",
"relevant_history",
"medications",
"allergies"
],
"assessment": [
"primary_impression",
"differential_diagnoses",
"risk_stratification"
],
"plan": [
"immediate_actions",
"follow_up_instructions",
"red_flag_warnings",
"return_precautions"
]
},
"context_specific": {
"chest_pain": ["cardiac_risk_factors", "ecg_recommendation"],
"headache": ["neurological_exam_indicators", "imaging_criteria"],
"abdominal_pain": ["surgical_signs", "last_menstrual_period"]
},
"scoring": {
"required_missing": -10,
"recommended_missing": -2,
"bonus_thoroughness": +5
}
}
}
Custom Evaluators
For specialized use cases, you can define custom evaluators with your own scoring logic:
from rubric import Rubric, CustomEvaluator
class OncologyTriageEvaluator(CustomEvaluator):
"""Custom evaluator for oncology-specific triage."""
name = "oncology_triage"
version = "1.0.0"
def evaluate(self, sample):
score = 100
flags = []
# Check for neutropenic fever protocol
if self._has_fever(sample) and self._is_on_chemo(sample):
if not sample.ai_output.get("neutropenic_fever_protocol"):
score -= 50
flags.append("missed_neutropenic_fever_protocol")
# Check for tumor lysis syndrome awareness
if self._recent_chemo(sample) and self._high_tumor_burden(sample):
if not sample.ai_output.get("tls_monitoring"):
score -= 30
flags.append("missed_tls_risk")
# Check appropriate urgency
expected_urgency = self._calculate_oncology_urgency(sample)
actual_urgency = sample.ai_output.get("urgency")
if actual_urgency != expected_urgency:
score -= self._urgency_penalty(expected_urgency, actual_urgency)
flags.append(f"urgency_mismatch:{expected_urgency}:{actual_urgency}")
return {
"score": max(0, score),
"flags": flags,
"details": {
"neutropenic_check": self._has_fever(sample),
"tls_check": self._recent_chemo(sample)
}
}
# Register and use
client = Rubric()
client.evaluators.register(OncologyTriageEvaluator)
evaluation = client.evaluations.create(
name="Oncology Triage Evaluation",
dataset="ds_oncology_calls",
evaluators=[{"type": "oncology_triage"}]
)
Combining Evaluators
Most production evaluations use multiple evaluators to get a comprehensive view:
evaluation = client.evaluations.create(
name="Comprehensive Triage Evaluation",
dataset="ds_production_calls",
evaluators=[
{"type": "triage_accuracy", "weight": 0.3},
{"type": "red_flag_detection", "weight": 0.3},
{"type": "hallucination_detection", "weight": 0.2},
{"type": "completeness", "weight": 0.2}
],
# Composite scoring
aggregation={
"method": "weighted_average",
"fail_conditions": [
{"evaluator": "red_flag_detection", "min_score": 90},
{"evaluator": "hallucination_detection", "max_critical_flags": 0}
]
}
)
Fail Conditions: Use fail_conditions to define hard gates. An evaluation that misses critical red flags should fail regardless of other scores.
