Who Reviews Our expert network includes licensed healthcare professionals across specialties, each verified and matched to appropriate review tasks.
Physicians Board-certified MDs and DOs across specialties including emergency medicine, internal medicine, pediatrics, and more.
Nurses RNs and APRNs with clinical experience in triage, telehealth, and specialty care.
Medical Coders Certified coders (CPC, CCS) for documentation accuracy and coding validation.
Allied Health Dietitians, mental health counselors, pharmacists, and other specialists.
Reviewer Categories Category Credentials Use Cases Physicians MD, DO (Board Certified) Triage validation, diagnosis review, treatment recommendations, safety-critical cases Nurses RN, BSN, MSN, NP Triage screening, symptom assessment, patient education review Medical Coders CPC, CCS, RHIA, RHIT ICD-10 validation, CPT accuracy, documentation completeness Dietitians RD, RDN, LD Nutrition advice, dietary recommendations, meal planning AI Mental Health LCSW, LPC, Psychologist Crisis assessment, therapy recommendations, mental health triage Pharmacists PharmD, RPh Medication interactions, dosing validation, drug information
Credentialing & Verification Every reviewer in our network undergoes rigorous credentialing before accessing any review tasks.
Verification Process 1. Application & Documentation ├── License verification (primary source) ├── Board certification check ├── Education verification └── Background check 2. Skills Assessment ├── Clinical knowledge test ├── Calibration case review └── Platform training 3. Ongoing Monitoring ├── License status monitoring ├── Performance tracking └── Annual re-credentialing
Primary Source Verification : All licenses are verified directly with state medical boards and credentialing bodies—never relying solely on self-reported information.
Credential Tracking from rubric import Rubric client = Rubric() # View reviewer credentials for your evaluation reviewers = client.reviewers.list( evaluation_id = "eval_abc123" , include_credentials = True ) for reviewer in reviewers: print ( f """ Reviewer: { reviewer.id } Type: { reviewer.credential_type } # e.g., "physician" Specialty: { reviewer.specialty } License State: { reviewer.license_state } License Status: { reviewer.license_status } License Verified: { reviewer.license_verified_at } Board Certified: { reviewer.board_certified } Platform Stats: Reviews Completed: { reviewer.total_reviews } Agreement Rate: { reviewer.agreement_rate :.1%} Avg Review Time: { reviewer.avg_review_time_seconds :.0f} s """ )
Reviewer Assignment Logic Cases are matched to reviewers based on clinical requirements, expertise, and availability.
assignment_rules = { "matching_criteria" : { # Required credentials "credential_requirements" : { "emergency_triage" : [ "physician" , "emergency_nurse" ], "routine_triage" : [ "nurse" , "physician" ], "mental_health" : [ "mental_health_professional" ], "pediatric" : [ "pediatric_specialist" , "pediatric_nurse" ], "coding_review" : [ "certified_coder" ] }, # Specialty matching "specialty_matching" : { "cardiology_cases" : [ "cardiology" , "internal_medicine" , "emergency" ], "oncology_cases" : [ "oncology" , "hematology" ], "obstetric_cases" : [ "obstetrics" , "maternal_fetal" ] }, # Experience requirements "experience_thresholds" : { "safety_critical" : { "min_reviews" : 100 , "min_accuracy" : 0.95 }, "training_data" : { "min_reviews" : 50 , "min_accuracy" : 0.90 }, "standard" : { "min_reviews" : 10 , "min_accuracy" : 0.85 } } }, "load_balancing" : { "max_concurrent_assignments" : 10 , "max_daily_reviews" : 50 , "fatigue_cooldown_hours" : 8 , "specialty_rotation" : True # Prevent monotony }, "priority_factors" : { "expertise_match" : 0.4 , "availability" : 0.3 , "historical_accuracy" : 0.2 , "response_time" : 0.1 } }
Assignment API # Configure custom assignment rules client.evaluations.configure_assignment( evaluation_id = "eval_abc123" , assignment = { "mode" : "auto" , # auto, manual, hybrid # Require specific credentials "require_credentials" : [ "physician" ], # Prefer certain specialties "prefer_specialties" : [ "emergency_medicine" , "internal_medicine" ], # Exclude reviewers with conflicts "exclude_organizations" : [ "competitor_health_system" ], # Geographic requirements (for state-specific regulations) "require_state_license" : [ "CA" , "NY" , "TX" ] } )
Conflict-of-Interest Controls Maintaining objectivity requires systematic conflict-of-interest management.
Control Implementation Organizational separation Reviewers cannot evaluate AI from their employer Competitive exclusion Option to exclude reviewers from competitor organizations Random assignment Reviewers cannot select specific cases Blinding AI source/version hidden from reviewers when appropriate Financial disclosure Annual disclosure of relevant financial interests Rotating assignments Prevent patterns that could enable gaming
# Configure conflict-of-interest rules client.organizations.configure_coi( # Organizations that cannot review your AI excluded_organizations = [ "my_company" , # Internal employees "competitor_a" , "competitor_b" ], # Require conflict disclosure require_disclosure = True , # Blind reviewers to AI source blind_ai_source = True , # Prevent same reviewer from seeing related cases prevent_case_clustering = True )
Quality Assurance & Calibration Calibration Program All reviewers complete initial calibration and ongoing recalibration to ensure consistency.
calibration_config = { "initial_calibration" : { "required" : True , "gold_standard_cases" : 20 , "passing_score" : 0.85 , "feedback_provided" : True , "unlimited_attempts" : False , "max_attempts" : 3 }, "ongoing_calibration" : { "frequency" : "monthly" , "cases_per_session" : 10 , "passing_score" : 0.80 , "remediation_threshold" : 0.70 , "suspension_threshold" : 0.60 }, "gold_standard_insertion" : { "enabled" : True , "rate" : 0.05 , # 5% of cases are gold standards "flag_poor_performance" : True }, "inter_rater_reliability" : { "measure" : "cohens_kappa" , "target" : 0.8 , "alert_threshold" : 0.6 } }
# Monitor reviewer performance qa_dashboard = client.reviewers.quality_dashboard() print ( f """ Network Quality Metrics ======================= Active Reviewers: { qa_dashboard.active_reviewers } Avg Inter-rater Reliability (κ): { qa_dashboard.avg_kappa :.2f} Avg Agreement Rate: { qa_dashboard.avg_agreement :.1%} Performance Distribution: Excellent (>95%): { qa_dashboard.excellent_count } reviewers Good (85-95%): { qa_dashboard.good_count } reviewers Needs Improvement (70-85%): { qa_dashboard.needs_improvement_count } reviewers Under Review (<70%): { qa_dashboard.under_review_count } reviewers Recent Alerts: """ ) for alert in qa_dashboard.recent_alerts: print ( f " ⚠️ { alert.reviewer_id } : { alert.reason } " )
Reviewer Specialties Emergency Medicine Board-certified emergency physicians and emergency nurses for high-acuity triage review.
Credential Scope Emergency Medicine (ABEM) All emergency triage, trauma, resuscitation decisions Emergency Nurse (CEN) Triage screening, ESI level validation Pediatric Emergency (PEM) Pediatric emergency cases
Mental Health Licensed mental health professionals for crisis assessment and therapy AI review.
Credential Scope Psychiatrist (MD) Crisis assessment, medication recommendations Psychologist (PhD/PsyD) Therapy recommendations, assessment interpretation Licensed Clinical Social Worker Crisis screening, resource recommendations Licensed Professional Counselor General mental health triage
Coding & Documentation Certified coders for clinical documentation and coding accuracy review.
Credential Scope CPC (Certified Professional Coder) E&M coding, procedure codes CCS (Certified Coding Specialist) Inpatient coding, DRG validation RHIA/RHIT Documentation completeness, health information
Bring Your Own Reviewers You can also use your own clinical staff as reviewers within Rubric.
# Register internal reviewers internal_reviewer = client.reviewers.register( email = "[email protected] " , credentials = { "type" : "physician" , "specialty" : "internal_medicine" , "license_number" : "A123456" , "license_state" : "CA" , "npi" : "1234567890" }, # Organization context organization = "my_org" , internal = True , # Skip external verification (you vouch for them) skip_verification = True , # Restrict to your evaluations only restrict_to_org = True ) # Create internal-only evaluation evaluation = client.evaluations.create( name = "Internal QA Review" , dataset = "ds_production" , evaluators = [ ... ], human_review = { "enabled" : True , "reviewer_source" : "internal_only" , # Only use your reviewers "reviewer_pool" : "my_org" } )
Internal Reviewer Responsibility : When using internal reviewers with skip_verification, your organization is responsible for verifying credentials and maintaining compliance.
