Regulatory Requirements Overview
Healthcare AI systems must demonstrate safety and efficacy through documented testing. Different regulatory frameworks have specific requirements:
| Framework | Applies To | Key Requirements |
|---|
| FDA 510(k) / De Novo | Software as Medical Device (SaMD) | Clinical validation, performance testing, risk analysis |
| FDA AI/ML Guidance | Adaptive AI systems | Change management, ongoing monitoring, predetermined change protocol |
| HIPAA | Any PHI handling | Audit logs, access controls, encryption |
| SOC 2 Type II | Enterprise SaaS | Security controls, availability, processing integrity |
| CE Marking (MDR) | EU medical devices | Clinical evaluation, post-market surveillance |
| ISO 13485 | Medical device QMS | Design controls, risk management, traceability |
from rubric import Rubric
client = Rubric(api_key="your-api-key")
# Configure project for regulatory compliance
client.projects.update(
"patient-triage",
compliance_config={
# Regulatory frameworks you're targeting
"frameworks": ["fda_510k", "hipaa", "soc2"],
# Device classification (for FDA)
"device_classification": {
"product_code": "QAS", # Clinical decision support
"device_class": "II",
"intended_use": "AI-assisted patient triage for non-emergency symptoms"
},
# Risk management (ISO 14971)
"risk_classification": {
"severity": "serious", # Could delay necessary care
"probability": "remote",
"risk_level": "moderate"
},
# Data retention requirements
"retention": {
"evaluation_results": "7_years",
"audit_logs": "7_years",
"model_artifacts": "lifetime_of_device"
},
# Required signatories for exports
"authorized_signatories": [
{"name": "Dr. Jane Smith", "title": "Chief Medical Officer", "role": "clinical_validation"},
{"name": "John Doe", "title": "VP Engineering", "role": "technical_validation"},
{"name": "Sarah Johnson", "title": "Head of Compliance", "role": "regulatory_review"}
]
}
)
Step 2: Generate FDA Submission Package
Export a complete validation package for FDA 510(k) or De Novo submission:
# Generate FDA submission package
fda_package = client.exports.create_fda_package(
project="patient-triage",
# Evaluation runs to include
evaluations=[
"eval_clinical_validation_v2",
"eval_safety_testing_final",
"eval_edge_case_analysis"
],
# Package configuration
config={
"submission_type": "510k", # or "de_novo", "pma"
# Performance summary
"performance_summary": {
"primary_endpoint": "triage_accuracy",
"secondary_endpoints": ["safety_score", "sensitivity_by_condition"],
"include_confidence_intervals": True,
"confidence_level": 0.95
},
# Test dataset documentation
"dataset_documentation": {
"include_demographics": True,
"include_source_description": True,
"include_sampling_methodology": True,
"include_ground_truth_methodology": True
},
# Clinical validation
"clinical_validation": {
"include_human_review_results": True,
"include_reviewer_credentials": True,
"include_inter_rater_reliability": True
},
# Risk analysis
"risk_analysis": {
"include_failure_mode_analysis": True,
"include_hazard_assessment": True,
"include_mitigation_measures": True
}
}
)
print(f"Package ID: {fda_package.id}")
print(f"Status: {fda_package.status}")
# Download when ready
if fda_package.status == "complete":
client.exports.download(
fda_package.id,
destination="./fda_submission_package.zip"
)
FDA Package Contents: The FDA package includes: Executive Summary, Performance Testing Report, Clinical Validation Report, Risk Analysis (FMEA), Software Documentation, Labeling/Intended Use, and all supporting data files.
# Generate standalone performance report
performance_report = client.exports.create_performance_report(
project="patient-triage",
evaluation="eval_clinical_validation_v2",
config={
"format": "pdf", # or "docx", "html"
# Report sections
"sections": {
"executive_summary": True,
"methodology": True,
"dataset_description": True,
"results_summary": True,
"detailed_metrics": True,
"subgroup_analysis": True,
"failure_analysis": True,
"statistical_analysis": True,
"conclusions": True
},
# Metrics to include
"metrics": {
"primary": ["accuracy", "sensitivity", "specificity", "ppv", "npv"],
"by_condition": ["chest_pain", "stroke", "pediatric_fever", "psychiatric"],
"by_demographic": ["age_group", "sex"],
"confusion_matrix": True,
"roc_curves": True,
"calibration_plots": True
},
# Statistical rigor
"statistics": {
"confidence_intervals": True,
"confidence_level": 0.95,
"method": "wilson", # For proportions
"hypothesis_tests": ["mcnemar"], # vs baseline
"effect_sizes": True
},
# Failure analysis
"failure_analysis": {
"include_examples": True,
"max_examples": 10,
"redact_phi": True,
"categorize_by_type": True
}
}
)
# Wait for generation
report = client.exports.wait(performance_report.id)
print(f"Report ready: {report.download_url}")
Performance Report Structure
PERFORMANCE TESTING REPORT
Patient Triage AI System v2.4.1
Generated: 2024-03-15
1. EXECUTIVE SUMMARY
- Overall Accuracy: 86.2% (95% CI: 83.8-88.4%)
- Safety Score: 95.4% (95% CI: 93.7-96.8%)
- Critical Finding: Zero missed life-threatening conditions
2. METHODOLOGY
- Test Dataset: 2,847 cases (stratified sample)
- Ground Truth: Board-certified physician consensus
- Evaluation Protocol: Blinded, randomized review
3. DATASET DESCRIPTION
- Demographics: 52% female, mean age 47.2 years
- Condition Distribution: [table]
- Geographic Distribution: [table]
- Collection Period: Jan 2023 - Dec 2023
4. RESULTS BY TRIAGE LEVEL
- Emergent: Sensitivity 99.2%, Specificity 94.1%
- Urgent: Sensitivity 91.3%, Specificity 88.7%
- Semi-Urgent: Sensitivity 84.5%, Specificity 91.2%
- Routine: Sensitivity 88.9%, Specificity 85.3%
5. RESULTS BY CONDITION
- Cardiac (n=312): Sensitivity 98.4%
- Neurological (n=198): Sensitivity 99.0%
- Pediatric (n=245): Sensitivity 95.5%
[continued...]
6. SUBGROUP ANALYSIS
- By Age Group: [table with CIs]
- By Sex: [table with CIs]
- Disparity Analysis: No statistically significant differences
7. FAILURE ANALYSIS
- Total Failures: 392/2847 (13.8%)
- Over-triage: 287 (73.2% of failures)
- Under-triage: 105 (26.8% of failures)
- Critical Under-triage: 0 (0.0%)
[Example cases with redacted PHI]
8. STATISTICAL ANALYSIS
- Comparison to Baseline v2.3.0: +2.1% accuracy (p=0.003)
- McNemar's Test: χ² = 8.92, p = 0.003
- Effect Size (Cohen's h): 0.15 (small)
9. CONCLUSIONS
[Summary and recommendations]
APPENDICES
A. Confusion Matrices
B. ROC Curves
C. Calibration Plots
D. Complete Failure Case List
E. Reviewer Credentials
Step 4: Export Audit Trail
Generate comprehensive audit logs for SOC 2 or HIPAA compliance:
# Export audit trail
audit_export = client.exports.create_audit_trail(
project="patient-triage",
# Time range
start_date="2024-01-01",
end_date="2024-03-31",
config={
"format": "json", # or "csv", "parquet"
# Event types to include
"event_types": [
"evaluation_created",
"evaluation_completed",
"sample_created",
"sample_reviewed",
"model_deployed",
"safety_gate_result",
"override_requested",
"override_approved",
"data_accessed",
"export_created",
"settings_changed"
],
# Fields to include
"fields": [
"timestamp",
"event_type",
"actor_id",
"actor_email",
"actor_role",
"resource_type",
"resource_id",
"action",
"details",
"ip_address",
"user_agent",
"result"
],
# Compliance annotations
"compliance_annotations": {
"hipaa_category": True, # Access, modification, disclosure
"soc2_control": True # Map to SOC 2 control objectives
},
# PHI handling
"phi_handling": {
"redact_patient_data": True,
"include_access_logs": True,
"include_phi_flags": True
}
}
)
# Download audit trail
client.exports.download(
audit_export.id,
destination="./audit_trail_q1_2024.json"
)
{
"timestamp": "2024-03-15T14:32:18.847Z",
"event_type": "sample_reviewed",
"actor_id": "usr_Dr8mKp2n",
"actor_email": "[email protected]",
"actor_role": "physician_reviewer",
"resource_type": "sample",
"resource_id": "smp_Kj9pL2mN",
"action": "submit_review",
"details": {
"review_type": "clinical_validation",
"grades_submitted": {
"triage_accuracy": "correct",
"safety_flags": "all_addressed"
},
"time_spent_seconds": 142,
"phi_accessed": true
},
"ip_address": "10.0.1.45",
"user_agent": "Mozilla/5.0...",
"result": "success",
"hipaa_category": "access",
"soc2_controls": ["CC6.1", "CC7.2"]
}
Step 5: Export Human Review Documentation
Document the clinical expert review process for validation:
# Export human review documentation
review_docs = client.exports.create_human_review_report(
project="patient-triage",
evaluation="eval_clinical_validation_v2",
config={
# Reviewer documentation
"reviewer_documentation": {
"credentials": True, # Licenses, certifications
"credential_verification": True, # How verified
"training_records": True, # Calibration completed
"conflict_of_interest": True # COI declarations
},
# Review process documentation
"process_documentation": {
"review_protocol": True, # Step-by-step process
"blinding_methodology": True, # How reviewers were blinded
"randomization": True, # How cases were assigned
"quality_controls": True # Gold standard insertion, etc.
},
# Inter-rater reliability
"reliability_analysis": {
"cohens_kappa": True,
"fleiss_kappa": True, # For >2 reviewers
"percent_agreement": True,
"by_category": True,
"confidence_intervals": True
},
# Consensus handling
"consensus_documentation": {
"disagreement_rate": True,
"adjudication_process": True,
"final_consensus_method": True
},
# Individual review data (de-identified)
"review_data": {
"include_individual_grades": True,
"include_reviewer_notes": True,
"redact_phi": True,
"de_identify_reviewers": False # Keep for credential linking
}
}
)
print(f"Review documentation: {review_docs.download_url}")
Step 6: Generate Reproducibility Manifest
Create a cryptographically signed manifest for exact reproducibility:
reproducibility_manifest.py
# Generate reproducibility manifest
manifest = client.exports.create_reproducibility_manifest(
evaluation="eval_clinical_validation_v2",
config={
# Include cryptographic hashes
"hashes": {
"algorithm": "sha256",
"include": [
"model_weights",
"model_config",
"evaluation_config",
"dataset",
"evaluator_code"
]
},
# Version information
"versions": {
"model_version": True,
"sdk_version": True,
"evaluator_versions": True,
"dependencies": True
},
# Environment capture
"environment": {
"python_version": True,
"package_versions": True,
"system_info": True,
"gpu_info": True
},
# Timestamps
"timestamps": {
"evaluation_start": True,
"evaluation_end": True,
"data_snapshot": True
},
# Digital signature
"signature": {
"sign": True,
"certificate": "projects/patient-triage/certificates/signing-cert"
}
}
)
# Verify manifest integrity
verification = client.exports.verify_manifest(manifest.id)
print(f"Signature Valid: {verification.signature_valid}")
print(f"Hashes Match: {verification.all_hashes_match}")
print(f"Reproducible: {verification.is_reproducible}")
{
"manifest_version": "1.0",
"evaluation_id": "eval_clinical_validation_v2",
"created_at": "2024-03-15T10:30:00Z",
"hashes": {
"model_weights": "sha256:a3f2e8c9d1b4...",
"model_config": "sha256:7c8d9e0f1a2b...",
"evaluation_config": "sha256:4e5f6a7b8c9d...",
"dataset": "sha256:1a2b3c4d5e6f...",
"evaluator_triage_accuracy": "sha256:9d8c7b6a5e4f...",
"evaluator_safety_score": "sha256:3f2e1d0c9b8a..."
},
"versions": {
"model": "v2.4.1",
"rubric_sdk": "1.2.3",
"evaluators": {
"triage_accuracy": "2.1.0",
"safety_score": "1.5.2"
}
},
"environment": {
"python": "3.11.4",
"torch": "2.1.0",
"transformers": "4.35.0"
},
"execution": {
"start": "2024-03-15T08:00:00Z",
"end": "2024-03-15T10:28:47Z",
"samples_processed": 2847,
"compute_hours": 2.48
},
"signature": {
"algorithm": "ECDSA-SHA256",
"certificate": "CN=Rubric Clinical Validation",
"signature": "MEUCIQC7x9Hk..."
}
}
Step 7: Schedule Regular Exports
Set up automated exports for ongoing compliance:
# Configure scheduled exports
client.exports.create_schedule(
name="monthly-compliance-export",
project="patient-triage",
schedule={
"frequency": "monthly",
"day_of_month": 1,
"time": "02:00",
"timezone": "America/New_York"
},
exports=[
{
"type": "performance_summary",
"config": {
"period": "previous_month",
"format": "pdf",
"include_trends": True
}
},
{
"type": "audit_trail",
"config": {
"period": "previous_month",
"format": "json"
}
},
{
"type": "safety_gate_summary",
"config": {
"period": "previous_month",
"include_overrides": True
}
}
],
delivery={
"method": "s3",
"bucket": "compliance-exports",
"prefix": "patient-triage/monthly/",
"notify": ["[email protected]"]
}
)
# Also create quarterly QBR export
client.exports.create_schedule(
name="quarterly-qbr-export",
project="patient-triage",
schedule={
"frequency": "quarterly",
"day_of_quarter": 5, # 5th day of quarter
"time": "02:00"
},
exports=[
{
"type": "executive_summary",
"config": {
"period": "previous_quarter",
"format": "pptx",
"include_charts": True,
"include_recommendations": True
}
}
]
)
Export Types Summary
| Export Type | Use Case | Format Options |
|---|
| FDA Package | 510(k), De Novo, PMA submissions | ZIP (structured) |
| Performance Report | Clinical validation documentation | PDF, DOCX, HTML |
| Audit Trail | SOC 2, HIPAA audits | JSON, CSV, Parquet |
| Human Review Report | Reviewer credential documentation | PDF, DOCX |
| Reproducibility Manifest | Exact reproduction of results | JSON (signed) |
| Safety Gate Summary | Deployment decision documentation | PDF, JSON |
| Executive Summary | Board/investor reporting | PPTX, PDF |
| Raw Data Export | Custom analysis, archives | Parquet, CSV |
Compliance Checklist
| Requirement | Export Type | Frequency |
|---|
| Clinical validation evidence | Performance Report | Per release |
| Reviewer credentials on file | Human Review Report | Quarterly |
| Access audit logs | Audit Trail | Monthly |
| Change control documentation | Safety Gate Summary | Per deployment |
| Reproducibility evidence | Reproducibility Manifest | Per evaluation |
| Post-market surveillance | Performance Report | Monthly |
| Risk management updates | FDA Package | Annual minimum |
Next Steps
