Overview

Thalia reviewed how gender and racial/ethnic bias are currently detected in general-purpose and medical LLMs. She designed an exploratory study testing whether explainability methods (SHAP, LIME) give fair, reliable explanations for medical LLM decisions — using paired clinical vignettes where demographics vary but clinical facts stay constant. The study compares model outputs and explanations across patient groups, looking at attribution differences, consistency, and stability. Findings are interpreted through bioethical principles (justice, non-discrimination, transparency, potential clinical harm) to separate statistical differences from ethically meaningful ones.

Outcomes

• A literature map on bias detection and explainability in medical/biomedical LLMs
• A preliminary protocol for evaluating SHAP/LIME disparities via counterfactual clinical cases
• A bioethical framework for interpreting whether explanation differences could contribute to unequal treatment

Why it matters

Explainability tools are often treated as making AI more transparent — but an explanation can look convincing while being unstable or inconsistent across demographic groups. Evaluating explanations themselves, not just outputs, adds a needed layer to AHEAD’s technical and ethical approach.

Next steps

Implement the pilot and assess whether these fairness/explanation measures are suitable for medical LLM evaluation, feeding into more interpretable indicators for clinicians, policymakers, and patients.