OpenAI Releases MentalHealthBench, Built With 80+ Clinicians
Mental health AI evaluations have historically focused on crisis scenarios, leaving everyday and ambiguous conversations unmeasured. This benchmark provides a structured, clinician-grounded method for comparing model performance across that broader range.
OpenAI released MentalHealthBench, an open benchmark for evaluating AI model performance in realistic mental health conversations. The dataset contains 1,215 synthetic mental-health conversations, with evaluation criteria developed by more than 80 licensed mental health clinicians spanning 22 countries, 19 languages, and nearly 20 subspecialties. Each synthetic conversation uses a custom expert rubric requiring agreement from at least 2 of 3 reviewers with no contradiction from the third. GPT-6 Astra scores 57.3 on the benchmark compared to GPT-4o's 32.1. GPT-5.6 Sol serves as the LLM judge, grading model answers against human-written criteria. The benchmark addresses a gap noted in prior work: most mental health AI evaluations focused on emergencies and broad safety, leaving everyday and ambiguous conversations under-measured. OpenAI stated it is releasing the benchmark openly so other researchers can examine the methods, run their own evaluations, and build on the work.
A September 24 source put the dataset's size at 1,215 synthetic conversations, the one new detail to surface since OpenAI published the benchmark on September 23 with input from more than 80 clinicians across 22 countries and 19 languages to evaluate AI on mental health topics.
OpenAI released MentalHealthBench on September 23, an open benchmark for evaluating AI models on realistic mental health conversations, co-created with more than 80 licensed psychologists and psychiatrists spanning 22 countries, 19 languages, and nearly 20 subspecialties.
Each synthetic conversation uses a custom expert rubric requiring agreement from at least 2 of 3 reviewers. GPT-6 Astra scores 57.3 on the benchmark versus GPT-4o's 32.1, with GPT-5.6 Sol serving as the LLM judge. OpenAI said it is releasing the benchmark openly so other researchers can examine the methods and run their own evaluations, noting that prior mental health AI work left everyday and ambiguous conversations under-measured.
I send one email each morning with the stories that moved. If you would rather just read here, that works too.
Subscribe free