Nolla Visual Medicine Benchmark
Intro
Nolla Visual Medicine Benchmark is the visual-medicine benchmark Nolla Health formally published on 16 November 2026, for evaluating how well models interpret clinical images, gather information through conversation, and bring the two together to reach a diagnosis. Sean Geiger (@sean_geiger) posted at 2026-11-16T16:47:19Z (00:47:19 Beijing time on 17 November 2026): “Today, we’re introducing the Nolla Visual Medicine Benchmark. We’ve been using it internally for the past year to evaluate models for patient care. Over that year, we’ve seen frontier models make rapid progress. Now we’re sharing the results comparing frontier model performance”.
The official blog at nollahealth.com/blog/visual-medicine-benchmark is dated November 16, 2026 and titled “Introducing the Nolla Visual Medicine Benchmark.” It argues that when you see a doctor, they do not just ask questions: the pattern of a rash, the color of an eye, or the location of swelling can change what they ask next. The benchmark is built to evaluate that relationship. It has two related tasks. In the image-only task, a model or physician sees a single clinical photograph and writes a ranked list of possible diagnoses, with no history and no chance to ask questions. In the interview task, the encounter starts with a photograph and an initial message from a simulated patient, then continues through follow-up questions and ends with a ranked differential and a structured clinical note. Each simulated patient combines a clinical history with a behavioral persona — occupation, circumstances, concerns, and demeanor — and some are anxious, embarrassed, or inclined to minimize symptoms.
The maintained image inventory spans eleven specialties and 49 disease groups. Dermatology accounts for about 45.4% of cases (451), alongside pediatrics, emergency medicine, geriatrics, orthopedics, internal medicine, otolaryngology, ophthalmology, endocrinology, gynecology, and psychiatry. Disease-group examples include skin cancers and precancers, eczema, acne, fungal infections, burns, joint disease, eye disorders, and visible signs of systemic disease such as acromegaly or Cushing’s syndrome. The comparison study is registered as NCT07470463 and enrolled 23 physicians (11 board-certified, ten residents, and two fellows) with a median of five years of clinical experience; ten qualified for the image analysis and 21 for the interview analysis. The post says only 16.5% of image reviews and 21.9% of interviews matched the physician’s own specialty, so the results describe a mixed group working largely outside its specialty, rather than a specialist-to-specialist comparison. It also notes that in smaller unpublished trials, board-certified physicians still reliably beat frontier models when operating within their specialty.
On the study’s Level 3 top-ranked diagnostic concordance measure — a match to the reference diagnosis, or an answer judged to imply clinically equivalent management, scored by an automated language-model judge — the official table lists: Gemini 4 Argon at 46.5% image-only and 83.2% on simulated interviews; Claude Fable 5.2 at 43.1% / 86.4% (best on interviews); Gemini 3.8 Flash at 40.4% / 78.4%; Gemini 3.1 Pro at 39.6% / 72.8%; Claude Opus 5.5 at 39.1% / 77.6%; GPT-6 Astra at 35.5% / 80.0%; GPT-5.4 at 27.5% / 77.6%; Claude Sonnet 4.6 at 23.2% / 53.6%; all eligible physicians at 30.2% / 56.0%; and the board-certified subgroup at 34.1% / 78.8%. Newer models were evaluated, the post says, with the same prompts, simulated patient, and automated judge, with one scored output per case.
Key points:
- Official introduction: https://x.com/sean_geiger/status/2122370944899686902 ; official blog: https://www.nollahealth.com/blog/visual-medicine-benchmark ;
- This is a concrete new benchmark name, not Nolla’s existing NollaMD model, and not a generic medical visual-QA test. The figures in the table are company results under this study’s conditions; this page does not independently reproduce them.
Advertisement
Explain
Why it matters
This is a new benchmark name Nolla Health confirmed on 16 November 2026 with a team introduction post and an official blog post, not a rename of the existing NollaMD model or of a generic medical visual-QA test. The introduction post went out at 16:47:19Z (00:47:19 Beijing time on 17 November), inside this 00:32–04:32 scan window, so it is filed separately. Official materials say the benchmark has been used internally for about a year; this release shares results and a physician comparison.
This page only restates the official introduction post and the official blog. The top-ranked diagnostic concordance figures, the physician-comparison design, and the claim about unpublished within-specialty trials are company statements; this page does not independently reproduce them.
Limits
This is an observational study that uses simulated patients and an automated language-model judge. Official materials also say the physician cohort mostly worked outside its own specialties, so do not treat the scores as a general conclusion about real clinical care or specialist performance. This page is not a guarantee of any model’s clinical safety, and it does not describe the benchmark as a replacement for NollaMD or for public medical-imaging benchmarks.
Usage
- Read the official introduction at https://x.com/sean_geiger/status/2122370944899686902 and cross-check the official blog at https://www.nollahealth.com/blog/visual-medicine-benchmark ;
- When citing specific scores, the enrolled physician count, or NCT07470463, attribute them to the 16 November 2026 official blog and recheck the upstream source; how to obtain or use the benchmark is as stated by Nolla Health;
- When discussing whether “models beat physicians,” also note that the study uses simulated patients and an automated judge, and that the physician cohort mostly worked outside its own specialties;
- Do not describe Nolla Visual Medicine Benchmark as the existing NollaMD model, or as a generic medical visual-QA test; this page only restates official materials — recheck the upstream sources before citing them externally.