Physicians Are Bad at Math, Too

The probability issue

Much of medical care involves multi-step processes. At the simplest of levels, you first make a diagnosis and, second, initiate treatment. In the world of probability, we could assign a likelihood to both making the correct diagnosis, say 90%, and then the possibility of initiating the proper treatment, say 80%. According to probability’s rules, the likelihood of these two independent events occurring is the product of their probabilities, in this case, 72%. When asked, physicians often overestimate the probability of both these events being true; they are committing the conjunction fallacy.

How frequently does this physician overestimation occur? Here is the graphic from the study I am sharing with you.

The Study

The research involved a survey of 215 physicians who were asked to judge the probabilities in three scenarios. In one scenario, pulmonologists, physicians trained in the care of diseases of the lung, were asked about the independent possibilities that

• a “lung nodule” was cancerous
• and could be successfully diagnosed by a needle biopsy

They were also asked the probability of the two events taken together, the “overall probability of the conjunction.” [1]

Of the 84 pulmonologists, 84% were male, with an average age of 53.3, having graduated from medical school 27 years earlier – these were experienced individuals.

64% overestimated the individual probabilities that the nodule was cancer and that a needle biopsy would provide a conclusive diagnosis. I can spot them overestimating the diagnosis because we are trained to be “conservative,” and ignoring a cancer is a far more significant problem than an “unnecessary” biopsy – remember, “out of an abundance of caution.” Overestimating the diagnostic accuracy of a needle biopsy is perhaps less forgivable, but what is in the textbook and what is the overall accuracy of the physicians doing the test can vary – as Yogi Berra stated, "In theory, there is no difference between theory and practice - in practice there is."

Twenty percent of the pulmonologists went on to commit the conjunction fallacy, further overestimating the combined likelihoods. They didn’t seem to understand the underlying mathematics. The researchers were concerned that perhaps asking for the combined probability first threw the physician “off.” But in a different scenario, physicians, in this instance, obstetricians considering an obstetric complication, were asked about the two events' independent probabilities first, then the conjuncture. It made no difference; we still got it wrong.

For those who are curious, gender made no difference, nor did time since medical school graduation.

Talking Points

Much medical care consists of multiple steps. Prior research has shown that greater experience does not necessarily translate into better probability predictions. More importantly, as the researchers note, a survey is not a real-world situation but a model. And the model can be further flawed because it is unclear whether a physician’s diagnostic and treatment decisions are driven by algorithms, the Bayesian probabilities, or a “black box” intuitive process. The real question that the study only begins to chip away at is how we might improve our medical decision-making – we get it right most of the time, but like many other goals, those last few percentage points are the toughest.  

[1] Two other scenarios considered an obstetrical emergency involving the remaining 131 participants. 78% committed the conjunction fallacy in their calculations, reporting the overall likelihood of the two events being more significant than the likelihood of either taken separately.

Source: Analysis of Physicians’ Probability Estimates of a Medical Outcome Based on a Sequence of Events JAMA Network Open DOI: 10.1001/jamanetworkopen.2022.1880