It doesn't make sense, but that's how we do it. Usually, if you want to judge the calibration of a decision maker, you tally hits and misses both, not just the hits. If you fail to record misses, a high hit rate might just signify a lot of swings. But in medicine, we are taught to ignore misses. It's only hits that count. This doesn't make sense because it encourages overtesting and waste. If surgeons practiced this way, all heck would break loose, everybody would be cut open and organs wantonly plucked from them.
There is a physician at a Midwestern institution who is known to come on service and order an echo bubble study on several (all?) patients the first day he is on service. He has doubtless impressed a generation of medical students when the occasional PFO (patent foramen ovale) is found and he gets to wax prolific about shunts and other mechanisms of hypoxemia for a few minutes as they listen with fawning and rapt attention. What they do not know is that he finds PFOs about 10% of the time he orders his echos, and this is just above the rate of PFOs in the general population. He's basically finding PFOs at random.
There is a physician at a Midwestern institution who is known to come on service and order an echo bubble study on several (all?) patients the first day he is on service. He has doubtless impressed a generation of medical students when the occasional PFO (patent foramen ovale) is found and he gets to wax prolific about shunts and other mechanisms of hypoxemia for a few minutes as they listen with fawning and rapt attention. What they do not know is that he finds PFOs about 10% of the time he orders his echos, and this is just above the rate of PFOs in the general population. He's basically finding PFOs at random.
