We Don't Know What We Don't Know

The Gist:  Overestimation of our level of mastery is common and is referred to as the Dunning-Kruger effect [1-3].  This effect persists in medicine and may have deleterious effects both at the bedside and in the training process [4-6].  Use of practice testing or simulation, metacognition, and the push of information to an individual may mitigate the dangers of unconscious incompetence by identifying areas of weakness and keeping the practitioner skilled.

"Ignorance more frequently begets confidence than does knowledge" - Charles Darwin [6]

The Case:  A 48 year old female with hypertension presented to Janus General with the sensation of the room spinning and feeling off balance for the past day.  The resident performed a head impulse, nystagmus type, test of skew (HINTS) exam, which demonstrated the presence of a saccade during the head impulse component. and determined that the patient likely had a central cause.  The patient was admitted to the hospital for questionable TIA/CVA workup, which was negative.

• The problem: a saccade during the head impulse component is suggestive of a peripheral cause of vertigo.  While use of the HINTS exam by emergency providers has not been validated and is outside the bounds of this discussion, this example demonstrates the overconfidence of the resident in their skills and interpretation and the potential for perpetuation of this misunderstanding.  

The Dunning-Kruger Effect - These researchers demonstrated a large discrepancy between the way incompetent people actually perform and the way they perceive their own performance level.  This discrepancy appears smaller for highly skilled individuals, who may underestimate their mastery. The studies demonstrating this effect often result in a graph akin to the one below.

The Danger in Medicine - A little knowledge is a dangerous thing.  In medicine, a balance exists between the confidence to commit and act and the insight to see that we have uncertainty or lack the appropriate skill set.  In emergency medicine this line may be particularly difficult to walk as patients often require immediate intervention at times when data is limited, clouding certainty.

• Diagnostic Error -  Overconfidence may result in premature closure, confirmation bias, or failure to use clinical aids such as practice guidelines or decision aids.  It turns out that compared with tempered attendings and insecure medical students, residents may be most susceptible to diagnostic overconfidence [8]. 
• Unsafe Procedures - An attending once told me, "Procedures in emergency medicine exist to keep us humble."  After a novice learner's first string of successful intubations, it's tempting to believe we've become "good" at intubations.  However, this hubris may cause us to expose patients to harm if we take short cuts in preparation or fail to continue to maintain contingency plans.  Furthermore, in emergency medicine, we must be able to cue up live saving procedures at any moment that we may only see or perform once in a career.  Without accurate awareness of our competence, we may we fail to recognize that our skillsets are weak or a procedure/tools/kits have changed leading to suboptimal outcomes.  

“Real knowledge is to know the extent of one's ignorance” – Confucius

Mitigating the Effect

• Practice Testing (or retrieval practice) - While the quality of the evidence is poor, Davis et al write that physicians are particularly not keen at assessing themselves [6].  As a result, we are unlikely to identify our areas of incompetence ourselves.  Assessment through practice testing, however, may identify weak areas in our knowledge base or skill set that we may otherwise not identify. In this way, we may decrease the discrepancy between what we think we know and what we actually know.
• Simulation - Practice and rehearsal can identify weak areas in procedures or situations we may not often encounter or, conversely, give us the confidence in our ability to act when needed.  This doesn't require mannequins and computers - simply rehearsing procedures and scenarios in our minds works quite well.  Dr. Cliff Reid eloquently stated, “We have the most powerful, 3D, high-fidelity simulator in the known universe," reference the mental simulator.
• Life and Limb Saving Procedures - Cliff Reid
• How To Make It Happen - iTeachEM
• Information Push - Most of us only review topics we believe we have previously mastered when we discover a weak spot in our understanding or make a mistake.  However, when information is pushed to the us, we encounter a wider array of information.  
• Free Open Access Medical education (FOAM) serves as a prime example of the concept of information push, as topics collated in RSS aggregators such as Feedly and delivered by subscriptions to podcasts or twitter feeds cover information not intentionally sought by the learner.  So, while some find FOAM overwhelming, it can be used in an interesting way to strengthen the areas that we don’t know that we don’t know.
• Metacognition - Awareness of weak points in our reasoning or systemic traps may help us overcome overconfidence, particularly with regard to diagnostics.  The post, Metacognition For the Pragmatist, discusses ways in which we can overcome our cognitive biases.  
• Remain skeptical of the feeling of complete ease and mastery, which may cause to stop working and striving to improve.  In his talk “The Path to Insanity” at SMACC 2013, Dr. Scott Weingart advises that we "never assume your own excellence" [9]. 

What's the underpinning of the Dunning-Kruger effect?

• It's difficult to estimate the quality of one's performance.
• Self-enhancement bias - We tend to think we're better than we actually are.  For example, very few bad drivers actually recognize the fact that they, in fact, are the poor drivers.
• Regression to the mean - This explanation exists as one of the primary criticisms of the Dunning-Kruger Effect and argues that the effect may be, in part, statistical artifact.  When people are asked to evaluate themselves, incompetent people may simply regress toward the mean when asked to evaluate their own performance. On the other hand, people who have extremely high levels of performance, those who have mastery in a given area, are also less likely to underestimate their skills to the degree that those who are incompetent.

“I know that I know nothing” – Socrates

References:
1. Dunning D, Heath C, Suls JM. Flawed Self-Assessment: Implications for Health, Education, and the Workplace. Psychol Sci Public Interes. 2004;5(3):69–106. doi:10.1111/j.1529-1006.2004.00018.x.
2. Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psychol. 1999;77(6):1121–34. 
3. Dunning D, Johnson K, Ehrlinger J, Kruger J. Why people fail to recognize their own incompetence. Curr Dir Psychol Sci. 2003;12(3):83–87. doi:10.1111/1467-8721.01235.
4. Edwards RK, Kellner KR, Sistrom CL, Magyari EJ. Medical student self-assessment of performance on an obstetrics and gynecology clerkship. Am J Obstet Gynecol. 2003;188(4):1078–1082. 
5. Hodges B, Regehr G, Martin D. Difficulties in recognizing one’s own incompetence: novice physicians who are unskilled and unaware of it. Acad Med. 2001;76(10 Suppl):S87–9.
6.  Davis DA, Mazmanian PE, Fordis M, Harrison R Van, Thorpe KE, Perrier L. CLINICIAN ’ S CORNER Accuracy of Physician Self-assessment Compared A Systematic Review. 2006;296(9).
7. Darwin C. 1871. The Descent of Man, and Selection in Relation to Sex, 1st edition. London: John Murray.
8. Berner ES, Graber ML. Overconfidence as a cause of diagnostic error in medicine. Am J Med. 2008 May;121(5 Suppl):S2-23. doi: 10.1016/j.amjmed.2008.01.001.
9. Weingart S.  EMCrit.  Podcast 105.  August 20, 2013.  Available at: emcrit.org/podcasts/path-to-insanity/