Sunday, March 17, 2013

The Modern Matthew Effect

The Gist:  In medicine and science, regardless of the medium - traditional or Free Open Access Medical education (FOAM)- the Matthew effect exists, potentially perpetuating knowledge and dogma that doesn't necessarily reflect intrinsic worth.  Question the medical dogma, respectfully and, while it's easy to copy and paste a citation for a quote a popular figure, consider critically evaluating the source of information or primary literature.  In words borrowed from TheSGEM podcast, "Be skeptical of everything you learn.." (to a healthy, not pathologic degree) - it's another arrow in the metacognition quiver.

Conversations on the perils of FOAM at the Social Media and Critical Care Conference (SMACC) spurned the following, something I think is worth reminding ourselves of from time to time:
While many of us exercise healthy skepticism we can still fall victim to a common phenomenon because, in the words of Daniel Kahneman in Thinking Fast and Slow, we have "almost unlimited ability to ignore our ignorance."  We may think we are fully aware of our biases, but they are worked into the fabric of life.  

The Matthew Effect (with regard to references):  essentially, the greater number of times a paper is cited, the more citations it will receive.   Coined by Merton in this paper, but initially researched by Harriet Zuckerman, it is borrowed from the Gospel according to St. Matthew: “For to all those who have, more will be given, and they will have an abundance; but from those who have nothing, even what they have will be taken away” (Matthew 25:29) (1).  
  • The Matthew effect is partially a byproduct of quality.  A content expert likely becomes trusted and their work becomes highly regarded due to the merit of their prior work(s).  Thus, this is a sort of natural phenomenon in any field that has experts/masters in particular disciplines.  Zuckerman identified this in that Nobel Prize winners tended to generate/produce more awards compared with those who had shared equally in the project but were more junior researchers (1).
How does this manifest in medical literature?
High Impact Journals.  Impact factor (IF) - The IF is essentially the average number of times an article in the journal is cited within the previous two years.  journal’s prestige is a function of the quality of the articles appearing in it. 
  • What happens when the exact same article (title, author, etc) is published in two journals with disparate IFs?  This paper by Lariviere and Gingras (full text) took at look at this question and found that duplicate papers (4532 pairs of papers) in high impact journals obtain, on average, twice as many citations as their identical counterparts published in journals with lower IFs.  
  • The intrinsic value of a paper is not the only reason for the citation of a specific paper; there is a Matthew effect attached to journals.  Thus, a paper published in a high impact journal has an added value over its intrinsic quality and will generate more citations.
High Impact Authors.  A high profile author's paper is likely to carry more weight or gain more recognition.
  • Example:  In the Feb. 2013 edition of Emergency Medical Abstracts (subscription required), there's a little bit of banter about how this paper on incidence of contrast induced nephropathy was referred to ("the Kline paper").  Pulmonary embolism guru Dr. Jeff Kline is listed on the paper but the first author is actually Dr. AM Mitchell.  
This sounds like splitting hairs..  Perhaps a little, but not necessarily.
  • Implicit in the concept of the Matthew effect is the notion is that a piece of research is more valuable or important because of its association with an individual rather than the contents, quality, or implications of the research.
  • Without realizing it, we may become susceptible to a cognitive bias secondary to the "Halo Effect," which I first heard about in Thinking Fast and Slow by Daniel Kahneman.  For example, if an individual is widely regarded in the community for a podcast or publication, their institution may be looked upon more favorably.  
    • Dr. Weingart's tweet at the beginning of this post demonstrates potential implications of the halo effect - a positive/powerful reputation may have undue influence over whether we see that information as important or valid.  If someone we respect says an article is a "must read" or "garbage" we have formed an impression of the article prior to actually reading it.  They may very well be spot on, but this is something to keep in mind.
    • In an era of information overload, especially in medicine, we may deal with this cognitive load by perceiving a reputable person's recommendations as most/more important (known as positional cues).  This may skew our evidence base or perception of prevalence or importance of a medical problem.
Is FOAM impervious to this effect? No.
  • FOAM has a form of Impact Factor.  This can be quantified in retweets, blog hits, or a spot  in a Life in the Fast Lane Weekly Review.  Again, this is not necessarily a negative thing and can be harnessed "for good," introducing innovative or important ideas quickly and diffusely across the globe.  
  • Example:  
 
  • The social connections and the platforms associated with FOAM are intricate (hospital and professional networks, friends/families (social media), affiliations with societies, etc).  As a result, the Matthew effect may be less like the "Nobel Prize" effect noted by Zuckerman as age, rank, and location may not carry as much weight and the sources are vast.
  • Recently, Google announced that it would drop its RSS aggregating service, GoogleReader. This move immediately induced a Twitter frenzy regarding replacement services.  One focus of conversation on this topic from some members of the FOAM community was that Twitter has replaced the need for RSS.  This article discusses this notion, a debatable assertion that I don't personally find applicable to my use of RSS.  Should Twitter supplant RSS, individuals who use an RSS aggregator to review journals and/or medical blogs may have increased susceptibility to biases associated with a social media/recommendation system based system. 
So what do we do?
  • Question productively and respectfully.  
  • Check sources.  For example, while putting together this post on elevated blood pressure in the ED, I came across a statistic in Tintinalli:  3.8% of headaches in the ED have serious intracranial pathology (Ch. 159).  Initially, I copied this statistic and reference because Tintinalli is one of the core EM texts.  FOAM has inspired me to check things out further, and upon evaluating the study I found it underwhelming to support the rate quoted.  This study was referenced by others as well, including the famous Perry et al article on subarachnoid hemorrhage and others. 
  • Keep the Matthew effect in mind when evaluating articles, watching posts/ideas go "viral", or evaluating the validity of an assertion or claim.
Updated 3/18/13.

References:
1.  Zukerman H.  Scientific Elite:  Nobel Laureates in the United States. 2d ed. (New York:  Transaction Publishers, 1996). 

8 comments:

  1. Very observant post. While social media exponentially increases access to resources and opinions, it is sometimes as the expense of deliberate, thoughtful peer review. The current trend of social media "individual meta analysis" (i.e. I reviewed all the literature and came to conclusion X), although interesting to senior providers, can be dangerous to young learners who lack the foundation to understand the implications of changing practice based on opinion. Although FOAM continues to expand, it lacks the critical peer review important to facilitate practice changing guidelines. Perhaps a future project...

    Bill

    Bill Soares
    Chief
    Highland Hospital
    Alameda County Emergency Medicine Residency

    ReplyDelete
    Replies
    1. Thank you for the comment! You raise some great points. My amateur thoughts: It is certainly dangerous for young learners to establish a habit or expectation of changing practice based on novice opinion. Unfortunately, in medicine, I think we are all subject to changing practice based on opinion...whether it's ours, an attending's, an institution's, etc. Regardless of the medium (FOAM, text, lectures, or consensus guidelines), some degree of opinion skews the perception. This is often good because many individuals, like myself, lack the knowledge base and clinical expertise to properly interpret and incorporate literature or practice changes. I think the value of the "individual meta-analysis" for the novice learner lies in the process of learning to evaluate literature, build a evidentiary knowledge base, and learn applicability. It may seem unimportant in academic centers, but in many community and rural settings, change often comes slowly or in a skewed fashion. Establishing self-directed information seeking and sharing (from content experts/paper authors/professors, etc available via Twitter) can, I think, allow for "catches" before misguided information reaches the patients bedside. This is difficult to learn and prone to individual variation and error. There are certainly limitations and pitfalls, but part of the educational process involves teaching students to critically appraise information and learn how this is translated in real life. Now, the peer review process is an entirely separate beast and, I believe, there are many projects currently under way to broach this issue. In this vein- http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/

      Thanks again for your input, excellent points that I shall continue to ponder. I look forward to seeing how this process/phenomenon changes and grows. - Lauren

      Delete
    2. Great counterpoints. I agree with you on many aspects, that our clinical practice is an odd amalgamation of individual opinion and consensus, of evidence and eminence, and that social media offers increased access to a wealth of information that groups (rural / community / residents) did not have access to before. It is the future of our field.

      However, it is the speed of these opinions, the forcefulness with which they are presented which carries inherent risk, especially to the provider who lacks their own developed clinical expertise . i.e. - http://www.thennt.com/blog/2012/06/delusions-of-benefit-in-the-international-stroke-trial/.

      Study after study demonstrates we are poor at interpreting literature - http://www.ncbi.nlm.nih.gov/pubmed/17785646 -
      and that all EBM can be misinterpreted. This may always be a component of EBM.

      However, whereas in the past a novice opinion had a limited range, social media has expanded this audience infinitely. And while your impressive insight will guide you to look at the actual literature and decide for yourself, you will be amazed to see how many of your colleagues will simply take what they hear verbatim

      Ultimately, I agree with your last statement, the more people understand about the primary literature - the better equipped to utilize social media. Now we just have to figure out how to do it.

      thanks for the interesting discussion
      Bill

      Delete
  2. Ack...pet peeve time:

    You probably should avoid CNET for articles about technology...Twitter and RSS are not related products and it is maddening to see so many people confuse the two of them. Ben Parr, the author of the CNET article, has no idea what he is talking about. Moreover, the company he works for has a hashtag in its name, so this is an obvious astroturf masquerading as "in-the-know content". It looks bad when people reference ZDNet, ComputerWorld, CNET, or any of those style publications as "authoritative".

    Google likely dropped Reader because it was difficult to monetize.

    Sincerely,

    A Software Engineer.

    ReplyDelete
    Replies
    1. Point taken. Thank you for the feedback. My use of this article was not to demonstrate so much that it was an insider take on the reason Google is dropping Reader, but rather because the article (and the Reader drop in general), sparked a debate on Twitter amongst healthcare providers about potential "fixes." One group argued that they already use Twitter as an RSS feed replacement of sorts. Again, thank you for the feedback and I will clarify the intent! - Lauren

      Delete
    2. Twitter is a short conversation and RSS is reading a book. You can't really replace one with the other because they serve entirely different functions. Members of #FOAMEd who miss out on this distinction are probably missing out in general.

      Thank you!

      Delete
  3. Thanks for the post. I agree that healthy skepticism is crucial.

    I try and look at the lit cited in other peoples "reviews" and decide for myself if i support the same conclusions. This is easier as an academic attending as opposed to a trainee.

    Legitimate bloggers are identified in the "about section" or through their publications elsewhere [e.g. Mike Cadogan and Amal ... nuff said].

    Keep the posts coming.
    Nadim

    ReplyDelete
  4. very observant post. while social media exponiently increasse aceess to resourses and opinions,it sometime as the expense of delibrate thoughtful peer review.
    Corona Dentist

    ReplyDelete