Pribiag part of team tackling confirmation bias in condensed matter physics

Professor Vlad Pribiag, of the School of Physics and Astronomy, was part of a group of condensed matter physicists who recently demonstrated—through four case studies—how over-reliance on single pieces of evidence (so-called “smoking-gun” studies) can lead to the misinterpretation of observed effects. 

The group focused on a particular subfield of condensed-matter physics called topological physics, which has gained significant attention as a potential pathway for developing error-protected quantum computing. Lead author, Sergey Frolov, of the University of Pittsburgh, has been active in debunking some recent claims in this field. The researchers created four case studies, which Pribiag describes as “cautionary tales,” showing how a major discovery might be claimed at first glance, only to be revealed as incorrect or unlikely upon closer analysis.  

The group issued a set of recommendations to avoid these mistakes in the future—guidelines that apply to any “smoking-gun” study beyond the realm of topological physics.

Condensed matter experimentalists, who study the electronic properties of new materials and devices, are often guided by pre-existing theories. When a theoretical prediction has not yet been observed in a lab setting, there is a rush to be the first to claim it.  An experiment may be able to reproduce something that looks like the confirmation of the theory, a data plot that matches one presented by theorists, and declare it “smoking-gun” evidence that would instantly prove the theory.  

“It’s not as simple as that,” Pribiag says. “There’s rarely a single plot that can support an interpretation.” 

The group's first recommendation is to broaden the scope of studies. “Any sort of good theory will tell you what should happen if you vary parameters. You have to go do that in the experiment and then also seek to replicate it in multiple samples. As you explore the parameter space, if the observations diverge from the theory, then you need to report that and reassess the validity of your interpretation."

Pribiag compares narrow results to a partial fingerprint. “If you only take a piece of a person’s finger print, you can’t determine a match, because a certain portion of the population might match with that part of the print.”

In several case studies Pribiag and colleagues considered, there was one narrow data set that pointed to a very exciting conclusion. However, when they explored a broader data set from the experiment, they found that these contradicted the initial interpretation and pointed to a very different, less exciting interpretation for the observed data features. 

A major problem is the potential for confirmation bias, where once some evidence of a desired effect is seen, one might stop searching for counter-evidence. This can lead other researchers down an incorrect path. Even when mistakes are found, papers retracted and conclusions debunked, many people in the community remain unaware of this. This can lead to wasted effort or propagation of incorrect views, especially among those not directly in the center of that field of study.

The second recommendation was to share full raw data sets, so that other scientists can make up their minds about a paper’s conclusions. 

“These days it’s very easy to share data sets,” Pribiag says. “There are online repositories where you can upload your data. There isn’t any reason not to do it.” 

Data sharing prevents the most damaging form of bias—deliberate fabrication—but it also allows the community to catch honest mistakes or reassess results as new discoveries emerge.

The final recommendations were to discuss alternative interpretations, and disclose the volume of the study. The interpretation of “smoking-gun” type experimental studies typically involves some qualitative judgement. Pribiag and colleagues argue that even if the authors think they have strong evidence for one interpretation, it is helpful to suggest other scenarios that may potentially explain some of the observations, and explain why those alternative scenarios are in the authors’ judgement less likely. Similarly, they point to the importance of informing the readers of the published work about the number of devices studies (volume of study) and how representative the published results were of the totality of the observations (ideally including the raw data for these devices). 

Pribiag says that a lot of their findings and recommendations were common sense. “We teach these things to graduate students.” The School of Physics and Astronomy has a mandatory class for all new students that covers research ethics and best practices. “Yet, in the last few years, we have still seen examples of high-profile papers where some of these basic principles were not followed, leading to damaging retractions.”

The paper was co-authored by Colin Riggert and Dr. Mohit Gupta (both of Pribiag’s group), along with colleagues from the University of Pittsburgh and the University of Grenoble.