Wednesday, March 16, 2011

Avoiding the File Drawer

In many areas of research, there's a problem known as the "File Drawer" problem. It appears to be especially pervasive in pharmaceutical research, but it exists elsewhere. Essentially, it's this:

A group or researcher does a huge number of studies about a particular subject. The researcher could then theoretically handpick a small number of those studies that show a hoped for effect and file away the rest. To the outside world, this appears compelling even if the larger body of work clearly demonstrates that the phenomenon doesn't exist.

When psi research is done with apparent positive effects, Skeptics often cite the file drawer problem and suspect that it was one study done among many, but the others have been buried. This may or may not be true, but there's really no way to know for sure. Doubt is cast, not much has been learned.

I recommend that psi researchers (not that they'll take my advice) simply eliminate this argument that we Skeptics have in regards to their studies. Before doing any studies, register the study with an agreed upon authority. Make it a nuetral party that both the psi researchers and the Skeptics agree upon. This party is simply there to verify that the study is registered.

Registration must include proposed methodology and an explanation of what is being studied. After the study is concluded, all results must be published and submitted to the registrar before any are released to the media or proposed for publication. Critics will have ample time to suggest improvements or changes to the methodology. Researchers will not be allowed to register or publish any further studies until all past results have been published. Basically, no filing away of negative results.

This will have the combined effect of eliminating the File Drawer problem AND taking it away as an excuse for Skeptics. Most importantly, it moves us closer to transparency and honesty, which I believe is something the vast majority of us want. Thoughts?

No comments:

Post a Comment