Statistic evidence for the truth

MBAd6224, Decision Making and Data Analysis
0 Introduction
This assignment asks you to address questions about statistical evidence for truth. In
common with the previous written assignment, the topics to be addressed stem from another
provocative article, Why Most Published Research Findings Are False. (As usual, everything
in blue is a link.) After you read the paper itself, I recommend reading the comments about
it that follow.
1 Possible Additional Readings
Although the assignment can be completed by reading only the subject paper, you may be
interested in knowing that repeated occurrences of non-reproducible results has generated
much recent discussion about this general problem of statistical inference. I have referenced
most of these papers already in the Enjoy Reading file and on the open MBAd6224 open
website. I note several in particular:
? ?How can so many scientists have been so wrong?? asked Engber in Slate magazine
(2016).
? In response, Sanjay Srivastava, a psychology professor at the University of Oregon,
created a fictitious course in which Ioannidis?s paper is discussed in Week 8, about
scientific publishing.
? ?How scientists fool themselves … ?
? The concerns are so high that in June the American Statistical Association published a
paper, The ASA?s statement on p-values: context, process, and purpose.
Dr. Denis F. Cioffi 224 Fall 2016
2 The Assignment Itself
I have divided this assignment into two halves. As described immediately below, the first is
an explanation of the paper itself, and the second is a reaction to it.
1. Part 1 [50 points]. Explain the basics of the paper. The first three items are mandatory,
but I have also listed a few other topics as additional possibilities to address in your
discussion.
a. (Mandatory [8 points].) Using the definition of R, explain in a couple of sentences
why ?The pre-study probability of a relationship being true is R/(R + 1).?
b. (Mandatory [8 points].) Explain ?Positive Predictive Value,? PPV.
c. (Mandatory [8 points].) Discuss the author?s notion of bias.
i. You might relate it to any of the homework assignment problems, which were
largely sterile in that regard. Or not.
d. We have lately focused on p values, so you might comment upon the statement,
?Research is not most appropriately represented and summarized by p-values, but,
unfortunately, there is a widespread notion that medical research articles should
be interpreted based only on p-values.?
e. We would prefer remaining naive and not considering Corollary 5, but we must
accept its reality. (Mustn?t we?)
f. ?Empirical evidence on expert opinion shows that it is extremely unreliable.?
g. We have only perceived single-team studies for the homework assignments, but
the author writes, ?… it is misleading to emphasize the statistically significant
findings of any single team.?
Unlike last time, some answers now can be factually wrong: if you do not properly
explain the parts of the paper that are to be addressed specifically (above), you will
lose credit.
2. Part 2 [50 points]. Now that you have explained the paper?s fundamentals, turn and
discuss what it might mean to your worldview by addressing two of the following [25
points each].
a. Again we compare to the business world. Explain if the problems noted by this
author have the same potential for harming statistical data analysis in business as
they do in science. (Hint: the answer is ?yes.?) How should they be addressed?
b. How does one live as an individual in such an uncertain world? For example, how
do you react now to studies you hear about, especially ones that suggest changing
the way you live for health reasons?
c. What is your personal solution to the analysis of data for which you might be
responsible?
2 of
d. You can write what would be a comment to the journal (and then if you like it,
submit it).