PSY 555 Homework 21
1. The two kinds of error rates are pairwise, or per comparison, error rates (PC) and familywise error rates (FW). PC error rates are the probability of making a Type I error on any given comparison. The probability of making a PC error rate is your alpha (α) requirement (for multiple comparisons it would be the corrected alpha for a particular test). FW error rates refer to the probability that a family (group-on the same data) of conclusions will contain at least one Type I error. The FW error rate, if alpha is not corrected, is very dependent on the number of tests being run (or, in other words, included in a family).
(C=number of comparisons. This formula really only works for independent observations, but it is a decent estimate of FW when observations are dependent.)
2. One way to reduce the FW error rate would be to choose a select number of a priori tests to submit your data to. However, we often want to test a number of hypotheses, so the most common way to reduce the FW error rate is to use a more conservative level of α for each test. Such a correction is a Bonferroni correction whereby the alpha level is corrected to be more conservative by dividing the desired alpha requirement by the number of comparisons and then using the results for the new alpha requirement for a given comparison (e.x., original α=.05 using 5 tests; .05=5=.01; so the new alpha level a statistic/test must meet to be significant is .01). You also can reduce the FW error rate by running each test at a stringent alpha level (e.x., α=.001) that is subjectively derived (rather than the precisely calculated Bonferroni adjusted α).
4. Yes, we often adjust our alpha level for a priori tests. We do this, as was the case with post hoc corrections, to minimize the FW error rate. So, anytime, we know we will be running multiple comparisons, we known that our FW rate will be greater than our desired alpha unless we use a corrected, more conservative alpha level for each test. Thus, anytime we run multiple comparisons a priori we would determine the appropriate corrected α used to minimize FW error concerns.
5. A priori tests may always resemble post hoc tests in that the same statistical procedures may be used for either type of tests. Additionally, a priori tests may resemble post hoc tests when multiple a priori tests are chosen (as post hocs generally tend to be more numerous when conducted). For example, if you decide a priori to test all possible comparisons, there is little difference between the a priori and all the possible post hoc tests that could be done (so they are very similar in this instance).
6. FW: α=1-(1-.05)4
The approximate probability of committing a Type I (α) error (or, in other words, the FW error rate) is .1855.