Abstract

Background: Statistical hypothesis testing is much like gambling. If, with one statistical test, your chance of a significant result is 5%, then, after 20 tests, it will increase to 40%. This result is based on the play of chance. In current clinical trials, instead of a single efficacy-variable of one treatment, multiple efficacy-variables of more than one treatment are increasingly assessed. Methods: The current paper reviews some methods for reducing the problem of falsely positive results due to multiple testing. Results and Conclusions: These methods include (1) the Bonferroni test, (2) the least significant difference (LSD) test, (3) other less conservative, more rarely used methods like Tukeys honestly significant difference (HSD) test, Dunnetts test, Student-Neuman-Keuls test, Hochbergs adjustment, and the Hotelling Q-square test. Alternative approaches to the problem of multiple testing include (4) the construct of composite endpoints, (5) no adjustment at all, but a more philosophical approach to the interpretation of the p-values, and (6) the replacement of the traditional 5% rejection level with a 1% rejection level or less. Evidence-based medicine is under pressure, because trials do not adequately apply to their target populations. As long as the effects of multiple testing are not routinely assessed in the analysis of clinical trials, it can not be excluded as one of the mechanisms responsible.

Keywords: Clinical trials, Multiple testing, Multiple comparisons, Type I error, Evidence-based medicine