multicmp

A function that adjusts p-values to control the Type 1 error rate during multiple comparisons.

Function File: multicmp

USAGE: [padj,alpha] = multicmp (p,option,alpha)

A set of multiple comparison tools that apply procedures to control either the family-wise error rate (FWER) or the false discovery rate (FDR) when performing post-tests. The FWER is the probability of making at least one false discovery (type 1 error) among the entire set of hypotheses when multiple testing. In contrast, the FDR is the expected proportion of false discoveries as a fraction the number of null hypothesis rejected among a set of hypotheses. For example, for an experimental set of 1000 t-tests, 50 of the tests (5 ) could be false positive (<0.05). Typically, when one conducts a statistical test one expects that a Type I error (one or more false positives) will occur once in every 20 times the test is repeated. FWER control maintains this interpretation for a set of multiple tests, such that if one repeats the 1000-test experiment 20 times, at least one of the repeats could have a false positive. FDR control permits some false positives to occur at a tolerable level (e.g. 0.05). Thus, each time one runs the 1000-test experiment, 5 of the test results considered significant will be declared false positive.

The default method for controlling the FWER is the universally applicable step-down Bonferroni procedure of Holm. This function reports multiplicity adjusted p-values [3]. Other method options available are described in detail below. They are summarised in the following list, starting with the most conservative, then becoming more liberal as one descends:

- Family-wise Error Rate (FWER) control:

* Holm-Bonferroni (type: 'down')

* Hochberg-Bonferroni (type: 'up')

- False Discovery Rate (FDR) control

* Benjamini-Hochberg (type: 'fdr')

The input takes the following arguments

'p' A column vector of raw p-values to correct for multiple comparisons.

'type' Sets the type of stepwise Bonferroni procedure ('down' or 'up') to control the FWER, or uses a procedure to control the false discovery rate ('fdr').

By default, the type is 'down', which implements Holm's universally applicable sequentially rejective step-down Bonferroni procedure [4].

If type is set to 'up', Hochberg's step-up Bonferroni procedure is implemented [5,6]. This procedure is less conservative than Holm's procedure in controlling the FWER across comparisons of positively correlated measures.

If type is set to 'fdr', the Benjamini-Hochberg procedure to control the false discovery rate is implemented at a tolerance level set to alpha [7]. This procedure is less conservative than the step-wise Bonferroni procedures but will not maintain the FWER. The FDR is maintained where tests are independent or positively correlated. The FDR is not maintained when tests are negatively correlated.

'alpha' A scalar value defining the alpha level to obtain 100*(1-alpha)% coverage of confidence intervals after multiple testing.

The output takes the following arguments

'padj' Vector of P-values adjusted for multiple comparisons.

'alpha' Vector of adjusted alpha levels for setting corrected confidence intervals. Adjusted alpha levels are available for the Holm-Bonferroni [8] and the FDR procedures [9]. For the latter, adjusted confidence intervals can only be made for tests that were significant after controlling the FDR.

The algorithms for adjusting p-values are summarised in reference [10]

[1] Welch (1947) The generalization of "Student's" problem when

several different population variances are involved.

Biometrika. 34 (1–2): 28–35.

[2] Ruxton (2006) The unequal variance t-test is an underused

alternative to Student's t-test and Mann-Whitney U test.

Behavioral Ecology. 17(4):688.

[3] Wright (1992) Adjusted P-Values for Simultaneous Inference.

Biometrics. 48,1000-1013.

[4] Holm (1979) A simple sequentially rejective multiple test

procedure. Scandanavian Journal of Statistics, 6:65-70

[5] Holland and Copenhaver (1987) An improved sequentially

rejective Bonferroni test procedure. Biometrics. 43: 417-423

[6] Hochberg (1988) A sharper Bonferroni procedure for multiple

tests of significance. Biometrika. 75: 800-802

[7] Benjamini and Hochberg (1995) Controlling the False Discovery

Rate: A Practical and Powerful Approach to Multiple Testing.

Journal of the Royal Statistical Society. Series B. 57:289-300

[8] Serlin, R. (1993). Confidence intervals and the scientific method:

A case for Holm on the range. Journal of Experimental Education.

61(4), 350–360.

[9] Benjamini and Yekutieli (1995) False Discovery Rate–Adjusted Multiple

Confidence Intervals for Selected Parameters. JASA 100(469):71-81

[10] Westfall (1997) Multiple Testing of General Contrasts Using

Logical Constraints and Correlations. JASA 92(437):299-306

multicmp v1.1.0.1 (last updated: 23/12/2019