Effect size, statistical power of the test, and confidence interval (of hypothesis testing)

I am using the following two-sample tests for non-normal distributions:

1. chi2gof
2. kstest2
3. ranksum
4. kruskalwallis

All of them return a p-value, i.e. a p-value for chi2gof, a p-value for kstest2, a p-value for ranksum, and a p-value for kruskalwallis.

Since the p-value is not enough to understand the data/distributions (for example, please see Sullivan & Feinn (2012), Dunkler et al.(2020), Greenland (2016), and du Prel et al. (2009)), I would like to calculate the effect size, the statistical power of the test, and the confidence interval (of hypothesis testing).

I would use the following MATLAB functions:

1. meanEffectSize (two-sample effect size computations)
2. sampsizepwr (power of the test)
3. not found anything about the confidence interval (except for the Two-sample t-test (ttest2) that I am not using)..

However:

1. I am not sure if both functions, meanEffectSize and sampsizepwr, can be "compatible" with (i) non-normal distributions and (ii) with the 4 previously mentioned tests.
2. According to the documentation I found on sampsizepwr, it looks like that the calculation of the statistical power of the test works only with normal distributions, but I am not sure.
3. Those 4 mentioned tests do not provide the confidence interval. Maybe other MATLAB function do, but I was not able to find them.

I have then 2 questions:

Question 1. Is there anyone who could kindly enlighten me on the "compatibility" of meanEffectSize and sampsizepw with (i) non-normal distributions and (ii) with the 4 previously mentioned tests? In other words, can I use meanEffectSize and sampsizepw for non-normal distributions and for any of the 4 mentioned statistical tests? Or, meanEffectSize and sampsizepw can be used only in some cases / with some assumptions?

Question 2. Is there anyone who could kindly tell me if there are MATLAB functions to calculate the confidence intervals for the 4 mentioned statistical tests? (So far, I found only that the Two-sample t-test (ttest2) provides the confidence interval, although I am not using that tests)

Answer 1

[Gautam]

Hello Sim,

The functions “meanEffectSize”  can be used for the non-parametric tests that you’ve mentioned. By default, the function uses Cohen’s D for effect size measure, but you can set the “effect” argument of the function to “cliff” which uses the Cliff’s Delta method or “ktest” Kolmogorov-Smirnov test statis both of which can be used for non- normal distributions. You can use bootstrapping to estimate the confidence intervals, which “meanEffectSize” supports.

However, the function “sampsizepwr” supports tests for only normally distributed data as mentioned in the MathWorks documentation for the function. For the estimation of the statistical power, it’d be better to use bootstrapping methods.

There are no straightforward ways of calculating confidence interval for non-parametric tests. However, bootstrapping or other resampling methods could be used to approximate or derive confidence intervals for the tests you mentioned.

1. For Chi-Square Goodness-of-Fit Test, you can calculate confidence intervals for proportions if you are testing categorical data. Use the “binofit” function to get confidence intervals for proportions if applicable

2. For Two-Sample Kolmogorov-Smirnov Test bootstrapping can be used to approximate confidence intervals for the test statistic

3. For Wilcoxon Rank-Sum Test, again bootstrapping to estimate confidence intervals could be used. You can also consider using the “bootci” function to compute confidence intervals through resampling.

4. For Kruskal-Wallis Test, use bootstrapping to estimate confidence intervals for group medians or effect sizes. Use the “multcompare” function after “kruskalwallis” to perform multiple comparisons, which can help infer differences between groups.

Please refer to the following MathWorks documentation for more information of these functions.

1. “binofit” for confidence intervals for proportions:

https://www.mathworks.com/help/stats/binofit.html?searchHighlight=binofit&s_tid=srchtitle_support_results_1_binofit

2. “bootci” to compute confidence intervals using bootstrapped samples

https://www.mathworks.com/help/stats/bootci.html?searchHighlight=bootci&s_tid=srchtitle_support_results_1_bootci

3. “multcompare” to perform multiple comparisons for inference differences between groups

https://www.mathworks.com/help/stats/multcompare.html?searchHighlight=multcompare&s_tid=srchtitle_support_results_1_multcompare

Hope this helps.

[Sim]

Thanks! :-)