Matlab Ttest2 changing significance level

I have two sets of patient age data that I want to compare with a ttest. The sets of data are of different lengths (one vector is 300 rows, the other is 900 rows), and I know their averages are close (30 and 35). That said, when I run the ttest with the default 0.05 significance level it fails to reject. This would normally be fine, I'll just increase the significance level to see where it eventually agrees with the null hypothesis (that their means and variances are equal). Unfortunately, even if I increase the significance level to 0.9999 it does not change, and I'm wondering if its an issue with my data, with the line of code that I've written, or what.

This is what I wrote originally:

[ha,pa] = ttest2(agepen,agegsw);

I eventually changed it to:

[ha,pa] = ttest2(agepen,agegsw,'alpha',0.999);

And the answers for ha and pa were exactly the same (a ridiculously tiny p value by the way, something times 10^-12). Is it my data set or am I miswriting the code?

Thanks a lot!

Answer 1

[Star Strider]

One option would be to use the histift function on each data set to see how they are distributed. Other options exist to test that assumption, such as the chi2gof function and others linked to in that documentation.

The Wilcoxon rank-sum test (the ranksum function) is distribution-independent, assuming only that both groups are independent and identically distributed. If it agrees with ttest2, your two groups simply might not be different.

[Joseph Novack]

Yeah I ran ranksum and it gave the exact same p value... maybe my data sets are just super different. Thanks!

[Walter Roberson]

In a situation like that I would generate a couple of test data sets of known different distributions and make sure that ttest2 is basically working -- just in case somehow the wrong ttest2 was being picked up or something like that.

[Star Strider]

@Joseph Novack — My pleasure! If ttest2 and ranksum agree, that would convince me your groups are different. Your degrees-of-freedom are such that your data — if normally distributed — will be approximated by the normal distribution, since the t-distribution approaches the normal distribution at about 30 degrees-of-freedom. Doing histfit on both might give you some idea of how your data are distributed.

@Walter — That’s possible, however the agreement of ttest2 and ranksum is enough to convince me that the results are real.