Effect Sizes in the NSSE 2001 Means Summary Report

When we compare NC State results to those from all NSSE respondents or from all respondents from Doctoral/Research Extensive institutions, the analyses use data from a large number of respondents. In this situation, even small differences may be statistically significant. Effect size is a measure of difference that is not affected this way by the number of respondents used in the comparison.

The NSSE 2001 Overview document on the 2001 Table of Contents page says:

Because of the large numbers of students in NSSE 2001, we set a very high statistical significance threshold to reduce the probability that the differences noted are due to chance (p < .01 for consortia comparisons, p<.001 for Carnegie and national comparisons). Even so, the actual magnitude of some item score differences may seem trivial, even though they are highly reliable and statistically significant. For this reason we also report the effect size associated with those item comparisons that are statistically significant. The effect size represents the magnitude of the discrepancy in the student or institutional behavior represented by the item. When the effect size is large, or a pattern of moderate effect sizes exists, it's likely that the quality of the student experience represented by the survey question(s) is appreciably different and, therefore, may be of practical as well as statistical significance.

• .20 is a small effect
• .50 is a medium effect
• .80 is a large effect

(For technical information about effect sizes and their relationship to statistical significance measures, see Jacob Cohen, Statistical power analysis for the behavioral sciences, revised edition. Academic Press, 1977.)