PAPER OR PC?
DESIGN, IMPLEMENTATION,
AND SUCCESS
OF A
MIXED-MODE SURVEY
Nancy Whelchel, Coordinator for Survey Research
North Carolina State University
Ephraim Schechter, University Director of Assessment Poster Presentation for the 56th annual conference of the American Association for Public Opinion Research, Montreal, Quebec For more information contact: Graduating Senior Survey
North Carolina State University
May 19, 2001
Nancy Whelchel
Office of Institutional Planning and Research
Box 7002
North Carolina State University
Raleigh, NC 27695-7002
(919) 515-4184
nancy_whelchel@ncsu.edu
Survey Background
Graduating Senior Survey
Population: NCSU May 2001 graduating seniors
Proxy Measure: Those completing the Application for Degree process by May 7, 2001**
Sample: NA (Census of population)
Sample Frame: Academic department lists of graduating seniors
** The final 2001 May graduation file will not be available until mid-July, so the results presented here are preliminary. A small number of students who have not already applied to graduate may do so before graduation (May 19,2001), and some students who had applied to graduate will not actually graduate. In addition, some students will submit the survey past the May 7 deadline for this presentation.
Graduating Senior Survey
Survey Administration
Graduating Senior Survey
Survey Administration (continued)
Web versus Paper Survey:
Population
WEB |
PAPER |
TOTAL |
|
N Departments |
40 |
12 |
52 |
N students applying for May graduation* |
1829 (85.9%) |
300 (14.1%) |
2129 |
Gender** |
|
|
|
Male |
1035 (86.4%) |
163 (13.6%) |
1198 |
Female |
794 (85.3%) |
137 (14.7%) |
931 |
Race/Ethnicity*** |
|||
White |
1517 (85.3%) |
262 (14.7%) |
1779 |
Minority |
312 (89.1%) |
38 (10.9%) |
350 |
Grade Point Average**** |
|||
Mean |
3.10 |
2.92 |
3.08 |
st.dev. |
0.52 |
0.60 |
0.53 |
Median |
3.14 |
2.89 |
3.11 |
* by May 7 |
|||
** male vs female chi sq = .532 (df=1) P=.465 |
|||
*** white vs minority chi sq = 3.619 (df=1) P=.057 |
|||
**** GPA t-value = -5.71 (df=2127) P=.0001 |
Graduating Senior Survey Response Rate:
Web vs Paper
No overall difference in response rate
Chi sq = 0.077 (df=1) P=.781
Graduating Senior Survey Response Rate:
Web vs Paper (by Gender)
Web vs Paper (Male): chi sq=5.90 (df=1) P=.015
Web vs Paper (Female): chi sq=5.10 (df=1) P=.024
Web vs Paper (control for gender): CMH=0.095 (df=1) P=.758
Male vs Female (Web): chi sq=0.297 (df=1) P=.586
Male vs Female (Paper): chi sq=14.209 (df=1) P=.001
Male vs Female (control for survey mode): CMH=3.722 (df=1) P=.054
Graduating Senior Survey Response Rate:
Web vs Paper (by Race)
Marginally significant race difference, especially on web survey
Web vs Paper (White): chi sq=0.193 (df=1) P=.661
Web vs Paper (Minority): chi sq=0.019 (df=1) P=.891
Web vs Paper (control for race): CMH=0.128 (df=1) P=.721
White vs Minority (Web): chi sq=3.628 (df=1) P=.057
White vs Minority (Paper): chi sq=0.140 (df=1) P=.708
White vs Minority (control for survey mode): CMH=3.679 (df=1) P=.055
Graduating Senior Survey Response Rate:
Web vs Paper (by Grade Point Average)
(*3.11 = Median GPA)
Significant GPA difference, especially on web survey
Web vs Paper (Hi GPA): chi sq=0.025 (df=1) P=.875
Web vs Paper (Low GPA): chi sq=0.001 (df=1) P=.985
Web vs Paper (control for GPA): CMH=0.013 (df=1) P=.909
Hi vs Low GPA (Web): chi sq=16.235 (df=1) P=.001
Hi vs Low GPA (Paper): chi sq=2.894 (df=1) P=.089
Hi vs Low GPA (control for survey mode): CMH=19.102 (df=1) P=.001
Graduating Senior Survey:
Changes over time in Web vs Paper
Department Response Rates
Graduating Senior Survey:
Web vs Paper Survey Completion Rate*
*Among those submitting the survey.
We do not known how many graduating seniors partially completed the Web survey, then failed to submit it.
Graduating Senior Survey:
Web vs Paper Item Non-Response
Web (n=1125) |
Paper (n=182) |
|
Total number of possible responses (n items [77] * n respondents) |
86,625 |
14,014 |
Total number of missing values |
652 |
47 |
Item non-response rate |
0.3% |
0.8% |
N items with missing values |
77 (100.0%) |
28 (36.4%) |
Items with greatest number of missing values |
||
Q18. Did your major include a co-op, internship, practicum, student teaching, or other field experience? |
59 (5.2%) |
6 (3.3%) |
Q29b. If you could start all over would you choose the same major field of study? |
29 (2.6%) |
6 (3.3%) |
Q30. Would it be okay for us to contact you in the future if we would like more information about your responses to this survey? |
22 (2.0%) |
0 (0.0%) |
Q11c. (Rate) Research support: research support overall |
21 (1.9%) |
0 (0.0%) |
Q27b. All things considered, how would you rate the quality of instruction overall? |
20 (1.8%) |
2 (1.1%) |
Q14_12. (Rate college education contribution towards your) exercising public responsibility and community service. |
20 (1.8%) |
1 (0.6%) |
Q14_33. (Rate college education contribution towards your) personal growth. |
8 (0.7%) |
3 (1.6%) |
Graduating Senior Survey:
Web vs Paper
Number of Open-End Items Completed
(Respondents Only)
Pattern slightly different on web and paper: paper respondents most likely to complete all or no open-ended items.
Chi sq = 5.958 (df=3) P=.114
Graduating Senior Survey:
Web vs Paper
Number of Words in Open-End Responses
Web |
Paper |
||
What are the two most important things NC State could do to improve the quality of undergraduates' experience here? |
|||
1st response |
|||
% Responding |
62.8% |
68.1% |
|
Mean N words |
18.9 |
19.0 |
|
St.dev. |
21.7 |
14.3 |
|
Median N words |
12.0 |
13.5 |
|
2nd response |
|||
% Responding |
58.1% |
* |
50.5% |
Mean N words |
20.5 |
18.4 |
|
St.dev. |
27.6 |
14.5 |
|
Median N words |
13.0 |
13.0 |
|
Please use the space below to share any general comments you have about your experiences at NC State or to tell us why you were particularly satisfied or dissatisfied with any aspect of your education at NC State. |
|||
% Responding |
39.5% |
41.2% |
|
Mean N words |
55.9 |
49.8 |
|
St.dev. |
54.7 |
36.3 |
|
Median N words |
40.0 |
41.0 |
Overall pattern: No web/paper difference
*chi sq = 3.678 (df=1) P=.055
Nested College/Department Insert Surveys
(Colleges, Departments, and Curriculum Programs had the option to include their own insert survey along with the Graduating Senior Survey.)
Web |
Paper |
|
Total number of unique insert surveys |
21 |
12 |
Nested Web College/Department Surveys: |
||
Possible WEB Survey Combinations |
N |
|
Web Departments |
||
1 College insert: |
||
College of Agricultural and Life Sciences (CALS) |
5 |
|
College of Engineering (COE) |
8 |
|
1 Combined College Insert |
||
(CALS + COE + Department questions) |
1 |
|
1 College insert & 1 unique department insert |
3 |
|
1 Combined College & Dept insert |
||
(COE + Department questions) |
1 |
|
1 unique Department insert |
8 |
|
1 unique Department insert & Teacher Ed Prog insert |
2 |
|
1 unique Department insert & |
||
Teacher Ed Prog insert for specific curriculum |
3 |
|
Total Number of Web Departments with Insert Survey |
31 |
|
Total Number of Web Departments without Insert Survey |
9 |
Nested College/Department Insert Surveys:
Web vs Paper Response Rate
(among those completing the
Graduating Senior Survey)
Web (n=1125) |
Paper (n=182) |
|
Returned/submitted correct insert survey* |
92.7% |
95.6% |
Selected correct insert survey but did not successfully submit |
3.6% |
na |
Correct insert survey not done |
3.7% |
4.4% |
*Includes web respondents in departments without an insert survey who successfully selected the "none" option for the nested college/department surveys.
Summary of Results