PAPER OR PC?
DESIGN, IMPLEMENTATION,
AND SUCCESS
OF A
MIXED-MODE SURVEY

 

Nancy Whelchel, Coordinator for Survey Research
North Carolina State University

Ephraim Schechter, University Director of Assessment
North Carolina State University

Poster Presentation for the 56th annual conference of the American Association for Public Opinion Research,

Montreal, Quebec
May 19, 2001

For more information contact:
Nancy Whelchel
Office of Institutional Planning and Research
Box 7002
North Carolina State University
Raleigh, NC 27695-7002
(919) 515-4184
nancy_whelchel@ncsu.edu


Graduating Senior Survey

Survey Background


Graduating Senior Survey

Population: NCSU May 2001 graduating seniors

Proxy Measure: Those completing the Application for Degree process by May 7, 2001**

Sample: NA (Census of population)

Sample Frame: Academic department lists of graduating seniors

** The final 2001 May graduation file will not be available until mid-July, so the results presented here are preliminary. A small number of students who have not already applied to graduate may do so before graduation (May 19,2001), and some students who had applied to graduate will not actually graduate. In addition, some students will submit the survey past the May 7 deadline for this presentation.


Graduating Senior Survey

Survey Administration


Graduating Senior Survey

Survey Administration (continued)


Web versus Paper Survey:
Population

WEB

PAPER

TOTAL

N Departments

40

12

52

N students applying for May graduation*

1829 (85.9%)

300 (14.1%)

2129

Gender**

Male

1035 (86.4%)

163 (13.6%)

1198

Female

794 (85.3%)

137 (14.7%)

931

Race/Ethnicity***

White

1517 (85.3%)

262 (14.7%)

1779

Minority

312 (89.1%)

38 (10.9%)

350

Grade Point Average****

Mean

3.10

2.92

3.08

st.dev.

0.52

0.60

0.53

Median

3.14

2.89

3.11

* by May 7

** male vs female chi sq = .532 (df=1) P=.465

*** white vs minority chi sq = 3.619 (df=1) P=.057

**** GPA t-value = -5.71 (df=2127) P=.0001


Graduating Senior Survey Response Rate:
Web vs Paper

No overall difference in response rate
Chi sq = 0.077 (df=1) P=.781


Graduating Senior Survey Response Rate:
Web vs Paper (by Gender)


Significant gender difference on paper survey but not on web survey

Web vs Paper (Male): chi sq=5.90 (df=1) P=.015
Web vs Paper (Female): chi sq=5.10 (df=1) P=.024
Web vs Paper (control for gender): CMH=0.095 (df=1) P=.758
Male vs Female (Web): chi sq=0.297 (df=1) P=.586
Male vs Female (Paper): chi sq=14.209 (df=1) P=.001
Male vs Female (control for survey mode): CMH=3.722 (df=1) P=.054


Graduating Senior Survey Response Rate:
Web vs Paper (by Race)

Marginally significant race difference, especially on web survey

Web vs Paper (White): chi sq=0.193 (df=1) P=.661
Web vs Paper (Minority): chi sq=0.019 (df=1) P=.891
Web vs Paper (control for race): CMH=0.128 (df=1) P=.721
White vs Minority (Web): chi sq=3.628 (df=1) P=.057
White vs Minority (Paper): chi sq=0.140 (df=1) P=.708
White vs Minority (control for survey mode): CMH=3.679 (df=1) P=.055


Graduating Senior Survey Response Rate:
Web vs Paper (by Grade Point Average)

(*3.11 = Median GPA)

Significant GPA difference, especially on web survey

Web vs Paper (Hi GPA): chi sq=0.025 (df=1) P=.875
Web vs Paper (Low GPA): chi sq=0.001 (df=1) P=.985
Web vs Paper (control for GPA): CMH=0.013 (df=1) P=.909
Hi vs Low GPA (Web): chi sq=16.235 (df=1) P=.001
Hi vs Low GPA (Paper): chi sq=2.894 (df=1) P=.089
Hi vs Low GPA (control for survey mode): CMH=19.102 (df=1) P=.001


Graduating Senior Survey:
Changes over time in Web vs Paper
Department Response Rates


Graduating Senior Survey:
Web vs Paper Survey Completion Rate*

*Among those submitting the survey.
We do not known how many graduating seniors partially completed the Web survey, then failed to submit it.


Graduating Senior Survey:
Web vs Paper Item Non-Response

Web

(n=1125)

Paper

(n=182)

Total number of possible responses

(n items [77] * n respondents)

86,625

14,014

Total number of missing values

652

47

Item non-response rate

0.3%

0.8%

N items with missing values

77 (100.0%)

28 (36.4%)

Items with greatest number of missing values

Q18. Did your major include a co-op, internship, practicum, student teaching, or other field experience?

59 (5.2%)

6 (3.3%)

Q29b. If you could start all over would you choose the same major field of study?

29 (2.6%)

6 (3.3%)

Q30. Would it be okay for us to contact you in the future if we would like more information about your responses to this survey?

22 (2.0%)

0 (0.0%)

Q11c. (Rate) Research support: research support overall

21 (1.9%)

0 (0.0%)

Q27b. All things considered, how would you rate the quality of instruction overall?

20 (1.8%)

2 (1.1%)

Q14_12. (Rate college education contribution towards your) exercising public responsibility and community service.

20 (1.8%)

1 (0.6%)

Q14_33. (Rate college education contribution towards your) personal growth.

8 (0.7%)

3 (1.6%)


Graduating Senior Survey:
Web vs Paper
Number of Open-End Items Completed
(Respondents Only)

Pattern slightly different on web and paper: paper respondents most likely to complete all or no open-ended items.

Chi sq = 5.958 (df=3) P=.114


Graduating Senior Survey:
Web vs Paper
Number of Words in Open-End Responses

Web

Paper

What are the two most important things NC State could do to improve the quality of undergraduates' experience here?

1st response

% Responding

62.8%

68.1%

Mean N words

18.9

19.0

St.dev.

21.7

14.3

Median N words

12.0

13.5

2nd response

% Responding

58.1%

*

50.5%

Mean N words

20.5

18.4

St.dev.

27.6

14.5

Median N words

13.0

13.0

Please use the space below to share any general comments you have about your experiences at NC State or to tell us why you were particularly satisfied or dissatisfied with any aspect of your education at NC State.

% Responding

39.5%

41.2%

Mean N words

55.9

49.8

St.dev.

54.7

36.3

Median N words

40.0

41.0

Overall pattern: No web/paper difference
*chi sq = 3.678 (df=1) P=.055


Nested College/Department Insert Surveys

(Colleges, Departments, and Curriculum Programs had the option to include their own insert survey along with the Graduating Senior Survey.)

Web

Paper

Total number of unique insert surveys

21

12

Nested Web College/Department Surveys:

Possible WEB Survey Combinations

N

Web Departments

1 College insert:

College of Agricultural and Life Sciences (CALS)

5

College of Engineering (COE)

8

1 Combined College Insert

(CALS + COE + Department questions)

1

1 College insert & 1 unique department insert

3

1 Combined College & Dept insert

(COE + Department questions)

1

1 unique Department insert

8

1 unique Department insert & Teacher Ed Prog insert

2

1 unique Department insert &

Teacher Ed Prog insert for specific curriculum

3

Total Number of Web Departments with Insert Survey

31

Total Number of Web Departments without Insert Survey

9


Nested College/Department Insert Surveys:
Web vs Paper Response Rate
(among those completing the
Graduating Senior Survey)

Web

(n=1125)

Paper

(n=182)

Returned/submitted correct insert survey*

92.7%

95.6%

Selected correct insert survey but did not successfully submit

3.6%

na

Correct insert survey not done

3.7%

4.4%

*Includes web respondents in departments without an insert survey who successfully selected the "none" option for the nested college/department surveys.


Summary of Results

  • Response Rate
    • No overall difference in web vs paper
      • Gender
        • Women higher than men on paper
        • Women higher on paper than on web
        • Men higher on web than on paper
      • Race
        • Whites slightly higher than minorities on web
        • Whites similar on web and paper
        • Minorities similar on web and paper
      • Grade Point Average
        • Hi GPA higher than low GPA on web and paper
        • Hi GPA similar on web and paper
        • Low GPA similar on web and paper
  • Survey Completion Rate
    • No difference in web vs paper
  • Item Non-Response
    • Item non-response rate very low on both web and paper
    • More individual items skipped on web than paper
    • Most frequently skipped items same on web and paper
  • Open-End Item Responses
    • Similar number of open-end items answered on web and paper
    • Similar average number of words on web and paper
  • Nested College/Department Insert Surveys
    • No difference in response rates of web vs paper insert surveys