Assessing the Impact of Technology on Student Learning:
An Institutional Initiative
Copyright North Carolina State University, 2006. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.
Assessing the Impact of Technology on Student Learning:
An Institutional Initiative
Joni Spurlin, Stan North Martin,
Sharon Pitt, Karen Helm,
and Allen Dupont
NC State University 

Session Outcomes
•	By the end of the session, participants will be able to develop or improve their assessment efforts by: 
–	Defining issues that need to be addressed when assessing technology’s impact on student learning;
–	Identifying specific assessment methods that would be applicable to their institution; and
–	Using the available list/handout of resources.

Session Overview
•	Learning in a Technology-Rich Environment – (LITRE)– Pre-Planning for Quality Enhancement Plan, SACS (5 min)
•	The LITRE Plan (25 min)
–	Implementation Plan
–	Assessment Plan
•	Results of Assessment (10 min)
•	Lessons Learned (10 min)
•	Issues (10 min)

Session Overview
•	LITRE as a Quality Enhancement Plan
•	The LITRE Plan
–	Implementation
–	Assessment
•	Results of Assessment
•	Lessons Learned
•	Issues
•	Questions
LITRE Planning
•	LITRE as an outgrowth of accreditation review
–	Plan for transformative, institutional improvement
–	Crucial to enhancing educational quality
–	Directly related to student learning
–	Based on comprehensive analysis of institutional effectiveness
LITRE Planning
•	Initial Benchmarking
–	Defining the “learning with technology” environment
–	Focus Groups
–	Critical Infrastructure Needs (we’ll discuss later...)
–	2003 Faculty Survey
LITRE Faculty Survey (2003)
•	Why:  Inform recommendations of LITRE and provide baseline for future LITRE efforts
•	Who and What:  Faculty were surveyed about their experiences with computer-based instructional and learning aids. 1,790 faculty were invited to participate in the survey. 983 did—a response rate of 55%.
•	Indicator:  Respondents were asked what would make it easier to use the technologies that they did use in their courses:  “If they were available and supported in the classrooms in which I typically teach” was chosen most often, 37% of the time.

•	See http://litre.ncsu.edu for survey report and instrument.

The Essence of the LITRE Plan
•	Scholarly inquiry focused on enhancing the technology-rich learning environment
•	Investigative process through which new approaches to student learning, using technology, are proposed, vetted, empirically evaluated, and if the evaluation results indicate, deployed and routinely assessed
•	Evidence would be collected and analyzed to inform future projects
First Wave Initiatives
•	Classroom and Laboratory Improvements
–	University-wide Classroom Improvement Plan
–	Student Group Collaboration (FlySpace)
–	ClassTech projects
–	SCALE-UP Classroom 
•	Faculty Innovation Grants
–	LITRE grants

LITRE Assessment
•	Overarching Goals
•	Focused on assessment related to student learning
•	Assessment Methods
•	Support through LITRE Assessment Committee
LITRE Goals
•	Improve student learning
•	Systematically investigate effectiveness of technology-based innovations in learning and teaching
•	Use results to scale our successes, shape future investigations and inform campus decision making
Student Learning Assessed: 
Four Dimensions Defined
•	Problem Solving
•	Empirical Inquiry
•	Research from Sources
•	Performance in the discipline
•	See LITRE Plan, Appendix A for definitions: http://litre.ncsu.edu/

Assessment Methods for Goals
•	Faculty Survey
–	Was used in developing LITRE
–	Will be conducted periodically to look for improvements and other issues
–	Will include faculty perceptions of how student learning improved in 4 dimensions
•	Student Surveys
–	Student perceptions on the 4 dimensions of student learning
•	Alumni Survey
•	Summary of LITRE Projects (LITRE-Like Projects) & Grants
–	Lessons learned
–	Improvement of student learning

Discussion Questions
•	What issues, related to assessment, do you think our goals have raised?

•	If you were given the task of developing goals for an institution-wide assessment of technology related to student learning – what goals would you develop?

LITRE Assessment Committee
•	Faculty
•	Assessment Professionals
•	Computer/Information Technology Professionals
•	Available to help with assessment design of Individual Faculty Proposals (http://litre.ncsu.edu/dfiles/people.html)
Assessment of Each of LITRE’s Projects Summarized in Annual Report
Projects:
•	Improving Classroom Technology throughout campus
•	ClassTech
•	Flyspace
•	G108 – Scale-UP
•	Individual Faculty Grants
Annual Report:
•	What have we learned?  What are the best technologies, processes, pedagogies, etc. to move forward?
•	In what areas has student learning improved?  Can we tie to LITRE projects?
•	Use information to plan next steps …close the loop!
Assessment of Individual 
Faculty Grants
•	Each Faculty Grant MUST include assessment activities
–	“How will I know if I accomplished my goals?”
–	“How does the technology and pedagogy affect student learning?”
•	PI does the assessment work, but the LITRE Assessment Committee members are available to consult with PIs
Assessment of ClassTech
•	How does the technology help students achieve course/program objectives?
•	How does the technology affect how faculty members teach and how students learn?
•	Determine needs and adequate support and training for faculty members to use the technology effectively.

•	Initial Data to include:
–	Equipment usage
–	Operability/Maintenance
–	Training 
–	Faculty and student perceptions 
on workload and learning
–	Course-based data on improvement 
in student learning
Modified ClassTech Assessment 2005-2006
•	How does use of technology impact course’s: 
–	pedagogy 
–	faculty workload 
–	faculty attitudes 
–	amount of material delivered
•	How does having the technology in the classroom affect:
–	use of class time and assigned coursework
–	how students learn (following LITRE defined outcomes)
–	student achievement of course and program objectives
•	Are students, faculty and technical staff satisfied with the use of this technology in academic settings?  
•	What are the challenges of using technology in the classroom for students, faculty, and technical staff?

Assessment Methods 2005-2006
•	ClassTech Assessment Team to define model that incorporates 
–	technology
–	pedagogy
–	learners
–	environment 
–	outcomes
•	Survey faculty who use rooms (More in-depth in spring)
•	Automatic tracking of usage in some of the rooms

•	Interviews with faculty and classroom observations in 20 courses 
•	Sample of student work in observed classes to assess student learning
•	Student surveys in selected classrooms
•	Focus group with 
technical staff
•	Review of problem call tracking logs
ClassTech results – 2004-2005
•	Overall favorable perceptions of technology use on faculty workloads
•	No significant evidence of direct impact on pedagogy or assessment methods
•	Almost half (48%) of faculty respondents felt pace was faster with technology
•	Half said technology allowed for wider variety of topics
•	Most (61%) said they were able to cover material in more depth
ClassTech results – 2004-2005
•	Most faculty felt students were more engaged in class when teaching with technology compared to teaching without it
ClassTech results – 2004-2005
•	In-class surveys indicated 81% felt it positively affected their learning
•	Most students surveyed in classes prefer 
moderate (71%) or extensive (21%) use of instructional technology
•	2004 Sophomore/Senior surveys generally indicate effect on learning is either same or better, depending on methods used
•	See www.ncsu.edu/classtech/survey_results/ 
Critical Infrastructure Needs
•	Classroom Improvement
•	Faculty Computing
•	File Space Quota
•	Software Licensing
•	Learning Management Systems
•	Digital Asset Management
•	Student E-Portfolios
•	Technology Support for Students

•	Faculty Innovation Grants
•	Information Exchange
•	Accessibility and Universal Design
•	Wireless Data Connectivity and Mobile Computing/Communication Systems
•	Advanced Remote Access Services
Results Related to Student Learning
Senior student survey: 
•	The highest increase was seen in use of computerized exams:
–	USE: 23% in 03/04 increased to 30% in 04/05 
–	LEARNED BETTER: 18% in 03/04 increase to 28% in 04/05
From LITRE projects: 
•	Only a few results related to student learning from 2004-2005 efforts because:
–	Time needed for Infrastructure
–	Time for faculty to incorporate technology into coursework
–	Time to develop DIRECT assessment methods of student learning (not rely on just indirect methods such as surveys)
•	Majority of faculty felt that the pace, variety and depth of their course has been increased and students were more involved in learning. 
•	Need to Modify Question: the interaction of technology as a tool, faculty’s pedagogy and student use of the technology on student learning. 

Lessons Learned - Overall:
•	Enables innovations in teaching and learning
•	Diffuses innovation among mainstream faculty
•	LITRE, LITRE, LITRE on the label, label, label 
•	Helps improve teaching
•	Helps improve understanding of assessment
•	Involve everyone!

Lessons Learned: Our Next Steps
•	Modifications:
–	Revisit overarching questions: What pedagogical issues are we trying to solve?  How can technology help address challenges? 
–	Focus assessment: Conduct fewer projects well and do meta-syntheses
–	Increase time and resources in assessment
Lessons Learned—Assessment 
•	Obtain baseline data
•	Develop Assessment Committee that actively engages community in discussions
•	ASK: What do we want to learn from assessment? What will it tell us?
•	Overestimate time and resources
•	Get faculty involved in assessment
•	Provide support and training for assessment

Success Factors
•	Institutional investment
•	External driver (SACS)
•	Team of professionals
•	One or two champions of the process
•	Communication
•	Develop culture
•	Brand identification – LITRE
Issues From Each of Our Perspectives
•	Joni – Assessment Issues
•	Stan – Classroom Improvement
•	Sharon – Learning Technologies
Questions?
Resources
•	Quality Enhancement Plan for Learning in a Technology-Rich Environment at NC State:
http://litre.ncsu.edu/pdf/litre_qep.pdf
•	LITRE Goals and Assessment Plan: http://litre.ncsu.edu/dfiles/goals_short.html
•	2003 LITRE Faculty Survey Report:
http://litre.ncsu.edu/fac_sur.pdf
•	Classroom Technology @ NC State:
http://www.ncsu.edu/classtech/
•	Other Session Documents:
http://www.ncsu.edu/classtech/workshops/eli2006/
•	Resources on assessment of technology related to student learning: http://www2.acs.ncsu.edu/UPA/assmt/litre/
Contact Information
Joni E. Spurlin, Ph.D.
University Director of Assessment
University Planning and Analysis
Joni_Spurlin@ncsu.edu
http://www2.acs.ncsu.edu/UPA/assmt/

Stan North Martin
Associate Director, Computing Services
Information Technology Division
stan_martin@ncsu.edu
http://itd.ncsu.edu/

Sharon P. Pitt
Associate Vice Provost
Learning Technologies
sharon_pitt@ncsu.edu
http://lts.ncsu.edu

Karen Helm
Director of University Planning and Analysis
Karen_Helm@ncsu.edu
http://www2.acs.ncsu.edu/UPA/

Allen Dupont, Ph.D.
Director of Assessment,
Division of Undergraduate Academic  Programs
Allen_Dupont@ncsu.edu
http://www.ncsu.edu/assessment/assess.htm