Adapt Courseware at GTC

Evaluation Report
Holly Sanders, Russ Palmer, Michael Smith, Sarah Bolen Marston

ABSTRACT This report documents an evaluation of Adapt Courseware as implemented at Gwinnett Technical College, and includes background information on the project, an analysis of our findings, presentation and discussion of the results, and recommendations for the client.

[ADAPT COURSEWARE AT GTC] 2 Table of Contents Executive Summary ...................................................................................................... 4 Introduction ................................................................................................................... 5 Background ................................................................................................................... 5
Adapt Courseware .............................................................................................................. 6

Purpose .......................................................................................................................... 9 Stakeholders .................................................................................................................. 9 Decisions ........................................................................................................................ 9 Questions ....................................................................................................................... 9 Methods ....................................................................................................................... 10
Time ..................................................................................................................................... 11 Budget ................................................................................................................................ 11 Communication ................................................................................................................. 11 Student Participation ......................................................................................................... 12 Reliability and Validity of Instruments .............................................................................. 12

Analysis ........................................................................................................................ 12
Data Analysis: Student Survey .......................................................................................... 12 Data Analysis: Instructor Survey....................................................................................... 12 Data Analysis: Final Exam Scores .................................................................................... 13 Data Analysis: Course Data Review ................................................................................. 13

Results........................................................................................................................... 13
Instructor Survey................................................................................................................. 14 Final Exam Scores .............................................................................................................. 16 Mean .................................................................................................................................... 17 Standard Deviation ............................................................................................................ 17 t-Test ..................................................................................................................................... 18 Course Data Review: Retention and Success Rates from Summer 2013 ....................... 18

Discussion .................................................................................................................... 20
Affordability ........................................................................................................................ 20 Student and Instructor Reactions ...................................................................................... 20 Effectiveness of the Product .............................................................................................. 21

Recommendations ..................................................................................................... 21
Recommendations based on the question: Is Adapt Courseware an affordable product for students enrolled at Gwinnett Technical College? ..................................... 22 Recommendations based on the question: What are the faculty and student reactions to the product thus far? ..................................................................................................... 22 Recommendations based on the question: How effective has the product been thus far? ...................................................................................................................................... 22 Recommendations Summary ........................................................................................... 23

Appendix A: Original GTC Grant Proposal ............................................................... 24

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 3 Appendix B: Student Survey....................................................................................... 24

Survey Questions................................................................................................................ 24 Full Student Survey Results................................................................................................. 26 Questions ............................................................................................................................ 34 Reaction .............................................................................................................................. 34 Effectiveness........................................................................................................................ 34 Cost ...................................................................................................................................... 35 Full Instructor Questionnaire Responses ........................................................................... 35

Appendix C: Instructor Questionnaire ...................................................................... 34

Appendix D: Student Survey Matrix.......................................................................... 42 Appendix E: Instructor Questionnaire Matrix........................................................... 44

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 4

Executive Summary
The purpose of this evaluation was to determine the appropriateness of continued use of the Adapt Courseware software currently in use at Gwinnett Technical College (GTC). Primary stakeholders of this evaluation include Sociology and Psychology faculty and students at GTC. This evaluation focused on one primary decision: Should the Sociology and Psychology departments at GTC continue or discontinue the use of Adapt Courseware beyond the initial grant period? In order to facilitate this decision, the evaluation considered the following questions: Is Adapt Courseware an affordable product for students enrolled at Gwinnett Technical College? How effective has the product been thus far? What are the faculty and student reactions to the product thus far? The evaluation team utilized a mixed methods approach in order to address those questions, including the following data collection methods: instructor questionnaire student survey review of general course data (focusing on student retention and student success rates) review of final exam scores both from courses utilizing Adapt Courseware and those using more traditional online instruction in use at GTC The results of the evaluation indicate that: Adapt Courseware does not offer significant cost savings over other online courses in use at GTC. Adapt Courseware is actually less effective than other online courses in use at GTC. Although student reactions to the Adapt Courseware were quite positive overall, instructor reactions were extremely negative. Based on these results, the evaluation team does not recommend the continued use of the Adapt Courseware software as its primary online learning tool for either Sociology or Psychology courses. Given the potential benefits to some students of the remediation and practice offered by Adapt Courseware, however, we do recommend that GTC explore possibilities for offering the product as a supplementary learning tool for practice and definition mastery.

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 5

Introduction
This evaluation report has been developed for the faculty of the Sociology and Psychology departments at Gwinnett Technical College. The report describes the purpose, stakeholders, decisions, questions, methods, instrumentation and constraints that guided the evaluation as well as a detailed analysis or our findings, a presentation of results, and recommendations for the client. The central questions that guided this project are based on concerns expressed by the GTC Sociology and Psychology faculty as well as information gathered from the Adapt Coursework proposal from Gwinnett Technical College. The evaluation team consists of Sarah Bolen Marston, Russell Palmer, Michael Smith, and Holly Sanders.

Background
Our team evaluated the effectiveness of the Adapt Courseware software program that is currently being used by the Sociology department at Gwinnett Technical College. Adapt Courseware was implemented into GTCs Introduction to Sociology course (SOCI 1101) in the summer of 2013. The courseware was introduced as part of a research grant in which GTC will offer outcome data back to the software company, including the results of their research of the product. While this research grant is an exciting opportunity for GTC and its students, the needs and thoughts of the Sociology faculty were not considered in the Adapt acquisition and implementation. The project leader (Eric Rodriquez) has voiced concerns that the Adapt Courseware does not offer an improved learning environment over the already established courses. Additionally, the entire faculty has expressed concerns that Adapt Courseware does not allow for enough instructor control and is too time consuming for students, leaving the faculty unable to add any supplemental assignments. Another concern is that Adapt Courseware offers only definitions and does not contain any problem solving or critical thinking activities. Finally, some instructors feel that Adapt Courseware is more expensive for students than the established course materials.

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 6

Adapt Courseware
Adapt Courseware is housed within the GTCs ANGEL learning management system. It consists of six units, which are broken into twenty-one sections that contain a total of eighty lessons. The courseware was created to generate mastery learning of each section before students can move to the next section. The basic design idea is to support students that need more time with the material while letting faster moving students progress at a pace that keeps them engaged. Opening screen:

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 7 The following images present the interface of the Adapt Courseware.

Each lesson contains three instructional elements; Read, Watch, and Practice. The following images contain examples of each element. Watch Example:

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 8 Read Example:

Practice Example:

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 9

Purpose
The purpose of this evaluation is to provide GTC Sociology and Psychology faculty members with information required to support decisions regarding the continued use of Adapt Courseware after the completion of the research study.

Stakeholders
The clients for this evaluation are the faculty at GTC who are using the Adapt Courseware for their SOCI 1101 class. Other primary stakeholders include the GTC students enrolled in SOCI 1101 courses using Adapt Courseware. The Academic Affairs department at Gwinnett Technical College is a secondary stakeholder in this evaluation. Although the evaluation focused on the faculty and students involved, Academic Affairs (which made the decision to implement the research grant) will also be involved in the decision to continue or discontinue use of Adapt Courseware after the grant.

Decisions
The information, analysis, and recommendations within this evaluation will assist Gwinnett Technical College in deciding whether it is worthwhile to continue using Adapt Courseware once the current grant for the project has ended. This was identified as the primary decision to be made about the product, and it guided the design and implementation of this evaluation.

Questions
In order to support informed decision-making about the continued use of Adapt Courseware, the following questions were addressed during this evaluation. 1. Is Adapt Courseware an affordable product for students enrolled at Gwinnett Technical College? a. What is the cost compared to alternative media (including textbooks) that students would need to purchase for the course if Adapt Courseware was not being used? b. What are faculty and student reactions to the cost of Adapt courseware?

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 10 2. How effective has the product been thus far? a. What were the original goals when the product was implemented, and have those goals been met? b. How has student performance been affected by the product? c. How has the implementation of the product affected student retention (the number of students who remain in the course compared to the number who drop the course)? 3. What are the faculty and student reactions to the product thus far? a. Do faculty feel that the product is effective? i. Do faculty feel that students learn from the product? ii. Do faculty feel that the product is easy to use? b. Do students feel that the product is effective? i. Do students feel that they learn from the product? ii. Do students feel that the product is easy to use?

Methods
A mixed methods evaluation design and multiple data collection strategies were used to address these questions. The data collection strategies include: 1. 2. 3. 4. Student Survey Instructor Questionnaire Final Exam Scores Course Data Review Is Adapt Courseware an affordable product for students? x x x x How effective has the product been thus far? What are the faculty and student reactions to the product thus far? x x

Questions

Methods
Student Survey Instructor Questionnaire Final Exam Scores Course Data Review

Instrumentation

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 11 For this evaluation the evaluation team analyzed retention rates and test scores (provided by GTC) (see Results: Final Exam Scores) before and after the implementation of Adapt Courseware. Additionally, we used an online survey created with Qualtrics to collect reactionary data from students using Adapt Courseware (see Appendix B). This survey used a combination of Likert scale ranking questions and open-ended survey questions to collect data on students perceived success and difficulties using the software. A survey consisting of several free response questions was used to collect reactionary data from the four instructors participating in the implementation (see Appendix C). The instructor survey was distributed via email and was designed to collect details on the challenges instructors face, as well as the benefits of using the software in their classrooms.

Limitations
Several limiting factors impacted this evaluation.

Time
Ideally, an evaluation of this nature would be conducted over an entire semester, or possibly more than one. Certain challenges and benefits may not present themselves until after the Courseware has been fully adopted and in use for some time. Since we were limited to the timeframe of our current semester, we had to keep the scope of this evaluation fairly narrow, providing analysis of a snapshot from the middle of the Adapt Courseware trial. Additionally, our participants had limited time to spend on the evaluation, as they were all busy leading their own classrooms, or participating in their course work.

Budget
In-depth evaluations, especially those that collect qualitative data, are often quite costly. Since we had no budget to work with, we were limited to the tools and techniques we could implement for free and the work we could afford to complete pro bono.

Communication
Communication with all students needed to be conducted electronically, as the students and evaluators were separated by distance. Since the course in which the software is being tested is a fully online course, we could not meet in person with these students to conduct focus groups or face-to-face interviews. November 16th, 2013

[ADAPT COURSEWARE AT GTC] 12

Student Participation
We received survey responses from 9 students out of a population of 37. Although this is close to a 25% response rate, more responses would have provided a more reliable analysis. Furthermore, we could not gather feedback from students who had already dropped the course, or incentivize feedback from students who were unmotivated to complete the survey, or busy struggling to catch up. This may have skewed results of the student survey.

Reliability and Validity of Instruments

Budget and time constraints prevented our team from conducting a scientific evaluation of the reliability and validity of the instruments used in this evaluation.

Analysis
We used a mixed-methods evaluation approach to analyze Adapt courseware. We analyzed the student survey, final exam scores, and retention rates using quantitative methods, and relied on a qualitative approach to analyzing the instructor survey. Using mixed methods allowed us to identify recurrent and divergent patterns across the data that helped guide the formation of our recommendations. A brief discussion of the analysis for each instrument follows.

Data Analysis: Student Survey

Qualtrics, an online data collection tool, provided valuable analytical data on the student survey results, including information on the mean, variance, and standard deviation for each question. Some manual calculation was also performed on the survey results to determine the percentage of responses in particular response categories or sets of response categories for the purpose of illustrating general trends.

Data Analysis: Instructor Survey

All instructor answers were collated and read through twice before further assumptions were made or actions taken. On the third reading, commonly occurring themes were highlighted and interesting trends noted. Following the highlighting of interesting trends, any occurrences of specific complaints or observations were noted and each instance was tallied. This allowed us to get a feel for the overall trends. Any dissenting views expressed were noted. Finally, we compared the trends mentioned in the instructor responses to those obtained by other evaluation methods, including test results, retention rates, and the students survey to gain a picture of the overall effectiveness of this software.

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 13

Data Analysis: Final Exam Scores

Final exam scores were analyzed from courses using Adapt and a control course from the summer of 2013. This consisted of determining the mean of each exam as well as the standard deviation and t-scores.

Data Analysis: Course Data Review

The Institutional Effectiveness group at GTC provided course data for the Adapt course and the control course from the summer of 2013. This data consisted of the distribution of letter grades for each course, the number of students who withdrew from each course, and the number of students who passed and failed each course. We analyzed this data to determine the retention rate (percentage of students who remained in the course until the end), and the success rate (percentage of students who passed the course).

Results
Student Survey
The student survey was sent out to 37 students, 9 of whom submitted responses before the survey was closed (providing a 25% response rate). Student survey responses were overwhelmingly positive; questions about the design and interface of the course software and about the effectiveness of the learning experience elicited no negative responses, with 67% - 89% of respondents answering either positively or very positively. More students felt that the online course readings and practice activities contributed to their learning than the online course videos, although feedback was still very positive overall for all course components. Most respondents agreed that they felt challenged to learn in the course, although as many were ambivalent about that question as strongly agreed. 67% of respondents said that they would be likely to take additional courses in this format compared to traditional online courses, and 78% said that they would be likely to recommend this course to other students. With regard to pricing, most respondents said that the cost of the course was comparable to other online courses, with only 34% saying that the cost was higher. For full student survey responses, see Appendix B.

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 14

Instructor Survey
The Instructor Survey was disseminated to all 4 instructors participating in the Adapt trial, and we received a 100% response rate. Thus, the following results include all four instructors feedback on all topics. All four instructors surveyed had similar complaints, concerns, and observations regarding the ADAPT Courseware used in their course. Though specific details in the nature of their frustration with the courseware varied from instructor to instructor, they all shared the following main observations most frequently: Lack of question depth, content depth, and feedback provided: Instructors all mentioned (in some cases several times throughout the survey) that the content and quiz questions lacked depth. They claim that the content contains only definitions with little real world application. They observe that when students try to apply this knowledge to a real world setting it does not translate. Also included with this concern is the lack of feedback and scaffolding provided by the course no coaching or remediation is included. A side issue associated with the problem of content depth was content accuracy some instructors expressed concerns that some questions contained errors. Increase in students dropping/quitting/failing all instructors agree that since implementing the ADAPT Courseware their drop out rates and the number of students failing the course have increased. Log-in/access issues, technical problems, and lack of response from help desk several instructors mentioned students having problems accessing or logging in to the software, and some mentioned an unhelpful or unavailable help desk staff at ADAPT. It is important to note that a few instructors mentioned that despite initial problems with access, once they were familiar with the software they did not have further technical difficulties, while others appeared to have consistent problems throughout. A subset of this complaint is the fact that there were browser and platform compatibility issues the courseware was not accessible on Internet Explorer or on iPads. Another issue related to this is that the software conflicts with the ANGEL LMS software in certain functions, such as their discussion boards. Unequal or unfair for diverse students/disproportionately difficult for some students Many times, instructors mentioned that any students lacking technical skills quickly fell behind. Also of note were observations that the Courseware prevents struggling students from moving ahead without providing adequate help and the possibility that students who are not majoring in Psychology may be at a disadvantage using the courseware. An instructor mentioned a huge gap between students with As and Bs in the course and the failing students (noting there was little middle ground).

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 15 Program is time consuming All instructors agree that the workload and the amount of busy work students had to complete is higher with Adapt than traditional courses, and was not beneficial in their eyes. Beneficial to certain students (self-motivated and who work hard from the beginning, or Psychology majors) an interesting positive observation on the courseware that appeared several times in instructor responses is that the Courseware does appear to be beneficial to certain students particularly those majoring in Psychology and those who are highly selfmotivated and able to begin the course with a great deal of diligence. However, one instructor noted that these students are able to fly through the courseware, which may suggest a lack of challenge. Repetitive several instructors observed that the program is repetitive. It seems this stems primarily from the fact that the program will repeat any questions the student fails several times throughout the course, even beyond the unit they missed the question in. Lack of opportunity to teach, add value, assign outside assignments, customize, etc. instructors expressed some concern over the fact that they cannot customize the Adapt content, or add to it. An instructor observed that this is made worse by the fact that the program takes so much of the students time that they cannot assign any outside work or application assignments. Cost We asked about cost and actually received conflicting responses. Some say it is more expensive than the current courseware, others say less expensive. Most instructors agreed that the courseware is not worth its cost, however.

The following table and bar graph contain the most common observations mentioned in the instructor surveys organized by the number of times the issue was mentioned:
# Instances
10 10 7 6 6 6 5 3 3 2 2

Observation
Lack of question depth, content depth, and feedback provided Increase in students dropping/quitting/failing Log-in/access issues, technical problems, and lack of response from help desk Unequal or unfair / disproportionately difficult for some students Program is time consuming Beneficial to certain students Repetitive Lack of opportunity to teach, add value, assign assignments, customize, etc. Browser/platform compatibility issues Program contains errors/incorrect information/poorly worded questions Program conflicts or causes confusion with LMS software (ANGEL)

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 16

Most Common Observations by # Times Mentioned

Program conflicts or causes confusion with Browser/platform compatibility issues Program is repetitive Program is time consuming Log-in/access issues, technical problems, Lack of question depth, content depth, and

10

15

For complete responses, see Appendix C.

Final Exam Scores

The following is a table showing the SOCI 1011 Final Exam scores from Summer 2012. Scores for both the Adapt course and the Control course are included:
Participant 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Final Exam Adapt 99% 98% 96% 96% 95% 95% 94% 94% 94% 93% 93% 93% 93% 93% 93% 93% 91% 91% 90% 89% 88% 86% 86% 86% Final Exam Control 100% 99% 98% 98% 98% 98% 96% 96% 96% 96% 95% 95% 94% 94% 94% 94% 93% 93% 93% 90% 89% 89% 88% 84%

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 17

25 26 27 28 29 30 31 32 33 34 35 36 37 86% 84% 83% 83% 83% 81% 79% 79% 76% 76% 73% 69% 68% 84% 83% 80% 78% 78% 78% 66%

These scores provided the data to perform a quantitative statistical analysis to compare the mean, standard deviation and t-test scores of the two courses. Mean The average of the final exam scores from the Adapt Course was 87.36%. The Control course had a final exam average of a 90.32%. The difference between the two courses was only 2.96%.

Mean of Final Exam Scores

Adapt Control

87.36%

90.32%

Standard Deviation The standard deviation for the Adapt course was 0.0795. The standard deviation for the Control course was 0.0794. There was only a difference of .0001.

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 18

Final Exam Scores Average Standard Deviation 87.36% 0.0795 90.32% 0.0794

Adapt Control

t-Test After compiling the data and calculating the statistics, we determined that there was no significant difference between the SOCI 1011 Adapt course and the Control course for Summer 2012. This determination was made by calculating a critical two-tailed t-test to compare the mean ratings between the two groups. The t-statistic proved not to be significant as it is much higher than the .05 critical level. (t critical two tail=1.99) The following is the t-Test statistics for the SOCI 1011 Summer 2012 Final Exams: Adapt Course Mean Observations t Stat t Critical two-tail 0.87 37 -1.5 1.99 Control Course 0.90 31

Course Data Review: Retention and Success Rates from Summer 2013
In order to show an overall course comparison between the Adapt Course and the Control Course, the following data was collected through Institutional Effectiveness from the SOCI 1011 Adapt course and control course from summer 2013:

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 19 Upon analysis of the SOCI 1011 course information from summer 2013 for both the Adapt course and the Control course, we determined there was a statistical difference between the retention rates of each course. The Adapt course started with 52 students and only ended with 45 students, meaning seven students withdrew from the course. This is a withdrawal rate of 9%. All thirty-five students who enrolled in the control course completed the course. According to the GTC research proposal, one of the main objectives for using the Adapt courseware was to improve retention.

Retention Percentage for Adapt Course

9% Completed the course Withdrew from the course 91 %

Additionally, the Adapt course had a much lower success rate than the Control course. The following chart is a graphical representation of the comparative course success rate between the Adapt Course and the Control Course:

Comparision of Course Success from Summer 2012

7 13% 10 19% W 5 14% D-F A-C

35 67%

30 86%

Adapt Course

Control Course

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 20 Out of the fifty-two students who started the SOCI 1011 Adapt course, only 67% successfully passed the course with a final grade of C or higher. In comparison, 86% of the students who enrolled in the SOCI 1011 control course completed the course with a final grade of C or higher. Again, a 67% pass rate is well below the GTC benchmark of 80%.

Discussion
The major questions addressed in the questions section of this evaluation have been inserted below to provide additional relevance and context to this discussion. 1. Is Adapt Courseware an affordable product for students enrolled at Gwinnett Technical College? 2. What are the faculty and student reactions to the product thus far? 3. How effective has the product been thus far? A synthesis of the results reveals the following items of significant interest:

Affordability
Students tended to view the product as similar in cost to what would be required from other courses, although 34% of respondents reported that the cost was higher. Students do not appear to save money by using Adapt Courseware. Though instructors were divided on whether or not the software costs more than traditional options, most instructors believed that Adapt Courseware was not worth the cost to students.

Student and Instructor Reactions

Student reactions to the course were highly positive. Students liked the user interface and the course material, and felt that they learned from the course and would use what they learned in the future. However, it is important to keep the limitations of the student survey results in mind. Only 9 students responded to the survey, which makes it difficult to generalize the results to the entire population of students who took the course. Further, students who dropped the course (those likely to have had the most difficulty with the software) were not included in the survey sample, which is a major limiting factor in determining reliable results. The instructors overall opinion of Adapt Courseware was unfavorable. They were mostly unified in their opinions of the coursewares failings and all seem ed to mention similar problems including technical issues, a high student drop out November 16th, 2013

[ADAPT COURSEWARE AT GTC] 21 rate, and the software failing to provide opportunities for more than surface learning. It was noted, however, that the product appeared to be beneficial to certain students, particularly those majoring in Psychology and those who were highly self-motivated and able to begin the course with a great deal of diligence. Due to the fact that instructors have been dealing with frustrations arising from use of the software and that a qualitative survey lends itself to emotional responses, the possibility that feedback is skewed toward the negative should be considered. This style of courseware is an obvious departure from what instructors are used to, and some degree of frustration may be attributable to natural adjustment to change. Certain observations and complaints (such as fewer median class scores, for example) conflict with results obtained from other instruments.

Effectiveness of the Product

Our evaluation sought to examine the effectiveness of the software in improving student performance and reducing the number of students who dropped the course compared with those who completed it. There was no significant difference found between the exam results for the control group and the Adapt Courseware group. This indicates that the use of Adapt Courseware is no more effective than the traditional instructor-led course model previously used. A finding of more significant importance, however, is the fact that 9% of students in the Adapt course withdrew compared with no withdrawals from the control course. The Adapt course also had a much lower rate of students who passed the course (67%) compared with 86% who passed the control course. This observation indicates that Adapt Courseware may be less effective than the instructor-led course model.

Recommendations
Our recommendations support the main decision of whether Gwinnett Technical College should continue to use the Adapt Courseware product after the research trial has ended based on the questions raised in this evaluation.

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 22

Recommendations based on the question: Is Adapt Courseware an affordable product for students enrolled at Gwinnett Technical College?
Although students view the cost of Adapt Courseware as similar to the cost of resources in other courses, they do not appear to be significantly saving money by using the product, and instructors feel that the product is not worth the cost to students. Thus, we do not recommend that GTC continue to use the product based on affordability.

Recommendations based on the question: What are the faculty and student reactions to the product thus far?
Although we were impressed by the positive student reactions gathered in the student survey instrument, we do not recommend the continued use of the software from this data alone. There were only nine student responses to the survey, and although this was 25% of the survey participants, we feel these nine students are not an appropriate representation of the student body at large. Limitations to consider from the student survey: Students who filled out the survey are likely high achieving students who are successful in the course. We were not able to gather data from students who either withdrew or stopped participating in the course. Whereas the student reactions seemed mostly positive, the instructor reactions were highly negative. Instructors reported not having enough autonomy over their course. Instructor survey responses indicated that the Adapt Courseware product did not offer any higher learning activities and was basically a practice and definition tool. Instructors also indicated that Adapt Courseware was so time-consuming there was not any space within the course to add supplemental activities to nurture higher learning. As a result, we do not recommend the continued use of the Adapt Courseware based on instructor reactions to the product as the GTC Sociology and Psychology departments have competent, innovative instructors who provide a richer learning environment for their students.

Recommendations based on the question: How effective has the product been thus far?
As has been demonstrated in the results and discussion sections, exam scores indicate that Adapt Courseware is no more effective than the traditional instructor led sections of the course previously offered by the department. Additionally, higher success rates and lower drop out rates indicate the control November 16th, 2013

[ADAPT COURSEWARE AT GTC] 23 course offers a more successful learning environment. Thus, we do not recommend that GTC continue to use Adapt Courseware based on the effectiveness of the product.

Recommendations Summary
After analyzing the data and comparing the results, we do not recommend that GTC use the Adapt Courseware as its primary online learning tool for either the Sociology or Psychology courses. Adapt Courseware was not as reliableas the GTC regular online courses in producing student success or retention in summer 2013. It was not well liked by any instructors, and although we documented favorable student responses, weve noted multiple problems with the student survey as an instrument in this evaluation. From the Student and Instructor Surveys, we did discover that Adapt Courseware offers a positive attribute for some students by offering remediation and practice. Therefore, as a secondary recommendation we suggest exploring possibilities for offering the product as a supplementary learning tool for practice and definition mastery. If Adapt Courseware is offered as an optional add-on to the standard online course then students who may benefit from the product would have a choice to use it, while others would be able to save the extra expense.

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 24

Appendix A: Original GTC Grant Proposal

The original grant proposal for GTC can be viewed here.

Appendix B: Student Survey

Survey Questions
Answers to these questions are to be selected from the following range: o o o o o o n/a strongly disagree somewhat disagree neither agree nor disagree somewhat agree strongly agree

1. The course site was easy to navigate 2. The online instructional software and learning tools were user friendly 3. The course material was presented in a clear and organized manner 4. The course objectives were clearly stated at the beginning of the course 5. The tests were directly related to the course objectives 6. Provide an explanation/further details for your answers above. Provide specific examples where possible: Answers to these questions are to be selected from the following range: o o o o o o n/a strongly disagree somewhat disagree neither agree nor disagree somewhat agree strongly agree

7. The online course readings contributed to my learning

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 25 8. The online course videos contributed to my learning 9. The online course practice activities contributed to my learning 10. The course material was presented in a way that maintained or increased my motivation to learn about the subject 11. I felt challenged to learn in this course 12. I feel that I learned a lot from this course 13. The knowledge that I learned in this course will be useful to me in the future 14. Provide an explanation/further details for your answers above. Provide specific examples where possible: Answers to these questions are to be selected from the following range: o o o o o o n/a significantly lower somewhat lower about the same somewhat higher significantly higher

15. How likely would you be to recommend this course to other students? 16. How likely would you be to take other courses in this format compared to traditional online courses? 17. Provide an explanation/further details for your answers above. Provide specific examples where possible: The answer to this question is to be selected from the following range: o o o o o o n/a significantly more somewhat more about the same somewhat less significantly less

22. Based on your experience, how would you rate the cost of the Adapt

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 26 Courseware compared to other media (including textbooks) used in other online courses? 23. Please add any additional comments that you have about the course.

Full Student Survey Results

Question 1 - The course site was easy to navigate.

0%

0% 11%

Strongly Disagree Disagree 22% Neither Agree Nor Disagree Agree Strongly Agree

67%

Question 2 - The online instructional software and learning tools were user friendly.
0% 0% 22%

Strongly Disagree Disagree Neither Agree Nor Disagree 22% Agree Strongly Agree

56%

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 27 Question 3 - The course material was presented in a clear and organized manner.
0% 0% 0% 11% 0% Strongly Disagree Disagree Neither Agree Nor Disagree Agree Strongly Agree 89% N/A

Question 4 - The course objectives were clearly stated at the beginning of the course.
0% 0% 0% 11% 0% Strongly Disagree Disagree Neither Agree Nor Disagree Agree Strongly Agree 89% N/A

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 28 Question 5 - The tests were directly related to the course objectives.
0% 0% 0% Strongly Disagree 11% Disagree 11% Neither Agree Nor Disagree Agree Strongly Agree 78% N/A

Comments: Course material was designed to make every student to read the material. Focused on making the student knowledgeable. Pictures are taken with great thought - every pictures speaks the whole story, very impressive. Question 6 - The online course readings contributed to my learning.
0% 0% 0% 11% 0% Strongly Disagree Disagree Neither Agree Nor Disagree Agree Strongly Agree 89% N/A

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 29 Question 7 - The online course videos contributed to my learning.
0% 0% 0% Strongly Disagree Disagree 33% Neither Agree Nor Disagree Agree 67% 0% N/A Strongly Agree

Question 8 - The online course practice activities contributed to my learning.

0% 0% 0% Strongly Disagree 11% Disagree Neither Agree Nor Disagree Agree 67% Strongly Agree N/A

22%

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 30 Question 9 - The course material was presented in a way that maintained or increased my motivation to learn about the subject.
0% 0% 0% Strongly Disagree 11% Disagree Neither Agree Nor Disagree 56% 33% Agree Strongly Agree N/A

Question 10 - I felt challenged to learn in this course.

0% 0% 11% 22% Disagree Neither Agree Nor Disagree Agree Strongly Agree 45% N/A

Strongly Disagree

22%

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 31 Question 11 - I feel that I learned a lot from this course.
0% 0% 0% Strongly Disagree 11% Disagree Neither Agree Nor Disagree 56% 33% Agree Strongly Agree N/A

Question 12 - The knowledge that I learned in this course will be useful to me in the future.
0% 0% 11% 11% Disagree Neither Agree Nor Disagree 33% 45% Agree Strongly Agree N/A

Strongly Disagree

Comments: I sure felt that this course had made me a better speaker. A knowledge complied can be used in day to day life.

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 32 Question 13 - How likely would you be to recommend this course to other students?
0% 0%

22% 45%

Very Unlikely Unlikely 0% Somewhat Unlikely Somewhat Likely Likely Very Likely

33%

Question 14 - How likely would you be to take other courses in this format compared to traditional online courses?
0% 11% Very Unlikely 11% 45% 11% Unlikely Somewhat Unlikely Somewhat Likely Likely Very Likely 22%

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 33 Question 15 - Based on your experience, how would you rate the cost of Adapt Courseware compared to other media (including textbooks) used in other online courses?
0% 0% 11% Significantly Less Somewhat Less 22% 56% 11% About the Same Somewhat More Significantly More N/A

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 34

Appendix C: Instructor Questionnaire

Questions
Reaction 1. What were the original goals for the use of Adapt Courseware within your course, and do you feel those goals have been met? 2. Do you feel that Adapt Courseware helps students meet the learning objectives for the course? 3. Is Adapt Courseware easy for students to use, or do they have difficulty using the program? 4. As an instructor, do you enjoy using Adapt Courseware in your course? 5. As an instructor, have you experienced any technical problems using the program? If so please explain 6. Are the time requirements within Adapt Courseware feasible for students? 7. How do these time requirements compare to the previous online version of the course? 8. Does Adapt Courseware offer multiple delivery options for different learning styles? 9. Does Adapt Courseware offer remediation and/or learning activities for struggling students? Effectiveness 1. Has there been an overall increase or decrease in student performance since the implementation of Adapt Courseware? 2. Has the number of students who drop out of the course increased or decreased since the implementation of Adapt Courseware?

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 35

Cost 1. How much does Adapt Courseware cost for each student? 2. What other resources must students in the course purchase in addition to Adapt Courseware?
3. What is the cost compared to alternative media (including textbooks) that students would need to purchase for the course if Adapt Courseware was not being used? 4. Do you consider Adapt Courseware to be an affordable product for students in your course?

Full Instructor Questionnaire Responses

1. What were the original goals for the use of Adapt Courseware within your course, and do you feel those goals have been met? Originally, we thought that maybe this type of program would help those students who were not doing well with the more traditional, textbook based curriculum. That has not turned out to be true. Far more of these students are dropping the class or failing it than in any other of the other delivery formats. The original goals for the use of ADAPT courseware in Intro to Psychology was to give students an alternative to purchasing a textbook, exposure to a new online program, curriculum customization, increased enrollment and an increase in student scores. The courseware has been beneficial to some students who are self motivated to complete online units daily/weekly. Overall, we have seen a larger number of students who drop the course or wind up failing due to falling behind in the courseware. There seems to be confusion about the use of ANGEL online learning system and an additional courseware. Many students have reported frequent technical issues with the ADAPT site and instructors are unable to fix these problems but instead must wait for ADAPT employees to correct the issues. The original goals were to have students show mastery on learning objectives. Those goals were not met. Adapt Courseware only had students do repetitive quizzes on the exact definitions of psychological terms. When they took the actual exams that needed critical thinking to solve, students were unable to correctly answer those types of questions. Higher retention, more successful students. The have not.

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 36 2. Do you feel that Adapt Courseware helps students meet the learning objectives for the course? Please explain why. No, it does not. Although the topics crossover, the content in the Adapt Courseware is deficient. It leaves out key information. It is almost 100% terminology with very few examples and no critical thinking at all. ADAPT Courseware only helps some of the online students meet the learning objectives. If a student has a computer or technical issue, they may fall behind. If a student is not computer savvy, they may struggle initially in the class with the absence of a paper textbook and the immediate need for completion of units online. I personally feel that some students are set up for failure with the ADAPT model. We serve a variety of students at Gwinnett Tech, and some students benefit from traditional textbook in addition to online courseware. It only allowed students to recognize the exact definitions of psychological terms and did not help students understand or apply those terms. It helps them regurgitate definitions, which can help on some tests, but it does very little to help students apply the material to real life situations. 3. Please describe any problems your students encounter with Adapt Courseware. The biggest problem is that it is extremely time consuming. The program forces the students to master the review questions at 90% and repeat the questions over and over again. Many of the questions are strange and almost impossible to figure out. If you get something wrong it haunts you for the entire course. Missed questions repeat in subsequent units as many as 5 times. It takes so much time that there is no time left for other activities outside the program. The most common problem my students have encountered would be the inability to access the courseware. I have received numerous emails from students saying they are being prompted for a username and password. A few times when the students called the ADAPT hotline, they were told it was a school issue that we needed to resolve. Other times, it may have been as simple as the student using the wrong browser. The limitations of the software (it doesn't work on tablets or Internet Explorer) can prove difficult to students who have limited access to computers. We have had numerous problems this semester with the Courseware. Almost every weekend, students were unable to get into the Courseware. This is a time when many students complete their work. Students requests for help were ignored by the Adapt Courseware IT help desk. Problems logging in, figuring out where to go once logged in, problems with the fickle nature of the mastery device.

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 37 4. As an instructor, do you enjoy using Adapt Courseware in your course? Why or why not. No, not at all. There is absolutely no teaching involved. You just babysit the program. The instructor has no opportunity to modify the course to add or delete content. Everything is controlled by the publisher. I like the concept of ADAPT Courseware allowing students to work at their own pace and learn material on their own, but I personally do not enjoy using ADAPT in my courses. I am very dissatisfied with the number o f technical issues and the lack of application of the material in the program. The courseware consists of matching and fill-in the blank activities which I consider to be busy work. You must achieve mastery in the units before you can move on, and sometimes students feel stuck and therefore get discouraged. We have so me students taking unit practice sections 20 and 30 times, and that seems excessive. In our traditional psychology courses, we strive to provide the students not only with information needed to meet course objectives, but also application. We include videos, assessments, and hands on activities that challenge the student to think and apply the terminology. Based on a lot of the content included in ADAPT, it also appears more suitable for students who intend to major in Psychology, rather than our GTC students, who only take one or two psychology courses for their degree. I did not enjoy using Adapt Courseware. It was repetitive and did not allow students to apply psychological terms to real life. It was essentially busy work, which prevented me from assigning extra articles that would allow for more critical thinking. I do not. I have seen courseware that looks fairly promising, but this is not it. This is quite rudimentary compared to what else is out there from other publishers. It really takes any sort of fun out of the material. It is essentially a series of dry definitions with very little context added. 5. As an instructor, have you experienced any technical problems using the program? If so please explain. The biggest problem is that the program is not compatible with Internet Explorer. Students are told to use Firefox or Chrome. It seems ridiculous that it is not compatible with the most commonly used browser. If I forget and log on with Explorer, I have back out and go into a different browser. I have not had any major technical problems in ADAPT. I do have to make sure I'm using the right browser and I feel that the inability to use ADAPT on a tablet or Internet Explorer is inconvenient. I have had to contact ADAPT several times on behalf of students, and for the most part they get back to me within 24 hours. A couple of times they were unable to resolve my issue, so I had to speak with a contact instead of the hotline

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 38 representatives. Also, our primary contact left her job in the middle of the semester, and it was unclear who was taking over her responsibilities. Many students were unable to get into the software throughout the semester. There were also several errors in the quizzes. Students were not allowed to move forward until passing the quiz which makes it difficult to move forward when the "correct" answer is wrong. Some problems logging in. Once I figured out how to use it, I haven't had many problems.

6. Are the time requirements within Adapt Courseware feasible for students? Please explain. No, it is far too time consuming and repetitive. That has been the biggest student complaint. The program was adjusted this fall to decrease the amount of time but it still takes an average 14 minutes to "master" a section. That is 34.5 hours just in the practice activities. It doesn't account for the time watching videos, reading the content, or studying. As long as a student begins diligently working the first week of class, they can meet the time requirements. Students who enter through drop/add or have technical difficulties from the beginning, may start to feel overwhelmed by the number of practice units that must be completed with a mastery of 80 or higher. The student can keep moving forward in the material as long as they achieve a mastery of 80 or higher on the practice units. If not, they can only read ahead. No! Students had to complete 1000 -1200 quizzes to get through all sections. This is a waste of the students' time. Especially since the "quizzes" only involved matching definitions. The research we've done shows that some students can fly through the courseware, but the majority spends an inordinate amount of time on the exercises and reading. 7. How do these time requirements compare to the previous online version of the course? I don't measure time requirements in the other online sections. There is really no way to. For years, students have reported on evaluations that the workload was just about right for the online class. In our traditional online course, students have 3-4 weeks to cover multiple chapters, which are divided into sections. In this time, they will read the chapters, complete an assignment, access a study guide and practice test, and then take a test over the material. The assignments are in essay format and students must access current articles, videos, and textbook information to complete the assignments. Students are given 2 weeks to submit an assignment and 2 weeks to submit a test.

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 39 Our regular online course has nowhere near this time requirement of trying to finish 1200 quizzes! This made us reduce our regular class load of completing 4 written assignments because that would be too much additional work on top of 1200 quizzes. Far more time consuming for the most part.

8. Does Adapt Courseware offer multiple delivery options for different learning styles? Please explain. There are text files and videos. However, the videos are just someone narrating the text files verbatim. ADAPT does offer video or lecture note options for the student. They also offer a study board for students to post questions and comments. I think the confusing part for the student is that we already use Angel as our primary mode of communication. Students can read the definitions for each chapter or watch a video that has a person reading the definitions. I believe that they think it does, but apart from having videos that speak the lecture and traditional written content, there isn't much else. 9. Does Adapt Courseware offer remediation and/or learning activities for struggling students? Please explain. Not really. The student goes back to the original files to review if they miss practice questions. I think they would consider this remediation but it is essentially repetition. It would be useful if, when you missed a question, you were given feedback. ADAPT will filter in questions from previous units if students have struggled with those sections. There is no other specific remediation that I'm aware of because the student has to achieve mastery to be able to move forward in the material. The students can go back anytime to access the lecture no tests or videos and they can retake practice units if needed. It does not allow students to move on until they show an 80% mastery level on that learning objective. Not particularly. If a student misses a question, the program continues to bring up the same question over and over. This isn't really remediation though. 10. Describe the overall impact on student performance since the implementation of Adapt Courseware. Far fewer students complete the course successfully. There are some students who are achieving mastery and making A's in the course. We have seen an increase in the number of students who withdraw with a W at the midpoint and overall many students fall behind in the course and their grade does not recover. When reviewing grade

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 40 book grades, there is a significant gap where students either have an A or B, or they are failing. Our performance had a HUGE decline. The dropout rate tripled and the failure rate for the course quadrupled! Grades are lower, retention is lower, general satisfaction seems to be lower. I don't really see an upside.

11. Has the number of students who drop out of the course increased or decreased since the implementation of Adapt Courseware? Increased. The number of students withdrawing has increased since I began teaching ADAPT courses. Last semester I taught the control group and I did not see as many students drop mid-semester. Increased dramatically Increased 12. How much does Adapt Courseware cost for each student? $100.00 The cost of the ADAPT Courseware is $135.75. $138 Somewhere around 90 to 100 dollars I think (far too much). 13. What other resources must students in the course purchase in addition to Adapt Courseware? None There are no other resources required for the course. Nothing else I think that is about it. It helps to have the book that the other students use but it isn't mandatory. 14. What is the cost compared to alternative media (including textbooks) that students would need to purchase for the course if Adapt Courseware was not being used? The cost of a new book is around $65.00. It is available used for as low as $20.00. Students also have this textbook to keep or resell at the end of the semester. They do not have access to the Adapt materials after the semester is over. The students in the traditional class pay $139.50 for a new textbook. The used option would be $104.79 and a textbook rental is $69.68. The digital eBook would be $52.99, which the student can access for 180 days. There are no other resources needed so the clear disadvantage is that students have only one option with ADAPT.

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 41 They charged the price of the new textbook so most students in the regular online class paid much less if they purchased a used book or rented the book. It's more expensive than what I use.

15. Do you consider Adapt Courseware to be an affordable product for students in your course? It would be if the quality was there. The cost of the ADAPT courseware is $135.75. This price is comparable to the average price of the textbooks we utilize, and the student can continue accessing the courseware after they complete the course. The disadvantage is that the students have nothing to resell at the end of the semester and this is a problem for some students, who count on selling their textbooks to purchase new ones. Absolutely not. It is way overpriced! Not really.

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 42

Appendix D: Student Survey Matrix

The following matrix correlates the primary evaluation questions with the student survey:
Survey Questions Evaluation Questions

Q1 What is your age range? What is your gender? What is your program type? What is your race? How easy was the course site to navigate? Were the online instructional software and learning tools user friendly? Was the course material presented in a clear and organized manner? Were the course objectives clearly stated at the beginning of the course? Were the tests directly related to the course objectives? Provide further explanation of your answers above. Did the online course readings contribute to your learning? Did the online course videos contribute to your learning? Did the online course practice activities contribute to your learning?

Q1a

Q1b

Q2

Q2a

Q2b

Q2c

Q3

Q3a

Q3b

Demos.

X X X X X X X X

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 43

Was the course material presented in a way that maintained or increased your motivation to learn about the subject? Did you feel challenged to learn in the course? Did you feel that you learned a lot from this course? Will the knowledge that you learned in this course be useful to you in the future? Provide an explanation to your answers above How would you rate the quality of this online course instruction? How would you rate the quality of this online course instruction compared to other online course instruction? Provide an explanation for your answers above How would you rate the cost of Adapt courseware compared to other media (including textbooks used in other online courses? Please add any additional comments that you have

X X

X X

X X

X X

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 44

Appendix E: Instructor Questionnaire Matrix

The following matrix correlates the primary evaluation questions with the instructor questionnaire:
Questions from Questionnaire Evaluation Questions

Q1 How much does Adapt Courseware cost for each student? What other resources must students in the course purchase in addition to Adapt Courseware? What is the cost compared to alternative media (including textbook) that students would need to purchase for the course if Adapt Courseware was not being used? Do you consider Adapt Courseware to be an affordable product for students in your course? Has there been an overall increase or decrease in student performance since the implementation of Adapt Courseware? Has the number of students who drop out of the course increased or decreased since the implementation of Adapt Courseware? What were the original goals for the use of Adapt Courseware within your course, and do you feel those goals have been met? Do you feel that Adapt Courseware helps students meet the learning

Q1a

Q1b

Q2

Q2a

Q2b

Q2c

Q3

Q3a

Q3b

November 16th, 2013

[ADAPT COURSEWARE AT GTC] 45

objectives for the course? Is Adapt Courseware easy for students to use, or do they have difficulty using the program? As an instructor, do you enjoy using Adapt Courseware in your course? As an instructor, have you experienced any technical problems using the program? If so, please explain Are the time requirements within Adapt Courseware feasible for students? How do these time requirements compare to the previous online version of the course? Does Adapt Courseware offer multiple delivery options for different learning styles? Does Adapt Courseware offer remediation and/or learning activities for struggling students?

November 16th, 2013