ONLINE GENERAL EDUCATION COURSES AT A RESEARCH UNIVERSITY

Barbara Sommer, Harry M. Matthews, Michael M. Maher and Curt Acredolo

Due to demography, globalization, and economic restructuring, the demand for postsecondary education is increasing at the same time that its costs are rising. New information technologies offer a means of addressing this situation. Some aspects of the modern university are probably more likely to change than others. The college experience is an important vehicle for socializing the young, particularly for the middle and upper social strata; and it continues to be seen as a source of upward social mobility for the working class. The residential college, or at least the classroom campus, is not likely to disappear soon. Few parents want their children to attend college via a computer at home, and the university experience represents a valued and important life stage for many adolescents and young adults.

When it comes to accomplishing the educational mission of the university, the traditional means of information transfer, the classroom lecture, is certainly at risk. Foundations such as Andrew W. Mellon, Pew Charitable Trusts, and Alfred P. Sloan are addressing this shift from the traditional college format by funding experimentation with new forms of instruction. A main focus of the educational paradigm is on the learning process – captured by the similes of “the guide on the side” replacing “the sage on the stage.” A major goal of the research on new educational technologies is to discover alternatives to the classroom lecture with the aim of reducing building construction and maintenance costs.

Substitutes for the classroom lecture have ranged from simply putting one’s syllabus and lecture notes online to an extensive restructuring of courses, such as that supported by the Pew Foundation (http://www.center.rpi.edu/PewGrant.html).

Studies of the effects of the technology transfer on student performance reveal a pattern of “no difference” (Phipps & Merisotis, 1999; Russell, 2002). Many investigators did not control for relevant variables. In a case study of an Introductory Nutrition course at the University of Arizona, Ricketts, Wolfe, Norvelle and Carpenter (2000), taking relevant variables into account, concluded that the self-selected students using a web version did at least as well and often better than students taking the course in the traditional format. Maki and Maki (2002) have consistently found improved examination performance in web-based Introductory Psychology courses. They summarize nine other studies in the social sciences that show mixed results as to the effectiveness of the online versions. Brown and Liedholm (2002) found that online students in a Fall and subsequent Spring offering of a Microeconomics course at Michigan State University did less well than those who attended class. The ambitious and ongoing effort by the Pew Grant Program in Course Redesign involves restructuring materials to fit the online environment. The courses vary with regard to the extent of reliance on the online material. Comparing the traditional lecture format with web-enhanced offerings, the results from the first round of ten courses, each at a different institution) showed inconclusive evidence at one, no difference at four, and improvement with web-enhancement at five (Jarmon, 2002).

The present study was done at the University of California, Davis, which is experiencing a major growth in enrollment at a time of diminishing financial resources. The Andrew W. Mellon Foundation’s program entitled "Cost-Effective Uses of Technology in Teaching" (CEUTT http://www.ceutt.org/) sponsored the research aimed at addressing the question “Can undergraduate general education courses be put online without a decrement in student learning?”

Method

Courses and participants

We studied traditional and online presentation for six large undergraduate general education courses from Fall 2000 through Spring 2002. Participating faculty volunteers received an additional stipend of $10,000 for their participation. One instructor already had a proprietary website with course materials online. A second instructor provided lectures in text form only (refusing to be videotaped). The remaining four instructors’ lectures were either videotaped during the preceding quarter, or recorded in the studio.

Several technologies were used to construct the online materials. The result was a set of more or less illustrated lectures delivered via Internet browser in both auditory (with the one exception) and text modes. Examples can be viewed at http://moby.ucdavis.edu/mellon/.

Five courses had parallel traditional and on-line offerings; that is, the instructor presented the course in the traditional lecture manner, but also made the course material available online to the students. The instructors attempted to keep the information comparable, so that students could take the entire course online. In four of these five courses, students self-selected during the 10-week quarter with regard to whether or not they used the online material. In the Statistics course, they initially enrolled in either the traditional or online version. The sixth class, Anthropology, was offered exclusively in a traditional mode in Winter 2000 and 2001, and exclusively online in Winter 2002. Table 1 lists the courses, their technical base, the number of students in each usage category, and the online amenities for each course.

Measures

Student performance. For comparison purposes, performance scores were standardized to a mean of 50 and standard deviation of 10 within each course. In four courses (Food Science, Art History, Psychology and Viticulture & Enology), performance was based on the total accumulation of points for examinations and assignments. For the Statistics class, performance was based on the multiple-choice section of the final examination that was identical for both groups. It excluded the final essay question and midterm examinations because they differed for the traditional and online students, and were graded by different Teaching Assistants. The Anthropology instructor had been on sabbatical the preceding year (the course is offered only once a year). Instead of a simultaneous comparison, the traditional group performance measure was two years prior to the online performance measure. Grading was based on written papers, and the point system from two years earlier was not comparable to the later offering. Thus, overall performance was calculated from the students’ final letter grades using a four-point system, (e.g., A = 4, B- = 2.7, D+ = 1.3).

Student evaluation. At the end of the quarter, before grades were assigned, students anonymously evaluated their course using a standard campus form.

Online logs. We had logon information for four courses – number of online sessions per student and number of students logging on by day.

Results

Student performance

On the final examinations, students indicated their degree of reliance on the web-based material. The students who attended class lectures and those who relied on the online materials were labeled “traditional” and “online,” respectively. “Both” refers to students who used both resources about equally. Given a choice, students were more likely to attend class than to rely on the online material (Table 1). The only instance where the numbers were similar is anthropology which was offered either entirely in class or entirely online.

We had useful logon information for two of the five parallel classes. In both cases there was a reliable correlation between number of logons and self-report of online usage [Art History, r (92) = .509, p<.001 and Viticulture & Enology r (259) = .573, p<.001]. In these two courses, the only rationale for logging on would be to access the lecture material. For the WebCT course (Psychology) we were unable to distinguish between access related to course content and that connected with reading class announcements, sending e-mail, or using a chat room. Not surprisingly, there was no relationship between the number of logons and self-reported online use.

We analyzed student use of the Internet (“little or none” = traditional student, “all or most” = online student, and “about equal” = both) as a predictor of performance (standard scores) for each of the six courses. Grade point average (GPA) was a strong predictor of performance, with correlations ranging from .477 to .825. Class level also made a difference with frosh tending to do less well.

Table 2 shows the estimated mean performance scores for the self-defined groups with grade point average (GPA) and class level included as covariates in the analysis. With the exception of Viticulture & Enology (where there was essentially no difference), the differences between means favored the in class students.

The analysis for each course can be considered a replicate. Following meta-analytic procedure (Rosenthal & Rosnow, 1984), the one-tailed probabilities were combined to produce a single z-value of 2.86 whose probability is .002, meaning that the students who attended lectures performed significantly better than those who relied mainly on the online material (see Table 3).

The effect size for relying on lectures versus relying on the online material was calculated using Eta-squared (h²). In a contrast where df (degrees of freedom) equals one, Eta is identical to r. Eta-squared shown on Table 3 is the proportion of the total variability in performance that is accounted for by variation in usage group (traditional versus online). Combining Eta-squared calculations (Rosenthal & Rosnow, 1984), the overall effect size is miniscule (r = .01). At most, the amount of variance in performance explained by usage group (traditional versus online) was 2.2 percent. In contrast the Eta-squared calculations for GPA (grade point average) ranged from .228 (22.8%) to .424 (42.4%).

We applied a curve to give a more concrete picture of what these obtained differences suggest for actual grades. If we use a hypothetical grading curve of 10 percent As, 20 percent Bs, 40 percent Cs, 20 percent Ds, and 10 percent Fs, there would be 10.5% fewer As and Bs for those using the web, but only 1.5% more of them would get Ds or Fs. A more realistic estimate would be 15% Ds and Fs, which yields a pass rate of 88.4% for the traditional students and 84.2% for the online students, a difference of 4.2%.

In order to explore whether or not the pattern of online use and reduced performance differed for the students with higher and lower ability, we analyzed the relationship of usage and performance for the top, middle, and bottom thirds on the Grade Point Average (GPA) distribution within each course. There were no significant interactions between GPA and usage group, suggesting that there was neither an advantage nor disadvantage to any particular ability group in using the online materials.

Log data

There was no relationship between how often students accessed the course website and how well they did in the class. We had class attendance records for the Art History course. Student performance correlated significantly with number of classes attended [r (92) = .314, p = .002]. The relationship remained significant, but declined to r = .262 when GPA (grade point average) (r = .680) was taken into account in the regression equation.

Students’ evaluation and attitudes

In addition to anonymously rating aspects of the course, students indicated whether they attended class or relied on the online materials, or used both about equally. The data are incomplete as that item was omitted from the Course E form. It was possible to reconstruct lecture attendance from reliance on the web on the basis of one of the items, but we were unable to discern those who may have relied equally on both information sources.

Figure 1 shows the mean ratings for the general item “This was a good course.” There are two sets of ratings for Course A. The performance data described earlier were obtained in Fall 01. However, the instructor had posted the online material the previous spring as a test run. We obtained evaluations for both quarters. The fall evaluations (second time through) showed a significant improvement. Combining probabilities (Rosenthal & Rosnow, 1984) across the seven offerings indicated that the traditional students were more favorable in their evaluation of the course (z = 2.77, p = .003).

The ratings of a second general item “Overall, the instructor is a good teacher” showed a similar pattern (Figure 2). Again, combining probabilities across courses showed that, although the magnitude of the difference is very small, traditional students tend to give the instructor a higher rating (z = 1.81, p = .04).

We ran eleven focus groups (ranging in size from 3 to 10) for the six online courses, concentrating on attracting those students who used the online material. Findings were consistent across all groups. Students very much liked the convenience and flexibility of having the lecture material available online. They used it when they missed class, and for study before exams. They commented on the variations in quality – in one case a student remarked that the online lectures constituted “a poorly-written textbook” and that a well-written bound text would be preferable. Despite their enthusiasm for flexibility and convenience, many expressed the desire for the traditional classroom lecture. There was a feeling of entitlement to a “real professor” that went along with going to college. Online courses appear to have some stigma attached and were perceived as “community college” level. Students also wanted to have a person from whom they could seek help if they had questions. Finally, there was considerable concern about the opportunity for procrastination. Having to attend class was more likely to keep students on track then the knowledge that the material was available to them whenever they chose to look at it.

Discussion

A major strength of the present study is that it covered six courses across a variety of disciplines. Our results suggest that students taking courses online tend to do slightly less well on examinations than those who attend class. The results for students using both sources of information was mixed, with means lower than those of the traditional and online groups in three out of four courses. One would expect that student availing themselves of both resources would do better. Perhaps having access to both produced procrastination – that students felt they did not need to attend all the lectures, but then failed to adequately master the online material.

Despite the grade performance decrement shown by the online students, our findings can be used to support the conclusion that online presentation does work. Given the extremely small magnitude of the differences, our results are more in accord with the four Pew course redesigns showing no difference in performance between the students in traditional classes and those in the web-enhanced versions. With the exception of Introductory Statistics, our redesigns were probably less extensive than theirs.

A number of factors may have contributed to lack of difference. This was the first attempt for most of these instructors at putting their material online. None of them had professional training in course development. Similarly, it was a first effort for many of the technical support staff. On the plus side, we found improved student responses to the second online offering of one of the courses.

The experiment was conducted at a Research 1 university, an institution where professors receive little recognition or reward for teaching. We recruited these faculty members; most were not “early adopters” eager to put their courses online. As such, they probably are typical of many professors at such universities for whom putting one’s course online is simply a matter of slightly modifying and posting lectures. Except for Statistics, the courses in the present study had little in the way of web enhancement. It is of note that the performance results in the Statistics class were practically identical between the traditional and online students (Utts, Sommer, Acredolo, Maher, & Matthews, 2002). The course showing the greatest difference (Anthropology) had the least reliable performance measure (a two-year interval for the comparison). The other course with the most reliable difference in performance between groups was Food Science. It was our first online attempt and was accompanied by numerous hardware and software problems that were frustrating for the instructor, as well as the students.

That there was no relationship between the logon data and performance was no surprise. Time on task is but one of many variables affecting performance (e.g., attention, ability, level of effort, related knowledge, study habits, etc.). The logon frequency was a good indicator of the validity of students’ self-reports of web usage.

From the students’ perspective, it is clear that they would like to have it both ways. They like the convenience of 24-hour access to course material. They dislike the perceived impersonality and coldness of online instruction. They feel they have an earned and paid right to a real professor. Going to class (as well as skipping class) is part of this tradition.

Instructors who venture into the online realm may suffer some loss with regard to their evaluation by students. From our data, the decline was very small. Sommer (2002) reported that an online lab section received higher ratings than the in-class versions. The instructor, who had never been seen by the students, received a lower evaluation than did the instructors of the in-person labs. Waschull (2001) found a decline in instructor evaluation only when students were assigned an online section, irrespective of choice.

The present study suggests that good students will learn under a variety of circumstances, so long as the material is made available to them. Improving the quality of learning and performance will require more than a simple change from one mode of delivery (the traditional lecture) to another (online presentation). It will require a more extensive commitment of resources to course design. Preliminary reports from the Mellon CEUTT program suggest that improved learning associated with online materials requires active involvement on the part of students , for example, with interactive tutorials, exercises, and quizzes (Kashy, Thoenessen, Albertelli, & Kashy, 2002; Scheines, 2002).

References

Brown, B. W., & Liedholm, C. E. (2002). Can web courses replace the classroom in principles of microeconomics? Retrieved from: http://www.msu.edu/~brownb/brown-liedholm%20aea%202002.pdf.

Jarmon, C. G., Editor. (2002, June). The Pew Learning and Technology Program Newsletter. Retrieved from: http://www.center.rpi.edu/PewNews/PLTP12.html.

Kashy, E., Thoenessen, M., Albertelli, G., & Kashy, D. (2002, November). Psychology and Natural Sciences with CAPA. Paper presented at the CEUTT 2002, Northwestern University, Evanston, IL.

Maki, W. S., & Maki, R. H. (2002). Multimedia coomprehension skill predicts differential outcomes of web-based and lecture courses. Journal of Experimental Psychology: Applied, 8(2), 85-98.

Phipps, R., & Merisotis, J. (1999). What's the difference? . Washington, DC: The Institute for Higher Education Policy.

Ricketts, J., Wolfe, F. H., Norvelle, E., & Carpenter, E. H. (2000). Asynchronous distributed education -- A review and case study. Social Science Computer Review, 18(2), 132-146.

Rosenthal, R., & Rosnow, R. L. (1991). Esssential of behavioral research: Methods and data analysis. (2nd ed.). New York: McGraw-Hill.

Russell, T. L. (2002). The no significant difference phenomenon. Retrieved from: http://teleeducation.nb.ca/nosignificantdifference/.

Scheines, R. (2002, November). Web-based modules on causal reasoning. Paper presented at the CEUTT 2002, Northwestern University, Evanston, IL.

Sommer, B., & Sommer, R. (in press). A virtual lab in research methods. Teaching of Psychology.

Utts, J., Sommer, B., Acredolo, C., Maher, M. W., & Matthews, H. M. (2002). A study comparing traditional and internet-based instruction in introductory statistics classes. Manuscript submitted for publication.

Waschull, S. B. (2001). The online delivery of psychology courses: Attrition, performance, and evaluation. Teaching of Psychology, 28(2), 143-146.

Figure 2. “Overall, the instructor is a good teacher.” Means and .95 confidence intervals for student ratings (5 = Strongly Agree, 1 = Strongly disagree) by usage group for each course.