• Home
• bwp@_Ausgaben
• Rezensionen
• Autoren
• Informativ
• Newsletter
• über_bwp@
• Suche

Since first offering online courses in 1997, the faculty in the Administration and Supervision Program has carried out a number of formative and summative evaluation activities. A basic assumption of the evaluation of the model is that multiple measures need to be used and where possible repeated over time. The following are examples of some of these activities.

In Spring, 1997, data on student satisfaction with their learning experiences were collected using a student satisfaction survey administered at the fifth week and again at the fourteenth week of the semester. The survey asked a series of questions regarding student experiences while taking an online course in comparison to traditional face-to-face courses. The survey was administered twice to determine if there was any change in student satisfaction during the course. In the data provided in Table 1, responses were based on a five-point Likert scale where 1= decreased, 2 = somewhat decreased, 3 = no change, 4 = somewhat increased, and 5 = increased. A sample question was: "In comparison to a traditional class, in this course the quantity (or quality) of interaction decreased -> increased?" The means of the student responses to the items are summarized in Table 1.

The data in Table 1 indicate that a high level of student satisfaction with their learning experiences was present at both intervals (fifth and fourteenth weeks). With the exception of the "quantity of your learning experiences" students were consistent in their responses on all questions. The "quantity of your learning experiences" was the only question that had a statistically significant difference in Week 5 compared to Week 14. One interpretation of this difference was that after fourteen weeks, the students were exposed to significantly more material than after five weeks and hence the "quantity of learning experiences" increased. The last item in this table "Overall experience - poor/excellent" indicates that students' overall satisfaction with the online course was high at 4.56. Responses to this question were based on: 1=poor, 2=satisfactory, 3=good, 4=very good and 5=excellent.
The student satisfaction survey has been administered in every online course offered in the program since 1997. Figure 2 provides a graph of the student responses to the question relating to their "Overall Experience - poor/excellent" for online courses offered between Spring 1997 and Spring 2001. Responses indicated that student satisfaction was rated high (between 4.00 and 5.00) throughout the period.

In Fall 1997, as part of the second asynchronous learning course (ADSUP 701) using the model described above, a quasi-experiment was conducted with students from another section of the same course taught in traditional mode by the same instructor. A two-week school law module on Brown v. Board of Education of Topeka, Kansas was conducted in both courses during the same period. In the traditional course section, the instruction was centered on a multimedia presentation using video clips from the award-winning Eyes on the Prize documentary series produced by the Public Broadcasting System (United States). In the asynchronous course section, the same content was covered without any multimedia presentation but using additional text-based materials available on the World Wide Web.
At the end of the two-week module, both classes were given the same assignment to write a short (3-5 pages) essay on the relevance of the Brown Decision to education in the 1990s. These papers were graded blindly by an outside evaluator who did not know any of the details of the experiment. The evaluator was asked to give a numerical grade for substance as well as for writing ability. The mean grades for the traditional class were 91.27 (Substance) and 92.90 (Writing Ability). The mean grades for the asynchronous class were 93.38 (Substance) and 95.0 (Writing Ability). A One-way Analysis of Variance indicated that there was not a statistically significant difference for the means for any of the grades. In this quasi-experiment, it was concluded that student performance using asynchronous learning was comparable to student performance in a media-rich traditional learning environment.

Hunter College requires all academic programs regardless of delivery mode to be evaluated on a cyclical (every five to seven years) basis. During the most recent evaluation of the Administration and Supervision Program in Fall 2000, three external evaluators were asked to evaluate plans for expanding our online program by responding to the following questions:
Does the asynchronous learning aspect of the program contribute to the learning curve of these future administrators?
Would a completely on line program enhance or inhibit students' learning?
The responses of the three evaluators were generally positive and indicated that with careful planning and monitoring the online component of the ADSUP program could and should expand. A sample of their comments follows:
"The fact that the 3 full time instructors are using technology as a teaching tool is a strength. Modeling effective use and knowledge of technology with students is a way to reach more students and teach students different modes of instruction. Students see their teachers using, not just talking about technology in teaching... It is important to analyze on a continuous basis, the positive and negatives of teaching each individual course through [online] technology. I encourage you to continue thinking about where it is a strength and where the face-to-face interaction is essential in developing a successful school leader."
-- Evaluator No. 1 (From California)
"This [online learning] is a strong aspect of the program. It will have an even greater value when the mapping is completed and there is a stronger assessment profile of incoming students.
A full on-line program is another avenue of services offered by the Administration and Supervision Program. The kinks will have to be worked out first. For example, who will assess and supervise these students? How does one overcome the face-to-face interaction? How do these on-line graduates compare to traditional graduates?"
-- Evaluator No. 2 (From New York)

"The technology component of your program has great merit. The business of schools is both "high tech and high touch." Many interactions that take place in a class designed to promote meaningful dialogue cannot be replicated on line at this time. [Technology] program enhancements, tutorials, chat rooms on topics of interest appeal to some, but not others."
-- Evaluator No. 3 (From Connecticut)
The comments from the three external evaluators were shared with administrators at Hunter College and the City University of New York. As a result of these and other evaluation results, a decision was made to seek formal approval from the New York State Education Department for approval of the online component of the Administration and Supervision Program in Fall 2001. York State has a policy which requires any academic program in which more than fifty percent of the course work can be taken in a distance learning mode be separately reviewed and approved.

The results from the above activities have been used for both summative and formative evaluation purposes. Where necessary, components of the model have been adjusted and fine-tuned. For example, students have voiced concern about the amount of work involved in reading and responding to colleagues' comments in an online course especially if enrollment is high. As a result, enrollment limits for online courses in the program are set at a maximum of twenty students.
While readily available in the BlackBoard course management system, the use of synchronous (chat) activities has been used sparingly in online courses since most students have consistently indicated that they prefer the freedom and convenience that the asynchronous nature of the courses provides.
Multimedia has been used sparingly as well because students do not have access to high-speed modem connections. However, it is anticipated that in the not too distant future, voice and video will become a more integral component of most of the online courses.

In Spring, 2002, the Administration and Supervision Program at Hunter College was the first program in the City University of New York formally approved by the CUNY Board of Trustees and the New York State Education Department to offer a majority of the courses in the program in an online mode. As a result of curriculum mapping completed in Spring 2001, the faculty in the program decided that two of the required courses will be offered only in traditional face-to-face mode. The remaining required courses including the internship experiences, as well as several elective courses can now be taken online.

While it is still a bit too early to determine how many students will complete the majority of their courses online, indications are that some will do so. Most students in the program now request and enroll in at least one or more online courses. Overall enrollment in the Administration and Supervision Program has also increased. Some of this increase is attributable to the availability of online courses.
This paper suggests that the development of an online course model and the expansion of courses into a program should be an evolutionary rather than a revolutionary process. Multiple evaluation techniques were described to adjust and fine tune the course model over time. The results of this activity were shared with colleagues at the City University of New York and other colleges and universities. While presently online courses are becoming commonplace throughout higher education in the United States, there will still need to be careful evaluation of online teaching techniques. As the technology changes, as it surely will, so will the need to examine and evaluate online course delivery. It is highly recommended that faculty teaching online and planning to expand their course offerings in this mode, carefully evaluate what techniques work pedagogically and what needs to be adjusted or improved.