The following document is an article which has appeared in the Journal of Chemical Education. Please give literature citations to this article as Journal of Chemical Education, 2000, 77 (2), 227-31.

Organic Chemistry and the Internet:

A Web-based Approach to Homework and Testing

Using the WE_LEARN System

by

John H. Penn,* Vincent M. Nedeff, and Gloria Gozdzik

Chemistry Department, West Virginia University, Morgantown, WV 26506-6045 USA

and

Horizon Research Consultants, Inc., 1534 Point Marion Road, Morgantown, WV 26505 USA

 

Abstract:

The development of the Web-Based Enhanced Learning And Resource Evaluation Network (WE_LEARN) system is described. A prototype model has been developed and applied to an organic chemistry course. In the WE_LEARN system, a modular approach for concept acquisition (i.e., a similar learning environment) is combined with a self-testing module (i.e., a dissimilar learning environment) to yield a student-friendly learning system. The success of this approach is demonstrated by higher class averages as compared to the class averages of previous years, student participation as measured by system usage, learning by increased practice test scores as a function of time and usage, and the nature of faculty/student interactions.

Keywords:

Organic Chemistry

Computer Assisted Instruction

Internet

Teaching/Learning Aids

Introduction:

The computer and the Internet have proven to be extremely versatile tools for a variety of applications. A complete citation listing of the uses of computers in chemical education is beyond the scope of this manuscript. However, a sampling of the recent uses of computers in chemical education which have been reported in this Journal are given here to indicate some of the various techniques now available (e.g., application of hypermedia and other Internet methods,1-3 kinetics,4,5 X-ray powder diffraction simulation6 mathematical software for chemical applications,7 visualization of structures,8,9 course delivery over the Internet,10 course delivery using Internet techniques11 ).

Although computer methods have achieved a measure of success when viewed from an instructional standpoint,12 we desired a computer-assisted tool that would enable the instructor to become part of the educational process throughout the entire course. A necessary component of any system for the instructor is a flexible tool that can be modified to meet the individual needs of both students and instructors. Several design features are critical to the ultimate success of the system. The learning system must provide immediate and adequate feedback. For the student, immediate feedback is well known to be essential for enhanced learning.13-17 For the instructor, feedback would allow for changes in instructional methodology or re-explanation of critical concepts. Currently, feedback to the instructor is normally available only after the completion of an in-class examination. As a second critical design feature, the system must engage the students in an active learning mode, since it is well-known that an active learning environment is superior to a passive learning environment.18 A third design goal is to ensure that learning occurs in both similar and dissimilar environments.19,20 In the similar learning environment, all problems related to a certain topic, area or idea are grouped together. Students are able to easily identify the concepts being tested/assigned and are able to improve their understanding and/or proficiency of this concept. An enhanced learning situation would be created by having a practice area whereby students could personally assess their learning prior to an exam. This situation corresponds to the dissimilar learning situation as various concepts, topics, and problems are randomized out of the rote memorization context. A final design feature is to ensure a high degree of flexibility within the system, since no two students learn the same way and no two instructors teach the same way.

These design requirements are easily met with today's available technology by combining Internet capabilities with the availability of computers to this generation of student. Following these design requirements, we have developed the Web-based Enhanced Learning Evaluation And Resource Network (WE_LEARN) system. This system is composed of a software engine (designed as an Internet server) and a database of questions which is easily controlled and manipulated by the software engine. Almost instantaneous feedback is provided to the students when they "submit" answers from a series of questions provided to them via an Internet server. Feedback to the instructor is provided by information concerning student performance on questions which are related to class material. The system provides an active learning mode since students select the correct answer(s) from a list of potential answers. Many individualized "modules", which focus on similar concepts, create a similar learning environment. The software maintains the capacity for randomly selecting questions from a larger subset of questions already learned within the individualized "modules" to provide a dissimilar learning environment. Flexibility for students is achieved by using the World Wide Web for distribution of materials. Web principles allow anyone to use the WE_LEARN system at any place (e.g., on campus, in the privacy of their own room) at any time of day. Research has shown that students study at different times and places than do university administrators and professors.21 We describe here our development and implementation of the first prototype of the WE_LEARN system for a section of organic chemistry.

Experimental Methodology

Software Engine and Internet Server:

The Cyberexam program from Virtual Learning Technologies, LLC (1401 20th Street South, Suite 300, Birmingham, AL 35205, URL: http://www.vlearning.com) was used. This engine provides an Internet server and maintains a database of statistics on students and their assignments. This server was chosen because it allows for any HTML commands (e.g., graphics files, multi-media support), on-line grading, a wide range of question types (e.g., multiple-choice, multiple-select, true/false, short answer, fill-in-the-blank, selection from a list), and random test generation. Flexibility for a large number of test administration techniques is also provided by this server software.

Question Database:

A multiple-select question database was created which was similar in style to the homework assigned from the textbook which was then used in the author's classroom. In fact, the vast majority of the questions were taken directly from the textbook to ensure that students who did not elect to utilize the WE_LEARN system did not suffer an unfair advantage relative to students who did utilize the new technology. Multiple-select questions were chosen as the primary question delivery mode, since organic chemistry requires a large amount of structural information that is best represented in a graphical format. Although the system had the capability for the students to draw structures and to submit them via Internet methods, we believed that chemical structure drawing and submission was not the aim of our study. Therefore, structures were generated for the students and were presented as part of a multiple-select format. The multiple-select format (where zero, any, or all of the answer choices are potentially correct) was chosen to prevent students from eliminating answers simply by choosing one correct answer (i.e., all answers must be considered). Incorrect (i.e., distracting) answers were generated by choosing the incorrect answers that students had submitted in written homework assignments during earlier semesters of teaching this course.

Assignment Methodology:

The homework was assigned in both a similar learning environment and a dissimilar learning environment. In the similar learning environment, examples of an assignment might be a series of questions relating to the nomenclature of aromatic compounds or a series of questions relating to the Grignard reactions of aldehydes and ketones. These assignments were made after the material had been covered in the lecture portion of the course. Both the question order and the presentation order of the answers were randomized in order to lessen the possibility that answers were memorized. Each assignment was structured to enable the student to complete the assignment in approximately ten minutes in length. This would allow students without their own internet access to be able to complete assignments in university computer labs, where there is a high background noise and activity level. When the assignments were completed, the students submitted their assignment for grading by clicking on the "Submit" button. Students received immediate feedback about their correct and their incorrect responses. Students were permitted to take the assignment a maximum of ten times. Each time that the assignment was taken, a different set of questions in a randomly generated order was obtained. For each in-class exam, there would be a minimum of eight individualized assignments that contained an average of 120 questions.

Prior to each exam, the students were moved into the dissimilar learning environment. This dissimilar learning environment was created by placing all of the questions that had been previously assigned as single concept questions (e.g., nomenclature, Grignard) into a "database" of questions which consisted of all the concepts taught for that exam. When students accessed this practice test, the Cyberexam server would select ten questions at random from the practice test "database". These practice tests could have multiple concepts and material unrelated to another question selected for students' assessment. Knowledge and proficiency of the material was assessed prior to the actual examination by submitting responses to these questions for grading. The computer immediately graded the questions and would indicate to students which questions were answered correctly and which questions were answered incorrectly. Because the material included in each practice exam was random, we hypothesized that if the students consistently received a grade of 100% on each practice test, then they would perform well on the in-class examination.

Samples of the question methodology and a sample practice test are maintained at our WE_LEARN server (http://www.access3.wvu.edu). Access to these sample assignments are available by using the account loginID of "guest" and password of "guest". The server is case-sensitive. Therefore, be sure to use lowercase letters.

Results and Discussion

Higher Test Scores:

A critical issue in this discussion is the change in exam style from a fill-in-the-blank testing strategy, which was used in all previous years, to a multiple-choice or multiple-select testing strategy which was used in the WE_LEARN system. Despite a large volume of literature to the contrary,22 many students and instructors believe that multiple-choice testing is easier and less educational than a fill-in-the-blank examination. For this purpose, the testing strategy for this course was changed one semester early to verify that similar grades are obtained in multiple-choice tests and fill-in-the-blank tests.

Figure 1 depicts data concerning the scores obtained in multiple choice exams as compared to fill-in-the-blank exams. The same textbook has been used for a number of years in this instructor's course. Having used the same textbook for a number of years, and having structured the course similarly each year allowed a comparison of the tests given over a several year timeframe. Therefore, in the fall '97 semester, a multiple-choice testing format was used. As seen in Figure 1, there was no significant change in the class averages for each test as the semester proceeds. Only exam 1 showed an increase. However, this exam covered only the first two chapters of the textbook as opposed to three chapters in each of the proceeding years. For all other exams, the class average obtained in the multiple-choice exam format was less than or equal to the class average obtained in previous years. Thus, these results are in agreement with previous studies that indicate that properly designed multiple-choice tests provide an adequate evaluation tool for student knowledge when compared to fill-in-the-blank methods.22 Examining the class average for the entire semester provides a further check of these data. The semester averages in the fall semesters (i.e., prior to the WE_LEARN system) are computed to be 63, 64, 60 and 60 for the years shown in Figure 1. This is in agreement with our assertion that no increase in class average was observed in shifting from fill-in-the-blank testing format to a multiple-choice testing format.

Data for the spring '98 semester, in which the WE_LEARN system was implemented, is shown in Figure 2. For all exams, an increase in the class average of 5-10% is observed. This increase occurred despite the multiple-select format of the exams of the WE_LEARN system. The variability in scores for exam 3 (1996) is related to subtle changes in exam coverage for the 1996 and 1997 years and the coverage of easier material in the 1996 spring semester. Examining the class average for the entire semester provides a further check of these data. In contrast to the data for the fall semesters, the semester averages in the spring semesters (Figure 2) are computed to be 66, 70, 65, and 74 in 1995-1998, respectively. The increased average of 74 for the spring '98 semester clearly shows that the change to the WE_LEARN system has had a significant positive impact on the class.

 The observed trend towards higher test scores could be challenged by questioning the make-up of the in-class exams and source of the questions for these exams. Prior to the introduction of the WE_LEARN system, a typical percentage of questions that were on the in-class exam which came directly from the assigned homework questions was 75%. Answers for these questions was readily available, both in the back of the textbook and in a solutions manual which could either be purchased by the students or was available on reserve in the library. After the introduction of the WE_LEARN system, the percentage of questions that were taken directly from the homework assignments was reduced to 62.5% after the first exam. The percentage of questions directly from the homework assignments has been further reduced with no concomitant decrease in exam scores. We believe that the observed increase in test scores originates from an enhanced conceptualization of the key concepts. However, we cannot rule out an explanation in which we have enabled the students to study better because the answers to all of the assigned problems have been gathered together to make for more ready student access and subsequent study. With either explanation, the system was responsible for the trend towards higher test scores.

Average Practice Exam Score Increases Prior to an In-class Exam:

Feedback concerning student performance is readily obtained from this system and provides a feedback mechanism to evaluate the system effectiveness. In addition to our average test score data presented above, a cumulative average of all students having taken a given assignment is available. As an alternative method for evaluating the WE_LEARN system effectiveness, we have recorded the average score for a practice test as a function of time or as a function of the number of completed attempts on the practice exam. Representative data are shown in Figures 3 and 4 for exam 4. Although data for only one exam is shown here, similar trends were observed for all other in-class exams throughout the course of the semester.

 A plot of the cumulative average for the practice test for exam 4 as a function of the number of times that the practice test was taken is shown in Figure 3. As can be seen in this figure, the average score starts at a value of 61% and appears to reach an asymptotic value of ca. 90%. This is an indication that performance on the practice exam increases with the number of attempts that students make on the system.

 

An alternative, but similar, view of this data is obtained by comparing the average practice exam score to the time prior to the in-class exam (Figure 4). Again, the same trend is observed, in that the practice exam average score rises from an initial value of 61% to a value approaching 90% as the in-class exam time approaches. Although only one practice exam's data are presented here, similar data were obtained for all other practice exams administered during the course of the semester.

One might ask if this data is exceptional, since an increased amount of review of the material should, in fact, result in an increase in the exam scores. The computer system provides a means for documentation of the amount of review time that was previously unavailable under the old system. We believe that the ability to document student effort represents a potential pedagogical advantage in comparison to previous methods. The cumulative class average and the cumulative amount of time spent on review could be used as an important diagnostic for the instructor. In the same way that cumulative class data is available, the Web-based method described in this manuscript also allows for examination of individual student effort.

Growth in Student Usage as a Function of Time:

Data related to the usage of the practice exam feature for this course are shown in Table 1. These data are presented to show how the students themselves reacted to the system. This increase in level of usage was not an effect mandated by the instructor nor was it induced by incentives (e.g., extra credit). The only incentive offered to use the system came at the beginning of the semester. Students could earn up to a maximum score of 1/4 of a letter grade, based upon their maximum percentage score for each assignment. There was no modification to this incentive at a later time, nor were there announcements about this incentive during the classroom lectures.

Each practice exam represents ten questions selected at random from the total database of questions related to material to be contained on the real exam. Each attempt represents a student actually completing a practice test and submitting it for grading. The trend is clearly towards an ever-increasing number of attempts prior to each exam. Only practice exam 3 showed a decrease in the number of attempts. This decrease is related to an administrative change in the location of the Cyberexam server to a location that gave a lower overall rate of data transmission over the Internet. This change resulted in long loading times (ca. 5-10 minutes per ten-question practice exam during peak usage periods). The overall system response became so slow that both instructor and students became frustrated and used it less often than desired from an instructional standpoint. This problem was rectified prior to the following exam (i.e., exam 4) by placing the server on a faster Ethernet hub.

 The percentage of the class that was using the WE_LEARN system shows a trend that is similar to the total number of attempts on the practice exam. The percentage of the class that was using the WE_LEARN system increased as the semester proceeded (Table 1). The downturn in the usage of the system for practice exam 3 must be related to the poor system response as described above. Since there was no instructor-driven reason for the students to use this system (i.e., no mandate nor incentive), we conclude that these student usage statistics show that the students themselves found the system to be worthwhile and used it as a worthwhile study aid.

 

Table 1. Usage During the Semester

Practice Exam Number

Number of Attempts

Percentage of Class Using the System

1

578

55.5%

2

891

57.0%

3

458

49.2%

4

1000

65.6%

4

1144

66.9%

 

Qualitative Data Relating to Faculty/Student Interactions:

In addition to the positive changes in the quantitative assessment tools originating from the class after implementation of the WE_LEARN system, qualitative data also indicate that positive changes have occurred in the learning environment. These qualitative indicators include the number of students coming to the instructor's office or to a help-session during the week of an in-class exam, the types of questions asked, and the air of student confidence exuded during the in-class exams.

 Prior to the implementation of the WE_LEARN system, long lines of students outside the instructor's office were typical during an exam week. Although no actual statistics of the number of students requesting office help were kept, an estimate of the average number of students coming to the office for help was 20-30 students or about 15% of the class. An estimate of the time required to answer the questions from these students is 20 hours. After the introduction of the WE_LEARN system, a dramatic change in the number of students requiring office help took place. The change was so dramatic that quantifying the magnitude of the change was rather easy. After the WE_LEARN system was implemented, an average of only 3 students/exam, requiring only 15 minutes of total time, came for office assistance. A similar dramatic effect was noted in the attendance at help sessions, which this instructor has given on the night prior to an exam for many years. Prior to the introduction of the WE_LEARN system, a typical attendance at the help session was ca. 75% of the class enrollment. After the introduction of the WE_LEARN system, attendance at help sessions dropped to ca. 33% of the class enrollment.

The types of questions that were asked by students were also of a different nature following the introduction of the WE_LEARN system. Before the implementation of the WE_LEARN system, the majority of the questions posed by students showed almost no comprehension of the basic concepts. This phenomenon was observed in the questions that were posed in the office and in the questions that were posed at the help sessions prior to the exam. After the implementation of the WE_LEARN system, this type of question disappeared almost completely. In fact, the routine type of question came from the students with the strongest critical thinking skills and were very specific to the types of reaction conditions or subtle structural variations which might lead to a different set of products.

A final qualitative indication of the change in student attitude/perception was the change in classroom atmosphere during the in-class exams. Prior to the WE_LEARN system, there was an atmosphere of tension and anxiety. Almost all students utilized a minimum of one hour to complete the exam and ca. 10% of the class utilized the entire two hour time period which was allocated for the exam. This distribution of time to complete the exam occurred in both the fill-in-the-blank questions of prior years and in the multiple-choice questions of the Fall '97 semester. After the introduction of the WE_LEARN system, most students took much less than one hour to complete the exam and no one took the entire examination period of two hours. The faces of the students appeared much more confident. These data indicate that it is the WE_LEARN system that is responsible for the change in the tension level and the time required to complete the exam and not a change to a multiple-select format.

Summary and Conclusions

Taken together, these quantitative and qualitative data clearly indicate that a positive change has occurred in the classroom upon the introduction of the Web-based Enhanced Learning Evaluation And Resource Network (WE_LEARN) system. In comparison to previous years, students obtained higher test scores and were more confident when taking exams. In addition, students used the system as a learning tool, as demonstrated quantitatively in the increase of the average score of the practice exams and by the increased usage as the semester progressed. Qualitative data on the types of questions and the number of questions asked by the students indicate that the students have a better feeling about their level of knowledge prior to the exams when using the WE_LEARN system. It is tempting to conclude that the higher test scores indicate that the students have learned more when using this system. However, this conclusion requires a firm definition of the meaning of learning and interpretation of variables that are beyond the level of the current study. We conclude that a positive change has occurred in the classroom environment and are seeking to further elaborate the reasons behind the positive results.

Acknowledgements

Many people have been involved behind the scenes in providing the technical expertise to accomplish the implementation of the WE_LEARN system. The West Virginia University Office of Academic Computing (Donald McLaughlin, Sherrie Kelly, and Bradley Forbes) have been extensively involved in the discussions concerning system requirements and hardware maintenance. Virtual Learning Technologies, LLC (Gary Hulse, James Johnson, and Morgan Sapp) was an invaluable asset on those moments when software problems threatened to ruin this classroom experiment. Finally, the 130 students in the major author's section of Organic Chemistry (Spring '98) are gratefully thanked for their efforts and patience when problems arose.

Literature Cited

1. Tissue, B. M. J. Chem. Ed. 1996, 73 ( 1), January, 65-8.
2. Tissue, B. M. Trends Anal. Chem. 1997, 16, 490-95.
3. Smith, S.; Stovall, I. J. Chem. Ed. 1996, 73 ( 10), October, 911.
4. Guzmān, M.; Ortega, P.; Vera, L. J. Chem. Ed. 1996, 73 ( 10), October, 921.
5. Bruist, M. F. J. Chem. Ed. 1998, 75 ( 3), March, 372-3.
6. Masson, B. L. J. Chem. Ed. 1996, 73 ( 10), October, 918.
7. Smith, W. R.; Missen, R. W. J. Chem. Ed. 1997, 74 ( 11), November, 1369.
8. Crouch, R. D.; Holden, M. S.; Samet, C. J. Chem. Ed. 1996, 73 ( 10), October, 916.
9. Childs, S. L.; Hagen, K. S. J. Chem. Ed. 1996, 73 ( 10), October, 917.
10. Liu, D.; Walter, L. J.; Brooks, D. W. J. Chem. Ed. 1998, 75 ( 1), January, 123-4.
11. Matthews, F. J. J. Chem. Ed. 1997, 74 ( 8), August, 1011-14.
12. Mehan, H. Anthropology and Education Quarterly 1989, 20 ( 1), 4-22.
13. Dubinskas, F. A. NAPA Bulletin 1993, 12, 1-11.
14. Clark, K.; Dwyer, F. Int. J. Instructional Media 1998, 25 ( 1), 55-63.
15. Carter, J. The Computing Teacher 1984 12, 53-55.
16. Alessi, S. M. Computer-Based Instruction Methods and Development: Prentice Hall: New Jersey, 1991.
17. Mory, E. Educational Technology Research and Development 1992, 40 ( 3), 5.
18. Lagowski, J. J. J. Chem. Ed. 1998, 75 ( 4), April, 425-36.
19. Osin, L.; Lesgold, A. Review of Educational Research 1996, 66 ( 4), Winter, 621-656.
20. Farnham-Diggory, S. Review of Educational Research 1994, 64 ( 3), Fall, 463-77.
21. Anderson, K.; McClard, A. Anthropology and Education Quarterly 1993, 24 ( 2), June, 159-177.
22. Haladyna, T. M. Developing and Validating Multiple-Choice Test Items: Lawrence Erlbaum Associates: Hillsdale, New Jersey, 1994.