Case Study 1
Home Up






Health Care Informatics

Spring 2000 Evaluation

Rosalee J. Seymour (Dr. Roz), EdD, RN

East Tennessee State University

In 1998 the National Education Association (NEA) and Blackboard Inc., jointly commissioned The Institute for Higher Education Policy to examine the multitude of quality assurances (QA) benchmarks which have been produced since 1994. These benchmarks or QA strategies for distance education have been in existence for many years. The question the commissioning organizations wanted to answer was are there specific benchmarks applicable to Internet-based distance education. The answer was found in the literature, interviews with faculty, administration, and students with substantial experience in distance education (as in offering more than one degree program via online distance learning). Of the 45 identified benchmarks found using these methods, 24 were determined to be essential to ensure quality in Internet-based distance education. They are recommended for use by all levels of administration, accrediting bodies, state legislatures as well as faculty and students to make reasonable and informed judgments with regard to the quality of Internet-based distance education both before and after it is offered.

In an effort to assess the degree to which the benchmarks for Internet-based distance education where met I adapted the program benchmarks to be able to use them to evaluate one course offered at ETSU. The course will be presented here as a case study using all 24 benchmarks, resulting from the commissioned study, as evaluation questions.

 Course Description

This course is open to all graduate or undergraduate students interested in health care informatics. The course consists of one in-classroom face-to-face meeting on the first night of class, four scheduled field trips (one is optional the other three are required), and a final face-to-face meeting on the last day of class, at a local restaurant, for exit interviews.

The course incorporated WebBoard for asynchronous conferences and Chat for synchronous conferences. Three small group, peer moderated, asynchronous conferences consisting of no fewer than three students were established by week three of the course. Peer moderators evolved as leaders do in any group. Specific rules were set up for moderator duties. A plenary, faculty moderated, asynchronous conference was established in week one. This conference was the constant for all members, a place where all small conference teams reported in summary fashion, on their discussions, every three weeks.  There were five synchronous chats. Three chats with experts in a variety of informatics areas (library database and search engine use by two librarians, clinical information systems by a nurse computer expert, OASIS by the state manager and the state nurse consultant to all home care sites implementing it). The other two chats where with the faculty and topics for these where established in the course calendar. One of these chats was a discussion of a case study, posted two weeks before the chat, regarding the implementation of a health information system in a clinic setting.

Course assignments consisted of reading chapters in the required text. Chapters were not assigned but selected by student as appropriate to the content listed on the calendar for that week. Answering questions, in the class conference, on the first three chapters of the text reading, was assigned during the first three weeks to get students used to the chat space as where introductions of each other to the class. Students were required to take part in the class and small conferences at least once a week, posting meaningful content.

Completion of a Web Site Evaluation for any site used in any part of the course (referring class to useful site, used as reference in a project or paper) using a Web Site Evaluation Tool posted on the class web page.

Two evaluations of health information systems, visited on field trips to local facilities, were completed, by students in pairs, using a tool posted on the class web site. The first was a report of the first visit and the second incorporated the first visits evaluation with the second including a discussion comparing the system capabilities at the two sites. The sites where a large veterans administration hospital and a private, acute care facility. The other two field trips, one to visit a classroom using A.D.A.M. and other computer based software for the teaching and learning of anatomy and physiology and the second to the Advanced Visualization Laboratory (AVL). The AVL which is part of a degree program on the ETSU campus is one of only nine in the world doing virtual three dimensional work. The student could chose not to attend one of these last two field trips. No other assignment were associated with these last two field trips. Student only had to post follow up discussion on WebBoard regarding their value to the achievement of course outcomes.

The final assignment was a five page,  health informatics ethical/social issue paper,  submitted using creative electronic means (most used Microsoft Power Point), to every student in the course. Each student was required to read, comment on, post relevant questions and responses in discussion on the class conference, and lastly post a grade on each paper to the faculty. The ethical/social issues paper must have been publication ready using guidelines for publication from the On-line Journal of Nursing Informatics All papers/products could be submitted to faculty via attachment in email, in person, or Faxed and required use of the appropriate publications format for the students discipline (APA format for nursing).

Students were provided access to a help system via the university distance education department and in addition to open access to faculty via e-mail, phone, fax and/or office hours for which a 48 hour turn around time was established. Each signed a university wide permission form to participate in chat.

A large reference list was placed on the course web site including web addresses (URLs), print journal articles, books, personal email addresses for outside resources, and other documents. Both hypertext and embedded links were placed throughout the reference list for ease of use.

Evaluation was conducted at each face to face field trip and using an evaluation tool from the course web space during week six of the course. Evaluation was also conducted on the last week of the course via the same tool and via exit interviews and students’ writing letters to students needing to take this course next semester telling them how it went and how to get ready. Paper and projects where graded using the preset guidelines and class participation was evaluated by monitoring the value of comments placed in class conference, chats with experts, small group conferences and the statistics on the number of times a student was on the chat space.

The Web Site consisted of a welcome page with the faculty email address, dates of class, number and time of face-to-face meetings and sites for field trips,  links to a pre-course assessment of readiness to take an online course and a link to an article about computer rage. Class conference was explained as was required participation in WebBoard.

The welcome page was a modified listed version of the course described above and provided a hyperlink to a course registration site, a link to acquire a password to enter the course web site itself, and an access point to WebBoard. WebBoard was set up by faculty, after students registered for the course. Web Board sends out an automatic Welcome message with the individual, faculty assigned password for access to chat and conferences. WebBoard has a large help component. During the first three weeks the assignments on WebBoard where designed to increase comfort with use and to begin to build a community of online learners for this course, such as introductions and required reading of the text and responding to specific questions on the reading. Other course requirements such as email addresses, types of computer hardware and software needs were spelled out.


I will proceed to describe in seven categories: a) institutional, b) course development, c) teaching/learning, d) course structure, e) student support, f) faculty support, and g) evaluation and assessment if and how each of the aforementioned 24 benchmarks under those categories were achieved during the offering of this course.

 Institutional Support Benchmarks

According to the benchmarks study there are three components to examine under institutional support when evaluation is implemented. The questions that must be answered yes t establish high quality in this area are: 1) Is a documented technology plan including electronic security measures such as  password protection, encryption, and back-up system in place and operational and does it insure quality, integrity, and validity of information, 2) Is the reliability of the technology delivery system as failsafe as possible, and 3) is there a centralized system which provided support for building and maintaining the distance education infrastructure?

Documented technology plan. As can be seen by the course description both the faculty and students had log in and passwords assigned which were private to them. The password assigned to students for use in WebBoard could be changed by the student if desired. Participation on the chat space required signatures with an explanation of the public nature of any chat space so students would be aware others would be seeing all posts made to the site.

At one point during the course a student from another technology intensive course was visiting and participating in this class (because of many factors she was unable to participate in the course chat for her course enough times). The students in this course agreed she could participate in our discussions. They gave permission for this stranger to take part. Faculty entered this student and gave her a password and log in to chat space only.

Security measures documented and in place. After about the third week of the course the course administrator in the distance education department notified the faculty that several students had dropped the course and that the password for entrance into the course web site should be changed. The course faculty chose a new password and sent an email to all students, those who dropped and those remaining, to announce that the new password would go into effect in one week and why. This email also explained that students no longer in the course would also be dropped from WebBoard and it requested that they complete one of the posted evaluation tools and send me an email explaining how I could have helped them to stay in the course. A second email was sent to only those remaining in the course, with the new password and log in and the exact date for the change. At this time the dropped students where also deleted from the WebBoard conferences and students in small conferences were notified of missing members names.

Failsafe reliability. Whenever the entire system was to be down for any reason the faculty was notified by the distance education administrator. The faculty passed this information on to students via email. This administrator was available on what seemed like a 24 hour a day basis for answering any faculty questions and had a posted help site for students to which the course had an embedded link. On one occasion, one student had used the administrator help site over five times in two days and the administrator made the faculty aware of this. This allowed the faculty to intervene and counsel the student along with the help person.

At one time during the course the administrator thought students who were not officially in the course had entered and completed mid-course evaluations. Again this allowed the faculty to investigate. It was found that these were the students who had dropped the course complying with a request to complete this tool, made in the final email to them.

At one time during the course a student not in this course but in the Introduction to Professional Nursing course, and completing an informatics assignment on the topic of computer use in education, chose to request to enter the health informatics web site. He was given the password to the site and all students in the course where informed that this was happening. He was not given access to WebBoard so no breach of confidentiality occurred here.

Course Development Benchmarks

The benchmarks in this area require the evaluator to ask four questions: 1) Are there guidelines for minimum standards for course development, design, and delivery, 2) Do learning outcomes determine the technology to be used to deliver content rather than the availability of existing technology, 3) Are instructional materials reviewed, 4) Is the course designed to require students to engage themselves in analysis, synthesis, and evaluation?

Course development, design, and delivery. The guidelines that exist for the development of any course in the College of Nursing were followed in the development, design, and delivery of this course. It has a structured syllabus with all outcomes, teaching/learning methods and evaluation methods spelled out. It includes guidelines for projects with grading criteria.

Learning outcomes determining technology use. There was much more technology available to be used in this course, as in other software such as Real Audio, Real Video, and Acrobat Reader, which could have been used. Students were provided with links to these packages which could be downloaded free from online sites but they were not required because they were not needed to achieve learning outcomes. A minimum of computer technology was used: email, Internet, synchronous and asynchronous chat, fax, and phone. Students needed to know how to send attachments via email, how to search the web using various browsers and search engines, how to post to a conference or chat space, and how to access on line or in-library data bases to accomplish outcomes.

Instructional materials reviewed. Instructional materials were submitted to peers on other campuses, who are teaching a similar course. Feedback from them was excellent. Peers teaching other online courses at ETSU reviewed the course web pages and the strategies for learning. The distance education administrator reviewed the public page and made suggestions for improvement which were followed. The department chair of the PMNU department, the Associate Dean for Undergraduate and Graduate Programs, the Dean of the College of Nursing, and several faculty where sent the course web site and a password to review the pages prior to the course offering. The PMNU Chair gave feed back which was used. The National League for Nursing was making an accreditation visit to the college during the time this course was being offered. It in no way hindered that review.

Student engagement in analysis, synthesis, and evaluation. Students, as can be seen in the previously described assignments, were consistently engaged in analysis of text material and systems and readings; they conducted multiple syntheses of data collected from the field, and conducted evaluations of web sites, papers submitted by other students, self evaluation to assess outcome achievement, and course and faculty evaluation. Early in the course they complained that one other assignment (the preparation of several annotated bibliographies) would make the course a burden and demonstrated how other assignments incorporated the same tasks and outcomes. This assignment was dropped from the course and students determined how they wanted the grade points reassigned.

Teaching and Learning Benchmarks

The questions asked of a course, faculty, and students here are: 1) is student interaction with faculty and other students seen as an essential characteristic and is it facilitated in various ways, 2) Is feedback on assignments and questions constructive and provided in a timely manner, and 3) are students instructed in the proper methods of effective research, including assessment of the validity of resources.

Student interaction essentials. Student to student, faculty to student, and student to outside expert interactions were essential to meeting course outcomes. Interaction was facilitated by face-to-face, email, fax, office hours, online scheduled synchronous chat and unscheduled online conferences.

Constructive and timely feedback. Feedback on assignments was extensive, given by faculty and peers, revisions were allowed and encouraged. Feedback via email was guaranteed within 48 hours but was usually within minutes. The faculty member was on vacation during spring break but continued to respond to students who were participating in chat or with her via email.

Effective research methods and assessment of resources. Students were instructed in ways to collect, sort, and cluster data from a variety of sources to construct various types of reports and projects. They were instructed in synchronous chat with two librarians (one education librarian and one medical librarian) on how to use online data bases and search engines to conduct effective searches. They were instructed on how to use tools to assess data acquired from Web sites and documents using a tool designed for the course. They were instructed how to collect verbal data from health systems administrators, who had previously provided them with hands on experience with health information systems, to use in various reports. They were encouraged to check the validity of resources by calling or e-mailing experts such as members of the Nursing Informatics News group, the persons who gave them health systems information during field trips, and to use on line dictionaries and their text book to define computer terminology they were not familiar with. Every project required use of the skills of data collection, data analysis, synthesis, evaluation and revision.

Course Structure Benchmarks

The questions an evaluator must ask in this area are: 1) Do students pre-assess  to see if they are ready and  possess the self motivation and commitment to learn at a distance, 2) do they have access to the technology required by the course design, 3) Are they provided with supplemental information and learning outcomes for the entire course and each course assignment, 4) Is there access to sufficient library resources including virtual library resources, 5) have faculty and students agreed to time lines for assignment due dates and faculty responses?.

Pre-assessment for distance learning. A readiness scale which helps students assess their readiness to take part in an online course appears on the welcome page for the course. It is a tool used by many colleges for the same purpose. Students are asked to respond to the tool, on-line and email the results to the faculty member before registering for the course. Students who achieve less than a score of 20 on the instrument are reminded that they may want to reconsider signing up for an online course at this time. All of the students did not take the tool so much of this data is missing. It is clear from the attrition rate and the comments from those who dropped the course that many where not ready (this information will be elaborated on under the last benchmark section, evaluation). Students were also encouraged too hyperlink to and read an article on Computer Rage from the USA Today web site. No feedback is available to assess whether any read the article. A meeting with the advisor with many of the RN/BSN students who enrolled in but did not complete the course, later validated that they had not taken even the basic computer course required of all ETSU graduates.

Access to technology.  The course design required minimal technology. It was assumed that students wishing to take the course would read the welcome page which had specific technology requirements listed. Any student registered at ETSU has an e-mail address and access to computers on any of its campuses. However,  access was not an issue but where the computer being accessed for this course was located was an issue. For example, it was discovered in week three that several students did not have computers at home. These students seemed to think they could take this entire course from work computers. They soon discovered that firewalls on work computers would not let them use synchronous chat or email for course work. Many also discovered they could not get into the course from work computers because it required a password. Many realized they would not have the time to participate via work computers only, at the level required. Several students had never owned nor used a computer. They purchased them during the first week of the course and so, even though they had access, had severely miscalculated the steep learning curve.  The students with all of the required hardware and skill using word processing, presentation software as well as e-mail and Internet access had little difficulty (these issues will be addressed in more detail under assessment benchmarks).

Supplemental course information. All course requirements were spelled out in detail in an online syllabus, calendar, assignment guidelines, and reference list. WebBoard was explained and practice sessions were set up in the first three weeks. On the first class night the course met face-to-face and at that time some articles were distributed, several free journals were recommended as were ways to acquire them made available.

Library resource sufficiency. Students had email access to the Sherrod Library Education Librarian and the online library services with access to multiple data bases. They were instructed as to how to use data bases and search engines by the education librarian and a medical librarian in a ‘chat with the experts’ session. Pubmed was available to them in the virtual world as well. Many resources were emailed as links by faculty. All students had easy access to interlibrary loan from the medical library as well as Sherrod and many local hospital and town libraries. One student commented on the overwhelming amount of available resources. 

Agreement on completion of assignment times and faculty responses. It was agreed early in the course that due dates were to be hard and fast but that extensions would be given where needed. The field trips were not scheduled until the first night of class in order to get agreement from all students as to the best time. These were scheduled during the pre-set class time of Tuesdays from 6-9 so that a regular schedule could be adhered to. All students were working full time. One field trip was optional and the student could choose, from two, which one they could not attend. It was agreed that faculty would respond to email no later than 48 hours. Faculty office hours appeared on the class calendar along with email, phone, and voice mail information. It was agreed that a first draft of any project would gain high critique with in one week and there would be one opportunity for revising. Revisions would be resubmitted within one week and receive a final grade within one week of re-submission.

The course motto which was stated early on the calendar was, “Blessed are the flexible for they shall not be bent out of shape.” This motto was used often as in when asking for extensions, when asking for one project to be taken out of the course, when asking for more time for anything.

Student Support Benchmarks

Student support is a broad area and requires the evaluator to ask: 1) do students receive information including admission, tuition and fees, books and supplies, technical and proctoring advice, and student support services, 2) do students get hands-on training and information to aid them in securing material through electronic databases, interlibrary loans, government archives, news services, and other sources, 3) do students have access to technical assistance, detailed instruction about the electronic media used, practice sessions prior to beginning of the course and convenient access to technical support staff, 4) are questions directed to students service personnel answered accurately and quickly and are student complaints handled in a structured way?

Students receive information. There is a catalogue published, there is a home page for the distance education learner and there is a course welcome web site. Each of these sources describes and provides access to information or direct links to access for admissions, the book store or online text book sellers. Student support services are in place as for any campus student.

Hands-on training and information. Librarians conducted a two hour ‘chat with the experts’ session to assist students with all issues regarding access and use of library and on line services, data bases and other sources. The faculty provided access to free related journals. Students were asked to join the nursing informatics listserv and were given directions as to how to accomplish this. Students at ETSU are all required to take a computer hands on course during the lower division component of the curriculum. Practice sessions were held in chat spaces during the first three weeks of class by having students introduce themselves and respond to questions, posted by the faculty, from the first three chapters of the text book. No other practice sessions were conducted. This will be commented on again in evaluating segment of paper.

Access to technical assistance. Technical assistance was available from the distance education department via a hot link for questions. These were responded to with in a few hours. Faculty provided technical assistance as well which speaks to the need for faculty to be knowledgeable not only about content of the course but also about the courseware they are using.

Student services. In the ETSU College of Nursing there is a student services department. The staff in this department recruited the majority of the students who signed up for this course. They provide ongoing academic counseling for all students in the nursing program regardless of what type of course they are taking. Students from other majors also have university student services departments to get assistance from. Each of the student services groups and the college of nursing has formalized ways to address student complaints no matter what courses they are in or what method those courses are offered in.

Faculty Support Benchmarks

It is in this area that one must be pro-active as well as with students. If faculty do not have the support needed they cannot do justice to any curriculum in any format. In this area it is suggested that the following questions be asked in evaluation: 1) do faculty have technical assistance in course development and are they encouraged to use it, 2) are faculty assisted in making a transition from classroom teaching to online instruction and are they assessed during the process, 3) is there training and assistance including peer mentoring continuous and ongoing throughout the course, 4) are faculty provided with written resources to deal with issues arising from student use of electronically-accessed data?

Technical assistance in course development. The only technical assistance this faculty member required was how to use Front Page to design the course Welcome and Web site, how to set up, manage, and use WebBoard for synchronous and asynchronous chat. This was provided by request of the faculty member to the distance education department via three private tutorial sessions during the summer, on the faculty’s own time. The department provided support by upgrading the faculty’s office computer and providing Front Page 2000 for installation at home and on campus for ease of managing the course. Access to the web on campus was by high speed ISDN line and at home the faculty changed from a slow 56K modem, which tied up the phone lines to a more expensive, faster, cable modem with no financial support, even after a formal request. There was little encouragement to use technical support. The faculty member often became the technical support for others attempting to use one or two online techniques in their courses.

Transition from classroom to online. Little help was needed in this area. This course had been taught in traditional classrooms three times previous to the online version. The course remained virtually the same only the space and distance changed.

Training, peer mentoring continuous. There would have been plenty of help had I needed it. As a matter of fact a year prior to offering the class online I attend a formal session at ETSU on another online course system which was helpful. I also read volumes about online teaching and learning in various print and online journals and consulted with peers on other campuses who are teaching similar courses. I also feel this will be one of the hardest areas for many faculty. It bears mentioning that this course faculty member attended an on campus faculty development session  provided by two peers who had just completed an online course. This session provided some clues to the need for clearly stated project guidelines and descriptions of how to conduct the course. Technical support also came from a faculty list set up by the Dean of Distance Learning and a list set up by the nine peer faculty from other colleges, currently teaching online courses at ETSU. 

Written resources regarding student use of electronically-accessed data issues. I was never provided with anything in this category. I did however use the web to find out this information on my own and so perhaps did not seek it on campus. That does not mean it does not exist. A policy was developed for the university, during this semester and that is now in writing and will be very helpful in the future regarding what constitutes a totally online course versus a partially on line course and other policy areas. However I still see nothing on the horizon on this area regarding issues related to student use of electronically-accessed data.  At this time I cannot say I am certain what this benchmark means. I will need to conduct a literature search. 

Evaluation and Assessment Benchmarks

It is here that one sees that the benchmarks are for programs and not one course but I feel I can still demonstrate my success or lack of success in one course using the following questions from this area: 1) is the educational effectiveness and teaching/learning process assessed through an evaluation process that uses several methods and applies specific standards, 2) are data on enrollment, costs, and successful/innovative uses of technology used to evaluate program effectiveness, and 3) are intended learning outcomes reviewed to ensure clarity, utility, and appropriateness?

An evaluation process using several methods and applying specific standards. In this course several methods were used to evaluate the teaching/learning process and its educational effectiveness. There were two evaluation tools made available on line. Students were asked to complete the tool at mid-course and changes were made based on this evaluation. They were asked to evaluate if they dropped the course. These results will be used to revise the course for future offerings, where appropriate. Students were asked to write a letter to a friend they new would be taking the course next semester to tell them how to get ready for it. Standards established as guidelines for projects were matched with student accomplishment of the projects and students in exit interviews were asked if all projects helped them to achieve course outcomes. 

Exit interviews also were conducted on the last day of class to ask for feedback on ways to change the course for the next offering. Suggestions were do not put the Annotated Bibliography assignment back in, continue to require the final paper in publishable form but do not require submission of it as part of the course, during the first three weeks make it clear that the questions will be put in chat to assist us to learn how to use the system, cut down the reference list, especially web sites they are overwhelming, plan synchronous chats more thoroughly with experts, keep all field trips, keep the text book and continue to not have assigned readings just keep using the discussion questions, discontinue small conferences if less than two people remain in them, create a small conference or space for current links, keep posting calendar changes in a different color and keep sending email when changes or additions are posted. Students felt the chats went well and that the class conference was a valuable and interesting space that took much more time than they had allotted for in their schedules. This time factor was true in general. Most commented the course required more than they expected. Some were taking another online course in the science department and felt the two together were not possible to take because of the time. Not that the time was greater than other class time but that it seemed to be when spread out rather than being in two or three hour time blocks in the week. They felt like the work was never done because they had to check the site and the class conference on WebBoard everyday in order not to miss anything. They felt they established a unique class, made friends, had good faculty support. They were concerned about all of the drop outs but seemed to understand why they occurred. Many admitted they were taking this course as an elective and so dropped it when they found it requiring more time than they expected it would.

Exit interviews were also conducted with the five online experts who took part in synchronous chats. Their feedback was that they felt they went well, they wished they had done as I encouraged them which was to post an outline to me ahead of time along with a web site or two for students to visit prior to the chat. The students felt these went well but would have liked faster bandwidth to increase the speed so the time lag between responses did not get them off track. They also wanted a bit more direction from the experts. They felt the best asynchronous chats were with the course faculty but even then they go kicked offline by their service provider several times during one hour chats. 

Data on enrollment, costs and innovation. I really have no data on cost but have been asked to give report to the Dean of Distance Education for his use to evaluate these aspects. We actually enrolled 12 paid students who started the course in the first week, six (6) more late enrolled and started the course in the second week for a total of 18. Of those 18, 9 were undergraduate RN/BSN students taking the course as an elective, two (2) were non-health related majors and seven (7) were nursing masters students taking this as a required course. Six students completed the course: two undergraduate RN/BSN, three graduate students and one undeclared non-health related student. There are many recommendations in the literature about the need for small size classes in online environments. Students commented at the end of the course that they do not see how they would have kept up with the discussion on the WebBoard in class conference if everyone who signed up had remained in the course. Students dropped for illness in the family, conflicts with other courses, because it was a non-required elective and as such they thought it was too time consuming. Two dropped because of lack of knowledge about use or lack of ownership of a computer. The students who enrolled late all dropped. Two stated they could never get caught up. I feel the course was innovative but that no high level or high speed computer was needed to complete it successfully.

Regular review of learning outcomes. Like most courses there were times when students did not understand guidelines for projects and so became frustrated trying to successfully achieve outcomes. It is worth noting here that to nursing students only a grade of A+++ seems to mean successful achievement. The most common example was in the health information system evaluation comparison project. Most felt that it could be done in pairs even after the first assessment was conducted. This was never said and was not true but I had to allow two pairs to complete the entire project this way because of the lack of clarity or understanding.

Students did not know the difference between a web page, web document, or web site. This made it impossible for them to accurately cite sources, in text and on reference lists, using APA or any other paper format. There was much frustration expressed about APA.

Students did not want to conduct complete web site assessments until later in the course when they began to see the value as they found some very unreliable patient education sites. Learning outcomes will be reviewed.


This course met most if not all 24 benchmarks successfully. All students who stuck it out stated we need to offer more online courses. This appears to be a testament to establishing a cadre of students familiar with this learning environment who are or can becomes self motivated learners and can manage time in fundamental ways. The faculty was intimately knowledgeable about the software and hardware and its use in the course which probably was responsible for keeping down student complaints about the web site, software, and hardware requirements. Faculty was available in multiple ways as in office hours, email, fax, synchronous chat, asynchronous class conference, and telephone. Students took advantage of each of these as they felt they needed them.

Students were available to faculty and each other in most of the same ways. They used each other frequently for various consultations about this and other courses they were taking at the same time. They gave support to each other and to the faculty via many virtual mechanisms. The innovative method of ‘chat with the expert’ was only partially successful. In the future, use of the same experts will help. The experts now know how important it is to post an outline ahead of the chat and can anticipate some of the problems inherent in this format. Most common problem was response time delays making the chat a bit disjointed at times.

The other innovative method of field trips, with some being not required, was very successful. The field trips took us to exciting implementation sites where students could see informatics in action. The pharmacy robot, the VA medication administration using scanners, ADAM as a learning tool, and the Advanced Visualization lab have been much discussed. A video tape about the future impact of technology on health care and one on a telehealth session in progress were circulated and viewed be each student and discussed on WebBoard. This learning tool gained high praise.

Executive Summary

Problems in a list. Problems are broken down into I. technological, II. computer literacy, and III. pedagogical teacher and student. Section IV is a list of problem prevention strategies used by faculty but not by students. Section V lists strategies used during the course to try to reduce student time. Section VI is a list of the results from the exit interviews with students on the last night of the course. Section VII are the changes planned for the next course offering.

I.                  The most common technological problems were:

1. getting kicked off chat by service provider during one hour chat sessions.

2. changes of email addresses for both students and faculty.

3. computer down time (although this was minimal and on weekends).

II.          The most common computer literacy problems were:

1. students never participated in chat space before so many students did not enter  the class  conference space until week three finding themselves very far behind. Even when given time to catch up, they did or could not.

2. reading online is hard to do if large amount of material. Students did not begin to print until late in course

3. no computer at home

4. new computer at home with no knowledge of how to use it

5. did not know how to send attachments via email

6.  no knowledge of how to cut and paste into email or WebBoard 

from another software program.

7. no knowledge of how to create hot links in email or WebBoard

8. no knowledge of how to format printer to print in landscape or to create charts in word processing package.

9. no knowledge of how to download and install free software from the web (although no downloads were required to succeed in the course options to do this were offered).

III.      The most common pedagogical issues for faculty and students

1. too many assignments

2. first two weeks in class conference seemed to have no purpose.

3. scheduling of a class day and time caused conflict. Faculty expected students to set aside that time for face-to-face options and students expected to have that time free if they wanted to.

4. you never know if the excuses are real such as my computer was down, my email crashed, I have to work so I cannot do that live chat or field trip. Students did not seem to see this as cutting class as they would have in a regular classroom.

5. students lacked awareness of time and organization skills needed to take course on-line and to do off-line reading

6. lack of knowledge of use of APA or any other standard for formatting papers.

7. students expressed frustration with lack of ability and skill to learn on their own.

IV         Prevention strategies that were not used by students:

1. pre-course survey regarding readiness to take distance course via online.

2. article about computer rage.

3. web site hyperlink to a free, short course on how to use the Web including chat.

4. help mechanism in WebBoard

5. Link to APA Web site to download APA Helper 2.0 free.

V.                  Strategies used during the course to reduce time use, based on student input

1. Cut one assignment,.the annotated bibliography

2. allowed students to work in pairs on Health Information Site Visit report and comparison.

3. accepted issues paper ready to be submitted for publication with minor revisions

VI.        Exit interview with students: results

1.  break up course calendar into more pages.

2.  tell students early to print and read

3.  keep conferences and chats

4.  give a bit more instruction on chat space in a synchronous environment

5.  keep chat with experts

6.  reduce the number of required website evaluations or shorten the format

7.  keep required sign on to informatics listserv very helpful for external sources

8.  all nursing students need this course and need to take it on online for need to learn content and method

9.  offer more on-line courses

10. in first face-to- face class put all students on web or computer and go over course online together as in logging on, passwords etc.

11. embedded links at top of calendar to each week were great

12. reference list was great, especially links to web sites

13. text excellent

14. field trips excellent

15. allowing pairs for one project great

16. APA awful

17. requirement for graduate students to submit issues paper for publication excellent but cannot be accomplished until course is over.

VII.      Changes projected for next course offering. Based on student, peer, expert, and self feedback I plan to make the following changes in this course for the next offering.

1.  take a careful look at all project guidelines

2.  keep the annotated bibliography requirement out of the course

3.  try to find out sooner who does not have a computer at home and keep them from taking the course (for this I will meet with student services person who recruited the RN students without computers for the course)

4.  switch the course from Front Page to Blackboard. Spread the Calendar over more pages instead of just using embedded links to sections

5.  require everyone to take the pre-course readiness for online learning tool

6.  not allow late registration without meeting with the students in person

7.  try to keep the same experts

8.  try to keep the same field trips

9.  make known early to print for reading

10. make known early to reserve class night and time until after first night of class is over and schedule for field trips and chats are set to reduce conflicts with student work schedules

11. make known early that skipping a required chat or other session is equivalent to skipping a class

12. make known the first three weeks in chat will be to introduce selves, to answer specific questions about assigned readings, to help with using chat, and understanding computers as tools 

VIII.     Attachments that can be or are made available:

1. course Welcome page (online)       http://de.etsu.edu/001pmnu40175017/Public/index.htm

2. copy of executive summary of Online Benchmarks study

3. this course will be moving from Front Page 2000 to BlackBoard as the delivery software for the online format. 


Institute for Higher Education Policy  (April, 2000). Quality on the Line: Benchmarks for success in Internet-based distance education. Executive summary p. 1-5. [Available: the Institute for Higher Education Policy , 1320 19th St., NW, suite 400, Washington, DC 20036. Internet: http://www.ihep.com/qualityonline.pdf].