EVALUATING WEB-BASED CONTINUING EDUCATION COURSES: COMMENTS ON CREATING AN EVALUATIVE SUBJECT AREA GUIDE

Dale Mueller, University of Phoeniz Online Campus <dmueller@pepperdine.edu>

URL: http://www.love2learn.com

PURPOSE OF THE PROJECT
The purpose of this project was to create an evaluative subject area guide regarding continuing education offerings published on the Internet for health care professionals. "Offerings" included Web-based courses, conferences, chat rooms and discussion lists, and excluded calendar listings and event guides. "Continuing education" criteria were established, and thus did not limit review to those resources offering formal continuing education contact hours. Where criteria were met, the offering was visited and evaluated. Each site was visited and reviewed according to the process and criteria discussed in this presentation. Content experts as well as consumer visitors offered comments, and this feedback is included here. The published subject area guide and the course offering evaluations can be viewed at http://www.love2learn.com. The supporting research and discussion on Web-based evaluation criteria, criteria and need for continuing education in the health care professions, and product creation of a subject area guide are contained in more detail as part of a doctoral dissertation in partial fulfillment of the Ed.D. degree, Pepperdine University, Graduate School of Education and Psychology, Malibu, USA. Oral defense of this project and dissertation contents is scheduled for April 29, 1998.

CRITERIA
Final criteria for evaluation of Web-based continuing education were derived from three sources, as no single comprehensive source was found in the literature:

1. Widener University/Wolfgram Memorial Library guidelines for Informational Web Pages
   
2. Argus Clearinghouse ratings system guidelines for level of resource evaluation
   
3. Literature on professional continuing education, used to qualify the resource for inclusion in the subject area guide

The Wolfgram Memorial Library guidelines http://science.widener.edu/~withers/inform.htm contain five criteria. While these criteria pertain to informational web pages, these same criteria are applicable to educational offerings, as consumers and students believe continuing education offerings will be presented as factual information. These five criteria are: authority, accuracy, objectivity, currency, and coverage.

"Authority" concerns sponsorship, verification of sponsorship legitimacy, author qualifications, and copyright.

"Accuracy" concerns verification of factual information, free from typographical errors, who is responsible for accuracy of content, and readability of charts and graphs.

"Objectivity" concerns whether the information is presented as a public service, and whether advertising is clearly labeled as such.

"Currency" concerns when the material was first published, and whether it has been updated

"Coverage" concerns whether there are print equivalents available, whether the site is under construction, and whether the work is the most current available.

The Argus Clearinghouse guidelines http://www.clearinghouse.net/ratings.html#eval add both subjective and objective information, and are used to supplement the Widener criteria. Descriptive information includes content, traffic levels, whether lists are moderated, intended audience, access/technology instructions where necessary, and technical performance concerning browsers and servers. Subjective information includes quality of content and assessment of usability regarding layout, readability and organization.

Continuing education criteria for course inclusion was based on Houle (1980) and subsequent studies in education on models of professional practice (Schon, 1983, 1987; Munby, 1987; Kissam, 1986; Carroll, 1985; Cervero, 1988). Key elements of broad criteria for professional continuing education are: information or technical skill beyond competency required for entry-level; contribution to the theoretical or scientific body of knowledge; exploration and integration of contemporary issues and challenges in the professional practice environment; authorship, construction, and content derived from a substantiated and authoritative scientific base.

See final evaluation checklists used for Web sites and discussion lists in creation of the subject area guide, Exhibits A and B.

PROCESS USED TO CREATE THE GUIDE
The process employed in creation of the evaluative subject area guide was based on the fundamentals of the 12-Step Program (Rosenfeld, 1996), while adding the site evaluation component discussed in the preceeding section. These "steps" do not necessarily sequence, but this author found their differentiation and identification extremely helpful in structuring an approach for creation of the subject area guide.

1. Determine the subject and audience The subject was Web-based continuing education offerings for health care professionals; the audience was defined as limited to particular health care professionals (administrators, nurses, therapists).
   
2. Determine what you already know Background research included industry trends and needs for continuing education, existing resources including journals, seminars, trade associations, and electronic (Web-based) offerings, and general status of industry computer literacy. Trends and needs supported potential for new offerings.
   
3. Do a brief search The most obvious key words were applied with general search engines for a first look at what exists on the Web (nursing, therapy, continuing education, health care).
   
4. Identify knowledgeable people, and ask them for help Industry journal editors were contacted, as well as query notices on industry related discussion lists, and announcements at trade conferences. Many sites not identified through search engines were gathered for review from referrals obtained through these outreach efforts.
   
5. Use the tools you know best, then use the rest Search tools provided within individual sites were explored, as well as email to online journals and sources where accessible via Web site. Announcements and queries were included in professional meetings and dialog where appropriate.
   
6. Keep a journal of your travels Information on list subscriptions, especially how to unsubscribe, was kept in an electronic journal. Contacts and queries not directly related to the project, as well as opportunities for further professional development, were recorded along with response. Information and comments regarding guide development was also recorded as well as actions taken. Difficulties with site construction, HTML formatting, and server/Internet Service Provider were noted, along with work-arounds or other solutions. A production time schedule log was kept, as well as alternatives to project flow for contingency planning.
   
7. Choose an appropriate categorization scheme A three-level site map scheme was created, and later refined for navigability based on piloting prior to public industry announcement. The index (home) page was the first level page. Informational pages and gateway listing pages were the second level pages (about the author; about LIST; reply/guestbook; and gateway listings to the individual site evaluations broken out by practic: nursing, therapy, administration). Each site being evaluated was listed individually along with complete evaluation information on its own third level page. Internal links were provided to all levels on all pages. Hyperlinks to the sites being evaluated were also included.
   
8. Know what you're looking for with each resource Evaluation criteria established and notes kept for each site/list. See Exhibits A and B. In addition to evaluation criteria, a statistical access counter was established on the Internet Provider Server to monitor various user attributes such as country, type (university, government, commercial), time of day, day of week, number of hits by page.
   
9. Keep your community informed Industry Press Releases were prepared at various stages of preparation and inquiry, delivered both electronically and via fax.
   
10. Make your guide more usable by adding meta-information Each site/list included in the subject area guide has a separate page presenting the evaluations. Information about the project and the project author are also included. A separate page discussing LISTs and how to subscribe was also included, in addition to a guestbook for feedback.
    
11. Announce publication to user communities A final press release was prepared and distributed to lists and journals on February 8 and 9, 1998. A domain name was created and registered to distinguish this site, registered in November 1997 and launched by the Internet Service Provider in January, 1998.
    
12. Update, update, update Plans for periodic updating and review, as well as expansion of the subject area guide topic areas within health care are planned.

ISSUES ENCOUNTERED
Thematic issues raised during the site construction are presented here for consideration by future site authors. Where practical, the response implemented for this current project is indicated.

1. What if the Internet Service Provider goes out of business during site construction? Back up all text and graphics on a separate hard drive and where possible, on a large capacity floppy such as a zip disk.
   
2. What if a course provider disagrees with the evaluation provided in the subject area guide? A caveat statement that evaluations are done according to criteria appears on the index (home page. The criteria used appear on a mid-level page in more detail for readers to review. The date of the site review is included on the evaluation detail.
   
3. Given that most health care professionals don't have personal computers, or if they do have one it may be their school-age child that is the literate user, how is the subject area guide a contribution? This author has launched an industry initiative to present educational materials at trade conferences, as well authoring and scheduling a demonstration on-ground classroom-style Introductory Internet for Health Care Professionals course. The course, offered for 6 contact hours, will launch in Southern California in June, 1998, and will be offered in sites throughout the country if well accepted by potential attendees.
   
4. What if providers use information provided on this site as a competitive edge against other providers?
   
5. What if individuals committed to aspects of the project do not fulfill their duties in the required time?
   
6. What if providers request inclusion in the guide but the offerings do not meet the criteria?
   
7. What if the project becomes bigger and more demanding than you anticipated at the outset?

FEEDBACK FROM CONTITUENCY
Five content reviewers were selected for industry expertise, established roles as educators, and familiarity with electronic media distance education. Feedback from the constituency served is ongoing, and is derived from ongoing discussion list dialog regarding project development, industry press announcements, and the reply/email mechanism on the site itself. Feedback for this conference presentation incorporates aggregate comments from all of these sources.

1. Easy to navigate, useful resource. Many respondents stated they "bookmarked" the site, and found centralization of resources very helpful.
   
2. Not International enough. Although all sites fitting the criteria were reviewed, the subject area guide retains the flavor of being North American-dominated. Either sites from other areas are not locatable through the search techniques employed, or such sites are non-existent at the present time. All sites suggested from visitor feedback were visited and evaluated for inclusion, and this process will continue as long as the site is being maintained.
   
3. Moving graphics are great/annoying. Apparently the user community is divided on the topic of animated graphics, such as a book with pages turning. Those in favor state such graphics are attractive, attention-getting, and add appeal. Those against state such graphics are distractions and express the notion that Web designers should consider users with poor or low vision, and users with attention deficit disorders. Excellent points, especially for a health care oriented site.
   
4. Statistics regarding access. Two statistical access monitoring programs were employed for data gathering. None of this information reveals user name or address, as this is solely aggregate information gathered to understand the traffic patterns and interest associated with particular pages within the site. In the first two weeks following press announcement, twice as many users accessed from Canada (n=105) than the second place locations: US (46) and Switzerland (38). In descending order, the other countries were: Britain (32), Japan (29), Australia (18), New Zealand (15), Israel (14), Germany (10). Five or less visits were recorded from the following: Malaysia, Thailand, Singapore, Brazil, Hungary, Italy, Belgium, The Netherlands, Greece, France, Sweden, South Africa, Spain, Finland, Norway, Estonia, and South Korea. These numbers indicate the total logins from separate sources, not the number of pages viewed at the site. Total number of visitors (including single sources and return visits) during the first two weeks was 1131.

The most popular pages during the first two weeks following press announcement (not counting the home page) were, in descending order by number of times accessed: Nursing Courses gateway page (n=417), Therapy Courses gateway page (224), the Author bio page (186), About LISTs page (183), Administrator Courses gateway page (155), Response page (128). Total visits to these pages is 1293. These numbers indicate the number of times each of these pages were visited, therefore indicating to this author which pages attracted the highest interest. These numbers do not differentiate single access logon (one time user) from repeat visitors. Numbers higher than total number of site visitors are derived by repeat access to the gateway pages by a single user navigating back to the gateway page to visit one or more of the 53 courses listed within.

Industry initiatives for health care industry user education regarding computer literacy and continuing education access on the Internet is planned, beginning June 1998.

EXHIBIT A

Data Gathering Sheet

Web Sites

Site URL:

Date searched

Authority

whose site is it?

Sponsorship

are there sponsors?

Intended audience

who is the site for?

Accuracy

source of info

Objectivity

academic, commercial

ads clearly separated

Currency

how often updated

Browser

links updated, working

load time

table of contents

Browser used

Continuing Ed

hours

for whom

EXHIBIT B

Data Gathering Sheet

Lists

Name of List:

Dates

Authority

whose LIST is it?

Sponsorship

are there sponsors?

Intended audience

who is the LIST for?

Accuracy

source of info

questions answered well?

Objectivity

academic

promotional

collegial

moderated/unmoderated

Frequency

Avg # posts per day

Day #1 Day#2 Day #3 Day #4 Day #5

How to subscribe

open or membership only?

List address:

body text:

Continuing Ed

hours

for whom

new info offered for Cont Ed?

REFERENCES
Alexander, J. & Tate, M. (1996). Critical evaluation skills for World Wide Web resources. Computers in Libraries, 16(10), 49-54.

American Health Care Association. (1996). Facts and trends: The Nursing Facility Sourcebook. Washington, D.C.: author.

Appleby, C. (1995). Consumers: Doc shopping. Hospitals 1995, 69(14), p. 49-50.

Arthur Andersen, The Guide to the Nursing Home Industry, 1995. By HCIA Inc and Arthur Anderson LLP

Bergman, R. (1994). The world at their fingertips: Rural providers turn to Internet. Hospitals 1994, 68(14), p. 52.

Boyd, D. (1994). Enhancing continuing ed through partnerships. -- association for continuing higher education.

Brandt, D. S. (1996). Evaluating information on the Internet. Computers in Libraries, 16(5), 44-46.

Brockett & Darkenwald. Trends in research on the adult learner. New Directions for Continuing Education, 36, p 29-40, winter 1987

Bunch, W., and Dvonch, V. (1994). The "value" of functional independence measure scores. American Journal of Physical Medicine and Rehabilitation, 73(1), p. 40-43.

Cervero, R.M. (1988). Effective continuing education for professionals. Jossey-Bass: San Francisco.

Cherry, B. (1987). Problems in planning, implementing and evaluating CE programs for health care providers. Lifelong Learning, 10(8), Jun 1987, p 4-10

Courtois, M.P. (1996). Cool tools for web searching: An update. Online, May/June, 29-36.

Davis, D. The impact of CME. Evaluation and the Health Professions. 7(3) Sept 1984, p 251-84.

DeCesaris, T. (1996). Medicare home health growth disproportionate Home health line, April 22, 1996, p 7-8.

Denton, D. (1990). Caring for an aging society -- Southern Regional Education Board, Atlanta, Ga.

Deshler, D. Involving program constituencies in the evaluation process. New Directions for Continuing Education, 24, Dec 1984, p. 15022

Deshler, D. Techniques for generating futures perspectives. New directions for Continuing Education, 36, winter 1987, p 79-92

Deshler, D. Why wait until the year 2010? Journal of Extension, 26, p29-30, fall 1988.

ed. (1995) Info management. Hospitals 1995, 69(4), p. 14-15.

ed. (1995). Human resources. Hospitals 1995, 69(4), p. 19.

Escovetz, A, et al. Survey of the CE needs of Ohio health care professions. Journal of Continuing Education in the Health Professions, 11(2),, Jul 1991, p. 139-53

Filipczak, B. Trainers on the net. Training, December 1994, p. 42-51.

Fisher, C. The outlook for '96: Providers diversify to stay healthy. Provider, January 1996, 22(1), p 36-42.

Fisher, C. The road ahead: Health care reform will be a driving force in long term care in 1997. Provider, January 1997, 23(1), p. 26-32.

Flarey, D. (1993). Quality improvement through data analysis. Journal of Nursing Administration, 23(12), 21-30.

Guiliano, K., & Poirier, C. (1991). Nursing case management: Critical pathways to desirable outcomes. Nursing Management, 22(3), 52-55.

Hauck, R. (1995). Training in the use of the internet. paper -- National Convention of the Association of educational Communications and Technology

Hawryluk, M. (1996). In like a lion, out like a lamb: The 104th Congress in review. Provider, 22(12), 25-26.

Heinrich, V. Professional Development using the Internet. Adult Learning, 7(3), Jan-Feb 1996. p. 9-10.

Hekelman, F., Niles, S., and Brennan, P. Gerontologic Home Care: A prescription for distance continuing education. Computers in Nursing, 12(2), p. 106-109.

Hert, C.A. Learning organization perspective on training: critical success factors for Internet implementation. Internet Research, 4(3), Fall 1994, p. 36-44

Hohmann, L. Interorganizational collaboration in CE education. New Directions for Continuing Education, 27, Sept 1985, p. 75-85

Houle, C.O. (1980). Continuing learning in the professions. Jossey-Bass: San Francisco.

Klevans, D. Developing a needs assessment plan in continuing professional education. Eric document ed297092, 1987.

Kontos, G, et al. (1995). Electronic publications can help build a global community. -- paper presented at the International Conference on Technology and Education

Krol, E. (1994). The Whole Internet User's Guide and Catalog, Second Edition. O'Reilly & Associates: Sebastopol, CA

Kuster, J. M. Internet: Commands to remember. ASHA, 38(1), Winter 1996, p. 19.

Kuster, J. The internet: Providing links for practicing professionals. 37(4), April 1995, ASHA, p. 29-30.

Kuster, J.M. & Kuster, T. Finding treasures on the internet: Gopher the gold!. ASHA, 37(2) February 1995, p. 42-7.

Lay, M., et al. (1995). Technical communication. Irwin: Chicago.

Lindberg, D.A.B. (1996). The NLM and Grateful Med: Promise, public health, and policy. Public Health Reports, 3, p. 552-555.

Madden, M.J., and Ponte, P.R. Advanced practice roles in the managed care environment. Journal of Nursing Administration, 24(1), January 1994, p. 56-62.

Manion, J. (1997), Teams 101: The manager's role. Seminars for Nurse Managers, 5(1), 31-38.

Montague, J. & Pitman, H. (1996). Currents: Internet resources. Hospitals 1996, 70(9), p. 12.

Montague, J. (1996). Currents. Hospitals 1996, 70(21), p. 12-14, 16, 18, 20-23.

Moore, D., Green, J., Jay, S., Leist, J. & Maitland, F. Creating a new paradigm for CME. Journal of Continuing Education in the Health Professions, 14(1), Oct 1994, p. 4-31.

Negroponte, N. (1995). Being digital. Alfred A. Knopf: New York.

Nyberg, D., and Marschke, P. (1993). Critical pathways: Tools for continuous quality improvement. Nursing Administration Quarterly, 17(3), p. 62-9.

O'Connor, A. (1979). Reasons nurses participate in continuing education. Nursing Research, 28(6),p. 354-359.

Onkvisit, S. & Shaw, J. (1989). Product life cycles and product management. Quorum Books: New York

Parsons, Y. From Albany to Tallahassee: the future of long term care is being decided in state capitals nationwide. Contemporary Long Term Care, April 1996, p.35-44.

Peterson, S. Designing continuing education programming based upon futurist literature. Journal of Continuing Education in the Health Professions, 9(2), 1989, p 87-93.

Powanda, W.C. (1997). Patient power. Hospitals 1997, 71(1), p. 8.

Requirements for Long Term Care Facilities, 56 Fed. Reg. 48867 ß483 (1991).

Rizzolo, M.A. & DuBois, K. (1994). Developing AJN Network: Transforming information to meet the needs of the future. Proceedings of the Annual Symposium of Computer Applications in Medical Care, 1994; p. 27-31.

Rothert, M., et al. Joining forces to meet the needs of the RN learner. Nursing and Health Care, 9(5), May 1988, p. 260-62.

Sanddal, N. Using needs assessment to design continuing education. EMT Journal, 1981 Dec 5, p436-9.

Schnieder, E., and Guralnik (1990). The aging of America: Impact on health care costs, Journal of the American Medical Association, 263(17), p 2335-2340.

Shelton, E., et al Leadership through professional associations. New Directions for Adult and Continuing Education, 51, fall 1991, p 71-78.

Smith, I, et al. Needs Assessment: An overview for health educators. MOBIUS, 2(2), Apr 1982, p 52-59

Spooner, S.A. (1996). The pediatric internet. Pediatrics, December 1996, 98, 1185-1192.

Stahl, D. (1995). Subacute care: Creating alternatives, critical pathways. Nursing Management, 26(9), p. 16-18.

Stewart, R. Distance learning technology.... New Directions for Adult and Continuing Education, 67, Fall 1995, p. 11-18.

Stivers, C., Bentley, and Meccouri, L. Internet: The contemporary health educator's most versatile tool. Journal of Health Education, 26(4), July-August 1995, p. 196-9.

Sunquist, J. The HealthVillage pilot: Tuning up an internet solution. Health Management Technology, 18(12), November 1997, p. 38-9.

Tomaiuolo, N., and Packer, J. (1996). An analysis of Internet search engines. Computers in Libraries, 16(6), 58-62.

Towne, V. (1990). 1989 management seminars for the New York Assn of LTC Administrators.

Turner, S. (1996). Plugged in to the community, Hospitals 1996, 70(2), p. 48.

Wagner, J. (1995). Using the internet in vocational education. ERIC digest no. 160.

Wechsler, J. & Campbell, R. NCQA issues HEDIS 3.0: Employers, MCOs may comment on the draft version's 75 standardized performance measures through September 5., Managed Healthcare, August 1996, p. 10, 16.

Wiener, J. M. Managed care and long-term care: The integration of financing and services, Generations, Summer 1996, p. 47-52

Williams, A. (1996). Staff skills key to computer effectiveness. Provider, 22(12), 56-57.

Ziomek, R. (1992). Commentary on case management with the nurse manager in the role of case manager in an interventional cardiology unit. Critical Care Nurse, 3(4), 749-787.

Zorn, P., Emanoil, M., & Marshall, L. (1996). Web searching: Tricks of the trade. Online, May/June, 15-28.