An authentic learning tool for teachers: The OII WWW Site Evaluation Form

Jason Ravitz
Syracuse University
 
 

Dan Lake
Onondaga-Cortland-Madison BOCES, Syracuse, NY

Suggested citation: Ravitz, J. and Lake, D. (1996). An authentic tool for teacher: The OII WWW site evaluation form. Proceedings of the FSU/AECT Distance Education Conference, Talahassee, FL. June 20-23, 1996. (Available: http://www.bie.org/Ravitz/authentic.html)

Introduction

It is increasingly common today to see World Wide Web (WWW) sites displaying icons that proclaim they are "award winning" sites. Some of the more popular sites, such as AskERIC, mention numerous awards provided by organizations such as Point Communications and PC Computing Online. While these awards are well-deserved and meaningful, the criteria for selection are not usually evident at the site displaying the awards. The process of developing criteria and rating sites is typically a generic closed process which does little to advance the understanding by teacher/practitioners about either the characteristics of good web sites, or how they are useful for specific purposes.

 Without being critical of the honors described above, this project has attempted to provide a very different model that delivers on the potential of the technology for people to communicate with each other, and to share their work. Providing this model and format has been a daunting task since Internet resources are constantly being built. Some of the purposes of the form have been to do the following:

In short, the purpose has been to address the professional concerns of teachers and provide a process with meaningful outcomes. In addition, the form can also serve a number of research purposes, including some of the following: This paper is intended to introduce the survey form as it has been used and developed over the last year, and to discuss implications for future research using data generated by the form.

Background

The form started out as a response to a specific problem -- lack of resources to help teachers explore and learn about the educational potential of the Web. It was originally intended as a generic learning tool to help teachers understand the possibilities of the World Wide Web and to think critically about what resources are available and their implications for the classroom. As the Web has grown, the number of educational sites, the variety of resources, and the depth of material one finds has stretched the use of this form. For example, there is clearly a need for special consideration to be given to school-based sites. The original purpose here was not to focus on evaluation of these sites, but this became one implementation.

 A second implementation that arose quickly was that of a resource location tool. As it provides rather detailed analysis by peers of specific aspects of sites, the resulting data could be linked specifically to learning standards published online, allowing others to easily access reviews along with the sites reviewed.

 Finally, the form, as originally intended, provided a setting where reflective online dialogue between teachers could take place about the design features and possible uses of the Web in the classroom. The hypertext capabilities of the review archive allows teachers to share their explorations of the web with each other.

 Thus, while the tool started out as a generic way of helping teachers analyze the web as an educational tool, it has evolved into being multi-faceted (and perhaps too generic). As the Web itself has grown and sites have become richer and richer, the form itself has become more rich, and may have to be adapted to particular uses. Specifically, one issue we address later involves the question of whether sites being reviewed are pure "resource" sites, or school-sites containing useful instructional materials, whether the entire site is being reviewed, or a particular resource within the site.

What are some distinguishing characteristics of the project?

1. Various Uses. The project is intended to support a number of uses.
a. Learning about the Web. It provides teachers with a focused, authentic, hands on learning activity as they begin to explore the Web. Instead of placing the focus on the technology, or all the wonders of the Web, it requires teachers to focus on how resources on the Web relate to their professional concerns in the classroom.

b. Resource location. It provides teachers with an opportunity to locate resources that will be personally useful to them, and have undergone peer review. The result is shareable with others who may have the same interests.

c. Discussion. It provides a discussion tool for teachers to share their findings and respond to what others have seen and said in their explorations. It appears that a lot more work is needed to make this discussion happen, and some suggestions are provided below concerning this issue.

2. Audience for Writing. The ability to share with others is motivational, and the process generates enthusiasm and captures the imagination of users. The work of reviewing a site becomes meaningful since it is delivered to a wide audience of readers. In general, the presence of an audience is motivating, although one issue involves whether teachers might be inhibited by their work being put on display, particularly when they are novice users of the technology.

3. Use by Groups. While the form can be used by individuals, it appears to be best suited for use within teacher training classes or curriculum development teams, either pre-service or in-service, where it stimulates discussion in relation to curriculum and design, and provides learning opportunities that may not be as available to isolated users.

4. Asynchronous. Any discussion between distant viewers/reviewers takes place in an archived asynchronous environment. This provides opportunities for additional reflection before commentary is passed among the viewers/reviewers.

5. Feedback to Site Developers. The developers of sites can benefit by reading their reviews, and dialogue with reviewers is supported. It is expected that creators of sites will request reviews using this tool. An issue that arises is the presence of possibly negative reviews.

6. Living Archive. Reviews of sites that have changed can be re-reviewed and updated. This moving-target problem is one which needs further attention. The proliferation and constant construction of different types of education-oriented web sites has been an ongoing issue for this project.

7. Smaller school sites can be recognized. Unlike the current set of criteria and awards one sees, this project allows educators to identify the usefulness of WWW sites to them personally and to professionals who share their interests. Additionally, it provides opportunities for reviewers to identify a specific subset of a site for review, e.g., a school project containing a collection of political cartoons that is noteworthy. In order for a site to win a prestigious award, a site must provide a rich array of services to a wide audience. Thus, a carefully developed school web site that is of interest, for example, to art educators, or provides some unique resources to a particular audience, is usually not a candidate for such an award.

8. Individual interests and differences are supported, instead of a universal rating being established. Use of narrative and commentary by reviewers, coupled with individual background data provide the reviewer and the reader with opportunities to understand the unique perspective of any given submission.

9. Creative uses are supported. The number of ways the form has been used continues to grow. It was originally a tool for learning about the web, but evolved into a number of implementations that are still being refined.

10. A searchable database is possible. Text-based searching of all the reviews that have been submitted, or data-based searching for specific characteristics of sites, is a reachable goal. For example, one might find all of the sites that feature online portfolio reviews, or that provide feedback mechanisms.

11. User design. It encourages reflection and suggestions for improvements about the tool itself. Interactions with the developer of the form is encouraged. The form has been useful even though some aspects of the design are still being formatively evaluated. For example, each list of options is followed by an "Add/Say More:" text field. Many reviewers have provided suggestions that have been incorporated into improving the form. This represents a process of participatory design that is worth noting.

12. Researcher opportunities. The rich data provided by the form and the opportunities for testing modification of the form provides a rich set of research questions, some of which are referenced later in this writing.

13. Communication with trainees. It can be used as a management tool for instructors. Direct email contact is available for administrative activities that are part of the learning process. For example, a group instructor can use the form to provide direct personal or public replies to each reviewer.

Details about the Form and the Process

The OII WWW Site Evaluation Form, maintained on a server at BBN, consists of five sections: Once the form is submitted, it goes to a customized Hypermail(tm) archive at the OII that allows others to review the submission and respond to the archive via the Web, thus supporting discussion and modifications. Additionally, the email and http: links entered through the form become "hot" in the archive allowing others to see not only the review, but noteworthy pages within the reviewed site as well. This is a result of a customization of the software, and indicates the need for programmers to support the creation of educational applications.

 This tool was developed with the novice (in terms of technology) educator in mind. In most cases it supports the process of learning about the Web by providing checkboxes of suggestions relating to educational value and general design. Every time there is a list of choices provided, an additional Add/Say More: field is provided for participants to add to the list of choices, discuss the choices, or comment freely. Additionally, the form contains a section about the reviewer so that, once posted to the archive, a viewer can respond either to the archive listing, or to the original reviewer (assuming the reviewer has an email address and provides it -- not always the case with trainees).

Use with Pre-Service Teachers in Central New York

The authors used the form in an introductory graduate-level course at Syracuse University called Computers in the Classroom. The context was to familiarize learners with characteristics of a school-based Web before constructing their own Web pages. The class of twenty pre-service and in-service teachers consisted of people who were new to the Internet, and generally unfamiliar with the Web. They were provided with an assignment to review two school web pages in the State of New York using the Web66 directory, a popular listing of school sites. Additionally, they were required to identify one site that they believed provided material that would support a standard set forth in the NY State Frameworks that have recently come on line. This required analysis of the Frameworks and of the activity taking place at the schools. They received an hour of hands-on group instruction about how to use the site feedback form. The process was modeled by the primary author, while the students followed along at individual workstations. This allowed students to become familiar, for example, with moving between multiple Netscape windows. Each student received up to fifteen minutes of individual attention to address any problems he or she was experiencing.

 Three weeks later, the primary author was able to review the results of the assignment online and use the Reply To: function of the archive to send feedback to each student, and assign a grade. This use indicates the management potential of this tool.

Use with In-Service Teachers in Central New York

Co-author Dan Lake used the form to support after-hours training to teachers wishing to learn "Netscape" and the uses of the Web. Rather than focus on the Netscape browsing software, he developed a process to tie Web resources to four curricular areas: Language Arts; Math, Science and Technology; Social Studies; and Languages Other Than English. Each of these areas is represented by learning standards posted at the Central New York Regional Information Center's Web site (www.cnyric.org/Frameworks).

 Each student was actually given TWO activities to accomplish on two successive class sessions. The first activity, outside the scope of this article, taught the participants to capture imagery and text from a Web site and immediately place it within a word processing program that allowed an instant "page display" slide show to be run. In this way, the teachers became familiar with the ease with which they (and by extension, their students) could create multimedia screen displays on a variety of topics, using information gleaned from a variety of web sites. Using topics such as "James Fenimore Cooper" for language arts and "Abraham Lincoln" for social studies, each teacher walked away from the first session being able to directly integrate "the Internet' within his/her classroom activities.

 The second session followed on this, focusing on an all-online evaluation process. This activity utilized the same subject matter and booklists of sites as the first day activity. The booklists were organized so that ONLY the specific learning standards page, the evaluation form page, and 5-7 bookmark links to the subject (e.g. Cooper or Lincoln) were displayed. Class members were given time to peruse the learning standards and view previous evaluations as models. The process invited class members to display a specific learning standard side by side with the evaluation form, and then choose a web resource from the provided booklist as a third page to display and evaluate. Once the evaluation was completed and submitted, the teachers all instantly accessed and inspected their evaluations placed within the OII archive.

 Within a week, the reviews were manually linked by a "Resource" button to a particular standard on the www.cnyric.org/frameworks/(subject area) page. In this way, others can locate resources that support specific standards, resources that were evaluated by working professionals. The class activities resulted in aligning web resources with specific New York State standards.

Future Implications

The nature of the web is getting more and more complex. While a uniform evaluation tool for educational web sites may have been a good idea a year ago, it is apparent that as the web expands, so must the capacity of our evaluation tools.

 Online feedback forms must respond to the evolution of the Net as a whole. One vision is for evaluation tools to become multi-dimensional, providing a variety of viewpoints depending on answers to initial questions. Alternatively, they might become customizable to the user over time. The most practical solution, in the near team, may be to develop feedback forms that provide greater and greater specificity in particular areas, e.g., a review form specifically for university or K-12 web sites.

Research questions that could be addressed using the form

While one outcome of the form may be to provide a service to the Internet community to help organize information on the world wide web, the form can also be used to ask and answer various research questions. It is important to learn lessons from the work that we do, as we attempt to provide new types of functionality and service with the tools we develop. Otherwise, we risk being driven by the tools we create, without evaluating their effectiveness or learning from their use. These are some of the research issues we want to address as the form is used:
 
 
  • Inter-rater reliability. Ask a large group to evaluate the same sites using the form, or some variant of the form. Compare response patterns based on display of the form, and reviews based on individual backgrounds.

    •  
  • Impact of earlier evaluations on later evaluations. If multiple evaluations of the same site occur, do the earlier reviews have an impact on later reviews?

    •  
  • Is there critical review of the content of sites, or a pro-reviewer bias? How will this impact the way the form is perceived and used? How well might the form support discussions of educational content?

    •  
  • Impact of posting reviews on the Web, as opposed to in the background, or anonymously. What approach will best foster discussion? Will discussion areas with more specific topics, e.g., concerning content or criteria items, encourage discussion more than general discussion areas about sites? Very little is known about the impact of Web-based discussions as they are posted more publicly than other discussion tools.

    •  
  • Analyze the moving target problem. How do web sites change? What mechanisms can support ongoing review of the reviews?

    •  
  • Do people perceive this as a process or product? Which do people value more, filling out the form, reading other's reviews, or interacting with others? How can the benefits of each of these explored?

    •  
  • How does form layout effect responses, e.g., if white space for comments were to appear before suggested checkboxes would people say more about the site?

    • Possible methods of data collection

    In order to pursue the above research questions, a few approaches to data collection are being considered. Some of the options include the following.

    In Conclusion

    The authors would like to suggest that an award of "Best 5% of Great Sites We Like" assumes a great deal about who "We" are and may not be an indicator of a site's usefulness to educators. Our teachers should not have to rely on the judgments of nebulous "We's", but rather make and share their own professional judgments as part of a process that teaches them the characteristics of good web sites and how these sites may be used to support learning and teaching. While the "Award Winning" designation may point to some outstanding resources, the evaluation process established within this project is just one step toward a future in which teachers can rely on each other to share links to specific curricular resources that benefit their students.

    For more information

    The OII WWW Site Evaluation Form was designed and developed as part of the Online Internet Institute (OII) with support from the National School Network Testbed (NSNT). Both OII and NSNT are funded by the National Science Foundation.

    If you are interested in using the form in any organized way that might help address the above questions, or build on the ideas presented here, please contact the authors to inform them of your work. For more information or further discussion, please contact Jason Ravitz at jason@bie.org