Ravitz, J. and Serim, F. (1997). Summary of First Year Evaluation Report for the Online Internet Institute. Paper presented at Edward F. Kelly Evaluation Conference. State University of New York, Albany, April 11, 1997. Available: http://www.bie.org/Ravitz/oii_summary.html

 This paper discusses the evaluation of the Online Internet Institute (OII). It briefly summarizes the project being evaluated, then presents the evaluation findings. Finally, it discusses issues concerning the evaluation process.
 
 

Overview of the Project

Created by classroom educators and supported by proponents of systemic reform, the OII is the product of a one-year planning grant (NSF Grant No.REC-9554232) to develop an effective, efficient, and economically feasible model for providing training. It is intended to enable teachers to implement systemic change in science, mathematics, and technology instruction through the use of Internet resources. It combines regional face-to-face collaborative learning with an online support system.

 Small groups of individuals at local sites collaborate and share with the online education community. Participants share, reflect, and collaborate as they learn and practice new skills, working together across time, place, and organizational boundaries. Information on their experiences and the understandings they gain are being collected and made available to help the next cadre of teachers learn online (see http://oii.org). At regional sessions during 1995-96 in New Jersey, New Mexico, Oregon and Texas, 1,100 educators invested 28,000 hours in their own professional growth, and experienced inquiry-based technology-enhanced learning so they could effectively support this type of learning for their students.
 
 

Summary of First Year Report

 OII has expanded from a summer professional development activity to a year round opportunity linking educators across the nation. Recent interest in hosting additional OII face-to-face sites across the nation and applications have exceeded all expectations, as well as the capacity to support these additional clients. Three lessons have emerged:
 
 

* Organizational support is critical
* Local support is critical
* Content support is critical

Organizational Support is Critical

 Funding was inadequate to professionally support a national implementation of a model that was simultaneously being developed. Many FTE worth of hours were donated by project leaders. While it may be inspiring to see the contributions and achievements of volunteer work, it doesn't make for reliable replicable research. There is a core team of nationally distributed people who must be in place for a project like OII to attain sufficient, sustainable momentum.

 OII enjoyed considerable donated organizational support from partners, including the Princeton Regional Schools, who permitted the PI to redefine his professional responsibilities so that when the NSF funded level of effort was exceeded, work could continue. Similarly, BBN Educational Technologies (see http://copernicus.bbn.com) contributed hardware, software and systems support to keep the project online as part of the National School Network project (http://nsn.bbn.com). Certain nuts and bolts such as clerical support, which are critically understaffed in the district, could not be covered in this way, and the PI's responsiveness suffered from the resulting uneven performance in this respect.

Local support is critical

Following the expiration of the planning grant, BBN has contracted with the PI to synthesize the experience of the last two years and develop a "local model" for OII that could assist school districts in creating their own online professional development opportunities. Building a business plan to make such relationships self sustaining is a key goal for this effort. OII expects to cover the costs of regional facilitators and other key project staff with a properly designed economic model that meets the needs of schools in a cost effective way.
 
 

Content support is critical

 OII's strategy is to work more closely with content area experts, returning to the original OII vision of mentoring accomplished by subject area experts. As OII matures, it intends to develop more mature organizational relationships with content rich entities such as the Smithsonian, National Geographic, Discovery Channel, as well as other established NSF funded projects. This high level work requires that OII develop sufficient support for full time, professionally led efforts to broker and nurture such collaborations.
 
 

Problems Encountered

Getting people to use the Internet has become much easier, with a reduced learning curve for the technical elements involved. However, building communities of reflective practice and the trust level leading to relationships that are a prerequisite to effective online collaboration is a more challenging activity. Much effort has been expended, and new entry points reflecting the diversity of skill/knowledge level of the people OII attracts as participants have been tried. Still only a small percentage (approximately 10% to 15%) of people who complete the 30 hours of structured activity go on to collaborate online, i.e., after completion of the local face-to-face or formal online sessions. While the work of these few people is exemplary, how does OII bring the majority of participants to this level?
 
 

First Year Evaluation Findings

 The task of evaluating this work requires multiple forms of data collection, and the construction of criteria for assessing project work. The following points are addressed based on analysis of the Great Matchup database, the online discussion archives, and the online portfolio of projects:
 
 

1) The original vision
2) Participant goals
3) What happened?
4) Synthesis of discussions
5) Analysis of projects
6) Value to teachers
7) Project Limitations
8) Recommendations
9) Meta-evaluation

 The mission of OII has been to create a vehicle for teachers to integrate the Internet into their classrooms. This was to be accomplished over the course of several years. The first year goal was to create a process for building a cadre of teachers capable of leading groups in their local districts and fostering collaboration across the network. The work of these local groups and their site facilitators would be supported through links with online mentors, OII staff and peers via the Internet. Thus, OII would become a community of local and distant collaborators, sharing their work and discussions.
 
 

Participant Goals

The Great Matchup database was constructed from a survey of all participants and made available online for searching. It provided the first round of teachers and planners an opportunity to share their visions and questions, and to find others with similar interests for collaborating. We have analyzed qualitative data from the Great Matchup survey about the types of questions OII teachers are asking and their curriculum goals. The following illustrates the variety of agendas different teachers bring to the OII. To some extent the evaluation must address the effectiveness of OII in meeting teachers different goals.
 
 

How will you know when you have succeeded?
 
 

Project Activities: "When I see URL's, project plans and announcements, and lessons that will have students engaged and communicating next school year.

Student Attitudes: "Little faces will beam with delight as they walk through the doors of the technology center and "guess what I found" will soon be replaced by "guess what I did" in describing their Internet accomplishments."
 
 

Collaboration: "I will feel that I have accomplished my goal when teachers communicate with me via email and share their ideas."
 
 

Utilitarian: "I will be able to "converse" with others about management, create a usable outline for teaching Internet search engines and evaluation of resources, and to create my own home page."
 
 

Personal Development: "When I gain the confidence to instruct my students in the use of the Internet."
 
 

Replication: "I will know when I come across ideas or plans that have worked for others and I was able to integrate it into my own teaching.; When I can take the information back to my school and show teachers how to use specific resources in their subject area."
 
 

Systemic change: "When I get feedback from parents and administrators and my students enjoy coming to school. I hope to spark an interest in telecommunications that will make parents demand for more and give more to the school district so more teachers can do this. I call it the 'Ripple Effect'.
 
 

 The Great Matchup provided the central process which structured people's entry into the project in a way that highlighted the goals of OII and of the individual participants. People were asked detailed information about their goals, their educational setting and issues, and how their work would benefit others. This data was stored and accessible via the World Wide Web (WWW) in a searchable database that facilitated the matching of mentors and the formation of groups. The originator of one exemplary project (Team Web) describes how the process worked for him:
 
 

The team was formed by me. Using the Great Matchup I looked for people who were interested in developing web pages for the training of teachers and/or students. I sent out a letter of invitation to join the group. Once I found enough people who were interested in the project I opened it up to others by using OIIALL listserv. At that point others offered assistance and support.
 
 

Other collaborative tools were also investigated and made available through the work of the OII staff, including a real-time conferencing tool (Co-Motion) that linked to the WWW with synchronous and asynchronous communication. In one instance, 25 teachers from Princeton and this evaluator practiced using the Site Evaluation Form (Ravitz and Lake, 1995) as an educational activity. Additionally, a WWW-based messaging tool provided by BBN was available for sharing discussions within and between groups.

 In sum, advanced tools helped to foster collaboration, keeping the interest of some of the more experienced (and otherwise busy) users, while serving to stimulate the newer users about the possibilities of internetworking. Most importantly, from an evaluative perspective, the Great Matchup structured people's entry into the project in a way that systematically collected data about new participants and their goals.
 
 

Summer of '96: What happened?

 By the time the second generation of OII groups came aboard, in the summer of 1996, a number of changes had occurred. The Great Matchup was no longer available, and neither was the real-time conferencing tool. This illustrates how the withdrawal of donated support by an organization, support that OII had come to rely on, seriously hampered development efforts. Both of these were removed from the project because of inadequate funding, and because organizations that had volunteered their services withdrew their support as time went on. This contributed to the following problems:
 
 

* loss of tools for collaboration
* unavailability of mentor pool
* ad hoc restructuring
* shift to WWW-based listserv as primary collaborative tool
* ad hoc collection of information from participants

 Efforts were made to compensate. The "Entry Point" pages were developed in the fall of '95 and refined in the Spring of '96 in response to participants' requests for more structure. Later, a year after OII had started, a volunteer team consisting of early participants worked to improve the OII home page. The re-development of the WWW pages created a structure where information could be found, but it did not go far enough in re-building the process whereby people would become engaged. This is an area where OII will have to continue to work to recreate itself.

 The redesign effort did not benefit from the large scale support that occurred in the original planning sessions. Instead, more burden fell on site facilitators to organize and support their local groups, and on the small OII staff. Later in the year, one site facilitator expressed the difficulties that had arisen:
 
 

"I think our biggest challenge is still to identify dedicated mentors who will provide guidance and leadership to a group of newbies interested in pursuing a particular topic of interest - people who will serve the way Art did to Team Web last summer. Those of us who are site facilitators cannot also effectively mentor a group - or at least I found I can't."

 Finally, another important change which occurred in the Summer of '96 is that groups met for shorter periods of time, and had less access than previous participants. While the Summer '95 groups had met for 8-10 weeks, many of the Summer '96 sites only met for a week over the summer. When their sessions were over many did not have access to practice their skills. This also hurt the ability to promote longer-term sharing.
 
 

Synthesis of Discussions

 As OII came increasingly to rely on WWW-based listserv discussions, it is important to examine the value of these discussions and how they can be improved. Were OII discussions useful? What for? These are critical questions, as email remains a central communications tool.

 The original vision was to get teacher participants and mentors to synthesize their discussions and share their findings with others. There appears to have been a lot of sharing on an "as needed" basis, but little effort to synthesize what was learned for the benefit of others.

One staff member wrote that making connections between the various discussions is a key challenge for site facilitators. "They do need facilitation and synthesizing and that will come as site facilitators learn to delegate and model that responsibility". Independently, two site facilitators indicated that the discussions had become an active concern. The first wrote: "Learning how to label or really describe the subject heading (of email messages) is being modeled by the site facilitators and is vital for the success of (discussion) archive". The second, indicated that "The site coordinators need to work on getting participants to become the discussion leaders and synthesizers."

Analysis of Projects

 The other component of OII which is available for review is the online display of projects from the various sites. For every product created by OII participants, a significant exchange of information and collaboration happened behind the scenes, and in many cases there were numerous online planning messages for the various sessions. The attention has been more on creating projects, not analysis of them There was not as much feedback and discussion about the displayed projects as had

 been expected. The process of inviting and receiving feedback is thus an issue for consideration. A rough count of email messages related to finished projects indicates that such discussion comprised less than 10% of the total email utilization.
 
 

Content analysis of the projects

In order to evaluate the projects, it is necessary to develop some set of criteria. Participants and OII staff have not been as involved in this process, and there is some disagreement as to the amount of structure, for example, that the presentation of project work should follow. To some extent for participants, meeting their own needs is criteria enough, but from an evaluator's perspective, even this is not always easy to see. The following is a set of criteria that was developed before the projects were reviewed. These are based on the notion that collaboration and shared inquiry was to be demonstrated:

Proposed Criteria:
 
 

* indicates intended audience
* explains project's origins & people responsible
* defines the educational goals
* explains the process of development
* shows results of implementation
* reports benefits and lessons learned
* is adopted by others
* responds to peer or expert review
* draws on distant expertise

Reflection-on-action (Schon, 1993) may be a critical component needed to realize the original vision of OII. First, if participants don't pay attention to process they may not be able to repeat it, or teach it to someone else. Second, OII cannot tell if it could have supported projects better. Third, would-be collaborators cannot contribute if they don't know what others are doing. Moreover, from an evaluators' perspective, lack of process information in the project portfolio makes evaluation virtually impossible.

For example, a group of teachers created Cyberographies, e.g., a list of Spanish resources on the WWW. If someone already knew HTML and how to use Search Engines these could have been constructed in less than an hour. It is more likely that someone spent hours and learned multiple new skills, but this is not discussed. Here are some questions that might have been addressed in presenting this work:
 
 

In the course of developing your project...

 What issues came up along the way that challenged you?
Did you need to learn about search engines?
Did you find magazines that were useful in identifying resources?
Did you send email to people who could help you?
Did you reject some sites for inclusion? If so, why? What criteria was used?
Did you have to learn how to make a WWW page?
How did you go about doing each of these things?
 
 

Value to Teachers

 No comparative judgments can be made about OII as a teacher development program. Lack of funding prohibited an extensive follow-up survey that had originally been planned. Instead, this section looks at the benefits that one can infer based on what teachers have demonstrated and said about the program.

 It appears that teachers have been encouraged and have found value in their participation. This is evidenced by the continued and growing participation, the numbers of projects and discussions undertaken, and informal assessments that were provided by participants. The value to teachers is assessed in terms of the following:
 
 

 
* sharing knowledge and skills
* collaborative projects
* dissemination
* development of skills

Sharing knowledge and skills

Collaborative projects

Dissemination

Skills development

Analysis of data captured by the Great Matchup survey indicates that the average teaching experience was 14 years; this includes the 12 people who had no teaching experience. Among those with teaching experience, the average teaching experience was over 15 years. Thus, the original OII participants as a whole represented a total of 2123 years of teaching experience! The average years of experience with telecommunications technology was 3 years. However, over half of the participants (76) had 1 year or less telecommunications experience.

 No attempt was made to test skills development. We can infer that skills were developed based on the development of projects, and based on very positive responses from participants about the usefulness of OII in developing skills. Furthermore, one participant, discussing the development of skills stated ""It's relative! If my skills haven't increased significantly, the domain in which I can apply them has!" Thus the likelihood of this participant applying what she learned in the formation of her curriculum planning is greatly expanded, although her score on a skills-based assessment might show little growth.
 
 

Summary of Project Limitations

 By now, the reader should be aware of these limitations that OII has faced:
 
 

1. Extensive and often total reliance on volunteer efforts and donated services
2. Failure to keep the original Great Matchup database online
3. Discontinuity in data collection efforts
4. No structure for synthesis and presentation of work relative to goals
5. Shorter sessions in the later stages of the project, i.e, Summer '96

The following section illustrates a further limitation -- lack of access for participants after completion of the shorter Summer '96 sessions. This was a problem faced by teachers who participated in OII summer sessions, and then returned to settings where access was not available. One staff member wrote, "We had several discussions about refining the process of forming working groups and providing follow-up once people finished the onsite session. One reason this was not completely successful is that some participants (SC, SF and Southern NJ) had only temporary e-mail accounts."
 
 

Highlighting the importance of connectivity: Where has OII happened?

OII happened both onsite (face-to-face) and online. From the chart above, one can see the impact connectivity has upon online participation. Although four times as many participants from New Mexico worked with OII, the greater availability of connectivity (both in school and at home) allowed the New Jersey people to spend 50% more time pursuing their learning online. These insights are being considered as OII plans for work with statewide initiatives in Oregon, Texas and Pennsylvania. In New Mexico, this finding resulted in the formation of the Regional Educational Technology Assistance program (RETA).
 
 

Commitment by Participants

 Another unexpected result includes the disproportionately large contributions of the Oregon group, considering their small numbers and limited time online as evident on this chart (see above). Beyond the advantage of capable leadership provided by site facilitators (both veterans of the first OII sessions), the facts that Oregon participants "auditioned" for a limited number of spaces, were required to publish on the WWW, and received graduate credit, may account for their success-- their participation accounts for more than twice the portion of hours as their numbers represent, thus helping to explain their productivity.
 
 

Recommendations

 The following represent the key recommendations of this report based on the previous discussion.

 
* work on synthesizing discussions
* projects should be linked to personal and OII goals
* re-institute plans for collecting participant data on entry and exit
* improve entry-level WWW pages as a "doorway" to participation
* prepare to measure student impacts in future
* focus on local support and implementation, i.e., on sites prepared to make larger commitments of time and resources< br>* seek strategies to foster more reflective conversation and sharing, including a way to share information on the process of project development and to request and receive feedback on projects from others.

Conclusion

 If the purpose of OII's first year was to demonstrate the ability of the project to create an idea and bring it to light, this was clearly accomplished in the early stages of development. Even if the estimates of geometric growth have been overly optimistic, there has been a fanning out of sites. More importantly, the idea of fostering leadership and expertise appears to have been realized. Several facilitators have presented OII activities at conferences, and provided leadership in their local communities, as evidenced in their communications, projects, and construction of WWW sites. Whereas the original body of expertise rested with nationally known mentors, much of this expertise has been transferred to educators who have taken up the leadership roles in their districts. It appear that sites are planning extensive implementations, with local support being promised in several states.
 
 

Meta-Evaluation

 This evaluator has worked closely with the PI concerning the design of the project. My close participation has not prevented me from voicing constructive criticism at times. However, it may mean that I have been more accepting and less critical of the apparent successes reported here.

 The time commitment needed to evaluate OII and funding availability have been entirely disproportionate as the project has extended from a test of a concept to a full-fledged implementation. Two of the three original evaluators left the project in its early stages because of other work commitments and scheduling conflicts. With it they took their expertise, and their knowledge of OII's processes and ideas about how to evaluate such a project. Their presence at the sites where OII happened (in New Jersey and New Mexico) would have been very valuable and permitted more direct observation of OII activities. Support for direct observation is needed for future implementations in order to assess the sessions more directly. For this report, the evaluator has primarily been in electronic contact and has attended a few national meetings where staff and participants have gathered to discuss issues.

The inability to quickly replace the functionality of the Great Matchup resulted in a severe discontinuity in the data as new data collection mechanisms were created at different sites ad hoc. Because the fate of the Great Matchup was not known, the extent of the problem was not realized and alternative plans for data collection were not developed in a timely manner. Data on self-rating of skills was collected when people joined, but not when they completed OII activities. An extensive follow-up survey was planned, but never implemented because of lack of funding. In the future, smaller data collection activities should be instituted at individual sites at the conclusion of face-to-face sessions. This will provide formative data for session leaders. However, longitudinal survey data after completion of sessions is also needed, because the incubation period for participants to integrate their learning within their districts can take considerable time. One original Summer 95 participant, created her own courses using the, and now in December 96 has involved her department chairperson in beginning to move the entire curriculum to the WWW, moving from classroom to district wide impact. Long-term impacts such as these do require follow-up surveys to be conducted.

Finally, no systematic transcript analysis has been conducted with the archived listservs. While content analysis of the 6,677 email messages was conducted, it was in a cursory manner to roughly categorize the purposes of communications. Whenever additional funds become available, it would be a worthwhile undertaking to understand better the role that the online discussions are playing, and the participation patterns of mentors, facilitators, and participants. Some initial analysis of the email discussions follows in Appendix B. This data generally supports the above discussion and is provided here for purposes of illustration.
 
 

Discussion of Evaluation Theory

 The attention to the original vision as a separate component of the evaluation is consistent with the discussion in Shadish, et al. (1994) regarding the evaluation of prototypes and "super-realizations". Superrealization is the stage "when a suitable design has been constructed and a demonstration study is initiated under conditions that maximize the likelihood of success" (p. 352). The slippage that occurred as initial funding ran out and larger numbers of clients became involved should not be the basis for judging the potential value of the project. Moreover, the achievement of the original vision may not be as important an indicator of success as the perceived benefits by the consumers (Scriven, 1991, p. 99).

 The author has taken a very service-oriented approach, employing components of a "responsive evaluation" (Shadish, et al., p. 308) approach in reporting issues to project staff. Most of the evaluation work has been in concert with the Principal Investigator and OII staff. The recommendations are given with the assumption that the decision to provide further funding is conditional on demonstrated successes and lessons having been learned from the pilot that can lead to project improvement. The evaluator has attempted to contribute to the first part of this, while implementation of the suggestions is left to the consideration of OII staff.

 Furthermore, the question of scalability and the ability to generalize findings from the original participants to larger numbers of clients is one that must be addressed. "It is never certain whether any observed advantage is attributable to the educational innovation as such, or to the greater energy that teachers and students put forth when a method is fresh and "experimental" (Shadish, et al., p. 333).

 Finally, this report has demonstrated three types of data collection that are useful in online environments: surveys of distant users, analysis of email transcripts, and analysis of online portfolios of work. In each of these areas, particularly the last, there is extensive work to be done. One is referred to other sources (Kozma and Quellmalz, 1996; Ravitz, 1997a; Ravitz 1997b; Riel and Harasim, 1994) for further discussion of data collection issues in online environments.
 
 

Kozma, R. and Quellmalz, E. (1996). Issues and Needs in Evaluating the Educational Impact of the National Information Infrastructure. Paper commissioned by the U.S. Department of Education's Office of Educational Technology. Available: http://www.ed.gov/Technology/ Futures/kozma.html.

Ravitz, J. (1997a). Evaluating learning networks: A special challenge for Web-based instruction? Chapter in Web-based Instruction. Badrul Khan (Ed.). Englewood Cliffs, NJ: Educational Technology Publications.
 
 

Ravitz, J. (1997b). Supplemental information. Available: http://nsn.bbn.com/Ravitz/ipv.html.
 
 

Ravitz, J. and Lake, D. (1996). An authentic learning tool for teachers: the OII WWW Site Evaluation Form. Proceedings of the FSU/AECT Conference on Distance Learning, June 20-23, 1996 Tallahassee, FL. (Available: http://copernicus.bbn.com/Ravitz/authentic.html).
 
 

Riel, M. and Harasim, L. (1994). Research Perspectives on Network Learning. Machine-Mediated Learning, 4 (2-3), 91-113.
 
 

Schon, D. (1983). The Reflective Practitioner. USA: Basic Books.
 
 

Scriven, M. (1991). Evaluation Thesaurus (4th Edition). Newbury Park, CA: Sage Publications.
 
 

Shadish, W.R., Jr., Cook, T.D., and Leviton, L.C. (Eds.). (1991) Foundations of Program Evaluation. Sage Publications: Newbury Park, CA.

Spitzer, B., et al. (1994). Fostering reflective dialogues for teacher professional development. Cambridge, MA: TERC. Available: http://hub.terc.edu/terc/LabNet/Guide/Fostering_Refl_Dialogues.html.
 
 

 
 
 

Using the Web	Designing Web Pages	Cyberographies	Community-related Projects
Team-Web's Downloadable, Customizable  Course	Janet Murray's  Hot Tips,  Many Links	Math (2)  Wonderful world  of Math;  Middle School  Math Bookmarks	Children, Chernobyl and the  (NJ Assembly votes  to allow student to  remain in US)
OII Oregon's  Searching the 'Net	OII New Mexico  HTML Primer	Environment(2)  Wetlands Project;  Virtually  Disappearing Zoo	New Mexican sixth  graders compiled information about remedios  (healing herbs) to preserve an oral  cultural tradition  via the WWW
Virtual Library  Research	Andy Carvin's  Crash Course for Educators	Biology (1)  AP Biology  Research Project	F.X. McGraw  Elementary School,  Camden, NJ
Avoiding Ethical Potholes: Student Navigators on the Information  Highway	NSN Site Feedback  Form	Chemistry (2)  Chemistry  Resources;  Chemistry  Syllabus;	Erving, MA students  create Business  Directory, winning  Cyberfair 96

Further analysis of E-mail discussions, contrasting 1995 and 1996 projects follows here. In working to develop online projects, small groups of users collaborated via WWW pages and discussion archives. Two sample cases examined here contrast year long archives of ongoing groups: Team-Web and Technology Based Emergent Literacy (TBEL) with ad hoc quick response projects (like Chernobyl).

Group	Messages	Size(K)	Start	End	Avg msg /month	Contributors
TBEL	57	190	8/24/95	9/15/96	4	3
Team-Web	77	301	7/30/95	2/11/96	12	6
Chernobyl	81	155	5/3/96	10/4/96	16	17

In the ongoing groups, most postings were initiated by the moderator, and served as a communications medium with members of the group. Shorter duration groups tended to compress the intensity of discussion and involve larger numbers of readers as participants. Archived data will permit subsequent analysis of WWW site visits during the period to track the proportion of "lurkers" to "workers".