A “distance scholarship” model for teaching and learning about technology
supported assessments
Jason Ravitz
Buck Institute for Education
Paper presented at Society for Information Technology In Education
(SITE). Nashville, TN. March, 2002.
Available: http://www.bie.org/Ravitz/SITE2002.html
This paper advances the notion of “scholarship”
as a primary goal of distance education. It describes an online course
taught in Blackboard™ that attempted to be accountable not only to the
learning of individual students in the class, but to learning of various
stakeholder groups including developers of new assessments for learning.
With the rapid pace of development in educational technology it is unacceptable
to focus on producing individual knowledge among users with no sense accountability
to the broader educational technology community (Shulman, 1999).
It is important for pioneering technology-using educators to make clear
to others where well-worn paths are leading and where new possibilities
for teaching and learning are emerging. It is also important to improve
communication between educators and developers about what types of tools
and services are most valuable in the classroom. Online courses offer
unique opportunities to support the advancement of scholarship within the
educational technology community.
Technology Supported Assessment (TSA) lasted 6 weeks and ran twice,
in the spring and summer of 2001, as part of a U.S. Department of Education
PT3 Catalyst Grant involving the Concord Consortium in Concord, MA and
the University of Virginia.
Students in the course included instructional technologists and educators;
many had responsibility for advancing technology use at their institutions.
The conceptual foundation of the course rested on two points: 1) It
is the formative assessment of learners – providing helpful feedback
to learners about how to improve - that allows students to become self-correcting
and to learn;
and, 2) Technology-supported tools offer exciting opportunities for improving
formative assessment for teaching and learning.
Technology-supported assessments that were reviewed by the class via
the Internet included IMMEX (Underdahl, Palacio-Cayetano, & Stevens,
2001), Intelligent Essay Assessor (Foltz, Laham, & Landauer, 1999),
CRESST’s Knowledge Mapper (Baker, 1998), and the Analysis Toolkit for Knowledge
Forum (Lamon, Reeve, & Scardamalia, 2001). In each case, there
is reason to believe that technology can be used to support teachers and
learners in obtaining better formative assessments of their learning.
Goals for the course included the following:
-
making it possible for educators to learn about and try different research-based
tools with their colleagues and students;
-
sparking discussions to better understand the availability of assessment
tools and their applicability to teaching in different contexts;
-
supporting reuse of the assessments, course materials and activities,
in local settings; and,
-
informing tool developers and researchers about the results of the activity
Designing the Course
The course was structured around weekly Readings,
Activities and Discussions (RAD) that were initiated by the instructor.
Each week of the course the class read about and made use of one of these
research-based tools and discussed their applicability to their own work.
The activities and discussion prompts used methods
described by Collison et al. (1999). The understanding was that in
online settings teacher-focused discussions will be severely overwhelming
the teacher and severely underwhelming for the student. As a result, it
is imperative to get the students involved in the discussion by “moving
out of the middle” as an instructor.
It is also important to give time for reflection on major readings, activities
and discussions. The course followed what Collison et al. call a
“structured asynchronous” format that gives students a week to complete
the assignments at their convenience.
Working with Developers
Tool developers took an active interest in the progress
of the class participants. Several of the tools did not yet have
complete user-documentation and training systems in place for online users.
For this reason, the developers were particularly keen to see how someone
else could teach others to use their tool via the Internet, or how someone
would use existing documentation.
For each tool that was selected, the author contacted
the developers in order to:
-
secure passwords and login access to the tools when necessary;
-
secure technical support during the week scheduled for use;
-
obtain or create instructions for using the tool;
and,
-
provide perspective on available readings and research
Generally, there were one or two people from each
research and development group who could free themselves to undertake this
effort. Sometimes the instructor had access to a high level researcher,
a technical support person, and a professional development person who could
help organize the week’s readings, activity and discussions. The
class then also had a chance to interact with these individuals.
Developers were also pleased to be invited to participate in discussions,
as “guest experts” concerning use of their tool. This provided a
unique opportunity to interact with educators who were using their tools.
Developers rarely get a chance to interact with their users in a meaningful
way, particularly at a distance. With little additional effort on
their part, developers could participate in conversations with a fresh
set of users over the course of a week or two.
This is far preferable to their having to track the progress of random
Internet users who use their online tool. Certainly, this does not
serve as a substitute to developing their own user communities, but use
of online students can supplement the costly and time consuming process
of developing their own pilot sites. The fact that students were engaged
in a pedagogically-oriented and structured inquiry adds to the value of
their contributions. In this small way, the course supported efforts
to improve delivery of online tools by developers.
Several students in the course were able to re-use
the tools as part of their own in-service offerings in their own institutions.
On reflection, mechanisms for tracking and discussing this use needed more
work. Most of the students used the tools themselves and with a few
colleagues, but use with groups of learners in local institutions is potentially
more interesting to study. This was problematic because the course
activities were set up in advance (before we knew which tools would be
used where). The course also had to be responsive to the group as
a whole and did not have resources in place to follow up on the few cases
where there was re-use of the tools in local settings (besides by the students
in the class themselves). More work is needed to determine the interest
of the researcher and developers to support and study these distant implementations.
In addition, re-use was not as easy as expected.
At the time of the course, copying of Blackboard modules for use by others
was somewhat laborious due to having to conform to archaic rules of html
code. Additional development was required to make the modules reusable.
Nonetheless, these processes will become easier and any additional effort
to promote reuse is worth undertaking. There growing interest in
re-use of instructional materials in order to contribute to scholarship
in educational technology (Bransford, 2001; Wiley, et al., 2001).
For some time I have been troubled by the sense
that there is too much activity in educational technology research and
development, and not enough knowledge being generated from this activity
(Ravitz & Serim, 1997) Educators require help identifying tools and
resources for student assessment and they trust the experiences of other
educators. Developers appreciated help obtaining feedback about the
experiences educators had when trying their tools. Prior work on
the Site Evaluation Form (Ravitz, 1995) also indicated that educators could
share reviews of online resources and that developers wanted to see these
and to respond to the perceptions of educators.
In conclusion, courses in educational technology have the potential
to generate a wellspring of knowledge about teaching and learning with
technology – not just among individual learners, but across the field of
study. It is plausible that use of tools by online educators
could provide substantial data for developers to improve their products
and their research. This would help meet the requirement for a decade
of rigorous scholarship described by Haertel & Means (2000).
One can imagine communities of educators developing knowledge around uses
of different online technologies by promoting trials of new tools in a
series of online courses that link practice to research.
Instead of only being accountable to the learning of individuals and
their classes, online learners, educators and instructional designers can
also contribute important knowledge to the field of educational technology
research. They can provide data, test cases, and reviews for researchers
and educators who want to hear how tools are used in different settings.
REFERENCES
Baker, E. (1998, November).
Understanding Educational
Quality: Where Validity Meets Technology. William H. Angoff Memorial
Lecture Series. Princeton, NJ: Educational Testing Service. Available:
http://www.ets.org/research/pic/angoff5.pdf
Bransford, J. (2001).
Toward the development of a stronger
community of educators:
New opportunities made possible by integrating
the learning sciences and technology. Paper submitted to the
PT3 Vision Quest on Assessment in e-Learning cultures.
http://www.pt3.org/VQ/html/bransford.html
Collison, G., Elbaum, B., Haavind S., & Tinker, B. (2000).
Facilitating online learning: Effective strategies for moderators.
Madison, WI:Atwood.
Foltz, P., Laham, D., & Landauer, T. (1999).
The Intelligent Essay Assessor: Applications to Educational Technology.
Interactive
Multimedia Electronic Journal of Computer-Enhanced Learning, 1(2).
Winston-Salem, NC: Wake Forest University. Available: http://imej.wfu.edu/articles/1999/2/04/printver.asp
Lamon, M., Reeve, R., & Scardamalia, M. (2001, April). Mapping
Learning and the Growth of Knowledge in a Knowledge Building Community.
In Scaradamalia (Chair), New Directions in Knowledge Building. Annual meetings
of the American Educational Research Association. Seattle, WA.
Ravitz, J. and Lake, D. (1996, June).
An authentic learning tool
for teachers: the OII WWW Site Evaluation Form. FSU/AECT Conference
on Distance Learning. Tallahassee, FL. Available:
http://www.bie.org/Ravitz/fsu_aect.html
Ravitz, J. & Serim, F. (1997, April).
Summary of First
Year Evaluation Report for the Online Internet Institute. Edward
F. Kelly Evaluation Conference. SUNY, Albany. Albany, NY. Available:
http://www.bie.org/Ravitz/oii_summary.html
Ruiz-Primo, M., Schultz, S., Li, M., & Shavelson, R. (1999, June)
On the cognitive validity of interpretations of scores from alternative
concept mapping techniques. CSE Technial Report 503. Center for
Study of Evaluation. Los Angeles, CA: UCLA Center for the Study
of Evaluation, [WWW Document]. Available:
http://www.cse.ucla.edu/CRESST/Reports/TECH503.PDF
Shulman, L. (1999, May). The scholarship of teaching for meaningful
learning. Plenary address at annual meeting of the Center for
Innovative Learning Technologies. San Jose, CA.
Stiggins, R. (1997). Student-centered classroom assessment (2nd
Ed.). Upper Saddle River, NJ: Merril.
For more information, please contact:
Jason Ravitz
Associate Director of Research
Buck Institute for Education
18 Commercial Blvd.
Novato, CA 94949
415-883-0122 x 310
jason@bie.org