Will Technology Pass the
Test?
Jason Ravitz, Ph.D.
Center for Innovative
Learning Technologies
UC Berkeley / SRI
International
PT3 Goals: To improve
the knowledge and ability of future teachers to use technology in improved
teaching practices and student learning opportunities, and to improve the
quality of teacher preparation programs (ESEA, Title III, Subpart 1, Sec.
3122).
This
paper focuses on the critical juncture we have reached now that
Internet-delivered technologies have become increasingly available in
classrooms for use by students and teachers.
We are at a fork in the road; our assessment and evaluation methods and
our ways of communicating about technology uses have not kept up with the pace
of innovation in the learning technologies.
This gap in knowledge keeps teachers from taking full advantage of the
tools that have become available and keeps developers from communicating
effectively about the possible benefits of using their tools.
A
process needs to be developed that will improve knowledge creation and
utilization in the areas of technology-related assessment and evaluation. This process would utilize technology to
bridge the work of teachers and researchers and would provide information about
the impacts of educational technologies in varied educational settings. Improved communications can lead to both
better assessments of individual learning and more useful knowledge creation
among educators.
Walking
in Each Other’s Shoes as a New Form of Distance Scholarship
In
our Internet-connected world, you don’t just have to read about how technology
is being used in education, you can see it and try it yourself! We can use other people’s ideas, tools,
rubrics, or analytic frameworks, thanks to the Internet. Distance scholarship is the communication
among different users (teachers and researchers) about what seems to work best
in different settings. This communication
can be coordinated and “mapped” (Horn, 1997) so that knowledge can be shared
and retrieved more easily.
A
new form of “distance scholarship” is proposed where teachers and researchers
become responsible for seeing how their favorite ideas and work can be
utilized, challenged, and improved upon by others. Schulman’s
(1999) plenary address at CILT’99 provides a view of scholarship that serves
here. A key test of a good idea is when
news of its return arrives – i.e., when it proves to be valued by others who
are dealing with similar issues or problems.
In many ways a good idea in educational technology is identified not by
its success in an isolated local or individual context, but when it results in
benefits in comparable settings in a way that can be confirmed. This is a twist on the familiar ideas of
“replication” or “transfer, but in this approach teachers and schools select
which ideas are most useful to “replicate” given their understanding of their
situation and goals. Educators will
benefit from collections and analyses of strategies for using technology that
have withstood the test of multiple instantiations and that have proven their
usefulness for large numbers of educators in similar contexts. A distance scholarship approach to examining
Internet-related technologies will improve the ability of educators to identify
resources that more closely match their requirements.
In
the past, teachers had to wait for an article to be published or attend
conferences to find out about the latest uses of technology, and even then
hands-on opportunities were limited.
Now, teachers can find new technologies for assessment and try these via
the Web or download. The same communications
technologies that allow delivery of instruction and assessment activities via
the Internet can be designed to allow teachers to share stories and data about
what tools have been most worthwhile for their specific purposes and
situations. Teachers and researchers can
give feedback to developers about how their tools worked in different settings
and pass this information on to the next round of developers and users. When this type of review of tools is done
within an instructional setting this feedback can be organized within a
conceptual framework and placed within a learning context that can also be
shared.
Formative
Assessment
Formative
assessment is one area where scholarship has seen benefits from
technology. Examples of
technology-supported assessments that have been available for distant learners
via the Internet include IMMEX (Underdahl, Palacio-Cayetano, & Stevens,
2001), Intelligent Essay Assessor (Foltz, Laham, & Landauer, 1999),
Knowledge Mapper (Baker, 1998), and the Analysis Toolkit for Knowledge Forum
(Lamon, Reeve, & Scardamalia, 2001).
In each case, there is evidence that assessment-related information can
be provided to teachers and learners in a way that supports understanding of
learning outcomes.
Technology
can be an extremely effective tool for giving feedback during the learning
process to learners and their teachers.
The ability of technology to rapidly provide feedback to learners is one
of its most salient features for education.
Consider how quickly young people learn complex video games by being
bombarded with constant feedback on their performance that proves to also be
highly motivational. Examples from the educational research community
include simulations, dynamic modeling tools, teachable agents, and other tools
that give students and teachers tools they can use as feedback on their thinking. Some allow students to view consequences of
their actions on the screen in near real-time and then provide tools for
students to modify their thinking and test the results again. The assessments that are provided are valued
over traditional “right vs. wrong” judgments because they facilitate analysis
of student reasoning processes.
Feedback
can be more or less prescriptive, directing the learner towards a next step or
simply providing information that can support self-correcting behavior or a
teacher’s intervention. Tools that make
thinking visual offer opportunities for students and teachers to review their
work and develop strategies for improvement based on what they see. The teacher can play a key role in
interpreting the data, or the data can provide the basis for student
reflection. Technology affords a more
rapid and individualized “stock taking” that provides learners and teachers
with the ability to pursue new avenues of learning.
Automated
analysis of writing via the Internet can let learners know if they are covering
the main topics in an essay before they turn their work in to the teacher. Students and teachers can look together at
problem-solving strategies using a visual representation of the information
they used. A constrained knowledge-mapping
exercise can be scored in real-time using pre-assigned ratings given by experts
of each proposition that is used. More examples of technology-supported
assessments include supports for mentoring and peer review, teachers using
rapid polling technology to see if students are following a lecture, or using
handheld electronic devices to gather classroom data for review with students.
Computer-based instruction methods also have a long-standing emphasis on
formative assessment and feedback mechanisms.
Improving
formative assessment of learning – providing feedback to help guide teachers
and learners – should be acknowledged as perhaps the single best way we have to
improve learning. It is the formative
assessment practices that accompany some interventions that are perhaps their
most valuable features. A large-scale
British study found effects of formative assessment practices that exceeded
those of many of the interventions themselves.
One reason offered by the study authors is that formative assessment
practices offer the greatest support to the students who need it the most,
raising the performance of the lowest performing students the most, while
raising the scores of higher performing students as well (Black & Wiliam,
1998). In contrast, traditional tests and grades often are given after
the performance is completed and without opportunities for incorporating the
feedback into new performances. In this
country, Bransford (2001), Stiggins (1997), and others also emphasize the
centrality of formative assessment for effective teaching and learning.
As
more formative assessment instruments become available, will knowledge
resulting from their use lead to better opportunities for teaching and
learning? One opportunity for “distance
scholarship” concerns the abundance of self-assessment surveys that are
available for review via the Web.
Profiler (http://profiler.hprtec.org) requires developers of these
surveys to categorize questions in a way that provides an indication of different
topics that are being addressed. In a
setting where there are large numbers of people employing similar measures – in
this case a self-assessment by teachers of conditions in their schools – the
ability to test the robustness of different conceptual schemes based on the
data is very valuable. Districts and
schools that use this data for decision-making can benefit from knowing what
patterns have appeared elsewhere along with the interpretations and actions that
have been advanced to respond to the data.
Methods for stripping identifying data and making it available for
secondary analysis are not hard to envision.
There
is the potential for a wellspring of knowledge about teaching and learning to
be created, based on the use of different tools for assessment that have become
available online. As more people begin
using technology for other types of data collection and analysis, it may be
possible to share collections of student performance assessments (Quellmalz
& Schank, 1998) and video examples of teaching practices (Fredericksen,
Sipusic, Sherin, & Wolfe, 1998) for analysis by others. Creating a system for sharing related
instruments and discussing their usefulness in different settings would be a major
contribution to assessment and evaluation practices. One can imagine communities of educators
developing knowledge around shared uses of technology and trials of new tools
that can inform their ongoing work. This
could support larger-scale implementations of technology innovations by
allowing people to connect more directly with useful information and ideas that
have helped others in similar situations.
Examples of problematic areas that require development include the
assessment of online conversations, and the assessment of group work and projects
using technology-supported tools.
Assessing
Online Discussions
The
assessment of online conversations is ripe for an influx of “distance
scholarship.” It is difficult to assess
the educational value of technology-supported learning activities, but finding
ways to assess learning in online conversations may be one possibility. Tools for online discussion routinely include
analysis features showing the extent of participation of different members. It is not clear how these tools will
contribute to better teaching and learning, however.
Additional
tools are emerging that allow participants to map online discussions as they
evolve –to label questions within a discussion, to link possible answers to
evidence and to creating unique views of discussions that can be compared to
others’. By incorporating these features
of discussion tools we might begin to provide more complex analyses that address
individual knowledge and participants' contributions to the knowledge of a
group (Lamon, et al., 2001).
One
can envision educators revisiting online discussions to analyze and synthesize
what has occurred, using frameworks like Mason’s (1992) to determine level of
independence and initiative: To what extent do students build on previous
messages; draw on their own experiences; refer to course materials; refer to
relevant outside material or initiate new ideas; and control, direct or
facilitate discussions? By having
teachers practice applying and responding to different sets of rubrics for
judging online interactions we might progressively develop the ability to
understand and assess online interactions. This will also support development
of strategies for interventions. In
response to shared analysis of situations, teachers might experiment with
trying different voices and tones in their composition of messages in order to
elicit students’ pragmatic intellectual strategies (Collison, Elbaum, Haavind,
& Tinker, 2000).
Obtaining
Better Information About Group Work and Projects
Another
area for urgent development includes the ability of teachers to assess student
group work and projects. When
implementing complex innovations involving student group work and projects,
teachers may find themselves having to focus on directing students
procedurally, and having less time for substantive feedback to individuals or
groups. Use of technology-supported
formative assessments can inform student self-assessments and help guide their
work without relying exclusively on the teacher until necessary. Technology also provides students with ways
of providing peer feedback or getting feedback from others – interactions that
can also be assessed when such interactions are one focus of the teaching and learning
process.
New
tools to help teachers assess their students’ project-related work and group
work include: dynamic display of online survey data collected from distant
learners, assessments of student work in local groups using handheld devices,
online annotation systems for the development of online projects, peer- and
expert-review systems, social network analyses, and other methods which
technology facilitates well.
These
methods can be done without technology, but they are quite laborious. Often, we do not really know what students
are learning when they incorporate technology into their intellectual and
creative work (CEO Forum, 2001). A
renewed effort to develop and validate performance assessments and assessment
strategies for online discussions, group work, and projects is envisioned. Unless helpful assessment mechanisms remain
in place, many group and online projects are simply impossible to assess and
evaluate (Ravitz & Serim, 1997).
They may show a product without any indication of the underlying thinking
that went into its production. While
this can be portrayed as a failure of pedagogy and task structure, it is a
problem that can be addressed through development of appropriate technology-supported
designs that structure inquiry, discussions and group processes in ways that
allow better assessments and evaluations.
Finally,
many researchers are questioning the usefulness of standardized tests as
measures of technology’s impact. Quellmalz
and Schank (1998) and others have called for a renewed effort to create
performance assessments and to build the capacity of teachers to use these with
students. Although standardized tests
are good at testing what a student knows, it is hard to see how they can
address what students are able to do with that knowledge, especially using
technology. In short, important
knowledge and skills are not adequately measured using existing tests. This represents a validity threat for use of
standardized tests in technology-related studies (Messick, 1998).
EVALUATION
CHALLENGES
Establishing
a Context for Making Comparisons and Causal Claims
So
far this paper has focused on applying the idea of distance scholarship to
improve the practice of formative assessment of student learners. Just as traditional assessments are unable to
account for learning in online settings, so too are traditional evaluation
methods (Ravitz, 1997). Many projects
assume a positive effect for technology, but how do we know technology is
making a difference? The assumption that technology use is somehow
meaningfully tied to teaching and learning opportunities is supported by
correlations resulting from large-scale survey studies (Becker & Ravitz, 1999). However these “snapshots” of self-reported
practices are insufficient for uncovering causal mechanisms by which a change
occurs. Technology is only one piece of
a complex puzzle that includes infrastructure, changes in direction and
practice over time, the moderating effects of family, school and district
variables, and many factors changing concurrently (Haertel & Means, 2000;
Means, 2000).
Evaluation
requires careful measurement or control of implementation and context variables
before one can claim any effect for a certain use of technology. Effective technology use is intertwined with
the characteristics of the people and environments where it is used. Some of the success of technology innovations
in a school will depend on the teachers who are involved and factors such as
why they are using a tool, and whether or not they have had opportunities to
observe benefits of different uses and develop their palette of uses. Teacher background characteristics that are
related to technology use include: philosophy of teaching prior teaching
practices, prior technology use, gender, academic background, and professional
involvement. Other contextual variables
include time and flexibility in the schedule and curriculum, full-time
technology resources and support, class size, and, not least, the level and
subject taught. Teacher
perceptions of student prior achievement are also strong predictors of teaching
practices and computer use, with evidence that high performing and low
performing students within a school receive substantially different
opportunities to use technology. These findings are based on a series of
reports from Teaching, Learning & Computing, a national survey of teacher
pedagogy and computer use conducted by Henry Jay Becker at the University of
California, Irvine with the author (see Becker, Ravitz, & Wong, 1999).
For
PT3, an additional set of contextual and teacher background variables can be
constructed: what is the faculty member’s position in the university; how was
this faculty member using technology at home, in his or her professional work
(e.g., collaborating with colleagues), or in the classroom prior to when PT3
started? Was the person involved in planning at the outset of the
project? How subject-specific is the use that was developed? What
other software does the person have and know how to use? Is the person
given paid time to develop technology uses?
Without answering these questions, or at least having a guiding
framework to ask the questions, one cannot confidently attribute value to
technology-related developments.
Conclusions
One
measure of PT3 success will be its impact in U.S. classrooms over the next
several years as pre-service teachers go into classrooms and we see how
technology influences their teaching.
When one reviews the stated PT3 goals in light of the ideas expressed in
this paper, the goals that seem most likely to result in effective uses of
technology are those that encourage teacher preparation programs to pursue
inter-disciplinary work with schools of arts and sciences, to improve
communications between colleges of education and K-12 schools, and to develop
and share best practices. These goals
reflect a renewed emphasis on improving formative assessment and scholarship in
educational technology. In contrast,
calls for general technology proficiencies may inadvertently encourage routine
assessments and uses of technology.
There may be a “pedagogical” divide that develops, as well as a digital
one among users of technology based on the varying abilities of teachers, students,
and schools to apply innovative tools for learning and assessment.
This
paper suggests that shared or “distance” scholarship can be used to address
widespread and problematic issues in the areas of assessment and evaluation,
including new forms of assessment and treatment of complex evaluation issues in
educational technology. The scholarship
that is envisioned will advance the understanding of the best technology uses
for a given context, what other important variables are at play, and how to identify
appropriate measures of student learning that correspond to the project work,
group work, and online discussions.
A
renewed focus on improving scholarship within and across educational technology
efforts to address these issues will help meet the needs of teachers,
researchers and policy makers as they attempt to communicate about the effects
of technology on learning. In many ways
the technological barriers to the described vision have come down faster than
the social barriers. Incentives for
sharing and collaborating, and norms for obtaining and giving useful feedback,
have yet to be adequately established.
The
goal of this paper is to focus attention on emerging uses of technology to
improve formative assessments of learners and to increase the cumulative
contribution to knowledge about teaching and learning arising from
technology-supported assessments and evaluations. The inclusion of technology-supported
assessments and evaluations within the context of online educational projects
offers immense opportunities for knowledge creation and utilization. We have a long way to go before we will
understand how the Internet is changing the way we teach and learn. New technologies are impacting not only
teaching and learning, but they also have the potential to transform the way
educators and researchers produce and use knowledge.
References
Baker,
E. (1998, November). Understanding Educational Quality: Where Validity Meets
Technology. William H. Angoff Memorial Lecture Series. Princeton, NJ:
Educational Testing Service. Available: http://www.ets.org/research/pic/angoff5.pdf
Becker,
H., & Lovitts, B. (2000). A Project-Based Assessment Model for
Judging the Effects of Technology Use in Comparison Group Studies. In Haertel
& Means (Eds.), Stronger Designs for Research on Educational Uses of
Technology: Conclusion and Implications. Menlo Park, CA: SRI
International. Available: http://www.sri.com/policy/designkt/found.html
Becker,
H. & Ravitz, J. (1999). The Influence of Computer and Internet Use on
Teachers' Pedagogical Practices and Perceptions. Journal of Research on
Computing in Education, 31(4).
Becker,
H., Ravitz, J., & Wong, Y. (1999). Teacher and teacher-directed student use
of computers and software. Report #3. Teaching Learning and Computing: 1998
National Survey. Irvine, CA: University
of California at Irvine. Available: http://www.crito.uci.edu/TLC/html/
findings.html
Black,
P., & Wiliam, D. (1998). Inside the
black box: Raising standards through classroom assessment. Phi Delta Kappan, October. Available: http://www.pdkintl.org/kappan/
kbla9810.htm
Bransford,
J. (2001). Toward the development of
a stronger community of educators: New opportunities made possible by
integrating the learning sciences and technology. Paper submitted to the PT3 Vision Quest on
Assessment in e-Learning cultures.
CEO
Forum. (2001). Key building blocks
for student success in the 21st century. The CEO Forum on Educational Technology. Available:
http://www.ceoforum.org/downloads/report4.pdf
Collison,
G., Elbaum, B., Haavind S., & Tinker, B. (2000). Facilitating online learning: Effective
strategies for moderators. Madison,
WI: Atwood.
Foltz,
P., Laham, D., & Landauer, T. (1999). The Intelligent Essay Assessor:
Applications to Educational Technology. Interactive Multimedia Electronic
Journal of Computer-Enhanced Learning, 1(2). Winston-Salem, NC: Wake Forest
University. Available: http://imej.wfu.edu/
articles/1999/2/04/printver.asp
Fredericksen,
J., Sipusic, M., Sherin, M., & Wolfe, E. (1998). Video portfolio
assessment: Creating a framework for viewing the functions of teaching. Educational
Assessment, 5(4), 225-297.
Haertel,
G., & Means, B. (2000). Stronger Designs for Research on Educational
Uses
of Technology: Conclusion and Implications.
Menlo Park, CA: SRI International.
Available: http://www.sri.com/policy/designkt/found.html
Horn,
R.E. (1997). Structured Writing as a
Paradigm. In A. Romiszowski and C. Dills
(Eds.), Instructional Development: State of the Art. Englewood Cliffs,
NJ: Educational Technology Publications.
Mason, R.
(1992). Methodologies for evaluating
applications of computer conferencing.
In A.R. Kaye (Ed.), Collaborative learning through computer
conferencing: The Najaden papers 105-116. New York, NY: Stringer-Verlag.
Means,
B. (2000, July). The impact of learning technologies: Penetrating the cloud
of confusion. Presented at the
annual meeting of the Consortium for Research on Educational Accountability and
Teacher Evaluation. San Jose, CA.
Messick,
S. (1998). The interplay of evidence and consequences in the validation of
performance assessments. Educational Researcher, 23(2), 13-23.
Quellmalz,
E., & Schank, P. (1998, April). Performance Assessment Links in Science
(PALS): On-line, Interactive Resources.
Presented at the annual meeting of the American Education Research
Association in San Diego, CA.
Available: http://www.pals.sri.com/papers/aera98/
index.html
Ravitz,
J., & Serim, F. (1997, April). Summary of first year evaluation report
for the Online Internet Institute.
Paper presented at Edward F. Kelly Evaluation Conference. State University
of New York, Albany. Available:
http://www.gse.uci.edu/Ravitz/oii_summary.html
Ravitz,
J. (1997). Evaluating Learning Networks:
A special challenge for Web-based Instruction? In Badrul Khan (Ed.)., Web-based
Instruction, 361-368. Englewood
Cliffs, NJ: Educational Technology Publications.
Russell,
M. (2000). It’s Time to Upgrade: Tests and Administration Procedures
for
the New Millennium.
Secretary’s Conference on Educational Technology.
Washington,
DC: U.S. Department of Education. Available: http://www.ed.gov/
Technology/techconf/2000/russell_paper.html
Shulman,
L. (1999, May). The scholarship of
teaching for meaningful learning.
Plenary address at annual meeting of the Center for Innovative Learning
Technologies. San Jose, CA.
Stiggins,
R. (1997). Student-centered classroom assessment (2nd Ed.). Upper
Saddle
River, NJ: Merril.
Trilling,
B., & Hood, P. (1999). Learning, technology, and education reform in the
knowledge
age or “we’re wired, webbed, and windowed, now
what?” Educational
Technology, 39(3), 5-18.
Underdahl,
J., Palacio-Cayetano, J., & Stevens, R. (2001). Practice makes perfect:
Assessing and enhancing knowledge and problem-solving skills with IMMEX
software. Eugene, OR: International Society for Technology in Education.
Author
Notes
Jason
Ravitz recently was appointed Associate Director of Research for the Buck
Institute for Education, in Novato, CA. The Buck Institute for Education is a non
profit research and evaluation organization dedicated to improving schools by
advancing knowledge about the practice of teaching and the process of learning.
The Institute develops innovative educational practices in collaboration with
teachers and refines these practices in classrooms. Please direct correspondence to the
following:
Jason Ravitz, Ph.D.
Associate Director of Research
Buck Institute for Education
18 Commercial Blvd.
Novato, CA 94949
Voice: 415-883-0122 x 310
Fax:
415-883-0260
Email: Jason@bie.org
Links
to online materials and examples are available:
http://www.bie.org/Ravitz/Assess/pt3.htm