In preparation for our meeting next week, Merrilee has asked us to draft
a statement regarding MOA-2 project evaluation. We will be joining you in
the morning and presenting in the afternoon session on Tuesday, September
29 at the NY Public Library.
During our presentation, we plan to spend a few minutes talking about our
background and similar evaluation projects we have coordinated. For more
information, you can visit us on the web at http://www.img.cornell.edu.
The coordinating evaluation team members are:
1) Dr. Geri Gay, Associate Professor, Communication Dept. and Director,
Interactive Media Group (IMG);
2) Robert Rieger, Assistant Director, IMG;
3) Oya Rieger, Digital Projects Librarian, Department of Preservation and
Conservation, Cornell University Library
In past evaluation efforts conducted by our lab, we try to review
carefully all the planning documents and correspondence, listen to
participants particular interests and concerns, come up with a draft
plan, then present it for feedback. To keep this process moving, we will
present a general evaluation strategy below and at the meeting, then
amend it after September 29.
The evaluation population identified includes both MOA-2 participants and
the larger community:
<bold>MOA-2 Evaluation Audiences
</bold>1. MOA-2 project participants (organizers, system designers,
technical teams)
2. Scholars (faculty, grad students and other researchers)
3. Technical consultants (systems analysts, programmers, etc.
unaffiliated with MOA-2)
4. Library community (librarians, archivists, curators, etc.)
The evaluation section of the proposal calls for ongoing feedback via
informal exchange among participants followed by a traditional,
end-of-project evaluation and report.
<bold>Informal Evaluation
</bold>For rapid prototyping, we are accustomed to integrating evaluation
throughout project development. That is, it's important to provide
continual information back to developers as soon as it becomes available,
rather than sitting on important feedback to share in the final report.
We think the MOA-2 listserv will serve that role efficiently, allowing
users to pass along anecdotal findings. We do, however, suggest that
developers build in a couple reality checks by enlisting the help of
audience members to gather their input and reaction at a couple key
stages along the way. We plan to chime in on the listserv at a couple of
designated times and ask participants to comment on the process and
product and also encourage participants to get and report feedback from
prospective end users.
<bold>Formal Evaluation
</bold>Based on our conversation with Bernard Hurley, the Interactive
Media Group is looking at the first two evaluation issues as outlined in
the proposal. <bold><underline>Issue One</underline></bold> involves
selection of digital archival classes and associated metadata, and how
this provides value to scholars (and those involved in creating DL
collections). <bold><underline>Issue Two</underline></bold> examines the
effectiveness of the electronic environment as a research and learning
tool. It also covers the effectiveness of the methods developed-how
archival objects are discovered, displayed and navigated. We like the
major evaluation question, which is:
"Did this tool encourage new ways of searching and understanding?"
We see evaluation Issues One and Two as very closely linked and plan to
rely on the same kinds of testers for both issues.
We do not expect to present a finely detailed evaluation plan until later
in the project development when we see exactly what is emerging. We have
had the most success with data collection through focus groups,
interviews and on-line surveys, in this case, directed at both scholar
users (professors and grad students) and professional users (archivists
and librarians). Most of this can be done at Cornell (for lack of a
travel budget), although the online surveys will obviously be cast to the
widest possible audience. We will probably call on a few participants to
help us formulate and review our methods and areas of questioning. We
call on everyone to help recruit potential evaluators.
<bold>Some questions that we will be asking the group at the upcoming
meeting are:
</bold>
1) Who will be the evaluation contact at your institution? Depending on
the particular strategy, this person responds to information requests,
recommends users, helps channel data, etc.
2) Do you have a pool of users, surrogate or real, that we can draw upon
for ongoing feedback and evaluation? It may be especially helpful to tap
highly motivated users, such as transportation history scholars, for
their insights.
3) For planning purposes, we need to know if a revised timeline has been
developed.
4) How can we make this "informal" followed by "formal" evaluation
strategy work most efficiently?
5) Do we need to consider the wide range of prospective user groups at
this stage; that is, k-12 through scholars?
Congratulations on the success of the proposal. We look forward to
meeting and working with all of you.
Geri, Oya and Robert
****************
Robert Rieger, Asst. Director, Interactive Media Group
209 Kennedy Hall, Cornell University, Ithaca, NY 14853
Phone: 607-255-5530; Fax: 607-255-4346; http://www.img.cornell.edu