Evaluating Multimodal Scholarship
Excerpt from "Evaluating Multimodal Work, Revisited" in Journal of Digital Humanities Vol. 1, No. 4 Fall 2012. Online here: http://journalofdigitalhumanities.org/1-4/evaluating-multimodal-work-revisited-by-shannon-mattern/.

Used here for class purposes with permission from the generous Dr. Mattern

Evaluative Criteria

After reviewing a great deal of existing literature and assessment models — all of which, despite significant overlap, have their own distinctive vocabularies — I thought it best to consolidate all those models and test them against our on-the-ground experience in the classroom over the past several years, to develop a single, (relatively) manageable list of evaluative criteria.

Steve Anderson and Tara McPherson remind us of the importance of exercising flexibility in applying these criteria in our evaluation of “multimedia scholarship.” What follows should not be regarded as a checklist. Not all these criteria are appropriate for all projects, and there are good reasons some projects might choose to go against the grain. Referring to the MLA’s suggestion that projects be judged based on how they “link to other projects,” for instance, Anderson and McPherson note that linking may be a central goal for some projects, but, “linking itself should not be an inflexible standard for how multimedia scholarship gets evaluated.” Nor should the use of “open standards,” like open-source platforms — which, while generally desirable, isn’t always possible.[2]

The following is a mash-up up these sources, with some of my own insight mixed in: Steve Anderson & Tara McPherson, “Engaging Digital Scholarship: Thoughts on Evaluating Multimedia Scholarship,” Profession (2011): 136-151; Fred Gibbs, “Critical Discourse in the Digital Humanities,” FredGibbs.net (4 November 2011); Institute for Multimedia Literacy, “IML Project Parameters,” USC School of Cinematic Arts: IML (29 June 2009); Virginia Kuhn, “The Components of Scholarly Multimedia,” Kairos 12:3 (Summer 2008); MLA, “Short Guide to Evaluation of Digital Work,” wiki.mla.org (last updated 6 July 2010).

Concept & Content

- Is there a strong thesis or argument at the core of this project? Does the project clearly articulate, or some way make “experiential,” this conceptual “core”? Is this conceptual core effectively developed through the various segments or dimensions of the project?
- “Does the project display evidence of substantial research and thoughtful engagement with its subject?” (IML) Does it effectively “triangulate” a variety of sources and make use of a variety of media formats?
- Is the platform merely a substrate for a “cool data set” or a set of media objects — or are individual “pieces” of content (data and media in various formats, etc.) contextualized? Are they linked together into a compelling argument? (Of course students might be working to create their own data sets or digital archives, in which case we might apply different standards of evaluation.)
- Is the data sufficiently “enriched”? (MLA) Is it annotated, linked, cited, supplemented with support media, etc., where appropriate?
- Does the project exploit the “repurpose-ability” of data? Does it pull in, and effectively re-contextualize, data from other projects? (Students should also recognize that their own data can, and should, be similarly repurposed.) This recognition that individual records — a photo or video a student uploads, or a data-set they import, etc. — can serve different purposes in different projects offers students great insight into research methodology, into the politics of research, into questions regarding who gets to make knowledge, etc. As Fred Gibbs acknowledges, discussing how a project uses data also “encourages conversations about ownership [and] copyright.”

Concept/Content-Driven Design & Technique
- Does the project’s form suit its concept and content? “Do structural and formal elements of the project reinforce the conceptual core in a productive way?” (IML)
- Is the delivery system robust? Do the chosen platforms or modes of delivery “fit” and “do justice to” the subject matter? Need this have been a multimedia project, or could it just as easily have been executed on paper?
- Does the project “exhibit an understanding of the affordances of the tools used,” and does it exploit those affordances as best possible — and perhaps acknowledge and creatively “work around” known limitations?
- Is there a “graceful balance of familiar scholarly gestures and multimedia expression which mobilizes the scholarship in new ways?” (IML) A balance of the old and familiar, to help users feel that they can rely on their tried-and-true codes of consumption; and the new, to encourage engagement and promote reconsideration of our tradition ways of knowing?
- At the same time, do the project creators seem to exercise control over their technology? Or does technology seem to be used gratuitously or haphazardly? “Are design decisions deliberate and controlled?” Does the project “demonstrat[e] authorial intention by providing the user with a carefully planned structure, often made manifest through a navigation scheme?” (IML)
- Do the project creators seem to understand their potential users, and have they designed the project so it accommodates those various audiences and uses?
- How does the interface function “rhetorically” in the project? Does it inform user experience in a way that supports the project’s conceptual core and argument? Does it effectively organize the “real estate” of the screen to acknowledge and put into logical relationships the key components — subject content, technical tools, etc. — of the project?
- Has the project been tested? Are their plans for continual testing and iterative development? Is the project adaptable?


Transparent, Collaborative Development and Documentation

- Do project creators practice self-reflexivity? Do they “accoun[t] for the authorial understanding of the production choices made in constructing the project?” (IML)
- Do project creators document their research and creative processes, and describe how those processes contributed to their projects’ “formal structure and thematic concerns?” (IML) McPherson and Anderson (2011) also emphasize the importance of “finely grained accounts of the processes involved in the production of multimedia scholarship in order to evaluate properly the labor required in such research” (142).
- Do project developers document and/or otherwise communicate their process — perhaps through a “ticket” system like Trac or a service like GitHub — and make it transparent and understandable to students?
As Rory Solomon pointed out in a comment on my earlier blog post, the adoption of open standards also contributes to longevity: “…the open source movement provides means to help minimize these concerns in that open source projects provide many ways to evaluate a given software tool / format / platform. Any serious project will have an open, public web presence, including developer and user mailing lists, documentation, and etc. It is fairly easy then to evaluate the depth and breadth of the developer and user communities. It is useful to check, via wikipedia and other open source project websites, whether there are competing initiatives, whether the project is getting support from one of the larger foundations (eg, FSF, Apache, etc), and if there is competition then what trends there are in terms of which tools seem to be “winning out”. Once a critical mass is reached and/or once a certain level of standardization has been achieved (through things like IETF, ISO, RFC’s, etc), one can be fairly confident that a tool will be around for a very long time (eg, no one questions the particular voltage and amp levels coming out of our wall sockets) and even if a tool does become obsolete, there will be many users and developers also contending with this issue, and many well-defined and well-publicized “migration paths” to ensure continued functioning, accessibility, etc.”
- Are students involved in the platform’s development? Does this dialogue present an opportunity for students to learn about the process of technological development, to see “inside the black box” of their technical tools, to develop a skill set and critical vocabulary that will aid them not only in their own projects, but in the collaborative process?
Students should be asked for feedback on technical design; this conversation needs to happen as part of a structured dialogue, so it’s made clear to students what would be required to implement their requests — and whether or not such implementation is even feasible. Students should also be encouraged to translate their technical snafus — bugs, error messages, etc. — into opportunities to learn about how technology functions, about its limits, and about how to fix it when it’s not cooperating. Ideally, students should have a sense of ownership over not only their own projects, but also the platform on which they’re built.
I wrote about some of these frustrations-turned-into-positive learning-experiences in regard to my Fall 2011 Urban Media Archaeology class. Besides, these hiccups — and yes, on occasion, outright disasters — are an inevitable part of any technological development process. The error-laden development process defines every project out in the “real world”; why should a technological development project taking place within the context of an academic class be artificially “smoothed out” for students, artificially error-free?

Academic Integrity & Openness

- Does the project evidence sound scholarship, which upholds traditional codes of academic integrity (which, of course, might need to be adapted for an age in which “publishing” and “authorship” mean something quite different than they did when many of these standards were developed)?
- Does it credit sources where appropriate, and, if possible, link out to those sources? Does it acknowledge precedents and sources of conceptual or technical inspiration?
For my classes, I’ve made special arrangements with several institutions for copyright clearances and waiver of reproduction fees. In other cases, students will have to negotiate (with the collections’ and my assistance) copyright clearances; this is a good experience for them!
- Does the project include credits for all collaborators, including even those performing roles that might not traditionally be credited?
- “Is it accessible to the community of study?” (MLA) Is the final “product” available and functional for all its intended users – and open enough to accommodate even unexpected audiences? Is the process sufficiently well documented to make the intention behind and creation of the project accessible and intelligible to its publics?
Telling students that their work will be publicly accessible, and that it could have potential resonance in the greater world, can be a great motivator. Of course some students might feel vulnerable about trying out new ideas and skills in public view — and teachers should consider whether certain development stages should take place in a secure, off-line area.
- “Do others link to it? Does it link out well?” (MLA) Does the project make clear its web of influence and associations, and demonstrate generosity in giving credit where it’s due?
Emphasizing proper citations — of data, archival work, even human resources that have contributed to the project — reinforces the fact that academic integrity matters even within the context of a nontraditional research project, and it allows both the students and the collaborating institutions and individuals to benefit from their affiliation — e.g., the archives can show that researchers are using their material, and the students can take pride in being associated with these external organizations.
Review & Critique

- “Have there been any expert consultations? Has this been shown to others for expert opinion?” (MLA) Given the myriad domains of expertise that most multimodal projects draw upon, those “experts” might be of many varieties: experts in the subject matter, experts in graphic design, experts in motion graphics, experts in user experience, experts in database design, etc.
- “Has the work been reviewed? Can it be submitted for peer review?… Has the work been presented at conferences?… Have papers or reports about the project been published?” (MLA)  Writing up the work for publication or presentation at conferences elicits feedback. Grant-seeking also gives one an opportunity to subject the project to critique. There are also a few publications focusing on multimodal work — e.g., Vectors, Kairos, Sensate — that have developed, or are developing, their own evaluative criteria.
Individual students in my classes have presented their own projects at conferences, submitted them to multimodal journals, or written about their multimodal work for more traditional journals. More informal, though no less helpful, forms of “peer review” can take place in the classroom — through design critiques with external “experts,” student peer-review, etc.



Originally published by Shannon Christine Mattern on August 28, 2012.

Full Text: "Evaluating Multimodal Work, Revisited" in Journal of Digital Humanities Vol. 1, No. 4 Fall 2012. Online here: http://journalofdigitalhumanities.org/1-4/evaluating-multimodal-work-revisited-by-shannon-mattern/.

Thank you, Dr. Mattern!