4 Chapter Four | Assessment and Evaluation

I wake to sleep, and take my waking slow.
I feel my fate in what I cannot fear.
I learn by going where I have to go.^[1]

If you are eager to see rubrics, skip ahead to the “Rubrics” section of this chapter. Study several rubrics and holistic guidelines, and then amend them to work for your specific learning objectives.

After seven years of teaching with digital stories in diverse courses and iterations, assessment is still a work in progress, but I’ve learned by doing and through discussion with colleagues and tech specialists. There are also now many more online resources to turn to, from thoughtful articles shedding light on the evaluative process, to specific rubrics to choose from and modify.

Let’s consider assessment in light of backward design and participatory learning.

Backward Design

Begin with the end. What are the three most important learning objectives you have for the course, and what part does the digital story assignment play in helping students achieve those learning objectives? Critical thinking? Communicating effectively? Engaging their audience with their findings? Keep your objectives in mind as we consider the end, the true goal of the role assessment plays in our courses, and specifically in the digital story assignment.

Participatory Learning: Including students in the assessment process

Howard Gardner, Harvard Professor of Psychology, helps us dive into the thickets of why we assess and what really matters in our assessment process:

Howard Gardner and the Understanding by Design method both prompt me to travel back to the beginning of my courses to consider how I seed the assessment portion of my assignments. I look for ways to prepare the students so when they begin the digital story assignment, they have already inculcated a critical mass of knowledge of at least one learning objective through encounters with it earlier in different learning situations.

Take, for example, an art analysis course I teach where the digital story is an option for the culminating art project. “Effective interplay between use of images, music, and text” is a category in the class’ rubrics and assessments. In assignments earlier in the class, which include low–stakes exercises and larger projects, we explore the interaction between images and text; how an effective thoughtful integration of writing, and either images or music, creates a third kind of knowing – a creative expression that is greater than the parts. Through diverse assignments, I highlight examples from student work that demonstrate an adequate interaction, meaning that the image and writing relate, but in a didactic straightforward way where the image does not shed new light on the text and the text simply reinforces the meaning of the image (or music). I also discuss and show models of art where the images and writing create a frisson of meaning with aesthetic power. Since students have given and received peer feedback on the “Effective interplay between use of images/music and text” rubric category throughout the course, they approach this category in the rubric for the digital story assignment with a nuanced understanding of the objective.

Gardner’s point about assessment not being “something that’s done to you, it should be something where you are the most active agent” is a recognition that the participatory nature of student’s learning extends into the assessment arena. If one of our goals for the digital story assignment is to empower students, to give them agency over topic and creative choices, then shouldn’t they participate in the assessment process as well?

This is not to be confused with handing over the reins of the course. As the teacher, you establish the learning objectives; you’ve made clear throughout the semester what you feel is essential for them to learn about the subject matter and the other elements of learning that occur in a classroom. A thoughtful benevolent power–sharing in the assessment process can engender in students a deeper level of commitment to the evaluative part of the learning. If students take part in creating the criteria and, depending on your comfort level, the process of evaluation, they will be invested in, and grant consent to, the grading.

Depending on time, age of students, class size, and your comfort level, students could create or help create:

• The method of assessment – Rubric? Holistic assessment (i.e., a written or oral feedback from you)? Self assessment? Some combination of these?
• The criteria for grading – Each category that will be assessed, and how many points they will be worth.

For example, if you have a lot of students and you are most comfortable using grading rubrics, then give students three different rubric options and let them determine which one is most reflective of the work they will be doing.

Here is a strategy I have found beneficial to both including students in the assessment process and creating evaluative processes that feel like a more natural result of the process, rather than a giant pain for me to complete, and a traumatic
surprise to the students:

1. Whole class assessment: Model the kind of assessment you wish to achieve by having the whole class watch a digital story and participate in assessing it collectively. To do this, share a few different rubrics and discuss. Distribute sample rubrics and discuss. Then have the class watch a digital story made by a student who has taken the class in the past or from a student in a different class. Frame the viewing with a discussion of the key criteria in the rubrics. Have them take notes. I have found that students are often harder on each other than I am and if the environment is conducive, give tough feedback, or a different kind of feedback that wouldn’t have occurred to me but is meaningful to the student, as well as helpful for the quality of the digital story.
2. Put students in small groups: Each group is assigned a particular element of the assessment that they will be focusing on, and responsible for assessing. View a second digital story. Allow time for groups to discuss their notes within their group, and then share their results with the rest of the class. What did you discover?
3. Setting criteria: Consider creating with them a set of criteria for an A–level project, for a B–level project, and so on: Students can select which level they will aspire to. In this way they understand what they need in order
   to achieve that grade.

Suggestions for first time assessing of digital stories: Transparency & Risk–sharing

If this is your first time teaching a digital story assignment, in a sense, you are co–learners with the students. You are embarking on a new assignment and discovering what you value and find effective, and what you want to head off. Here are some suggestions for sharing the risk with the students:

• Give students as clear a set of guidelines, or a rubric, as you can. Require a complete rough draft of the digital story. Allow time for you to view and respond to rough drafts, if possible using a tool such as VideoANT^[3]
  for online video annotation.
• If you find that a fair number of students are not achieving a learning element in the rough drafts of their digital stories, revisit what you wrote in the assignment guidelines and what you stressed in your peer evaluation guidelines. Be willing to amend your rubric and share your thought processes with the students. Communicate the changes to the students.
• For the first round of digital stories, consider giving all students the same number of points or percentage for completing all of the steps and requirements you establish in your assignment guidelines. Once you’ve viewed the stories you’ll know what you want, what you value, how to shape your rubric or holistic assessment in the next round, and you’ll have models from this batch to share with the next round of students. If it turns out that students invested more time and energy than the points you dedicated to the project reflect, consider adding points after the fact. While this is hard on us as we have to find those points from somewhere else, and be transparent with the students, it conveys integrity, and tends to engender more respect and trust from the students.

In the first few years of integrating digital story assignments, I taught courses solo and sought out colleagues and tech fellows in the department, across campus, and online for assessment questions. One of the more satisfying experiences teaching digital stories was when I co–taught a First Year Inquiry course with a colleague who teaches in the social sciences. Prior to the semester we met several times and hashed out a thoughtful digital story assignment which would occur in the last third of the semester and which would allow students to focus on one of the many topics we addressed in the class. When our first batch of stories came through from 100 students, so did his flurry of email questions:

• “ How do I grade Tom’s? He did great in voice and personality, but there’s hardly any integration of ideas from class?”
• “Susan did well on music and photos, wow, and its funny, but I’m not sure what her topic is.”
• “Abdi spent hours learning how to use iMovie, then the night before it was due his computer crashed and he lost his story because he didn’t back it up and he had to do it over; the final product is ok, but I saw the rough draft of the first one and the story was really doing something important and well.”

I was reassured by having someone else battling through these thorny issues. It helped me to realize these challenges are endemic to the digital storytelling process. I can imagine a time in the future when our collective expectations for digital productions will be more refined. The more we can collaborate with colleagues, either in person or online, about assessment, the stronger we all will be.

A concluding thought about assessment and our desires for an ideal product. The digital story, even more than a research paper or other academic product, lends itself to being scrutinized to a different standard due to its visual and multi–medium, as well as the public nature of the audience, whether it be the classmates or the world wide web. While we want our students to do their very best, it’s important to remember what your learning objectives are for your course and digital story. Is your course about the production quality of videos? If not, then production quality should count for less in evaluation. More emphasis and points should be devoted to, for example, whether the student has a voice in her story, and has thought through how to frame her story, and indeed created an arc of a story. More points should be given to the power of the learning that occurred, and whether the story they are trying to tell has been communicated.

The assessment process for digital stories requires us as educators to think deeply about the nature of evaluation. In light of understanding best practices of assessment, the transformative learning that we intend for our students, may extend to us as well.

Digital Story Rubrics

A thoughtfully developed, concise rubric can be a boon to both you and your students, especially with larger classes, and if you have assistants who are helping you with grading. The more explicit you can be about your learning objectives, and the grade point value placed on each skill or concept, the more satisfying the learning process will be for all involved.

Here is a rubric I use for my digital storytelling assignments:

• Digital Narrative Rubric

Feel free to download the rubric and either use it as-is or modify it for your own needs.

Additional rubric resources:

• The “Assessment and Research” section of Kathy Schrock’s Guide to Everything has multiple assessment rubrics.
• The AACU VALUE Rubric development Project has excellent downloadable rubrics that hone in on specific learning outcomes including intercultural knowledge and social responsibility.

Student Work and Reflection

Matt’s digital story, “Living with Down Syndrome” is a revealing story about his cousins; his story of how he formulated the idea for his project is also powerful from a digital storytelling standpoint. Watch his digital story and his reflection video on how he made the story and see how Matt took the concepts he learned in class and applied them organically to make this moving digital story.

Student Matt Welch on the making of the digital story “Living with Down Syndrome”

A digital story by Matt Welch, “Living with Down Syndrome: More Alike Than Different”

Teacher Reflection

Assessing a digital story takes time and thoughtful planning. The Chapter 4 teacher reflection video shows the struggles and successes teachers have had, and continue to have, when it comes to this challenging yet rewarding part of the digital storytelling process.

Assessing Digital Stories