Case Study
Arts-based evaluation on a college campus
The following is a case study based on a compilation of our varied experiences evaluating and developing strategies, collaborating with students, administration, staff and faculty at various institutions and community organizations. It illustrates some of the challenges and common issues that we have observed and highlights the committment to dealing with rape culture on college campuses by so many amazing collaborators. This narrative brings together the ideas in this toolkit and demonstrates ways in which they can be used to strengthen the important work that is happening across campuses and in community organizations. This case study portrays a very comprehensive and large scale evaluation that brings together people from across the campus. However, it can be scaled down to whatever the capacity of the resources allows while still providing useful data.
Scenario: Implementing strategies to address rape culture and sexual violence on campus
The administration at a local college developed several initiatives to address concerns raised by the college community about rape culture and sexual violence on campus. The college administration consulted with administrators at other colleges and universities in order to decide which strategies to implement and where to devote their limited resources. The result was a one-day event for students on campus. Personnel from campus health services, student services, student groups, and community organizations that provide resources relating to sexual health were invited to reach out and build awareness and educate students about key issues and resources. The various groups set up tables, put up posters, handed out educational and promotional materials, (such as pamphlets, ‘consent awareness’ pins and stickers) and were present to speak to the campus community, providing information and resources. Additionally, the administration purchased a curriculum about consent which comprised of a 2-hour workshop which they offered to students living in residence and all students participating on sports teams.
In the hours and days following the event, the administration received some positive responses from teachers and students in the form of passing comments and congratulatory emails. However, they also received a significant amount of feedback from the campus community critiquing the initiatives. Feedback was incidental, often second-hand and not systematic enough to provide a clear sense of the overall effectiveness. The varied, anecdotal feedback revealed a campus divided, revealing that while some students, faculty and staff were very satisfied with the initiatives, others believed that the strategies to end rape culture and sexual violence on campus were not sufficient. Some were frustrated and even angry, complaining that the strategies were not inclusive and did not reflect the authentic needs and issues of the students on campus. Students and college personnel felt that they had been left out of the conversations about which issues were the most important to them. Some of the students suggested that the point of the activities was unclear, some believed that the strategies were not comprehensive enough, others believed that the wrong issues were being addressed.
The college administrators who had selected and implemented the strategies were understandably very disappointed at the mixed feedback that they received because their intention was to effect positive change and bring the college community together to end rape culture on campus. They were frustrated at not having specific information and data about what went wrong. They were dismayed to learn that despite their good intentions the initiative seemed to have backfired in the eyes of some of the very people they wanted to help. Determined to learn from this experience and to do better going forward, they initiated an in-depth analysis of the situation in order to inform future planning and action.
Scenario analysis: Lessons learned
What went wrong? What might have been done differently? What was lacking in the initial planning? There is a lot to unpack in this scenario which can help illustrate what is needed for an evaluation that is more participatory, inclusive, and effective. Incorporating evaluation into the strategies from the beginning would have provided clear feedback from the campus community so that organizers could respond and adapt their initiatives to better meet community needs.
This situation reflects the need for a participatory approach to evaluation. In choosing which strategies to implement, the college administration had consulted administrators at other colleges rather than consulting their own campus community and particularly the people within the community who were the most impacted. This was now acknowledged to be an unfortunate oversight. Going forward, the administration will take a participatory approach and make sure that both the strategy and the evaluation are both inclusive and intersectional.
Developing and following an evaluation plan might have helped include the wide range of voices that need to be consulted when addressing rape culture on campus. Developing the tools to measure the effectiveness of proposed activities requires including and collaborating with the groups you are seeking to target with your strategy.
Developing an evaluation plan would have benefitted the administration because the first step in an evaluation plan is to identify goals, objectives, and outcomes. This important step was not well considered. Having an all-day event on campus can be a great strategy if your clearly stated objective is to briefly introduce new students to the range of campus-based student services, but, on the other hand, if your goal is to have students challenge their own assumptions and think more deeply about rape culture, a different activity might be more appropriate. In other words, part of the problem was that the administrators’ stated and perhaps an overly ambitious goal, addressing rape culture on campus and ending sexual violence, misled participants into expecting more than the strategies that were implemented that day could deliver. Planning an evaluation can help clarify objectives and point to appropriate strategy selection. Making these decisions starts with considering goals, objectives, and outcomes.
Understanding what the objectives are helps in identifying which populations need to be targeted with the strategy. For instance, is it a strategy that works to engage groups that are at risk of perpetrating harm? Or are the strategies directed at providing services for survivors? Is the strategy’s objective focused on prevention of sexual violence, or responding to sexual violence? It is important to have a clear understanding of which populations are being addressed and likely to benefit from which aspects of a strategy. By asking these questions first and examining your goals and objectives there can be better alignment between the strategy, goal, objective and its effectiveness in addressing specific populations and needs. Having an evaluation plan helps in gathering concrete data about both the effectiveness of the strategy but also whether the campus community agrees that the chosen strategy addresses a key issue. This could go a long way towards initiating a more balanced, informed discussion on campus.
“What’s going strong? What’s going wrong?”
Planning and implementing an appropriate evaluation
Armed with new insights from their analysis of what went wrong, the administrators decided to take action. After consulting several key faculty members they decided to use a participatory approach to:
- Determine the overall effectiveness of their attempts to address rape culture on campus
- Find out which strategies the campus community might want more resources devoted to
- Uncover aspects of the issue that might not have been visible to them, the unintended outcomes or changes that may occur as “side effects” of the stated goal, and to learn more about what strategies are needed.
Creating a participatory plan that is inclusive, intersectional and trauma-informed
This time around, the administrators decided to try to include as many members of the campus community in evaluating their efforts thus far and in determining where to go next.
1. They brought together the key policymakers and community leaders:
This included people who were well placed to effect change in policy and practices addressing sexual violence and rape culture on the campus to participate in designing the evaluation and receiving and acting upon the results. This included the Head of the Gender Studies Department, Dean of Student Services, Counseling Services, Sexual Assault Resource Centre Coordinator, the coaches of the athletics teams, campus faith leaders, heads of student organizations, such as the student union, transgender support group, and other student groups that showed an interest.
2. They applied an intersectional lens:
They reached out to students and organizations on campus who were disproportionately impacted by the issue and made sure that their voices were privileged – specifically, LGBTQ+ and BIPOC community. Students and members of the campus community whose everyday lives were most impacted by the issue of rape culture and sexual violence were recruited. The group included survivors, indigenous student representatives, students who are on the spectrum, as well as student groups that support students with (dis)abilities. Because people in these groups are frequently overburdened with these types of requests while experiencing additional challenges on campus, the administration was able to offer an incentive for the student club groups in the form of gift cards to these collaborators and consultants. The administration was also able to collaborate with student services to make sure that they were reaching out to the various communities in a trauma-informed and survivor-centered way.
3. They employed a participatory approach in creating an
advisory committee for their evaluation.
This committee was convened to ensure that the greatest representation shaped the evaluation so that the findings would be representative. By including the perspectives and ideas from a wide range of stakeholders they ensured greater buy-in to the evaluation within the communities that the stakeholders represented. Additionally, through including stakeholders in the process of developing and implementing the evaluation the chances of the findings being viewed as credible and acted upon was increased.
4. They adopted a trauma-informed approach.
They consulted with counselors from the student services department who were trained in both active listening and trauma-informed approaches to evaluation in order to ensure that the evaluation was not triggering to participants and that they were ready to support any students or members of the campus community. They consulted the trauma-informed section of this toolkit to help guide them in their approach.
5. The advisory committee met to decide what the key evaluation questions were and what methodology to use.
During the first meeting, they decided that the evaluation should ask what is working on campus to address sexual violence and rape culture (what’s going strong?) and which issues need to be addressed (what’s going wrong?). A key consideration was to choose a methodology that would be interesting and motivating to participants, enabling creativity, and expression. They also wanted to use a method that would be able to capture the hidden issues on campus that evaluators might not have predicted or even known existed and thus were unable to formulate questions for. This led to a decision to use an arts-based approach to evaluation. A faculty member suggested using photovoice because it is open ended and enables participants to use photography in order to express, reflect, and communicate aspects of their everyday lives. In this cell phone age where so many people are accustomed to “snapping their lives”, the committee thought that this choice might be motivating and feasible, giving students agency and voice.
Implementation of Photovoice:
An Arts-based Evaluation Method
Some of the key goals of photovoice that made it a relevant choice for the evaluation were that it enables evaluators to:
1. Record and reflect the community or group’s strengths and concerns.
2. Promote critical dialogue and knowledge about personal and community issues through both small and large group discussions of photographs taken by participants.
3. Reach policymakers and bring issues and recommendations to their attention.
Some of the premises and concepts that underlie photovoice:
Photovoice is a participatory and arts-based method of conducting evaluation that involves groups of participants taking photographs or assembling photographs taken by others around a theme or issue of their choosing; meeting frequently to show, discuss, and analyze their photographs; preparing a photo essay or notated album or slide show; and then deciding on a suitable format and venue for presentation of their work to policymakers. This general protocol can be adapted to multiple digital formats, using for example, cell phone videos and social media. It can also be adapted to using drawings or other art forms in combination with or instead of photographs. Most students on campus have access to a device that can take pictures, therefore the approach is very inclusive and resources required in terms of equipment are low.
6.
Another meeting was scheduled to introduce photovoice methods to the participants and facilitate a group discussion. The details of how to create an intersectional representative group of participants to develop the photovoice prompts and protocol, to participate in the data gathering, and to analyse and code the data, and submit the findings was resolved.
7.
The college’s ethics review board was consulted and their instructions were followed. As a result, an online informed consent form that could be submitted via cell phone was developed and the participants considered the implications and parameters of taking pictures of other people on campus eventually resolving not to include identifiable people in the images.
8.
The participants who were answering the evaluation questions in the form of photovoice prompts were provided with some useful tips on taking images. For instance, how to frame a photo, not to always put the subject in the center, use of creative ways to protect identity if necessary by taking the picture from behind, focusing on feet or hands.
9.
After the photos were submitted, a series of meetings to select, contextualize, and codify the themes or issues that emerged from the images was organized.
b. Participants framed stories about their picture or took a critical stance on their photos in terms of questions like: What do you see here? What is really happening here? How does this relate to rape culture on campus? Why does this problem or situation exist? What might we want to do about it?
c. Through group discussions with the evaluation team the links between the photos and stories were analyzed. Participants identified themes, common concerns, and theories that arose from and across their images.
d. The initial discussions guided further rounds of photo taking.
10.
When the participants decided that they had covered enough ground, and gained enough insight, they prepared a presentation to synthesize and share their results.
11.
They organized a presentation for the campus community to share their findings. The presentation took the format of a vernissage. They invited the entire campus community along with key policymakers and community leaders who were positioned to effect change in policy and practices and who had been involved in the evaluation process throughout. The chosen photographs, along with the accompanying captions and explanations were printed and hung in the same auditorium where the initial ‘one-day sexual violence awareness strategy’ had taken place. The vernissage resulted in bringing the campus community together in conversations about what was working in addressing rape culture on campus and what issues still needed to be addressed. Several new initiatives to address some of the findings brought forward in the photovoice evaluation emerged through the event.
Conclusion:
Through developing an evaluation plan, this campus was able to get a better idea of what types of strategies were working to address the issues of rape culture and what was needed going forward. By taking a participatory approach and engaging in an evaluation process the people who are most impacted by the rape culture are included in identifying key issues, providing feedback about the effectiveness of the strategies, and given an opportunity to suggest solutions and ways forward.
Including the campus community, and particularly those most impacted in the evaluation resulted in bringing a variety of perspectives into conversation, giving voice to the ‘experts’ who are the people at the center of the issues who are so often not provided with authentic opportunities to shape the conversation. The photovoice strategy resulted in the students feeling more included and the administration feeling that they were able to use the information gathered to guide their strategic planning and program developments on campus.
Using a participatory arts-based approach to evaluation brought people together and placed the discussion and evaluation of the effectiveness of various strategies to the forefront of campus awareness. The evaluation plan helped people re-imagine how they think about evaluation and the role it can play throughout the development and implementation of strategies.