Putting Evaluation in the Mix: Freire, Youth, Arts, and Equity
Putting Evaluation in the Mix: Freire, Youth, Arts, and Equity
By: Justin Laing | November 8, 2022 | Antiracism, Critical Philanthropy, Evaluation, Program Summary
In August of 2020, Hillombo, LLC was invited to lead a Utilization-Focused Evaluation (UFE) of a funding strategy for youth arts programs and to do so using the Equitable Evaluation Initiative Framework (EEIF) created by the Equitable Evaluation Initiative. At the provocation of a participant to use “revolutionary pedagogy” and because of the way it supported the two approaches named above, we integrated Paulo Freire’s Pedagogy of the Oppressed1 into our approach. To see the particular ways we blended these methods and the relationship between Freire, UFE, and the Equitable Evaluation Framework, please refer to the two tables at the end of this post. As the findings belong to the funder, this post will focus on how we integrated and applied these methodologies rather than findings and recommendations.
Using the model of UFE, the project began with conversations between foundation staff, program staff, and young people (ages ranged from 10 – 22 years old), and defined the following “uses” or goals for the project:
- Improve participating programs, including the funding strategy for youth arts programs grantmaking process
- Shift power to Black and Brown young people
- Attract money to both the grantees and the funding strategy for youth arts programs
- Examine the ways in which issues of structural racism and arts education intersect
- Make equitable the evaluation processes of the Foundation
In collaboration with each of the organizations, Hillombo designed a programming hypothesis of change2, a research purpose, and an interview guide used to steer data collection. Seeking to “shift power to Black and Brown young people”, the funder staff advocated to get the resources to ensure young people were paid for their time at a rate of $20/hr. Over the course of the project, participants received training on the basics of research and interviewing, aggregating their work across several programs, coding data, and conducted more than 80 online interviews with participants, parents, and organizational staff. Each Zoom-based interview generated a transcript, which we then edited to feature only the voices of the interviewee. These edited transcripts were called “vignettes” and were used to center the voices of young people in the data. From here, vignettes were then coded in alignment with the research purpose, the hypothesis of change, and the purpose of the funding strategy for youth arts programs. Once complete, these coded interviews were shared in a searchable format with each organization along with the corresponding codebooks so that organizations could expand the analysis. Access to this data also creates marketing and grant-writing material for programs to make their own meanings and use quotes for their own purposes in an effort to meet the evaluation’s fundraising uses.
We used a method of discourse analysis offered by Teun Dijk3 to assess the relationship between structural racism, youth arts programs and funder strategy, supported by the work of Theodore Allen4. The term “Reproduction of White Group Dominance Through Text and Talk” came from Dijk’s work and was used as a code to reflect on the relationship between the way the funder described the program and its rationale and what students, parents, and organizational staff did and did not say regarding race and racism.
At the conclusion of this project, our reflections on the extent to which the evaluation met its intended uses are the following: more than half the participants reported that the process yielded insights that would be used to improve their programming, the dollars and experience led to at least a temporary shift in the power Black and Brown young people held, and we made clear connections between structural racism and arts education programs. We were not able to determine the contributions the project made to attracting dollars to the programs or the equity of the funder’s evaluation approach, because these measures are more long-term in nature, and not enough data is available. Our attempt to meet all of the “uses” became quite time-consuming and pushed the project beyond our labor projections. Still, the experience with young people was quite rewarding and the lessons learned in the discourse analysis were significant.
1 For reference, we used the 30th anniversary edition of Pedagogy of the Oppressed
2 We use the phrase “hypothesis of change” in response to the criticism of a young person that “theory of change” made a program plan more grounded in agreed upon science than, in fact, it was.
3 Stanfield, John H., and Rutledge M. Dennis, Race and Ethnicity in Research Methods. Vol. 157, Sage, 1993.
4 Allen, Theodore, The Invention of the White Race Vol 1. Verso, 2012.