Guest post by Justin Piff, a senior director at Equal Measure in Philadelphia, Penn.
Despite national and international attention on collective impact since 2011, few individuals and institutions have articulated what it looks like or ought to look like in practice. In 2015, Equal Measure and StriveTogether set out to understand how civic infrastructure — as outlined through StriveTogether’s Theory of Action — develops within and across communities using collective impact to help children and youth. StriveTogether’s Theory of Action lays out a set of complex community processes that, working together, build the civic infrastructure necessary to change systems and improve outcomes in education at scale.
Evaluators like a challenge. So, when StriveTogether first contacted us about conducting an evaluation of its national initiative, we were eager to get started. Fast forward to 2017, and for the past three years, Equal Measure has worked with StriveTogether to better understand how civic infrastructure develops within and across communities in the Cradle to Career Network. To do so, we created a survey that captures how communities work toward a shared vision and measured progress over time. We asked partners from these communities to rate the extent to which a set of 83 indicators of strong civic infrastructure — like whether data are driving decisions and partnerships embrace diverse perspectives — were present in their communities.
We then calculated ratings based on the percentage of who agreed these things were happening in their communities. Although 20 communities participated, 14 of them did so each of the past three years, allowing us to track progress through more than 3,000 surveys. This has been an exciting process, and the results are fascinating. While we’ve learned a lot through this work, three findings highlight some of our greatest insights.
Civic infrastructure develops according to StriveTogether’s Theory of Action “gateways.” StriveTogether’s framework assumes that civic infrastructure development follows four sequential stages, called “gateways.” When we first met with StriveTogether to plan for the evaluation, we pointed out that, if true, “ratings” of civic infrastructure would be higher for earlier stage gateways, then descend from there. This is exactly what we found, three years in a row. These findings tell us that, as a group, communities’ processes are consistent with the Theory of Action, showing the greatest progress in gateways associated with what StriveTogether describes as early-stage work, and the least progress toward “Systems Change.”
Partnerships take a similar approach to addressing the four “pillars” of work within each gateway. While the Theory of Action outlines a sequential process for developing civic infrastructure across four gateways, it is not prescriptive about the process for tackling four threads of work — referred to as “pillars” by StriveTogether — within each gateway. The chart below, affectionately referred to in our office as our “favorite chart,” expands on the data above by illustrating ratings for the four pillars within each gateway over the past three years.
When we saw the first year’s data, we were intrigued. When year two data rolled in, we were encouraged. We held our breaths this year, but saw the trend continue. What we saw were remarkably similar patterns that reflect a unique approach to civic infrastructure development within each gateway. These data reveal that communities exhibit a clear and consistent sequence of pillar development unique to each gateway, and provide insight into the nuance of civic infrastructure development, including where communities most commonly succeed or struggle within each stage of collective impact.
The presence of essential principles follows a consistent pattern across communities that remains constant over time. In addition to tracking civic infrastructure development, we captured incorporation of the four Theory of Action principles, along with equity, in communities’ work. According to the Theory of Action, these principles should be present throughout a community’s efforts. The chart below demonstrates the extent to which partners reported evidence of these principles in their communities. Similar to the trends among gateways and pillars, the principles follow a clear pattern of development; each year, eliminating locally defined disparities and developing a culture of continuous improvement emerged as strengths across communities, while communities reported the least evidence — despite consistent growth — of engaging the community. These findings point to where communities can build on strengths, and where they may need support.
In addition to understanding how collective impact unfolds, we are examining outcomes among a subset of communities, and going deeper through interviews. These portions of the evaluation are still under way. To date, findings from our survey have helped validate the StriveTogether Theory of Action while providing a more nuanced understanding of how communities come together around a shared vision. Importantly, these findings add rigor to our understanding of collective impact and provide a clearer, data-based picture of how collective impact actually happens on the ground. Ultimately, we hope this evaluation will help StriveTogether and its network of 70 communities target resources and energy to accelerate efforts on behalf of children and youth.
Justin Piff is a senior director at Equal Measure, a Philadelphia-based nonprofit firm that provides evaluation and strategy services to social sector organizations. At Equal Measure, Justin directs evaluation projects of systems change initiatives focused on improving outcomes across the cradle-to-career continuum.
