I had the opportunity to spend some additional time with some visitors from Germany who came to Cincinnati to take part in the recent StriveTogether Discovery Workshop. They have become very interested in the collective impact approach to a cradle-to-career initiative and wanted to learn more about what is taking place in Cincinnati. They had clearly been thinking about this work deeply and asked a number of great questions related to creating shared ownership, picking the right outcome indicators, incentivizing collaboration and choosing the right continuous improvement processes.
The question posed around how to choose the right outcome indicators was a really good one that I’ll explore a little here. And it’s one that comes up a lot. How do you know that you have the right indicators? What if there is another critical measure that you missed at sixth grade, for example? The reality is that there are a lot of important indicators that can be tracked and reported on. We started off with a list that was close to 100 potential indicators, but this is of course way too many. Not only is it too many to feasibly track on a regular basis, but the fewer indicators you have, the more focused you can become and the easier it will be to focus a wide collective impact effort to drive a core set of outcomes.
One of our mantras, espoused by the partners in creating the first StrivePartnership report card in 2008, was don’t let perfect be the enemy of good. This was a critical disposition, because for the outcome indicators that we settled on (we started with 10 in the first report), there were any number of reasons why a particular indicator didn’t really tell the whole story. As one example of many, the kindergarten readiness assessment data in Ohio only measures literacy. However, many will argue that we need to be measuring more than this to understand if children are prepared for school. So, we’ll work on advocating for a better measure (and the report card also becomes a good tool for this purpose), but we won’t let this stop us from publishing a good indicator that still gives a strong proxy to the ultimate goal that every child be prepared for school. We created a data committee to help develop the report and one of the very first things we did was to create a criteria for selecting indicators to help guide this process.
So how did we get from 100 indicators down to 10? We started with the Cradle to Career Roadmap to Success, which includes the benchmarks (the research-based competencies and experiences that act as predictors along the roadmap, indicating future success along the continuum). Then the key transition years are time periods along the continuum where students are often found to fall at risk if they are not developmentally on target. The combination of benchmarks and transition years led to the creation of the five big overarching goals (prepared, supported, succeeds, enrolls and graduates), and the indicators developed from these as measures of how well we, as a community, are achieving these goals.
Just because you don’t choose an indicator doesn’t mean that it’s not important, and it may be tracked in some of the strategic priority areas of the partnership. The outcome indicators that we are interested in are primarily the longer-term indicators that tell us if we’re getting to our goal. The short- and mid-term indicators are still important, but not the primary focus in choosing indicators for the report card.
And the development of the indicators was very much a collaborative process with all the partners. The data committee was comprised of the lead data people from each sector contributing data — early childhood, k-12, higher ed — as well as some key community partners, including United Way and Children’s Hospital. We had a series of many meetings in the first year in an iterative process to develop and refine our list of indicators. And we have refined (and hopefully improved) the indicators and reporting style in each subsequent year.
We have since reduced the 10 indicators down to eight. And the work of the entire StrivePartnership is focused on helping to move the dial on these core outcomes:
- Percent of children assessed as ready for school at kindergarten
- Fourth-grade reading achievement
- Eight-grade math achievement
- High school graduation
- ACT composite score
- Percent of students enrolling in college
- Postsecondary retention
- Postsecondary completion
Ultimately, choosing the right indicators is both a science and an art. A science in that you need to have a rigorous process and criteria for selecting indicators, and a good fundamental understanding of what data is available and what it can tell you. And an art in that it’s also about aligning this work under the shared vision and leadership of the overall effort, as well as building relationships with the data experts and owners in order to create the buy in and shared ownership to move the work forward together.
