Sound Partners Technical Assistance Conference Call

PRESENTATION
The call opened with a brief review by the facilitator of a Performance Measurement Model, selected Measurement Vocabulary, and an Evaluation Plan template. Participants were referred to page 2 of handout materials (distributed in advance of the call). It was explained that the purpose of the chart is to show one example of a context for doing evaluation work; the evaluations of Sound Partners grantees fit into the performance measures outlined on the left-side set of the chart. Participants were also referred to page 5 in the handout materials which provides definitions for the evaluation terms effort, quality, outcome and impact. It was suggested that participants may be familiar with other evaluation terms, but these seem to be the more common.
Participants were then referred to page 3 of the handout materials. This chart is an initial effort to apply the model outlined on page 2 specifically to the RWJ/Benton Foundations and Sound Partners work. In the performance measures box at the bottom of the chart we began to apply the language of Sound Partners' projects to define effort, quality, outcome and impact.
Participants were referred to page 4 of the handout material in the final part of the formal presentation. This chart illustrates the 4 basic components of the evaluation process–design, data gathering, data analysis, and report and presentation. The data gathering and data analysis tasks will be unique for each evaluation.
The facilitator concluded the presentation with the following suggestions for making an evaluation process effective:
· Spend time in the design phase thinking through the entire plan through the report–this will help insure all parts of the process are connected and you reduce the potential for unexpected problems that could derail the work (i.e. the data answers the questions that you want to answer, and you know who in advance will do the analysis, etc.)
· Before defining the evaluation questions, be clear about what it is you are evaluating–what is the project trying to accomplish (i.e. Smile Healthy is considering whether it wants to improve access to dental care or increase involvement in its community organizing effort–access and involvement are different things and would measured in different ways)
· Spend time in the design phase clarifying each question you want to answer and what data you need. If you can't get the data, consider a different question.
· Measuring program performance when you don't have direct access to participants can be very difficult–all of you will have to be pretty creative.

ANSWERS TO QUESTIONS SUBMITTED IN ADVANCE

1. How do you measure success? Success can be qualitative or quantitative–it depends how you define it. If success is defined as the number of people served–it is quantitative; it is the level of satisfaction with the program–the percent of people satisfied is quantitative–what they were satisfied with may be described in qualitative terms; if success is what the program accomplished–increased awareness about a medical issue; provided access to more people–if you know the percentage increase in awareness or how many more people got access it is quantitative; if you created a new collaboration of service providers it could be described with qualitatively; if success is what happened to the people who got more access you might either count or describe this depending the type of data you can get.

2. How can you quantify an increase in listeners? You would need a way to measure the number of listeners before and after the intervention. Without out this need to look at a different measure for effort such as the number of call as a result of a show.

3. How can survey research be more scientific, not just responses that are opinions? One way you make survey research more rigorous is to design questions that “test” whether people learned certain types of information from the programs and not just what they think the got out of it. A starting point is to identify what you hope people will get out of the programs, then design questions to determine the extent to which the got it.

4. How can we measure impact without losing our focus on the service? Measuring impact means you want to understand what difference the programs made for the listener. By understanding more about this, you begin to get clearer about how well you are serving them. Measuring impact is one of the hardest things to do. A starting point is to identify the difference you hope the program to make than consider whether you have a way to get the data to measure it.

5. How do we measure the effectiveness of our program? To measure effectiveness, you have to define what it means–just like the previous example about success. By effectiveness do you mean that more Spanish speaking people are aware of senior services–any services, or the particular set of services featured in your programming? One way you might consider measuring this is to engage the assistance of the service you feature in the programs and ask them for a period of 3, 6, 9 months–depending on how much time you have–to ask all first time callers if they heard about the service through your programs and document their response.

In this part of the call we explored the evaluation logic–related to effort, quality, outcome and impact–for a couple of grants (based on the abstract from the Sound Partners website) and offered a few recommendations for each one. This discussion is related to the design phase of the evaluation plan in particular.

Strengthening Community in the Four Corners Region – Ignacio, CO
The abstract suggests that this program will increase the capacity of the community to strengthen itself through a cross-cultural talking circle…that both project partners seek increased connectivity among health care providers, community leaders and residents in need, and that the partners hope the project empowers locals to make informed decisions related to area's economic emergency…
Issues for consideration:
· What is it that will happen in the talking circle that will increase the capacity of the community to strengthen itself? Have to be able to define this in order to measure it.
· Have to define connectivity more clearly in order to measure it.
· How does the idea of connectivity lead to the impact of empowering locals to make informed decisions? The connection between the 2 ideas isn't clear enough to evaluate.
· How could you know if people are making an informed decision–what is the definition of informed and how would you collect data that would be an indicator of this definition.

Caring for the Chronically Ill – Richmond, KY
The abstract suggests the program aims to improve the quality of care for the chronically ill, get civic and government leaders to take a more informed and positive approach to dealing with medical issues affecting the area, and motivate patients and their families to take a more active role in maintaining and restoring their own health.
Issues for consideration:
· Have to define what improved quality of care means in order to know how to assess the degree of improvement.
· Have to define what it is you expect leaders to do differently as a result of the information you share.
· Motivating people to be more active in maintaining or restoring their own health isn't about quality of care–it is about health care behavior or practices. Which one is the program interested in? Have to know this in order to know what to evaluate.