North Shore Roundtables: Outcomes and Evaluations

There is a great deal of discussion at the funder level - and the policy - about how programs should assess themselves and how the diverse field of Out-of-School Time (that many say spans all ages and types of programs from year around to summer and weekend) needs to be evaluated.

There are basically two modes of thought: One proposes that Out-of-School Time needs to be evaluated vis-a-vis the school day. That improving programs means aligning them with school methods, norms, and expectations. This is a de facto if not in sutu way of extending the school day - which is a fancy way of saying Out-of-School Time becomes like-school-time. The other mode of thought is that programs need to be assessed based on the specific and unique elements that define a Out-of-School Time program and that evaluation of programs need only be against other programs. If there is improvement in academics in school and that student is a child in an Out-of-School program that is good, but not a benchmark of program quality.

The group gathered on the North Shore were lively and there were many ways people took up the idea of assessment and evaluation. There was discussion that assessment and evaluation are often used interchangeably - or at times mistakenly - to mean a similar thing (just as many competencies or outcomes are not on students but part of elements of effective management) One program director remarked that there is a distinction and one way to avoid confusion is to consider any tool to be assessment as long as it measures specific aspects of the program (an arts project, family engagement, etc) and larger program-wide data to be part of evaluating the program or organization. An interesting discussion was also about what tools promoted an academic-focus or put links to the school day or subjects as major elements of quality - missing the focus on the social-emotional development of children or the long-term values added to the community. In all, most agreed that they were not professional evaluators and that sometimes they will just collect data and "let the funders make sense of this all."

Evaluation and assessment are part of our popular mind. They are enforced by a culture of customer service that has been so pervasive in the past thirty years many of us seem not to remember a time when our civil society gave us what was good for us rather than constantly asking us what we want. Whether this is a positive or negative development has yet to be seen. Years of viewing social services as customer driven and creating an attitude that the public does not have a commonwealth but is comprised of a diverse network of consumers has indeed changed our landscape and how we interact with one another and run our programs. The resources we have placed into assessment and evaluation may perhaps give us a better point of view - or at least numbers to toss around and fun data charts we can post on our walls. However, has this process of constant assessment and evaluation led to an increase of quality at our programs? This question may actually require further evaluation. Perhaps we need to ask only those questions we want to answer - we are consumers after all...