Quality and Accountability - and then there is what you do each day for the kids

When the question was posed to a room of out-of-school program directors, site-based coordinators, researchers, graduate students, and direct-care workers as to what is the intended outcome of their program, the room fell silent.

There was that uncomfortable pause - the one that if cut short would produce cut-and-paste answers, but if held too long, would cause a meltdown in energy and perhaps get participants of the Boston Roundtable heading prematurely for the door.

Then one voice spoke up. "We want to hear kids laugh. We want them to have fun." Then the answers came more easily. "We want to hear that our kids feel self-esteem and make healthy choices outside our program." "We want kids to ask whether the program is running on days it isn't." "I don't want to write kids up [behavior reports]." The entrance to the conversation on evaluation and outcomes in Out-of-School Time was about what people saw at their programs and a sense of what they knew was good for children. This was both something that occurred at their point-of-service as well as beyond their direct control. It was, however, in contrast to the many benchmarks that are increasingly being set for programs linking them to various outcomes and creating an increasing amount of assessment burden on staff already handing the daily tasks of care and enrichment in the face of dwindling resources.

The presentation on tools covered only a certain amount of evaluations and assessments. Some of these "assessments" were actually evaluations (were monitors for program quality or accountability) while other "evaluations" were actually assessments (focused on child outcomes over program competencies). There was discussion about what tools were mandated by which funder or agency and whether certain tools were still required or were about to be phased out in light of new regulations from the Department of Early Education and Care or increasing interest in ELT by the Department of Elementary and Secondary Education (formerly the DOE). This confusion was indicative of the issues of quality ratings system development and the increasing confusion - even cynical burn-out - of many in the field who look for answers and find a multitude of shifting concerns. Many in the group asked who was going to advocate for them in the face of these changes in funding and the increase of required assessment and evaluation alongside a stagnation or decrease of funding.

There were no easy answers.

The presentation of the Boston Roundtable was intended to provide an approach to shopping for the right assessment tools as well as a critical discussion on how programs need to stop consuming every tool that comes their way and to start educating funders (one attendee said "argue with funders" and that may be the case, but we'll stake our middle ground with "educate funders") as to the realities on the ground around key issues of quality improvement. Many people wanted to know which tool was the preferred or met approval by either this organization or that trainer, however, the presentation was not a sales pitch for any one tool - but a call for people to go back to basics and ask, "what are we trying to do with all these things?" Are we researching, or managing our program? If we are using assessment and evaluation as a management tool, who do we really need to be talking to? Is there more value in self assessments? More questionnaires to children and families? Is there a balance between using face validity (going with our personal/professional observations and "gut") and formal processes?

Participants of the Boston Roundtable raised a number of issues that linked to or were directly related to the topic. These included:
The new regulations by the EEC for school-based centers, family child care, and Afterschool and Out-of-School programs do not have a clear enough plan for how to train staff (provide funding for extra time) to administer the child assessments
That arts programs are expected to deliver academics rather than focusing on their individual expertise (this may be extended to sports programs but none were represented at the meeting)
The connections to school may be valid for one kind of program but may not be the model for all programs given the diversity of program type and approach in out-of-school time (perhaps there needs to be a taxonomy of programs was not mentioned but perhaps should be discussed)
Some programs are asked to do one kind of assessment on year as a requirement and then that is dropped for another product
Program directors and on-site coordinators need to focus on staff development and that the most useful tools are those that assess staff competencies and help reduce employee turn-over - especially direct service
Programs need to stop following the system and must look to become more self-reliant economically so that they can meet the needs of the community they are in rather than attempt to align with a standard way of "doing things"

Again and again, there was a call to get organizations to take a stand on obtaining funding for the field to meet new of increased outcomes or adjust expectations to a reasonable level - focusing on what programs have historically done and can provide to our children and youth - supporting quality environments. There seems to be the subject of "quality and accountability" as it is seen through the lenses of several assessment and evaluation tools, and then what you do each day with the kids. It looks like there is a great deal more that has to be done to create tools that assist in managing a program, monitoring quality at point-of-service, and helping stakeholders have confidence in the quality and value of the program.

Perhaps we can all stand up together to bridge this gap between what and why we measure and the job we do every day working with children in our various settings.

YOU CAN FIND OUR QUALITY SELF ASSESSMENT TOOL HERE