North Shore Roundtables: Outcomes and Evaluations

There is a great deal of discussion at the funder level - and the policy - about how programs should assess themselves and how the diverse field of Out-of-School Time (that many say spans all ages and types of programs from year around to summer and weekend) needs to be evaluated.

There are basically two modes of thought: One proposes that Out-of-School Time needs to be evaluated vis-a-vis the school day. That improving programs means aligning them with school methods, norms, and expectations. This is a de facto if not in sutu way of extending the school day - which is a fancy way of saying Out-of-School Time becomes like-school-time. The other mode of thought is that programs need to be assessed based on the specific and unique elements that define a Out-of-School Time program and that evaluation of programs need only be against other programs. If there is improvement in academics in school and that student is a child in an Out-of-School program that is good, but not a benchmark of program quality.

The group gathered on the North Shore were lively and there were many ways people took up the idea of assessment and evaluation. There was discussion that assessment and evaluation are often used interchangeably - or at times mistakenly - to mean a similar thing (just as many competencies or outcomes are not on students but part of elements of effective management) One program director remarked that there is a distinction and one way to avoid confusion is to consider any tool to be assessment as long as it measures specific aspects of the program (an arts project, family engagement, etc) and larger program-wide data to be part of evaluating the program or organization. An interesting discussion was also about what tools promoted an academic-focus or put links to the school day or subjects as major elements of quality - missing the focus on the social-emotional development of children or the long-term values added to the community. In all, most agreed that they were not professional evaluators and that sometimes they will just collect data and "let the funders make sense of this all."

Evaluation and assessment are part of our popular mind. They are enforced by a culture of customer service that has been so pervasive in the past thirty years many of us seem not to remember a time when our civil society gave us what was good for us rather than constantly asking us what we want. Whether this is a positive or negative development has yet to be seen. Years of viewing social services as customer driven and creating an attitude that the public does not have a commonwealth but is comprised of a diverse network of consumers has indeed changed our landscape and how we interact with one another and run our programs. The resources we have placed into assessment and evaluation may perhaps give us a better point of view - or at least numbers to toss around and fun data charts we can post on our walls. However, has this process of constant assessment and evaluation led to an increase of quality at our programs? This question may actually require further evaluation. Perhaps we need to ask only those questions we want to answer - we are consumers after all...

STEM in Out of School

In a time of rather depressing news - take your pick as to what that may be for you personally or for your program or organization, there is some really interesting movement in getting Science, Technology, Engineering, and Math projects into Out-of-School Programs.

Currently the United States is not faring too wall in science education when compared to other countries. This may be due to how it is hard to measure one country against another. The difficulty to uncover standards or norms when there are so many variables and the fluid nature of knowledge when measured by strict instruments. Perhaps comparing Finland and the United States is like comparing informal learning environments to formal education. Or holding informal learning environments responsible for what occurs during the school day. Nevertheless, many researches think we can measure Finland against the United States and clean the data to give us a clear picture of the quality of our science education in this nation. If that is the case, as so many educational statisticians argue, we are not doing too well when compared against that and other nations. In fact, we rank #29th of 57 nations. Finland is #1.

Leaving the debate over evaluation and assessment of nations or learning, we can always use an extra hand in science education. Some say that our nation's financial health can only be saved by increased investment in science education.

This has led some to identify Out-of-School Time programs as the logical place for expansion of learning - and a natural place for learning that is by its nature out of the classroom and outside of the tests and answer sheets associated with current educational systems and academic-focused Out-of-School Time environments. Out-of-School Time programs have used projects for years to enrich their offerings and raise their quality.

Nevertheless, we will have to see whether STEM can work in the OST environment and the latest pilot to do so runs now until the end of summer. There are two groups in Massachusetts one in the west managed by WESTmost and the other in the east. The Eastern Cohort Group is run though a partnership of Build the Out of School Time Network, The Department of Early Education and Care, Department of Higher Education, Massachusetts Afterschool Partnership, The Museum of Science, and the Donahue Institute. The four sites and representatives of the six partners gathered today at City Year in Boston to learn about how to teach engineering using the Museum of Science's Engineering is Elementary curriculum. The training covered the fundamentals of engineering - have goals and ask questions until you solve these challenges with the materials at hand. The group was surprised at how simple materials were all that was needed. They also learned that through projects many children who may get factual answers wrong on a science actually can innovate in projects and become engaged in ways they may not through text books and discussions.

Over the next coming weeks, this group of STEM pilots will launch their projects and we will see what potential there is for this work and what challenges programs face in implementing these projects looking at what supports are needed, whether "out of the box" curriculum works, child engagement, and if these projects are fun for staff and children.

At a point in time when science, technology and engineering are opening all these incredible potentials in our world from micro-computers to green technologies, the United States is falling behind in the production of graduates in these fields. We need to do something towards growing the next generation of STEM workers. Already "by 2010, only four years from now, more than 90 percent of all scientists and engineers in the world will live in Asia." We have our work cut out for use to bring our rating up from #29th in the world science community and to make this country again the innovation powerhouse we are accustomed to.

Field Work: ASPIRE

As part of BOSTnet's on-going field work, we are profiling programs as well as collecting individual voices from the field from leaders in Out-of-School Time. This allows unfiltered voices to come through, allowing program staff to speak about their experiences and views as well as making this work more visible to a general audience.

Last week ASPIRE After School Program staff were interviewed about their program. They did not respond to any re-scripted questions but spoke about what they did at the program as well as share some of their passion for youth work. We hope to increase the number of profiles in the coming weeks and to present this work at our Quality Forum in June.

Quality In the Face of Challenges

Recently there has been a great deal of talk about the economy. It is on everyone's minds as daily we hear a constant drumbeat of bad news. More jobs lost, the "market" down, this or that company looking for a bailout.

The Out-of-School Time field has not been immune. Many programs are worried about summer enrollment, private and government dollars, and the use of a constant or diminished number of resources becoming ever more contested by programs and organizations - many of which are currently burning through 2009 reserves prior to the 2010 fiscal year meltdown. This is a time when the cohesion of the field is being challenged as many program managers and directors are in open competition with their peers. Applications to every source of funding, no matter how small, are up. While this may lead to many fractures - often between people who have known one another or worked together for years - these economic conditions make it more important for the Out-of-School Time field to come together and settle on an identity and key areas of quality. Managers and organization leaders must also be frank about financing, funding, and how supporting each other can help sustain yourself.

Over the past several years there have been various large system-building or capacity-building initiatives. These initiatives have taken much energy and often funding away from the work at the ground. Considering where the economy is, large-scale system-building initiatives may have to be reviewed for what is useful - and what of these endeavors are boondoggles that can no longer be supported with private or public funds. Leaders in the field may have to reflect on what their practices and expectations have been and use this period to bring these in to line with our current realities. What are we asking of the direct service staff? What training are we providing them? What is the actual scope of the work and can we continue to measure Out-of-School program quality against individual child in-school performance?

Programs may need to openly discuss what it means to share resources. We hear a great deal about "share resources" but what does that mean?
Share staff (some programs have been doing this)?
Share costs for materials?
Share space (many programs have be doing this)?
Share program participants?
Write grants together?
Merge management and close some organizations?

The last item is perhaps the most sticky of the points. Sharing of craft sticks or direct-service staff always is more popular than closing down upper management and even consolidating the infrastructure of organizations that provide similar services. It begs the sticky question of who is taking over whom?

The market conditions have not been favorable even in times of plenty (or at least the past eight years by comparison to today) to smaller programs or individual actions by managers to adapt their program in the face of NCLB-style initiatives and educational and enrichment misadventures. Certainly there have been many more barriers to entry placed in the way by creeping and pervasive "professionalization" that has provided no back end infrastructure of consistent funding, agreed upon standards of quality, or a sold commitment to either placing Out-of-School Time within the area of Youth Development or adjunct to academic remediation. Gone are the days when an interested community member could launch a needed program for idle children around her. With start-up times, organizational bureaucracy and funding restrictions such as restricted or project-driven funds, many of these elementary-age children may be adults before the program gets off the ground. Can a field that has slowed down in innovation respond to economic challenges in a timely manner?

Much of our current situation did not arise all at once but came from years of unsuitable development and unsustainable investments. Despite the many conferences and committees, the field appears at a social level (colleague to colleague) to be more fractured than united. This is indeed a larger symptom of the problems in how funding was deployed. The amount of unrestricted funding has diminished or vanished, inspiring many programs to higher levels of activities often outside of their traditional keen. The inconsistent "investment" by private and public entities and individuals that created programs at whim and abandoned them inexplicably.

What may be needed now is for more members of the field to come together and a renewal in that spirit that launched the Out-of-School Time field in the first place. Individuals inspired by the idea that children and youth can benefit from an expanded number of developmental settings and that those settings are intentional and provide the context for the learning that occurs in formal settings and does not extend nor replace that formal system. Out-of-School Time is more than challenged by current market forces and that once over, business can resume as usual.

The field of Out-of-School Time faces a critical juncture. We must look at the old ways of thinking and shed them. We must brush aside our differences in philosophy and approach and look to the fundamentals of the work and then advocate methodically for consistent funding for the kind of work this field was meant for, not the sort of work we are often given or demanded of us. Increasingly on the ground there is a frustration in the way things are done. Change happens only when we are the first ones to make it in ourselves. Time will tell what direction each one of us chooses.

LOYD Guest Blogger: Donna Folan

As part of our ongoing Lead to Opportunities for Youth with Disabilities (LOYD) and quality discussion, we are posting guest authors who are contributing to the Out-of-School Time Field. As an artist, director, and teacher, Donna Folan, Director of Until Tomorrow Productions, embraces both the nontraditional and unconventional. She believes that the outcome of creative work should never be known in advance, so something new is always shared and learned. The process of exploration in all activities is always changing, engaging all the participants in new and unexpected ways.

There is not one central place that provides support for OST participants with disabilities. While economic times are difficult, not all solutions are financial.

Some important considerations are:
o What is your current organizations level of physical and programmatic access?
o What are the needs of all your participants.
What supports or accommodations needed for participants with disabilities?
o Does your organization have a line item in the budget dedicated to accommodations in grants? This is a clear indication of commitment to inclusion to funders and may provide a way to pay for some needed accommodations.

Many people can have the same disability that does not mean they all have the same assistance. Disability is not one size fits all. Accommodations need to be individually designed.

All people connect to learning in a variety of ways, when designing activities have as many ways "in" as possible for all participants. Think about a flexible model that grows up around your groups. Provide information in a variety of formats and Multi-Modal Approaches gives much support. Everyone can and will achieve goals and will do so in a different way. A lot of creativity and coloring outside the lines.


Donna has extensive experience in acting, directing, and knowledge of most other areas of performing arts. She utilizes this experience to assist group members to identify their areas of interest and passion. Donna has a personal understanding of how creativity and theater enable individuals to decide how large or small a part they want their disability to play in their artistic work. Donna is able to help youth with disabilities and those without gain a realization that regardless of the challenges, or obstacles they face, no one factor can solely decide who they are. Participating in creating theater teaches this valuable lesson. It is imperative to provide fully inclusive and accessible settings where all people can learn from each other in a positive and equal environment.

Point of Service Quality

There is a great deal of discussion about the impact and quality of out-of-school programs - about as much as there is continued discussion on the definition of what out-of-school programs are - Early care? Elementary school? Teenager programs? Weekends? Summer programs? Are we stuck in a world of dead ideas and ways of seeing things that are out modded as Fortune Magazine's Matt Miller claims our nation as a whole is? Can we still provide quality service and keep the process of delivering that simple and effective?

Many providers in this year's BOSTnet Cohort of programs have discussed their frustration as to the definition of quality and the many and often competing tools and opinions out there vying for recognition. "I don't find much use in [an outside quality assessment tool]" one program leader claimed. "I do those [outside quality assessment tools] because I have to. Then, I create my own to learn what I need to know to improve my program." Are these tools focused on measuring dead ideas?

Many programs are asked to be held accountable to improving school-day performances either surrendering their unique offerings and substituting increased formal academics ("classroom" style learning) and linking their outcomes not to those that a unique developmental setting can provide but those improvements that may be very much out of their control - those performances of the classroom of such-and-such public school or schools depending on the composition of the program.

"We need to focus on what we do, and what we do well" seems to be the opinion of many leaders within the cohort yet upper level management often stifles innovation by holding on to older ideas of management and definitions of the field. Many emerging leaders as well as veterans see themselves needing high-level supports yet do not feel this clearly expressed need is being met. Again, older ideas of training and professional development continue to send experienced people back to the same old trainings they had the year before. How many "refreshers" are needed? "I am not sure why I am expected to attend a [professional development training] and expect to cover the same material that entry level staff are" one long-time program manager said. It is strange that in this field so many of the trainings are expected to meet the needs of all staff from the upper levels to the line-staff - not to mention the unrealistic expectations that a single trainer can "adapt" the materials to meet all these different needs in a single contact period.

Focusing on the needs of each level of staff and increasing the number of upper level opportunities may be one way that more experienced leadership can continue to grow and to shape their programs. This experienced leadership seems ready now, perhaps after years of developing the field and watching the out-of-school time community grow, to focus on what they do well, and reach to measure the sort of Point-of-Service results that are representative of their true contribution to healthy and resilient youth.

A viral video from NASA may shed some light on this situation since many issues of innovation and management cross boundaries and perhaps what is good for healthy science is good for healthy out-of-school time. Perhaps we need to throw out dead ideas and ways of framing what outcomes we are measuring and replace that with a focus on measuring the quality at the point of service and only those things we have direct control over in the multi-layered and ever complex lives of the children and youth we serve.

Out-of-School Goes to the State House

Today was Advocacy Day for Out-of-School time programs up at the State House on Beacon Hill. This event brought together a diverse audience of many different programs and workers of many different ages and walks of life. The event was organized by Massachusetts Afterschool Partnership had speakers from government, programs, and youth voices.

The room was full of many professionals in the field of Out-of-School Time and one thing was certain from several of the conversations in the room - advocacy was not a single day event this year but will involve hard work and determination over the next few months to make sure that funding is either preserved or that lawmakers and the public see that often social programs need more government funding when times are tough not less.

Program coordinators are reporting an increase in the need for assistance to families who cannot pay fees they were used to paying just a year ago. Summer attendance of fee-for-service programs have also dropped last year and there seems to be worry that this year will see a substantial drop off in enrollment. One coordinator said that she posted a part time position at her program on craigslist and rather than a one-day bump in resumes, she has received about 25 per day for the past week and a half - many from people who claim they are looking for work because their program was closed or position eliminated. What was more shocking was that many of these resumes were coming from other locations in the state, and region.

These are hard times indeed, if these reports reflect the reality out there. Representative St. Fleur when she spoke said that she sees our glass as half-full. She said that attorneys are only as good as the last case they've won and that each year the field of Out-of-School Time must again "win" funding. This may indeed be the case. One person in the OST field joked that anyone who claims they know what to do in these financial times is either fooling themselves or others or both.

One good thing about Out-of-School programs is that they have already learned to operate with little funding and do extraordinary things using few resources. Staff are usually part time. Technology is kept to a minimum. It may be that the funding needed to continue these programs is actually small amount considering the overall state budget or the costs of operating other educational services, such as the Boston Public Schools or flat funding certain line items over others. Glue, craft sticks, some creative energy from staff have already provide numerous youth development opportunities. It may be that our ask is not that much. After all, give even 1/100th or 1/1000th of 900 billion dollars (the current number for the stimulus plan) and our field can change the lives of young people and give children places to play and learn.

BOSTnet on Television

Suffolk County Criminal Justice Connection hosted by Maura Hennigan interviewed Executive Director Maryellen Coffey and Project Director Peter C. Griffin on the current state of out-of-school time and the future of the field.

Maura Hennigan is Suffolk County Clerk of Court, Criminal Division and host of the weekly program where she interviews community and political leaders from in and around Boston. The program airs on Boston Neighborhood Network weekly.

The BOSTnet appearance will premiere Tuesday night at 10PM on Live Channel 9 and then rebroadcast according to their schedule. Learn more about BNN Here.