To make the work we do in the field more accessible and raise the bar for dialogue and freedom of ideas in the Out-of-School Field, we are posting many notes we use to compile our project reports on this space. We hope that these full reports will bring much needed transparency to not just the product, but the process of a project. Additionally, in this format, we welcome comments (anonymous if need be to protect yourself from your organization's politics and fear of funder opinion) positive and negative, hoping that some comments are critical analysis we can use to further our thinking and do so in a forum where we "think aloud" and "puncture the corporate veil" that blankets many non-for-profit organizations and especially those who work in the high stakes of youth development, academics, and the lightning rod of "education."
Promoting Positive Behavior is a project that is in its second year, having been funded by the
Department of Education and Early Childcare the previous pilot year.
The previous year's project was to develop research-based supports and tools specific to the Out-of-School Environment. This was achieved by working with
Program in Education, Afterschool & Resiliency and contracting several behavioral specialist organizations in the geographic regions in and around Boston. This past year the project was funded by the
Department of Education (now Department of Elementary and Secondary Education)to build on the investment of last year and create and refine supports around the tools and take the research and fold it into practice. The goals were to improve services to children with special needs (emotional and physical) and provide staff development. This work was funded in part by the
ASOST grants.
In the coming weeks this document may be edited to include more information or fold in other views or to refine the language to better describe the project. Names and identifying remarks have been removed to protect the privacy of the participants and facilitate intellectual freedom. All
edits in CAPS are there to cover identifying details the remainder of the report is based on what was submitted to whatever funder/s required it.
Project Overview:
Promoting Positive Behavior continued the work of a previous year of development of tools and supports designed to improve program ability to serve special needs populations and develop highly qualified staff. These supports are intended to get program staff to focus on problem behavior as part of overall management of their groups and an extension of a common programmatic language around how behavior is addressed and a culture of prevention rather than a punitive approach. BOSTnet selected seven programs in and around Boston representing a diversity of programmatic type, neighborhood, and a span of ages from elementary to middle school in which to deploy the tools and develop supports around Promoting Positive Behavior (PPB) as an approach specifically designed for the Out-of-School field. The primary tool developed and used for the 2007-2008 project cycle was Behavior and Emotional Support Assessment Tool. This tool is divided into four areas: Assets, Behavior Support, and Resiliency, Children’s Socio-Emotional, Behavioral & Mental Health, Mental Health Services, and Staff Self Care. Each section gets staff to rate current practices, prevalence of behavioral challenges, and identify common procedures and resources. Staff are then to create a priority list (either individually or as a group). These priorities could serve as the basis of an action plan that is developed as a group, however, as the project progressed, there seemed to be increased value in collecting assessments and identifying and reflecting back trends to staff and site coordinators. Five programs received services as part of the project. None of the programs had received the completed tool before while three of the programs had been part of the original project group (2006-2007). These programs were located in AND AROUND BOSTON. Collaborating organizations were TWO YOUTH SERVICES ORGANIZATIONS. AN ADDITIONAL YOUTH SERVICE ORGANIZATION had been on the original application but were not selected for the project to accommodate the budget changes.
After an initial contact, it was determined that the use of consultants would be excluded from the current project. The previous year of development necessitated the contracting of certain behavioral specialists to work in developing the Behavioral and Emotional Assessment Tool by collecting anecdotal evidence, expulsion data, and interviewing staff and children. Behavioral specialists also collected strategies from programs that contributed to the publication of Promising Practices, a supporting document in the PPB approach. In the process of developing the tool the previous year, several programs in that cohort had developed a reliance on the specialist rather then developing in-house competencies and improving the professional development of their staff. This model would be unsustainable and the Behavior and Emotional Support Assessment Tool was designed to allow site coordinators minimum support to achieve maximum outcomes in staff development and reaching special needs populations. In addition to the monitoring expulsion data, this current year (2007-2008) would use delivery as a way to develop supports that would assist program independence from expensive supports. BOSTnet adjusted the proposal to fit a truncated time line (funding and the hiring of qualified staff) to reach out to sites to make initial observations to serve as a baseline, train the site coordinators and convene them as a cohort, deploy the tool at respective site staffs and collect completed assessments for use as data, design and test supports (formal trainings, coaching, etc), and make adaptations to the tools and supports to reach a larger audience and create a model that could be released at the state or national level.
Initial Observations
BOSTnet staff visited sites individually to meet staff and make observations of the program in action prior to administration and assess the current supports in place at programs. After these initial observations it was determined that one site originally listed on the application should be substituted for another site within the same organization due to a higher reported number of behavioral issues at this alternate site. ONE site was housed in a school that has recently switched to an Extended Day Schedule. The site had several programs working in the school day but it was determined that only one program (a YOUTH SERVING ORGANIZATION) was appropriate for the work since it occurred after the extended day. The YOUTH SERVING ORGANIZATION ran the other LOCATION program although it operated in a mid-sized elementary school and had more space within the school than the LOCAION site. There, the majority of children had mobility issues or other physical impairments while the staff were majority young college students or older high school students. The site coordinator for that site was concerned with the training level of his young staff and their ability to not just care for disabled children but to “involve them and really engage them every day.” Several children with special needs were observed at other programs. The SCHOOL-BASED program had two special needs children. During the first observation these children were disengaged from the activities and it appeared other students too. The site director reported that these children had special services during the day but that they didn’t have access to these services. One child was suspected of having an attention deficit because he was active and often aggressive to other children. He was reported on the verge of expulsion because of behavioral problems and was at the time of the first observation on the verge of a “third strike” that would remove him from the program. Another special needs student at the SCHOOL-BASED program was suspected of being on the autism spectrum because he received supports during the day in an intensive classroom. The Child was a common fixture at the program but did not participate in all the activities with the others in his group and appeared marginalized – standing at the edge of large group activities, not being involved with peers. The SCHOOL-BASED programs were very different from each other even through they were operated by the same organization. All sites visited reported having mild behavioral issues with some severe issues on occasion. All sites reported that there had been no expulsion or suspension due to behavior the previous year although some site directors did mention that many “problem children” had existed. Many of these children left the program due to being removed by parents, moving neighborhoods, or non-attendance.
Foundational Training
The participating site directors were gathered together to meet as a cohort and discuss their expectations for the project as well as issues common to the programs. BOSTnet staff reviewed the past year’s PPB project and developed preliminary expectations and project activities. Some of the challenges of the project were identified such as the truncated time frame as well as getting sites to recognize the importance of building staff competencies over receiving direct assistance from outside Behavioral Specialists. BOSTnet staff convened site directors and discussed the goals of the program as well as the challenges. The site coordinators used the assessments and results were discussed. From these first assessments it seemed that there were no severe mental health issues (there was a discussion as to what information programs were given and what had to be constructed based on parent communication or connections to day school administrators and teachers) even though the directors thought it a pervasive issue in the school age population. One site director thought that there was an over-medicalization of the OST field and related a personal experience with a special needs child and his resistance of medication. Site coordinators tended to discuss specific children rather than identifying programmatic changes or training needs as a whole. Site coordinators believed that they already had behavioral plans and needed instead a few strategies for a very small minority of children.
Building the Action Plan
Using the assessments, observations from initial site visits and interviews with site coordinators and line staff, BOSTnet staff developed an action plan for each site. These action plans synthesized the self-reported goals along with observations and reflected back to the site coordinator gaps in service or areas that needed improvement. It was common that sites needed staff to receive more training on child and youth development as well as group management. Based on the assessments, staff across programs needed opportunities to develop self-awareness and strategies to deal with problem. Site coordinators were presented with action plans for discussion and goal setting. Assessments were gathered from line-staff as part of meetings in order to present additional perspectives for the action plans. Many gaps appeared between understandings of site coordinators and that of the staff (i.e. site coordinators felt that procedures were in place to work with community agencies while line staff thought this was non existent or needed improvement). The assessments provided a pattern on how the program staff viewed the work as well as identifying areas where the staff most or least agreed on challenges, strategies, and the need for mental health services. Most action plans created in the beginning of the project followed a format that was alter refined so as to better step out activities and to identify the responsibilities and expectations of all interested parties.
Refining of Training and Supports
BOSTnet staff reviewed the previous years training with an eye to making the work reflect more on the Out-of-School environment as well as underscoring how PPB fits into an overall strategy for creating a quality environment for youth development. The original training was considered somewhat technical and focused on a few strategies that did not address the key issues around the types of schedules, activities, and group compositions that are widespread in the Out-of-School field. The revised trainings were presented at a Boston Leadership Roundtable and a BEST training where evaluations were collected as well as anecdotes from participants. These trainings were revised and included as part of the sequence of PPB supports.
Trainings and Supports
Each site has a specific action plan as well as sequence of trainings and supports. Many of the sites found it difficult to meet with BOSTnet staff or commit too many staff to the trainings. BOSTnet staff conducted on-site support to direct-service staff, getting them to use the Self Assessment as the lead activity. These assessments were tabulated and their results were shared with the site coordinators. The majority of staff felt that they let their children know they are doing a good job (Q#1) but that other strategies were rated all across the scale from “not at all” to Always.” These first applications of the assessment always revealed a span of opinions. On the second section of “challenges” many of the staff felt that there was a “mild/moderate” amount of social or interpersonal/ family problems (Q#17) while there was “insignificant” issues around gang-related problems (Q#21), suicidal thoughts or behavior (Q#22), alcohol/drug problems (Q#23), and eating problems (Q#24) This may be because elementary school age students were over-represented as well as staff awareness to identify these issues in their populations. When it came to mental health services (section 3), the questions moved staff to think about the procedures of their program as well as get staff to reveal their familiarity with how their program deals with issues. Most program staff reported that there was no outside consultant and often wasn’t clarity on what procedures were in plce to get information (Q#38) or whether there was a referral system (Q#31). The majority of programs did not have consistent resources share among staff (Q27) or have funding for outside consultants (Q#29). There was often a disagreement with procedural questions as to whether the program had these procedures (Q#37, #40). Staff generally reported less frequency to practices of self care with many program staff reporting no or infrequent access to staff building time (Q #41 - #45 see attached). The Action plans looked to training as the primary formal exposure to developing staff competencies with follow-up visits conducted. SCHOOL BASED siteS coordinators combined their staff trainings. This was least effective as the issues at the sites were very different at each site even though the population of staff was similar (teen and college-age). The follow-up visits were also decided to be combined as during certain times both staff were serving their program populations at a single location (SCHOOL-BASED). This presented challenges since there were multiple sites in addition to the aforementioned. In addition, the children were out of their routine and environment making it more difficult to determine what behavioral issues came up during “typical schedules.” The SCHOOL-BASED ELEMENTARY received supports on a regular basis, with the most progress as children who were disengaged were folded into the program yet there was a schedule disruption as the site took on creating “zones” in accordance with common 21st Century Learning Center practices. The SCHOOL-BASED SITE was also going through changes in expanding to fill rooms that were given for use by the school so the program would not have to use as much shared space. The rooms were being designed including the participant’s voices and this program received supports around how to structure use of time as well as creating space where children had common rituals to limit issues with transitions. The STAND ALONE FACILITY site with its own facility had issues in that the local school was in the process of switching to an Extended Learning Time schedule and the program was threatened by being replaced with other vendors, other models within the school itself, and arguments over future access to children in the community. This impacted the operations of the program and many staff left the site. At the site was a student who showed behavior that was difficult for the staff to manage. This child was thought to exhibit behaviors of ADD or ADHD. The site staff were convened to discuss ways of making the entire program supportive in meeting the needs of all students, as well as creating a behavior chart to share with the student and his family. This chart was framed in a positive way and worked with the child to become more self-reflective on their mood. Staff also were to reinforce the concept of getting the child to give himself time outs or let staff know when he felt overwhelmed. All YOUTH SERVING ORGANIZATION "A" sites received a group cross site training and like the YOUTH SERVING ORGANIZATION "B" multi-site training, it is difficult to assess whether it was as effective as specific and customized work.
Post Observations
Site Coordinators were met with and the action plans discussed. Many changes to the format were suggested to make the action plans more “readable” by coordinators. As the initiative was coming to a culmination during the end of the program year, it was difficult to interview staff about their use of the tool but some reflections were gathered. Many staff reported positive experiences, however, it seemed that many staff still wanted more support and had additional questions about specific children. Data was collected on expulsions. Across the programs, there was no expulsion reported for the project year.
Evaluations of the training showed that it was well received, that the child development section was helpful, and that looking at specific behaviors against the Out-of-School environment made it helpful to ground staff in making judgments about what was acceptable for children to engage in within the context of youth development.
Next Steps
PPB Assessment tools were collected from the programs as well as a control group of outside programs and forwarded to Dr. Noam. Pending funding, these will be reviewed and modifications will be made to the tool incorporating insights from this program year. The trainings and supports developed can now be used with greater accuracy to assist other Out-of-School programs in creating a positive approach to behavior. The use of expulsion as a measure of the project, however, may have to be revisited and replaced with a more specific measure of project effectiveness and outcomes.
There were three measures to evaluate the work and monitor outcomes: collection and interpretation of Behavior and Emotional Support Assessment Tool as a pre and post test, collection of expulsion/ withdrawal data for 2006-2007 and 2007 – 2008, and collected observations and anecdotes.
Assessment Tool Measurements and outcomes:
Each program used the tool in two different ways. The first was a cohort of coordinators who used the tool after an initial contact with BOSTnet staff and then discussed what priorities they were going to set for the year. Site coordinators who used the tool as individual managers were more likely to say they had some individual children who cause issues at their program but felt secure that their programs had the capacity to meet children with special needs (defined as cognitive, emotional, behavioral, and mobility). This same tool was administered to staff at each site and a very different picture often came about. Staff were often unsure as to the issues or how to deal with them. Staffs seemed to think their site had “insignificant” challenges around most categories (Q#16 – Q#25) with most “moderate” or “severe” ratings given to “Social, interpersonal or family problems” (Q#17) and “Aggressive/disruptive behavior, bullying” (Q#20). However, staff were usually hesitant to place a priority as far as programmatic goals and when a site staff did set priorities, they seldom matched those of other workers at the same site. This disagreement on challenges, strategies, and identification of the divergent thinking on key issues was identified by the tool and was used for discussion among staff as well as reporting to the site coordinator and developing the action plan. Here, the tool allowed site coordinators to look at their own reactions of the tool, their ability as managers to understand the needs and competencies of their staff, and for the staff to discuss with one another in a non confrontational manner the skills or knowledge they lacked and how to better work as a team. Having identified gaps in understanding of children with special needs, BOSTnet staff was able to refine training in child development as well as offer more specific group management skills.
Expulsion/ withdrawal data for 2006-2007 and 2007 – 2008
In the collection of expulsion/ withdrawal data, BOSTnet staff collected information from 2006-2007 and then at the conclusion of the program for 2007-2008. Form this information it was learned the number of children in the project, demographic information, and the level of expulsion/ withdrawal for each site as well as aggregating the data to get a clearer understanding of the field. Programs in the initiative served 301 children. Of these, 163 received financial assistance form the Department of Early Education and Child Care (EEC). There were 164 males and 137 females. Programs reported between 72% - 100% low income. Within the programs as a whole 45% were Latino/a, 8% African American, 1% Asian, and the remainder multi-racial or European with no reported Native Americans, Cape Verdeans, Africans and 6% as “other.” The leading reported reason of children leaving the program in 2006 – 2007 was moving neighborhoods. This data remained the same for 2007-2008. The second reported reason was “affordability” or “lost voucher” (aged out of program). This remained the second place reported reason for 2007-2008 as well. Of the children who left the program voluntarily, 5 were reported in 2006 –2007 numbers as having IEPs and 3 in 2007 – 2008. There was 1 expulsion in 2006 – 2007 of a student who was reported “caught breaking into school property.” There were no reported expulsions in 2007 – 2008.
Collected observations and anecdotes
Over the course of the project, observations were made with anecdotes collected on observations of children and staff comments that gave insight into their professional development as well as the special needs of children within the program. Several staff were observed during the beginning of the project having difficulty with managing their groups or trouble with certain children. Staffs at one site were having trouble getting the group to follow instructions and staff members rather than solving the issues were bringing children in one by one or requesting the site coordinator solve group management issues. “They’re getting wet out there, jumping in puddles. The group isn’t listening to directions” one staff member pleaded with her coordinator (01/08/08). It was common to see these staff needing such assistance. At another program, the staffs were more confident, yet there were children who they did not have the confidence to engage. “We’re not sure what to do with him [a disengaged child many suspected of being on the Autism spectrum]. We want him to join in more but he is… in his own world” (11/28/07). “These kids need serious help” one site coordinator said, hoping that the project would work with individual children rather than raising the quality of group management and by extension the program (12/4/07). As the project progressed, staff at the programs saw that the tool led to discussions that they would not normally have. Many staff reported on changes in special needs children. “He is so confident now. I was like, where’d you get that walk” one staff member commented on a child that had for much of the year been withdrawn prior to changing the group schedule and reorganizing how activities were sequenced (03/24/08). Staff appeared more confident in their interactions with children and appeared to share and support with other staff more in working with challenges they were presented with. Rather than taking a strict discipline approach, staff worked to prevent problems and get children to communicate more. “Believe me, when they have a real problem, they know me so they come to me…” (04/18/08). Site coordinators in general felt that their staff had improved in working with children with special needs by seeing fewer children brought to them since they had identified self care (second 4 of the tool) as a gap in their program. “The other day [two male staff members] brought in Guitar Hero as an activity. It was a blast. I could see that the kids were having fun because [the staff] were just having a ball!” (5/20/08). Other programs were able to focus on raising the quality of the program in general as staff were able to discuss issues and see that the majority of behavioral challenges were not due to special needs but were the result of a need to improve skills around group management. These comments and observations demonstrated that the tool use created discussion and moved the process of staff development as well as increasing the quality of service to children with special needs.
Posts
