Professional Development AR2004 |
Previous | 1 of 1 | Next |
|
small (250x250 max)
medium (500x500 max)
Large
Extra Large
large ( > 500x500)
Full Resolution
|
This page
All
|
T 550.3 A6l5p 2003/04 c.l Oklahoma Commission for TeacherPreparadon --PROFESSIONAL DEVELOPMENT ANNUAL REPORT 2003-2004 Student Success Through Quality Teaching LITERACY FIRST PROFESSIONAL DEVELOPMENT INSTITUTE EAST CENTRAL UNIVERSITY IMPACT • This past year 2,650 participants participated in Literacy First Training; with 1,557 in Phase I, 918 in Phase II, and 124 in Phase III Leadership training. This represents a total of 493 (90%) of the state's school districts participating and 19,261 participants since inception. • High quality content and learning documented by participants. • 65% said their greatest need is more practice and ?6% wanted additional training. • Prior to the training, most respondents stated they were familiar with the knowledge and skill areas, and a few reported having a solid understanding. Three months after the training, the majority of respondents moved from being familiar to having a solid understanding and the ability to apply the knowledge to their teaching practice. For example, 75% of Phase I & II teachers and 94% of Phase N teachers are using Literacy strategies in their classroom. • 60-63% of teachers reported that increased content knowledge in phonological awareness, phonics, vocabulary, fluency, and comprehension is helping them teach reading more effectively. REFLECTION • Many more principals attended the Literacy First, Phase III trainings. This was a result of a successful effort to recruit more administrators to attend the PDI. The principal is the key to creating the changes needed. With our Phase IV schools this became even more apparent as those schools in which there is strong leadership tend to have the greatest gains in student achievement. • Change takes time. The first level of change is bringing about an awareness of the learning content. Next is implementation of the new skills and strategies within the context of the present system or changing one, and finally the institutionalization of the practices into the culture ofthe school and system. • Phase IV schools data indicates gains in content and teaching strategies at a higher rate of implementation than non phase IV schools. Executive Summary Literacy First POI The purpose of this report is to provide the Oklahoma Commission for Teacher Preparation (OCTP) with evaluation findings pertaining to the quality and impact of the literacy Professional Development Institutes (PDls) offered across the state during the 2003-2004 school year. The goal of OCTP is to provide teachers with the training and support they need for students to achieve grade-level mastery of reading skills. OCTP partnered with Literacy First to provide literacy PDls to the teachers in state of Oklahoma. In Oklahoma, Literacy First offers four "phases" of services to elementary school teachers and administrators. These phases are: • Phase I and Phase" for kindergarten through sixth-grade teachers, • Phase III for school administrators, and • Phase IV, a whole-school literacy program. During the 2003-2004 school year, the Southwest Educational Development Laboratory's (SEDL) Evaluation Services (ES) unit evaluated Phase I and Phase" PDls, as well as the Phase IV program. Information was gathered on five levels of evaluating professional development (Guskey, 2000): participants' satisfaction with the PDls (level 1), participants' gain of new knowledge and skills (level 2), organizational support participants receive for implementation (level 3), participants' implementation of new knowledge and skills (level 4), and student outcomes (level 5). To evaluate the Phase I and Phase" PDls, follow-up surveys were administered to past participants. To evaluate the Phase IV program, current and past Phase IV participants were contacted. Current Phase IV participants completed surveys and interviews, and site visits were conducted. Past Phase IV participants completed follow-up surveys. Student test data were also examined to determine whether student achievement had increased at a rate different from a matched comparison group of non-Phase IV schools and the state average. Findings from the surveys, interviews, and site visits showed that most of the past Phase I and Phase" participants were satisfied with the PDls and gained new knowledge or skills through their participation in the PDls (levels 1 and 2). Teachers indicated that they received some degree of organizational support (level 3), implemented the techniques they learned (level 4), and saw improvements in student reading (level 5). Gains were evident at these last three levels from teachers and principals at Phase IV schools and also from teachers who completed the Phase IV school program last year (2002-2003 school year). Although most of the Phase I and Phase" Follow-up Survey participants rated content and quality of the PDls highly, participants that responded from the Phase IV schools indicated that content and quality of the Phase I and Phase" PDls varied widely. Further, most of the teachers in Phase IV schools attended Phase I and" PDls approximately two years ago, with the range of attendance spanning from 1 to 5 years. As revisions to the Literacy First program occur periodically, some of these teachers were not up-to-date on the changes. Teachers also noted that they received conflicting information on implementing Literacy First strategies. Finally, teachers expressed strong needs for further training and practice using Literacy First strategies, suggesting that the above factors may be affecting the successful implementation of the Phase IV program at their schools. Feedback from the Phase IV school administrators and teachers revealed that about one-third of the teachers in their schools were resistant to the Phase IV program, although almost all of the administrators supported the implementation of Literacy First Phase IV program 1. The reasons for teacher reluctance to implement the process included: I Guskey argued that organizational support is necessary for teachers to implement what they learned at professional development institutes. • The lack of a full understanding of the Phase IV process as a whole, • Perceptions of administrators as tending to push the implementation of all aspects of the program at once, and • A lack of clarity about the role and responsibilities of the Literacy Resource Specialists. This was echoed by LRSs as well. • Spread Phase I and" POI meeting days out to provide more practice days for teachers back at their schools and feedback sessions in the POls to reflect on and discuss difficulties experienced in using the strategies. • Offer an orientation session to Phase IV school staff at the beginning of their entry into the Phase IV program to clarify the objectives of the program, the roles and responsibilities of teachers, LRSs, and administrators, and a review of the content and major strategies and techniques they will be expected to implement. • Provide more focus on the strategies that administrators and LRSs report having difficulty with or using only occasionally (e.g., observation, feedback, modeling). • Provide more focus on strategies that teachers report having difficulty with (e.g., constructing Literacy Centers, conducting and using student assessments, classroom management issues). For student achievement, scores from third grade norm-referenced tests of reading ability in Oklahoma for 2002- 2003 school year were compared between Phase IV schools, matched comparison schools, and state average. Phase IV schools showed a higher gain than did comparison schools and the state average. Based on all of these findings, ES offers the following recommendations: Overview The Southwest Educational Oevelopment Laboratory's Evaluation Services (ES) recently completed its second year of the evaluation of multiple professional development institutes (POlS) that the Oklahoma Commission for Teacher Preparation (OCTP) offers to Oklahoma teachers. The purpose of this report is to provide OCTP with evaluation findings pertaining to the quality and impact of the literacy POls offered in Oklahoma during the 2003-2004 school year. LITERACY PDI GOAls OCTP provides professional development for Oklahoma teachers so that the teachers have an opportunity and the tools to grow as educators and professionals (Oklahoma Commission for Teacher Preparation, 2002). Emphasizing the importance of elementary school teachers' knowledge of what to teach as well as howto teach, OCTP partnered with Literacy First to provide professional development in reading instruction. The goal of Literacy First is to provide support and training to teachers to facilitate students' grade-level mastery of reading skills. Literacy First trainings focus on increasing educators' knowledge of reading processes, such as phonics and comprehension, and pedagogical strategies that use reading assessments to inform instructional decisions. THE LITERACY FIRST PROGRAM Literacy First, located in Mill Creek, Washington, offers a reading program for primary and secondary teachers nationwide (Literacy First, 1999-2001). Program goals are to have 85 to 90% of students reading at grade level by the completion of a three-year school wide implementation process. Literacy First's approach to increasing student reading achievement is to staff schools with competent teachers who explicitly teach reading in a diagnostic and prescriptive manner. Literacy First trains teachers in the skills needed to assess student reading ability and to use student assessment data for developing individualized reading instruction. They also train teachers in instructional competencies pertaining to the essential components of reading instruction. In Oklahoma, Literacy First offers four phases of literacy training. Phase I The Phase I POI is a five-day training, that focuses on instructing teachers in a comprehensive, balanced reading process and is offered to teachers across the state of Oklahoma. The training includes hands-on activities and opportunities for teachers to develop practical classroom assessments. The five days are divided into two sets of consecutive days. The break between the two sets of training days allows teachers to try the strategies in their classrooms that they learned during the first session. Phase II The Phase 1/POI is a three-day training that builds on the Phase I training. It instructs teachers in using flexible skill groups and developing literacy centers. It also addresses topics such as cooperative learning, collaborative planning, and guided reading. Ouring Phase II, teachers have opportunities to discuss problems or concerns with their implementation of Phase I strategies. Teachers who complete the Phase I POI can attend the Phase II POls. Both Phase I and Phase II POls provide participants with resources for developing meaningful and fun reading activities. Phase I and Phase II POls offer two levels of training each: (1) Kindergarten through second grade and (2) third through sixth grades. The Grades K-2 curriculum includes: • Phonological awareness • Phonics • Spelling • Comprehension The Grades 3-6 POI curriculum includes: • Pre-requisite information • Phonological awareness • Phonics • Spelling • Comprehension • Vocabulary Phase III The phase III POI is a two-day training for school administrators. Phase III training provides school administrators with strategies for facilitating teachers' implementation of Literacy First strategies. An overview of the Phase I and the Phase II POls that their teachers attend is also provided. Phase IV Phase IV is a three-year process that promotes reading instruction reform at the school level. A school's commitment to this process includes the school administrator's support of their teachers' implementation of Literacy First strategies, and that at least 80% of their teachers have attended Phase I and Phase II POls.2 The commitment from the schools allows Literacy First program staff to collaborate with the schools to expand the impact of the Phase I and Phase II trainings. Schools that agree to participate in the Phase IV process are assigned a Literacy First consultant and are asked to appoint one teacher at the school to become their Literacy Resource Specialist (LRS). Once becoming a Phase IV school, principals and LRSs attend leadership trainings that are held six times each year during the three-year process. At leadership trainings, principals and LRSs discuss their roles in faCilitating teachers' implementation of 2 The Phase IV program itself does not contain trainings for teachers. The Phase I PDI and the Phase II PDI are considered to be the teacher trainings for Phase IV schools. Literacy First strategies, and have opportunities to collaborate with others (Literacy First consultants and other Phase IV school administrators and LRSs) on strategies for increasing implementation. To initiate the Phase IV process, Literacy First consultants conduct a comprehensive school assessment. From this information, consultants develop a Strategic Reading Plan in consultation with the principal and LRS. Consultants facilitate and support the implementation of schools' Strategic Reading Plans by developing the capacity of the principal and LRS to maintain the infrastructure and culture necessary to accomplish the goal of helping students to read at grade level. Consultants visit their assigned schools six times a year, during which time they analyze reading instructional practices, demonstrate reading lessons, identify intervention strategies for under-achieving students, model teacher conferencing procedures for the principal, and serve as a catalyst and facilitator of the change process. The LRS assists in the implementation of the process at the school by collaborating with the consultant, the principal, and the teachers. The Phase I, Phase II, and Phase III POls have been offered in Oklahoma through East Central University's continuing education program since March 1998. The POls have been offered at major cities in Oklahoma, allowing teachers in all regions of the state to participate. To accommodate teachers with busy schedules, POls are offered on many different dates. During the 2003-2004 academic year, a total of 32 Phase I POls and 19 Phase II POls were offered. More than a total of one thousand teachers and administrators participated in the Phase I and Phase II POls during this academic year. However, only four Phase III POls were offered during the 2003-2004 academic year due to small enrollment. During the fall of 2003, 14 schools started the Phase IV program, and in the spring of 2004, six schools started the program. Of those schools who had already started the process before the 2003-2004 school year, five continued the program for a second year and 13 schools continued for a third year. Three schools left the program before they completed the three-year process due to their districts' lack of funding. The number of 2003-2004 Phase IV schools totaled 38 in 2003-2004 school year3. Summary of Evaluation Findings The following chart displays participant respondents in the increased knowledge in relation to instructional outcomes. Instructional Outcomes , Instructional Outcomes TPehaacsheerIsV INVoTne-Pachhaesres My increased content knowledge in phonological awareness, phonics, vocabulary, fluency, and/or comprehension is helping me teach reading 63% 60% more effectivelv. iInadmividbueattleizreadbliensthtraunctiboenf.ore to use student assessment data to provide 61% 49% I am able to assess students' reading skills more accurately than before. 59% 46% I am able to literacy centers more, more efficiently, and more effectively. 59% 38% sMtryatiengstierusctiIolnearhnaesd.not changed; I have not incorporated the Literacy First 4% 3% Total Number of Respondents " ., . 56 37 To document Literacy First participant content knowledge and instructional strategy improvement, the Literacy First staff constructed 20-item tests of the POI participants' knowledge of the content offered in these sessions, administering the tests at the beginning of the institute sessions and at their conclusion. 3 Of the 38 schools, 2 schools are counted twice as each school runs two programs for different grade levels. As shown in Table 1, during the fall 2003 POls, 270 grades K-2 teachers completed the pre-test and 253 completed the post-test. Of those, a total of 230 teachers completed both pre- and post-tests. For the grades 3-6 POI participants, 156 teachers completed the pre-test, 151completed post-tests, and a total of 136 completed both pre-and post-tests. Grade K·2 Sessions4 Figure 1 shows the percentage of teachers who correctly answered items on the pre- and post-tests for the fall 2003 and spring 2004 institute training sessions combined. Percent of Teachers Correctly Answering Test Items (Fall 2003 and Spring 2004 Combined) .. CII .c Q9 E; , Z Q10 I E .2.1. Qll t; Q12 .C.I.I K-2 Teacher Percentage Correct by Item Total ~--~--- ------~- ----- ~-~-----~-~-~------~-----~~------- - --~--~ I Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 ,bo I I I 82 ~ ", I <;4 j j 84 I " I <;7 I Ij -I , , t;LI I I I 85 j l~ , I I 'f ,ko f9 -t I 93 , I IdR I I 1 I 81 I, I I I ':I" I 1 , I I I I 80 I I , , I '7~1 j 92 I , 'Lln I I I J I I I 62 ! I , , I II ; , I I Iko I b9 I I 8f -I , I I ':It; l I I -I I I I I 57I 171 -t 82 " I <4 1 "j I I 93 -l , I' I I' , , 12 -f r I "'( '.,n , i ,60 I 41 60 i I .,~I I , , 85 I I 1 , ~1 , I II 86 1 I I ! ,ko I I I 88 III I, I I l.pre-te~1 I Post-te?!JI I I 100 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 o 10 20 30 40 50 60 70 80 90 __~ ~_ ___ ~~~:cen~___ I As can be seen in the above Figure, the teachers' knowledge on the topics was lowest at the time of the pre-test on items 15 and 16, and highest on items 8, 10, and 13. For all but one test item, the percentage of teachers correctly responding to the items increased from the pre- test to the post-test. Highest gains were seen on items 14 and 18 (59%), item 1 (53%), and item 7 (48%). Grade 3·6 Sessions The following graph shows the percentage of teachers who correctly answered items on the pre- and post-tests for the fall 2003 and spring 2004 institute training sessions combined. Percent of Teachers Correctly Answering Test Items (Fall 2003 and Spring 2004) Grades 3-6 Teacher Percentage Correct by Item Total Q1 Q2 Q3 Q4 Q5 Q6 Q7 .. Q8 Q) .Q Q9 E ~ Q10 E •~.. Qll .•.. r:-3 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 0 iiiiiiiiiiiiil 9 • Pre-test iii Post-test ••• _ •• _iiiil88 20 40 60 80 100 Percent THE PHASE IV PROCESS The emphasis of Phase IV of the Literacy First program is on implementing the knowledge and skills teachers gained through attending Phase I and Phase II institutes. The information gathered through surveys and interviews indicated that teachers had gained knowledge and skills from the Phase I and II trainings. They reported that they were primarily pleased with the support they were receiving from their administrators, Literacy First consultants, and the LRSs. They further noted that their instructional practices were changing, which was subsequently impacting their students' reading performance. While teachers were generally satisfied with the support they received, many also noted that they received "mixed messages" from their LRSs, administrators, and consultants as to what they were supposed to be doing, and how. Some also mentioned feeling that the program was too prescriptive or rigid, not allowing for flexibility related to individual student needs or teaching styles. Perhaps the most stated need by teachers was for further training and/or more modeling of Literacy First techniques. Even principals and LRSs voiced their need for further training or "refreshers" in these areas as well as in observation and feedback skills, and developing and providing resources to their teachers. Year 4 School Teachers. Teachers at the schools that completed the Phase IV program at the end of the 2002- 2003 academic year were also asked about their continued use of Literacy First strategies. As seen in Table 15, 58- 78% of teachers reported currently using the strategies "all the time." Only a small percent of teachers indicated that they were not using the strategies. Year 4 Teachers' Current Use of Strategies Literacy First strategies Not at Some of All the Total All the Time Time (N) Assessment 1% 21% 78% 90 Use assessment data to plan lessons 0% 36% 64% 90 Phonological awareness 4% 32% 64% 89 Phonics 6% 28% 66% 89 Vocabulary 1% 41% 58% 88 Comprehension 1% 24% 75% 90 Fluency 3% 30% 67% 91 Flexible skill groups 2% 31% 67% 91 Literacy centers 4% 22% 74% 91 Note: Total number of surveys returned = 91; response rate = 34% The vast majority of Year 4 teachers that responded to this item (96-99%) also reported that they intended to continue using Literacy First strategies in the future. Comments by these teachers indicate their value of these strategies, and included: • I have various Literacy Centers. I will never go back to the traditional method of teaching. • I let the assessment determine the instruction; it saves a lot of otherwise wasted time. • Flex group time gives the teacher and students the time necessary to concentrate on specific needs. • I will continue to use Literacy Centers. I have also added centers for math. They give me more small group and one-on-one time. • I use assessment to guide instruction and group my students accordingly. I will maintain this - it works! In summary, more teachers in all years of the Phase IV program implemented Literacy First strategies at a higher level this year than last year. Further, in the third year of the program, a larger percentage of the teachers maintained their level of strategy use than did first or second year teachers, and under five percent of all teachers indicated using the Literacy First strategies less this year than last. A large majority of Year 4 teachers also continued to use Literacy First strategies even though they completed the program last year. Because of the benefits they have experienced, many of them expect to continue using them in the future. These findings may also reflect that through the routine use of many of these strategies, teachers integrate them into their instructional practices to the extent that they remain usable and valued methods for reading instruction even after a school has completed the Phase IV program. Conclusions The findings presented in this evaluation report are a synthesis of data collected from Phase I and II Literacy First POls as well as from the Phase IV program. They are presented to our client, the Oklahoma Commission for Teacher Preparation, to assist in their understanding of the quality and impact of the literacy professional development currently provided to Oklahoma teachers. The following is a summary of the findings and recommendations based on these findings to assist Literacy First in considering next steps for increasing the effectiveness of the program for Oklahoma teachers. PHASE I AND /I POls • Phases I and" are multiple day workshops where groups of teachers from different schools and districts are presented with information and skills focused on reading instruction. • The content of Phase I is a basic introduction to literacy instruction. Phase" workshops focus on implementing knowledge and skills that teachers acquired from Phase I. • Survey findings indicated that most of the teachers who attended Phase I and II workshops during the 2002-2003 academic year valued the content presented and believed it to be useful in improving reading instruction. • They agreed that they had learned new knowledge and skills from attending the POls, and that they were implementing them to various degrees. • Teachers who were currently in a Phase IV school tended to implement the knowledge and skills to a higher degree than teachers in non-Phase IV schools. • The majority of non-Phase IV teachers also reported that they implemented a fair portion of the content and strategies they gained at the POls. • Teachers who had taken the Phase I and II POls from Phase IV schools had more consistent support than those from non-Phase IV schools. 91% of the teachers from Phase IV schools perceived their administrators as more highly involved. • Large majority of non-Phase IV schoolteachers reported their administrators to be "interested" and "supportive," even if not being actively involved in assisting these teachers. • Both Phase I and II past participants indicated that their training had led to instructional changes, in that they were more knowledgeable about the various literacy topics, were more able to assess students' reading abilities, and use the assessment information to better individualize their instruction. • Teachers noted that they were perceiving student impacts such as improvements in reading ability, more interest in reading, and their enjoyment of Literacy Center activities. RECOMMENDA TlONS Literacy First achieves breadth in offering a large number of Phase I and II institutes each year to teachers in Oklahoma. Depth is also attained through focusing on a wide variety of concepts and skills related to literacy learning. In meeting the Oklahoma requirements for professional development, Literacy First provides 30-45 hours of training, separated into two blocks (Phase I) to allow teachers to practice the knowledge and skills they are learning. In addition, the three days of Phase II provides teachers with a forum for further reflection on the results of their implementation of program content back at their schools as well as further training in Literacy First skills. Teachers, however, continue to express the need for more practice and training to better implement the Phase I and II knowledge and skills at their schools. • Literacy First staff might consider restructuring Phase I and Phase" institutes to allow teachers more practice time at their schools in between POI meeting days, instead of only one block of practice between the two sections of the Phase I institute and none in the Phase II institute. • Literacy First staff might consider incorporating a school-wide one-day orientation into the Phase IV program. This orientation would occur prior to getting the program underway, and focus on an overview of school staff roles in implementing the Phase IV program (e.g., administrator, LRS, and teachers), a review of highlights of content and major strategies and techniques, and time for questions and answers by school staff. • The role and responsibilities of the LAS, the Literacy First Consultant, and the administrator(s) should be explicitly laid out, so that all staff will have an understanding of the type of support they can expect from each. OEAMATH PROFESSIONAL DEVELOPMENT INSTITUTE • OEA focused on recruiting districts rather than individual teachers for the Professional Development Institute (PDI). As a result, participation remained at a high percentage with school wide change and improvement expected. • 83% of participants felt more knowledgeable about the standards and were better able to integrate them into their teaching practice. • The highest gains were in Mathematical Reasonings and Proof, and in Using Writing About Mathematics. • Participants report that the PDI was the best professional development of their careers, and indicated that the skills and strategies would be transferred to their classrooms. IMPACT REFLECTIONS • These trainings validated what research has been telling us: long-term professional development utilizing a quality curriculum will have significant impact on teaching practice. Initial training with multiple follow-up sessions shows the importance of allowing participants time to reflect on and practice new knowledge and strategies. • Keeping current in the teacher's content area is important to the knowledge level of teaching. Students benefit from teachers who understand their content. • Administrators who participated in the training indicated they felt more prepared to encourage their teachers to implement the new math strategies from the PDI. OSU/STILLWATER PUBLIC SCHOOLS CONSORTIUM MATHEMATICS PROFESSIONAL DEVELOPMENT INSTITUTE IMPACT • Teachers reported an increased confidence in their ability to teach for conceptual understanding and gains in useful teaching strategies. • Positive feedback from school administrators was received as they assessed the impact on teachers and on student learning - "As a principal and former classroom teacher of 20+ years, I am always open to new and improved ways of learning. I have witnessed students with whom we have had trouble in the past become excited about math class. They appear confident in their new found ways of learning. I was very impressed with the new attitudes that seem to be surfacing among the students toward learning ." • Videotaping lessons are powerful evidence of teacher practice, even though initially there was resistance - "It was a wonderful experience .. .! learned a lot from watching not only myself but the others as well. I enjoyed the feedback from the other teachers because it helped me realize the good things I have been doing." • The largest gains for participants occurred in the area of using mathematical investigation as a method for student learning. REFLECTIONS • Having quality trainers who have implemented the Connected Math Program curriculum was vital to the learning. • Developing the capacity for understanding conceptual learning will require extended time. • Respondents indicated that the instructional content was useful, the organization was excellent, the materials were very helpful, the presentation was stimulating, and the involvement of the participants was excellent. • Teachers need resources to teach problem-based mathematics. They also need a book for each student, which was limited during the PDI. Executive Summary OEAlOSU Math POI The purpose of this report is to present evaluation findings to the Oklahoma Commission for Teacher Preparation (OCTP) regarding the effectiveness of the Mathematics Professional Development Institutes (POls) for FY03-04. The Mathematics POls are designed to provide middle school mathematics teachers with instruction and practice in implementing classroom strategies, which is intended to improve standards-based mathematics performance of middle school students in Oklahoma. The POls used the Connected Mathematics Project (CMP) curriculum, which is student-centered, hands-on, and is organized around several major themes such as the "big ideas" of mathematical concepts, making connections among different mathematics topics in meaningful ways, and teaching for student understanding by guiding students to use discovery methods in practical problem-solving situations. Two institutions, the Oklahoma Education Association (OEA) and Oklahoma State University (OSU), conducted mathematics POI's for the state. OEA held POls in Chickasha, Woodward, and Ardmore, Oklahoma. OSU's POls were located in Ada, Stillwater, and Tulsa, Oklahoma. OEA completed two of their POls in the Spring of 2004, and OSU completed only one. Due to the low number of participants at the one institute that OSU completed during the spring, and the even lower response rate to the end-of-institute survey, findings focus primarily on the OEA POI's, although a discussion of the OSU POI participants' responses is also provided. OCTP contracted with the Southwest Educational Development Laboratory (SEOL) in 2002 to evaluate the mathematics POls. This report presents findings from the second of a three-year evaluation. The evaluation design for the second year consisted of a survey of mathematics POI participants and observations of several OEA and OSU training sessions. The evaluation sought to address the following questions: 1. Did participants perceive the mathematics institutes to be valuable and of high quality? (Participant Reactions) 2. What knowledge and skills did participants gain as a result of the mathematics institutes? (Participant Learning) 3. What aspects of organizational support do teachers perceive they need to facilitate the application of their new skills and knowledge? (Organizational Support) 4. How do participants expect to use their new knowledge and skills following the institute? (Participants' Use of New Knowledge and Skills) 5. What impacts to student learning do teachers expect to see based on the implementation of the knowledge and skills learned at the POls? (Impact on Student Learning) Key Findings Key findings were based on survey responses from 60 of 100 teachers who participated in the OEA and OSU 2003-04 Mathematics POls. Participant Reactions ~ Teachers found the institutes to be valuable and well organized. Respondents: • rated the institutes high on content covered, organization, presentation, materials, and participant involvement. • gave highest ratings to having opportunities to share ideas and learn from other participants and to the instructors' knowledge. Participant Learning ~ Teachers increased their knowledge and skills after participating in the institutes. Respondents: • reported more confidence in their ability to teach for conceptual understanding. • reported increased ability to use diagnostic assessment as a tool to improve student learning. Regarding the learning of math content, respondents' highest gains were in using mathematical investigations and student discovery as methods for student learning. Administrators' Learning Administrators received valuable information and skills from their institute training. • Administrators gave highest ratings to gaining valuable knowledge about their teachers' math training and to feeling more able to support their teachers' implementation of the math curriculum that they are learning at the POls. Participants' Use of New Knowledge and Skills )- Teachers increased the use of knowledge and skills learned at POls. Respondents: • reported increases in their use of new knowledge and skills. • reported that they were more prepared to teach geometry and spatial sense and to use writing about mathematics in their classroom instruction. Impact on Student Learning ~ Teachers' changes in instructional practices are expected to impact their students. Respondents: • expect positive impacts on their students' performance. • felt optimism for using their newly acquired strategies for improving student learning in mathematics. Key Recommendations Respondent Recommendations ~ Teachers expressed the desire for more training, more planning time, and more time to collaborate with other teachers. They also perceived a need for more direct connection of the curriculum with the OKJNCTM PASS standards during the institute. Evaluator Recommendations ~ Promote the use of content tests for assessing the effectiveness of the math training in improving teachers' content knowledge, and for helping the trainers to understand what concepts teachers need additional help with learning. )- Promote the importance of the administrators' participation in the POls as school leaders are often key in the successful implementation of new instructional strategies or approaches aimed at improving student performance. )- Extend the evaluation of the Mathematics POls to include a more specific examination of the effects of teachers' implementation of POI knowledge and skills. OEA Mathematics POI Evaluation Findings This section includes evaluation findings from the OEA Mathematics POI. Findings are organized according to the five evaluation questions and are based on the post-institute surveys, teachers' mathematics pre-and post-content assessments, administrator survey responses, and observations of a sample of institute training sessions. OEA Survey Respondents • Site 1: Woodward (n=22) • Site 2: Chickasha (n=38) • Site 3: Ardmore (n=32) Site Pa#rtiocfipPaOnIts i #CoofmSpulervteedys ResRpaotnese Ardmore 32 14 44% Chickasha 38 26 68% Woodward 22 16 73% Total 92 56 61% OEA Participants I Reactions This section discusses findings on participants' reactions to OEA PDls that were collected using two methods: (1) a summary of responses from the post-institute survey and (2) a summary of observations made during two days of training in January 2004. Post-Institute Survey Responses In a survey administered at the end of the institute, participants were asked to rate their perceptions of the quality of the institute they attended on a scale of 1 to 5 as to its content, organization, presentation, materials, and participant involvement. Table 2 displays their responses. Survey respondents' overall reactions to the institute were positive. The majority of the respondents indicated that the institute content was very useful (57%), the organization was excellent (70%), the materials were very helpful (63%), the presentation was stimulating (66%), and the involvement of the participants was excellent (73%). Percentages of Overall OEA Institute Ratings Of little use 2 3 4 Very useful Content 0% 0% 5% 38% 57% Poor 2 3 4 Excellent Organization 0% 0% 4% 27% 70% Boring 2 3 4 Stimulating Presentation 0% 0% 5% 29% 66% Ohf eliltptle 2 3 4 Very helpful Materials 0% 0% 7% 30% 63% Poor 2 3 4 Excellent InPvaorltviceipmaenntt 0% 0% 4% 23% 73% The Mathematics POI participants were also asked to indicate whether they agreed or disagreed, on a 6- point scale, to statements regarding specific characteristics of the institute at the end of the training. These characteristics were related to the structure and organization of the training (e.g., whether objectives of the institute were clear, trainers were experienced, and active learning opportunities were provided). The following table presents the percentages of responses for each category: Percentages of Specific OEA Institute Ratings c,r' .~ , '" Strongly ,,' ". Strongly Il" Disagree 2 3 4 5 Agree The objectives were clear. 0% 4% 2% 13% 34% 48% tThheeoibnjsetcruticvteosr.s adequately addressed 0% 0% 4% 9% 23% 64% tIhnestrcuocntotersnt.were knowledgeable about 0% 0% 2% 2% 21% 75% The instructors had experience implementing strategies being 0% 0% 2% 2% 23% 73% discussed. aTchteiveselsesaiornnisngi.nvolved participants in 0% 2% 2% 7% 18% 70% pTrhaectiscees.sions provided opportunities to 0% 2% 2% 5% 20% 71% iPdaeratiscipaanndtsleahrandfroopmpoorttuhneirtiepsartticoipsahnatrse. 0% 0% 0% 7% 13% 80% Itewaochueldrsr.ecommend this training to other 0% 2% 4% 7% 27% 60% OSU Mathematics POI Evaluation Findings This section includes evaluation findings from the OSU Mathematics POI. OSU provided four POls during the 2003-2004 academic year. The POI offered in Ada, Oklahoma is the only one that was completed by the time of this report writing. Therefore, only the Ada POI data are summarized. Findings are organized according to the five evaluation questions and are based on the post-institute survey, the mathematics content pre- and post-assessments, and observation of an Ada institute training session. The post-institute survey asked for participants' perceptions of the quality of the training, participant learning, organizational support for their implementation of participant learning, expected use of new knowledge and skills, and the expected impact on participants' students. Overview of OSU Survey Respondents Oemographics and background information were collected from the OSU POI participants using the Demographic Survey. These data are reported for the following OSU mathematics POI participants (n=45). • Site 1: Tulsa (n=8) • Site 2: Ada (n=8) • Site 3: Tulsa IV (n=10) • Site 4: Stillwater II (n=19) Summary of OSU (Ada PDI) Findings The four Ada POI respondents indicated that overall, the institute content was useful to them, its organization was excellent, the materials were very helpful, the presentation was stimulating, and the involvement of the participants was excellent. Positive ratings were also given for having clearly and adequately addressed training objectives, involved participants in learning activities, solicited feedback, and provided opportunities for participants to share. Comments and content test results indicated that these participants gained new knowledge and strategies for teaching mathematics, and expected that they would change their instructional practices based on what they had learned. They also expected to see positive changes in their students' attitudes about learning math, confidence in their math ability, and performance on math assessments. Indeed, these reports of increased confidence in their ability to teach for conceptual understanding and gains in useful teaching strategies, illustrates the value of modeling instructional practices that participants can use with their own students as observed by the evaluator during the third day of training in the Ada institute. Similarly, having POI participants work in groups and share their problem-solving methods, tangibly demonstrated how teachers could structure mathematical investigations for their own students. Given that these findings are based on only a small number of respondents, it is premature to make any definitive statements about the progress of OSU's POls toward attaining their goals. Conclusions This report presented evaluation findings regarding the quality and impact of the OEA and OSU Mathematics POls for Oklahoma middle school mathematics teachers offered during the 2003-2004 school year. These Mathematics POls represent the first in a three-year program. Findings from this first year of the Mathematics POls addressed five evaluation questions: (a) did participants perceive the mathematics institutes to be valuable and of high quality; (b) what knowledge and skills did participants gain as a result of the mathematics institutes; (c) what aspects of organizational support exist at their schools to help or hinder the application of teachers' new knowledge and skills; (d) how do participants expect to use their new knowledge and skills following the institutes; and (e) what impacts to student learning do teachers expect to see based on the knowledge and skills they learned at the POls? A summary of evaluation findings is presented below according to these five questions. Because only one of the OSU institutes had taken place at the time this report was written, the majority of findings are based on participant responses to the OEA institutes. Participants' Reactions Survey respondents' overall reactions to the institutes were positive. They indicated that the institute content was useful, the organization was excellent, the materials were very helpful, the presentation was stimulating, and the involvement of the participants was excellent. They also provided positive ratings in terms of participant involvement in the training, receiving instructive feedback, and having opportunities for sharing with other teachers. Sixty percent of respondents strongly agreed that they would recommend the training to other teachers. The evaluator's observations made during one of the OEA POls reflected the survey findings. It was noted that the trainers used practical experiences to demonstrate conceptual and/or abstract thinking. Participants were guided through ways to introduce, clarify, and review concepts, which connect them to real-world experiences. In addition, the trainers modeled cooperative methods of learning that allowed participants to collaborate in pairs or teams to develop problem-solving strategies and find solutions to the problems. In both OEA and OSU POls, trainers modeled the instructional strategy that POI participants were expected to use in teaching their students. Participants'Learning Findings from surveys, content tests, and observations indicate that POI participants appreciated and valued the institutes and expected that the instructional strategies and skills they learned will be useful for instructing middle school students. Just over half of survey respondents strongly agreed that they had gained useful knowledge and 55% strongly agreed that they had gained useful strategies for teaching mathematics. Respondents reported that after the training they were most prepared to teach for conceptual understanding but least prepared to assess student knowledge and create standards-based lesson plans. This reflects the POI focus on mathematics concepts in the first year and the future focus on assessments and lesson plans in the second and third years. In terms of how prepared teachers were to apply specific skills and content knowledge in the classroom prior to and at the end of the math training, respondents generally felt more prepared in all areas, with largest gains for using mathematical investigations as a method for student learning, and learning through student discovery. With respect to teaching specific topics, largest gains were seen in teaching mathematical reasoning and proofs, writing about mathematics, and geometry and spatial sense. For many respondents, their perceived gains in content knowledge were corroborated by assessed gains in their content knowledge through a mathematics pre- and post-content test taken at the beginning and the end of the training. However, the gains were generally small. The highest percentage of participants experienced only a slight gain of 1 point in pre- to post-test scores. Although just under half of the participants had gains of 2 or more points, 16% did not experience any gains and 10% had lower scores on the post-test than on their pre-tests. These findings were perplexing and somewhat disconcerting given the emphasis on improving teachers' content knowledge in the first year of the POI. However, the POI vendors, through informal communications with the evaluators, indicated that the importance or relevance of these contents tests were minimized when tests were administered to teachers. From this we might conclude that participants may not have performed as well as they could have if they were aware of the role the tests may play in determining the effectiveness of the program. Nevertheless, more investigation is needed to determine the extent to which teachers are truly learning about and improving their knowledge of various mathematical concepts. Organizational Support Regarding the kinds of additional support that might help teachers implement the knowledge and strategies they learned in the institutes, more computers, calculators, mathematics manipulatives, and Internet access were mentioned. Respondents also noted a desire for more opportunities to observe other teachers or trainers in classroom settings, and for networking structures to facilitate collaboration and support for sharing ideas and information with teachers they met at the institutes as well as teachers at their schools. Other comments included having more teachers at their schools trained in the math curriculum and attending follow-up trainings. Administrators reported that they felt more able to support their implementation of the math curriculum as a result of participating in the administrator training. After the training, they were stated that they most prepared to encourage their teachers to implement the new math strategies from the POI, to provide teachers with the resources necessary to effectively teach the math curriculum, and to promote the effort to improve student math performance in meetings. These administrators also perceived small changes in teachers' behaviors beginning to occur. They also mentioned that they were providing various methods of support, such as helping with curriculum alignment and ensuring that teachers have the necessary materials to implement the math strategies. Most responded that student impacts had not occurred yet because it is still too soon to tell or predict. Several challenges to implementation were cited by the administrators, including insufficient funds for textbooks, time for teachers to plan lessons and implement training strategies in the classroom, and time during the school day for teachers to share ideas with one another. Use of New Knowledge and Skills Findings indicated that many teachers are able and willing to actively try new strategies for engaging their students in mathematics learning activities. For example, teachers felt confident in their abilities to effectively implement what they learned in the institutes and stated that they planned to use more hands-on exercises in their mathematics instruction. Many expected to emphasize discovery and hands-on learning and use small learning groups to explore and solve mathematical problems collectively using manipulatives and/or learning games. In contrast, the teachers expected that they would use fewer worksheets and bookwork to teach math, and less lecturing or '1elling" during lessons. These are positive indicators that teachers are well prepared to implement what they learned in the first year of the POls. Expected Student Impacts The POI participants were asked what changes they expect to see in their students if they implement the new instructional strategies. Many teachers expect to see positive changes in students' attitudes, confidence, higher-order thinking skills, and performance on assessments. They also expect that these results would be partly due to students working more collaboratively with each other. Several teachers mentioned that they expected their students to have more "interest" in learning math. While these predictions of student impacts are positive, further evidence will be needed to determine students' responses to the new instructional approaches in terms of such factors as motivation, engagement, and mastery of standards. Ultimately, it will be important to explore whether students are able to perform better on high-stakes assessments as a result of teachers' implementation of the new approaches to teaching mathematics. Summary Overall, we conclude that the Mathematics POls are making progress toward their goals (although limited information was gathered from the OSU participants due to the timing of training sessions in relation to the writing of this report). These findings are particularly important for gauging the progress that the POls are making in this first of the three-year program. As described in the overview, in this first year of the OEA and OSU POls the focus was primarily on developing teachers' content knowledge and understanding of mathematics concepts. Indeed, survey respondents reported that they gained useful and valuable information from the training and that they are eager to implement what they learned. One concern was raised by the small gains in scores on the content tests from the beginning to the end of training. Although one might expect more teachers to have mastered the content tests by the end of training, it is uncertain whether teachers took these assessments seriously and performed according to their true knowledge and ability. Nevertheless, this is something for OCTP and the POI vendors to consider as they continue with the second and third years of the training. The ultimate value of professional development programs is the ability to help teachers transform the learning environment for their students. Evidence clearly suggests that participants are learning from, and find utility in, the Mathematics POI training. One limitation to these findings, however, is that these are self-reported perceptions and, therefore, they do not reflect how well the POI learnings actually transfer to the classroom. Furthermore, limited information has been gathered regarding the long-term support that teachers receive to implement what they learn. In line with OEA's and OSU's plans for the second and third years to focus more on implementation and support, this evaluation will also explore the extent to which participants of the POls, including administrators, implement what they have learned and whether adequate supports are in place to facilitate implementation. Recommendations Recommendations for program improvement are discussed below, beginning with a summary of participants' recommendations reported in the post-institute survey, followed by a set of evaluator recommendations, which are based on the overall findings from the 2003-2004 evaluation of the Mathematics POls. Participant Recommendations While progress is being made toward the goals and objectives of the POls, participants offered suggestions for furthering the effectiveness of the POI trainings. Suggestions included: • More time to collaborate with other teachers, • More alignment of the POI curriculum with the OKINCTM PASS standards, and • More opportunities to observe trainers teach a classroom lesson. Teachers also expressed the need to maintain contact with their POI colleagues. They suggested the use of a ListServ and/or an interactive web site. It may be beneficial to explore this possibility. As an example, electronic mailing groups on commercial Internet sites, such as Yahoo, are a free resource that may be used to create a networking community for the exchange of information and ideas. Other helpful sites containing resources that Middle School Mathematics teachers might find useful are: • Middle Web (http://www.middleweb.com/mw/listserv/MWLarchive.html) • The Math Forum at Drexell University (http://mathforum.org/) • Michigan State University Connected Mathematics Math Web Site (http://www.mth.msu.edu/cmp/) Evaluator Recommendations Based on the overall findings discussed above, SEOL presents the following recommendation for OCTP to consider in its ongoing efforts to enhance and improve the Mathematics POls. • Promote the use of content tests for (a) assessing the effectiveness of the training in improving teachers' content knowledge and (b) helping the trainers to understand what concepts teachers need additional help with learning during the training. Research has shown that more highly qualified teachers, particularly those who are knowledgeable and certified in their content areas, are more effective in helping students learn and perform better on assessments. Raising the awareness and importance of teachers' content knowledge through the pre- and post-tests may serve the dual function of reinforcing teachers' learning and enhancing accountability of the POls to deliver what they purport to do. • Promote the importance of the administrators' participation in the POls. School leaders are often key to successful implementation of new instructional strategies or approaches because they can often find time for teachers to engage in activities that improve teachers' learning and effectiveness with students, such as planning lessons, analyzing student work, sharing strategies, and observing each other's classes. The administrators who responded to SEOL's survey indicated that they valued their participation in the training because they became familiar with what teachers were learning, but some indicated that they would like additional strategies in their roles as administrators for helping teachers implement what they learned in the POls. OCTP might consider requiring principal participation, or at least encouraging principals to participate. Further information about the benefits of principal participation will be useful for determining more precisely how teachers benefit from this type of principal support. • Extend the evaluation of the Mathematics POls to include a more specific examination of the effects of teachers' implementation of POI knowledge and strategies. For example, student performance in specific learning activities might be structured to elicit differences in students' mathematics learning and understanding before and after teachers attend the POls. While POI staff expects student products to be submitted by teachers attending the POls, they are currently not a part of the evaluation. Student data would aid in moving the their evaluation to a stronger position in assessing the overall quality of the mathematics POls. SUCCESS IN SCIENCE PROFESSIONAL DEVELOPMENT INSTITUTE IMPACT • The Science PDI had 258 participants with an average of 15 years teaching. • The average number of years since the respondents had taken a college-level science content course was 12 years. • 82% of the respondents "Strongly Agreed" they had gained useful knowledge and strategies for teaching science. • Huge growth was seen in teachers' understanding of Teaching Learning Through Student Discovery (2.11 % to 3.5%) and Using Hands-On Science Modules (1.5% to 3.7%). • Respondents reported seeing students improve in multiple areas including more positive attitudes, an eagerness and excitement to learn, greater interest, better understanding and retention, and overall better performance. • 95% of the participants recommend this Science PDI as quality professional development REFLECTION • Teachers involved with Success in Science PDI are reporting positive student impacts. There is a need to increase the availability to more teachers to impact learning at a greater rate. • The limited number of professional development days is an obstacle in promoting quality learning for teachers. • Even though the funding is not available to expand this PDI, school districts have been willing to fund the cost of resources if Success in Science PDI will provide the training. The results speak loudly. Executive Summary Success in Science Professional Development Institute The purpose of this report is to present evaluation findings to OCTP regarding the effectiveness of the Success in Science Professional Development Institute (SSPDI) offered during 2003-2004. The OCTP's goal for the science professional development institutes is to provide professional development that enables teachers to make science more accessible, understandable, and engaging for students. Key Findings from FY 03-04 ~ Participants continued to rate the SSPDI as satisfactory, valuable, and well designed. Respondents consistently. • rated the institutes high on the content covered, organization, presentation, and materials; • gave overall highest ratings to the instructors' knowledge of the content and experiences implementing institute strategies; and • indicated they would recommend this training to other teachers. ~ Teachers reportedly increased their knowledge and skills after participating in the science institutes. Respondents consistently: • reported highest ratings in knowledge in their confidence teaching for conceptual understanding in science; and • reported highest gains in their preparedness to use science module materials and notebooks or journals for assessment. ~ More than half of the respondents (63%) strongly agreed that they felt confident in their ability to effectively implement what they learned at the SSPDI. ~ Teachers (former SSPDI participants) reportedly applied the knowledge and skills gained during the science institutes in their instruction. Respondents: • reported highest ratings in using the science module materials and teaching using student discovery and science investigation as a method for learning; and • reported highest gains in students' motivation and active involvement in science learning. ~ Teacher responses to the open-ended questions provided strong evidence that the teachers felt confident in the knowledge and skills they learned from the SSPDls and that their new knowledge has impacted their instruction and their students in very positive ways. Overview Science Institute Goals The OCTP's goal for the science professional development institutes is to provide professional development that enables teachers to make science more accessible, understandable, and engaging for students. The OCTP aims to provide professional development that: • adheres to professional standards for science teacher professional development; • increases teachers' level of content knowledge in science; • improves teachers' ability to use inquiry-based instruction methods; and • trains teachers to value the role of communication, both verbal and written, as a tool to aid student learning. Approach to Attaining Goals Three members of the Oklahoma Teacher Education Collaborative (Southwestern Oklahoma State University, University of Tulsa, and Oklahoma State University's Center for Science Literacy) submitted a bid and were awarded the contract to conduct the Success in Science Professional Development Institute. The SSPDI collaborative proposed to provide Oklahoma K-6 elementary and 6-8 grade science teachers with professional development to increase the science literacy of Oklahoma students by engaging teachers in inquiry methodology for instruction of grade-specific science modules. The SSPDls use Full Option Science System (FOSS) and Science and Technology for Children (STC) curricula which are standards-based, science units that employ inquiry strategies to help students learn science topics in meaningful ways. The professional development emphasizes a student-centered approach with instruction focused on guiding students to use discovery methods in practical problem-solving situations. Previous success with this model of science instruction led to expectations for significant increases in student achievement in science and related courses. The SSPDI participants learn to use grade-specific science modules during days of training that occur during the school year. Teachers receive hands-on experience with the modules that are intended to promote their reflection, planning, and implementation of inquiry-centered strategies in elementary and middle school classroom instruction. The SSPDls are structured so that participants receive an initial intense multi-day period of methodology and curriculum training with several single days of follow-up for the Module 1 and Module 2 materials during the academic year. The Module 3 training involved teachers in training in pedagogical and science instruction strategies to enrich their use of the module kits. The teachers received a kit with instruction manual and necessary materials and supplies for approximately 16 lessons they use with their students throughout the module. Evaluation Questions The POI evaluation questions related to the science institutes are: (1) Did participants perceive the science institutes to be valuable and high quality? (2) What knowledge and skill did participants gain as a result of the science institutes? (3) How do participants perceive that school leaders are supporting and/or hindering the application of their new skills and knowledge? (4) Do participants expect to be able to integrate training knowledge and skills into classroom instructional practices? (5) Are the science institutes producing gains in student learning? The 2002-2003 Science POI evaluation centered on three of the evaluation questions and determined that last year's Science POI participants found the science institutes provided training that was valuable, high-quality, and useful for developing their knowledge and skills in science instruction. The participants also indicated that they felt prepared and expected to use the knowledge and skills they gained in their classroom instruction. This SSPOI report is based on two primary sources of data: (1) an end-of-institute survey for this year's (2003-2004) SSPOI participants, and (2) a follow-up survey of last year's (2002-2003) SSPOI participants. The report will first address information obtained from the current SSPDI participants to verify information gathered in the previous year. Then, the information from previous participants will be presented. The evaluation questions, methods, and data sources are listed in the following table: SSPDI Year 2 Evaluation Evaluation Question Method Data Source (1) Are SSPDI of high value and • End-of-Session Survey • 2003-2004 SSPDI (Module 3) quality? • Observations Participants (2) Do participants increase • End-ot-Session Survey • 2003-2004 SSPDI (Module 3) knowledge and skill during • Observations Participants Science PDI? • Follow-up Survey • 2002-2003 SSPDI Participants (3) Do participants have school • Follow-up Survey • 2002-2003 SSPDI Participants leadership support tor SSPDI instruction? (4) Are participants able to • End-ot-Session Survey • 2003-2004 SSPDI (Module 3) integrate SSPDI knowledge and • Observations Participants skills into classroom instruction? • Follow-up Survey • 2002-2003 SSPDI Participants (5) Are there gains in student • Follow-up Survey • 2002-2003 SSPDI Participants learning? Did participants perceive the science institutes to be valuable and high quality? At the end of the training, respondents were asked to rate the SSPDI on a scale of 1 to 5 whether (a) the content of the institute was useful, (b) the institute was organized, (c) the presentations were stimulating, (d) the materials were helpful, and (e) they were actively involved in learning. the following table indicates that the SSPDI respondents average ratings for the institute was high (between 4.5 and 4.8) for all five categories. 2003·2004 SSPDI Respondents' (N:56) Percentages of Overall Institute Ratings I," o Ofuslitetle 2 3 4 ." Veruyseful M(SeOan) a. Content -- -- -- 10 (18%) 46 (82%) 4.8 (.39) Poor 2 3 4 Mean Excellent (SO) b. Organization -- -- -- 14 (25%) 42 (75%) 4.8 (.44) Boring 2 3 4 Stimulating Mean (SO) c. Presentation -- -- 2 (3%) 15 (27%) 39 (70%) 4.7 (.55) Of little , Very Mean help 2 3 4 helpful (SO) d. Materials .. 1 (2%) 2 (3%) 23(41%) 30 (54%) 4.5 (.66) Poor 2 I~ 3 4 Excellent Mean (SD) e. Participant .. .. 1 (2%) 8 (14%) 47 (84%) 4.8 (.43) Involvement Note: Ratings on a 5-point Semantic Differential Scale. The scale for each item ranged from: (a) "of little use" to "very useful," (b) "poor" to "excellent," (c) "boring" to "stimulating," (d) "of little help" to ''very helpful," and (e) "poor" to "excellent." N=number of participant responses. SD=Standard Deviation What knowledge and skills did participants gain as a result of the science institutes? After responding to items on the survey about their reaction to the SSPDI sessions, the participants had an opportunity to indicate what knowledge and competencies they had obtained from the institute sessions. Participants were asked to respond to eight items upon which they could indicate how prepared they thought they were to use the instructional strategies that had been discussed in the SSPDI sessions. These strategies included instruction on how to use science investigation, cooperative groups, or problem solving strategies as a method for learning science. Participants rated these instructional strategy items (4-point scale, 1=not well prepared to 4=very well prepared) twice: the first time to represent how prepared they currently thought they were to use these instructional strategies now that they had participated in the SSPDI, and the second time to represent how prepared they thought they had been to use these strategies before they participated in the SSPDI. Average Ratings of SSPDI Module 3 Participant Preparedness Perceptions of Skills and Knowledge Gains Learning through student discovery ~ 12.1 T 13.6 1 I,", n' ., 12.2 .s. 13.6 I I 12.5 13.5 I 12.2 1 13,4 I 12.1 1- 13.5 I 11.6 j I 3.5 I 11.5 1 13,7 j I 12.1 I 13.4 . , -",,' ! losefore I ~ow Science investigation as a method for learning Cooperative student groups Problem-solving strategies Question strategies to promote reflection Notebooks or journals for assessment Science module materials Connecting content to real world and careers 234 Four-Point Scale: Not well prepared (1) to Very well prepared (4) N=112 Do participants expect to be able to integrate training knowledge and skills into classroom instructional practices? Another goal of the SSPDI is to determine how the institutes impact the participants' instructional actions after they complete the sessions 2003·2004 SSPDI Respondents' Expectations for Implementation DSitsroanggrelye' 2 3 4 5 SAtrgornegely M(SeDan) N tie. aI cfehelfomr ocroencceopntfuidaelntunindemrsytaanbdiilnityg to -- -- -- 2 17 37 5.6 (4%) (30%) (66%) (.56) 56 than I was prior to this training. oim. pIlfeemelencot nwfidheant tI hthaavteIlceaanrneefdfecattivtheilsy -- -- -- (42%) (3149%) (6335%) (5.5.67) 56 Science POI. . . Note: N=Number of respondents. SD=Standard Devianon. Scale range from Strongly Disagree (1) to Strongly Agree (6). Are the SSPDls producing gains in student learning? Although we have no direct measures of student impact from the survey, the participant responses to open-ended items (italicized questions in paragraphs below) describe an anticipated level of student impact as a result of what participants had experienced while implementing the science modules during the time period spanning the Science PDI. What Student Impact Have You Seen? In an open-ended item on the SSPDI survey, teachers (n=56) reported seeing improvements in their students during this year when they have begun to use the science modules in their instruction in multiple areas. These included seeing more positive student attitudes, an eagerness and excitement to learn, greater interest, better understanding and retention of what students learned, and overall better student performance. Parents, teachers from other subjects, and the students themselves were among those who reported to SSPDI participants that their science students "love to come to class" and had a new enthusiasm for learning. As one participant mentioned, students were more active in taking the initiative to learn, read, or create new problems to investigate. Students were also developing as cooperative learners and becoming "more respectful of others work," as one teacher reported. There were very few comments by teachers that showed a negative impact on students. One commented that her students "didn't like to write in notebooks," but that they were "still working" on using that learning method. The above teacher responses to the open-ended questions provide evidence that the teachers feel confident in the knowledge and skills they learned from the Science PDls and that their new knowledge has impacted their instruction and their students in very positive ways. The next section details information from the follow-up survey about teachers' perceptions about how the Science PDls have impacted their instruction, affected student performance in science, and been supported by administrators and other teachers at their schools. Are the science institutes prOducing gains in student learning? Most of the former SSPDI participants indicated that their students are more engaged in classroom activities (94% agreed or strongly agreed, Item f). Former SSPDI Respondents' Ratings ASfctieerncpeartPicDipIa..t.ing in the Success in DSitsroanggrelye Disaegre Agere, SAtrognregely M(SeDan) N f. cMlayssstruodoemntsacatirveitiemso.re engaged in (43%) (21%) (181%2 ) (7560%) (3.7.73) 66 Note: N=Number of respondents. SD=Standard Deviation. Scale range from Strongly Disagree (1) to Strongly Agree (4). Respondents also had the opportunity to respond to six items about the impact of implementing SSPOI instructional strategies with their students. The following table lists those responses: ormer espon en s a Ings 0 e mea 0 ns ruc Ion Noatll at vNeoryt Some mVuecryh M(SeOan) N much a. Mimyprsotuvedde.nts' science performance has (21%) -- (3232%) (6453%) (3.5.68) 66 b. Motyhesrtubdeetntetsr inintsecriaecntc/ceoocplaesrastees. with each (21%) -- (4371%) (5314%) (3.5.53) 66 c. Mscyiesntcued.ents are more motivated to study 7 59 3.9 66 -- -- (11%) (89%) (.31) d. Mleayrnstinugdensctsienacree. more actively involved in -- -- (64%) (9641%) (3.2.94) 65 e. sMcyiesntcuedecnltasssaessk. questions more often in 19 47 3.7 66 -- -- (29%) (71%) (.46) f. I learned from indirect sources (for example, the parents, other teachers, 4 1 27 34 3.4 students' friends) that my students' (6%) (2%) (41%) (51%) (.80) 66 science motivation has improved. F SSPDIR d t' R f f th I ct f SSPDII t tl Conclusions • This report provided confirmation that these three professional development levels continued to be fulfilled by the SSPOI (the value and quality of the institute, the knowledge and skill gained, and the expectations for integrating into classroom instruction). • Provides evidence that according to the perspectives of former SSPOI participants who responded, the final two levels of professional development (administrative support and gains in student learning) are also present as a result of the quality of professional development to the K-8 teachers who participated in the institute. • Respondents liked the sessions, believed the inquiry-based strategies and skills are useful for their classroom instruction, and expected to see changes in their instruction and in student learning in their classrooms. • Respondents reported that they thought they were more prepared now than they were before attending the science training to teach using key instructional strategies discussed during the institute. • Success in Science POI Module 3 respondents reported gains in their ability to use inquiry-based instructional practices that are student-centered, problem-based, and built around students' collaborative investigation of scientific phenomena. • Improved use of effective questioning strategies and methods for eliciting regular student verbal and written communication as an aid to learning and assessment were other areas of improvement that institute participants found especially promising for future instruction. • Former SSPOI participants reported continued use of the materials and instructional strategies they learned from the SSPOI with their students. The use of notebooking and journaling, assessment tools, and instructional strategies associated with their grade level modules were some of the specific SSPOI strategies mentioned by the teachers. • Respondents also indicated that they used cooperative groups, discovery learning, and scientific investigation techniques as a method for leading students in exploring the science ideas, principles, and information presented through the science modules. • The results indicated that generally, most respondents felt supported by their school leadership for using the Success in Science POI materials. • Overall, the teachers indicated that while they shared information and resources that they could from the Science POI with their colleagues, they wanted more teachers in their school to participate in the SSPOI so more students could benefit from these type of science learning experiences. • In general, respondents indicated that students benefited from instruction using the SSPOI modules and recommended strategies. Recommendations • Teachers want to participate in an additional SSPOI because they want to be prepared to teach another module for the same or a different grade level • The teachers also reported that they appreciated being able to spend more time and have more opportunity to work with different modules in preparation for instruction with students. • According to the former participants, participation in an additional SSPOI allowed them to come into the institute with an understanding of the process so that they could then concentrate on absorbing the content and information that would improve their ability to plan, prepare, and lead students in the science modules • It is recommended that, as much as is feasible, additional SSPOI sessions and materials should be provided for these teachers to continue their growth in inquiry-based science instruction. • Providing funds and opportunities for teachers to participate in follow-up sessions of the SSPOI may help solidify instruction, provide a way to deepen their content knowledge of the different science modules, and allow them to sharpen their skills at leading, assessing, and encouraging student learning of science. OEA MENTORING PROFESSIONAL DEVELOPMENT INSTITUTE • OEA developed partnerships with several school districts with the understanding that district support would increase successful implementation and outcomes. • Approximately 700 teachers have participated in a mentoring PDI since the inception. • Classroom management skills were enriched early in the training. • On average participants have formally mentored 1-3 teachers. • Three-year professional growth plans were written to enhance the teacher's practice. • 89% of participants agreed that beginning teachers and their own practice improved as a result of the institute. IMPACT REFLECTION • Aligning our evaluation system from pre-service to retirement is a result of the data received from this grant. A pre-service to retirement evaluation system is being proposed. Higher education institutes are being held accountable to the INTASC standards for their programs. These state adopted competencies will guide the development of a uniform system. • NCLB (The No Child Left Behind Act), OSRHE (Oklahoma State Regents for Higher Education), and NSDC (National Staff Development Council) concur that mentoring and/or coaching of teachers is the most vital and the most neglected aspect of assisting teachers to improve practice. The mentoring institutes must be expanded and supported to improve state wide student achievement. Executive Summary OEAlOSU Mentoring POI The Oklahoma Commission for Teacher Preparation (OCTP) is charged with creating a competency-based teacher preparation system to ensure that every Oklahoma classroom has a competent and qualified teacher. The OCTP contracts with independent vendors to provide professional development Institutes for teachers in the state of Oklahoma, including teacher mentoring. Since 1998, OCTP has provided two mentoring Institutes that provide training in essential skills necessary to support beginning teachers, which are applicable to supporting experienced colleagues as well. Two vendors were contracted to provide this training-the Oklahoma Education Association (OEA), using the Pathwise Induction model, and Oklahoma State University (OSU), using the Performance Learning System Coaching model. SEDL's Evaluation Services was contracted to conduct an external evaluation of the Mentoring Professional Development Institutes. This report presents findings from the second of a three-year evaluation contract, which began in April 2002. The purpose of this evaluation was to assess the progress that the Mentoring Institutes were making toward the goal of enhancing the induction process by providing consistent mentor training in the essential skills necessary to support beginning teachers. To do so, SEDL's evaluation worked from the following evaluation questions: 1. How many teachers participated in the training and how were they selected? 2. In what types of mentoring assignments and activities have trainees been engaged since participating in the Mentoring Institutes? 3. Have Mentoring Institute participants been supported in their use of the mentoring strategies? 4. What was the impact of the training on participants and beginning teachers? The evaluation findings are based on a survey that was administered to past participants of the OEA and OSU Mentoring Institute during the four-year period beginning with the 1999-2000 school year. The findings are discussed in terms of training participation and recruitment, mentoring assignments and activities, support for implementing mentoring strategies, and the impact of the training. Key Findings • Participation and Recruitment. An examination of participation rates and the recruitment process for the mentoring Institutes revealed that the purpose of training was effectively communicated although the training is reaching a relatively small number of teachers as a percentage of all teachers in the state. • Mentoring Assignments and Activities. The majority of survey respondents have mentored 1-3 beginning teachers although a number have still not had this opportunity. Survey respondents indicated that while most have mentored beginning teachers, the training did not result in being assigned to more Residency Teams. Based on reports of mentoring activities and assignments, the findings also suggest that the trained mentors are successfully implementing the mentoring strategies they learned in the Institutes and that they are providing substantial support to beginning teachers. • Support for Mentoring. Survey respondents reported that collaboration and follow-through are important to successful implementation of the mentoring strategies and that a number of participants reported having teams of teachers in their schools who were trained in the mentoring strategies. • Training Impact. Survey respondents suggested a variety of benefits for themselves and beginning teachers. In particular, respondents reported being more reflective about improving their own teaching practice and finding effective ways to help beginning teachers or other colleagues. The respondents' comments illustrate their appreciation of the training and the desire for continued support for implementing the strategies they learned. Key Recommendations Evaluation findings indicate that the Mentoring Institutes are beneficial to both the participants of the training and the beginning teachers (and colleagues) who they mentor. Findings also illuminate several areas where the Mentoring Institute or the implementation process in schools and districts could be strengthened. Based on these findings, several recommendations are suggested for ~CTP's consideration: • Increase the number of teams of trained mentors within districts or schools to promote collaboration and ongoing support for the use of mentoring strategies. • Promote principals' participation in the mentoring Institutes. • Promote the adoption of some or all of the mentoring strategies school-wide or district-wide. • Promote the benefits of assigning beginning teachers to trained mentors and encourage schools and districts to use trained mentors when possible. • Promote school and district policies that will ensure adequate numbers of trained mentors to serve as resources for beginning teachers as well as experienced colleagues throughout the state. Overview The Oklahoma Commission for Teacher Preparation (OCTP) is charged with creating a competency-based teacher preparation system to ensure that every Oklahoma classroom has a competent and qualified teacher. The OCTP's three-pronged approach toward attaining this goal includes the provision of competency-based teacher preparation and accreditation, candidate assessment, and ongoing growth and professional development of classroom teachers across the state. In support of federal legislation mandating that all states develop plans ensuring that all teachers of core academic subjects are highly qualified by the end of the 2005-2006 school year, the OCTP contracts with independent vendors to provide professional development Institutes for teachers in the state of Oklahoma. These Institutes emphasize both content and professional instructional practices in the areas of literacy, mathematics, science, and teacher mentoring. This report addresses this last emphasis, mentoring. Since 1998, the Oklahoma Commission for Teacher Preparation has provided two mentoring Professional Development Institutes that provide training in essential skills necessary to support beginning teachers, which are applicable to supporting experienced colleagues as well. Two vendors were contracted to provide this training-the Oklahoma Education Association (OEA), using the Pathwise Induction model, and Oklahoma State University (OSU), using the Performance Learning System Coaching model. This report presents findings from the second of a three-year evaluation contract, which began in April 2002. In Year 1 of the evaluation, SEDL's evaluation focused on the training provided by OSU during the 2002-2003 school year. Findings from the Year 1 evaluation are presented in the 2002-2003 Annual Report on Professional Development. Key findings regarding the first year evaluation of the OSU Mentoring Institute included the following: • Survey respondents reported being very satisfied with the training they received and that the OSU Mentor Institute was of high quality and contained useful content and materials that were presented effectively. • Three months following the end of training, survey respondents reported that they were applying their new skills and knowledge for mentoring beginning teachers and colleagues, indicating that they had increased their level of use of the new skills and strategies to the extent that the majority of respondents were now able to use the mentoring strategies most of the time. • Although survey respondents indicated that principals generally supported mentor strategies, information obtained from interviews indicated that implementation was significantly impacted only if the principals actively participated and promoted teachers' use of mentoring strategies. • Teachers reported that they still needed more time to practice and more involvement of other teachers at their school in order to increase their effectiveness as a mentor teacher. In the Year 2 evaluation, attention turned to both OEA and OSU Institutes and the overall quality and impact of the mentoring training over the four-year period of 1999-2000 and 2002-2003 school years. To do this, a survey was sent to all participants who took part in the OSU and OEA Mentoring Institutes during this time period. This report presents evaluation findings from the second year follow-up survey of the Mentoring Institute participants. The following sections of this report include a description of the two Mentoring Institutes, the evaluation approach, evaluation findings, conclusions, and recommendations. Mentor Professional Development Institute Goals and Objectives The goal of OCTP is to ensure that every student in Oklahoma has access to competent, caring, and qualified teachers. The Mentoring Institutes are expected to contribute to the OCTP goal by achieving the following: 1. Provide a systematic process for the support and development of mentor teachers for the Oklahoma Resident Teacher Program. 2. Provide quality mentor training that enables the mentor teacher to address the varied needs of the resident (beginning) teacher. 3. Create a framework for ensuring sustainability by developing and maintaining a cadre of mentor teachers throughout the state who are prepared to coach entry-level teachers and have the flexibility to coach colleagues interested in expanding instructional skills in the content area. OEA Mentoring Institute The OEA Mentor Institute uses the Pathwise model of mentor training developed by Education Testing Service (ETS). The institute aims to increase mentor teachers' ability to support beginning teachers in a variety of teaching practices. The training consists of the following components. • Classroom Management Skills • Introduction to Frameworks • An Assessment of Teacher Effectiveness • Performance-Based Work Products • Three-year Professional Growth Plan • The Colloquium The OEA Mentoring Institute consists of four days of training throughout the year followed by a celebration day in the spring, which is to recognize the accomplishments and professional growth of the beginning teachers who were mentored. OEA implemented a new approach to recruiting participants for the 2003-2004 school year. In this new approach, OEA partnered with several districts throughout the state to provide training to its mentor teachers with the understanding that district support for such a program would increase the likelihood of successful implementation and outcomes. In previous years, teachers from different districts participated in the training together. OSU Mentoring Institute The OSU Mentoring Institute is based on the Performance Learning System Coaching model. The project goals are to: • Provide a systematic process for support and development of mentor teachers for the Oklahoma Resident Teacher Program. • Provide quality mentor training that enables the mentor teacher to address the varied needs of the resident teacher. • Create a three-year framework for ensuring sustainability by developing and maintaining a cadre of mentor trainers who will be available throughout the state. • Provide a cadre of mentor teachers who are prepared to coach entry-level teachers and have the flexibility to coach colleagues interested in expanding instructional skills in the content area. OSU's Mentoring Institute consists of seven days of training held over a six-month period, with the first block of training days typically conducted in the summer or the beginning of the school year. The Mentoring Institute has been offered at various sites throughout the state. The 7 days of training focuses on a comprehensive set of mentoring strategies intended to help participants coach beginning teachers as well as experienced colleagues. In particular, the training consists of 10 coaching skills for successful teaching, which are presented over the course of the 7-day training and with which participants are given time to practice. The ten skills include the following: • Conducting Pre-observation Conferences for Effective Coaching • Using Open and Closed Questioning • Constructing Confirmatory Paraphrases • Using Positive Phrasing • Constructing Approval Statements • Conducting Post-observation Conferences • Selecting Modes of Questioning • Attending to Non-Verbal Communication • Using Pause Time • Selecting Data-Collection Formats The training involves formal presentation of materials as well as substantial opportunities to practice and receive feedback from peers and presenters. Participants are expected to videotape themselves during mentoring sessions at their schools in between training days. These videos are then shared and discussed with fellow participants during the following sessions. Participation According to OEA and OSU participation databases, a total of 695 teachers participated in both Mentoring Institutes over the four-year period beginning with the 1999-2000 school year. An average of 73 participants attended the OEA training each year and an average of 101 participants attended the OSU training each year. Table 2 shows the number of participants who attended the two Institutes each year. The total number of participants trained in both OEA and OSU mentoring programs represents roughly 2% of all Oklahoma public school teachers. Mentoring Institute Participation Rates 1999-00 2000-01 2001-02 2002-03 OEA participants 44 103 62 82 OSU participants 120 98 112 74 Mentoring activities To further explore the impact of the Mentoring Institute, SEDL asked training participants to indicate how often and in what types of activities they shared with their mentees. Additionally, mentor-training participants were asked to indicate how often they met with their mentee. The pattern was similar across all four years and across OSU and OEA training programs. Between 86% and 91% reported meeting with the beginning teacher at least once a week or more. Participants were also asked to report when they met with their mentees. Respondents indicated that they met most often after school (64%), but they also indicated meeting before school (38%), during a common planning time (38%), during lunch (31%), and other times such as during individual planning time or in workshops. (13%). These responses suggest that the mentors are providing a fairly substantial amount of support to beginning teachers on a regular basis throughout the school day. SEDL also asked the training participants to indicate what types of activities they have shared with their mentee. Table 6 shows the percentage of respondents who engage in the various mentoring activities. These activities, which respondents selected from a list of options, are presented in order of greatest to least engagement. As indicated in the table below, the most common type of activity reported was observing the mentee in the classroom and giving feedback. The response patterns were fairly similar between OEA and OSU training participants. The biggest difference in reports of mentoring activities between OSU and OEA (24 percentage points) related to planning lessons, units, and assessments. This difference may reflect the different approaches presented in the two Institutes. While this breakdown of activities may indicate that there is room for mentors to focus more on analyzing student work and test data, planning professional development, and co-teaching, it nevertheless suggests that there is still a fairly good balance of support activities provided to beginning teachers. en ortng C IVI ies OEA OSU (n=39) (n=41 ) Observing the mentee in the classroom and giving feedback 82% 90% Planning lessons, units, assessments, etc. 49% 73% Strategizing for communication with parents 64% 71% Referring the mentee to other people as resources 59% 71% Looking at student work 56% 54% Analyzing student test data 28% 44% Planning for professional development 33% 37% Co-teaching with the mentee 21% 29% IOphthileorso(peh.gy.,, deitscc.)ussions about classroom policies, teaching strategies or 5% 20% Mt' AfT Use of mentoring strategies Mentoring Institute participants were asked about the extent to which they have used a variety of mentoring strategies now that they have had time to implement what they learned in the training compared to the use of those strategies before training. A series of survey items focused on the implementation of specific mentoring strategies using a 4-point Lickert scale of 0 (not at all) to 3 (extensively). To assess the effect of the training, SEDL computed the difference in the ratings now and before training. The following table presents the mean differences on these items. The differences in reported use of the mentoring strategies now compared to before training was significant across both programs and for all strategies (p<.01). These responses provide a measure of the degree to which respondents perceive they have implemented what they learned in the mentoring training. It is notable that the mean differences between reported use of mentoring strategies now and before training were greater for OSU participants than OEA participants, although it is unclear why. Mean Differences in Mentoring Strategies Use Now and Before OEA OSU Mean N Mean N Observing classrooms 0.76 34 0.89 38 Providing feedback 0.86 35 1.21 38 Using questioning strategies 0.89 35 1.58 38 Conducting pre-observation conferences 0.91 34 1.26 38 Conducting post-observation conferences 0.79 34 1.16 38 Using data collection techniques 0.83 36 1.51 37 Using strategies for working with adults 0.62 34 1.21 38 Accepting feedback from colleagues 0.52 29 1 19 Being observed in my classroom 0.44 36 0.71 38 Reflecting on my own teachinc practice 0.86 36 1.03 38 Collaborating with my peers 0.63 35 0.89 38 The strategy for which OEA participants reported using more as a result of training (mean difference=.91) was conducting pre-observation conferences. The strategy for which OEA participants reported using least as a result of training (mean difference=.44) was being observed in the classroom. The strategy for which OSU participants reported using more as a result of training (mean difference=1.58) was using questioning strategies. The strategy for which OSU participants reported using least as a result of training (mean difference=.71) was being observed in the classroom. These findings may reflect the variations in training approaches used by OEA and OSU in their Mentoring Institutes. Nevertheless, they suggest that there is a significant increase in the reported use of mentoring strategies as a result of training, which provides evidence that trainees are indeed implementing the skills they learned. Impact of mentoring training on teaching practices SEDL asked participants to respond to a series of items about the ways in which the mentoring training has impacted beginning teachers' and their own teaching practice. Table 11 presents the percentages of respondents' ratings on this set of items. Overall, respondents generally agreed or strongly agreed that beginning teachers' and their own teaching practice improved as a result of the training. They also generally agreed that their working relationship with other teachers has improved and that they have learned more about supporting beginning teachers and supporting colleagues. Respondents were more divided regarding whether they learned more about district or state standards, student assessments, and planning for professional development. Finally, respondents generally disagreed or strongly disagreed that they already knew how to use the mentoring strategies prior to the institute training. The following table does not compare OSU and OEA responses because we found that the means were similar for OSU and OEA training participants for all items except ''The mentor institute has helped me learn more about supporting experienced COlleagues." The mean rating for OSU was higher (mean=2.54) than the mean rating for OEA (mean=2.06) on this item. This finding reflects the greater emphasis on working with colleagues in the OSU Mentoring Institute compared to the OEA training, which focuses almost exclusively on mentoring beginning teachers. Impact of Mentoring Training on Teaching Practices i.' , " , Strongly Strongly Disaaree Disagree Agree Aaree Mean N The teaching practice of the teacher I pmaerntictoipreadtionimpinrotvheedMaesnatorreTseualtcohfermy 1% 10% 65% 24% 2.1 72 Professional Development Institute. My working relationship with other colleagues has improved because of my participation in - 17 59 24 2.1 76 the Mentor Institute. My own teaching practice has improved as a result of my participation in the Mentor - 4 51 45 2.4 76 Institute. The Mentor Teacher Institute has helped me learn more about: Supporting beginning teachers - 3 42 55 2.5 74 Supporting experienced colleagues - 7 56 37 2.3 75 Instructional strategies · 10 53 37 2.3 73 The district or state standards · 30 55 15 1.9 74 Student assessment · 23 52 25 2.0 73 Planning for professional development 1 25 55 19 1.9 73 I already knew how to use the mentoring strategies we learned in the Institute 25 55 18 3 1.0 73 beforehand. Conclusions In this report we shared findings from the second of a three-year evaluation of the Mentoring Professional Development Institutes. The evaluation findings were based on the survey that was administered to participants of the OEA and OSU Mentoring Institute during the four-year period beginning with the 1999- 2000 school year. The findings were discussed in terms of training participation and recruitment, mentoring assignments and activities, support for implementing mentoring strategies, and the impact of the training. A brief summary of these findings is presented below, followed by a set of recommendations for OCTP's consideration. • Principal and/or district administration support and commitment at times seemed weak. • The majority of survey respondents have mentored 1-3 beginning teachers although a number have still not had this opportunity. • Respondents also indicated that while most have mentored beginning teachers, the majority has not been assigned to more Residency Teams as a result of training. • Respondents also reported engaging in a variety of mentoring activities, the most common were observing their mentee during a lesson and giving feedback, planning lessons, units, and assessments, strategizing for communication with parents, and referring the mentee to other resources. • The large majority of respondents reported meeting with their mentee at least once a week or more; this happens most often after school. • The trained mentors are using the mentoring strategies they learned in the Institutes and that they are providing substantial support to beginning teachers. • Collaboration and follow-through are important to successful implementation of the mentoring strategies and that a number of participants reported having teams of teachers in their schools who were trained in the mentoring strategies. • Respondents generally rated the support they receive from others to implement the mentoring strategies fairly high. The actions that helped them implement mentoring strategies the most focused on carving out time and opportunities during teachers regular schedule to observe each other and meet to discuss teaching practices. • Important actions also included clear expectations about the use of mentoring strategies and the importance of mentoring for school improvement efforts and the need to "scale-up" the mentor training. Respondents indicated that they would benefit from more staff participating in the Mentoring Institutes, including principals. • Respondents want more support for mentoring either with time to collaborate with other teachers or for more compensation to conduct these activities. • Respondents perceive they are much better teachers and better prepared to support beginning teachers. Recommendations Evaluation findings indicate that the Mentoring Institutes are beneficial to both the participants of the training and the beginning teachers (and colleagues) who they mentor. Findings also illuminate several areas where the Mentoring Institute or the implementation process in schools and districts could be strengthened. Based on these findings, several recommendations are suggested for OCTP's consideration. • Increase the number of teams of trained mentors within districts or schools to promote collaboration and ongoing support for the use of mentoring strategies. Findings indicated that teachers benefited from having other teachers in their schools go through the mentor training because they were able to help each other implement the mentoring strategies. • Promote principals' participation in the Mentoring Institute. Respondents indicated that the implementation of mentoring strategies would be enhanced even further by more support from principals and even their participation in the mentoring training. • Promote the adoption of some or all of the mentoring strategies school- or district-wide. • Promoting the use of mentoring/coaching strategies school- or district-wide would allow teachers to collaborate and support each other as they continue using the mentoring strategies with new teachers and experienced colleagues as well. • Promote adequate time to engage in mentoring activities, such as observing classrooms and meeting to discuss observations and other teaching practices. • Promote the benefits of assigning beginning teachers to trained mentors. • Promote school and district policies that will ensure adequate numbers of trained mentors to serve as resources for beginning teachers as well as experienced colleagues throughout the state. • Promote the assignment of trained mentors to beginning teachers. In conclusion, the evaluation findings indicate that the mentor training positively impacted participants of the training and that the new teachers whom they mentored were positively impacted as well. There is generally support for the use of these mentoring strategies but some respondents still reported not having the opportunity to work with beginning teachers. However, there are aspects of induction programs that we have not yet addressed in this evaluation and may be potential topics for the third year evaluation. For example, how well are Mentoring Institute trainees prepared to provide advice to beginning teachers about serving diverse student populations and students with a considerable learning needs (e.g., struggling readers). Many beginning teachers leave because they are overwhelmed by the challenging circumstances in which they find themselves, such as in high-poverty, low-performing schools, or in schools where administrators do not support teachers. We also know little about how stipends or other motivational factors influence teachers to participate in the training or on the Residency teams. These are potential questions for future consideration. EDUCATION LEADERSHIP OKLAHOMA HIGHLIGHTS FROM 2003·2004 NATIONAL IMPACT Independent Studies Confirms Effectiveness of National Board Certification ~ An independent study of 600,000 student records from students in North Carolina schools has found that children learn more from National Board Certified Teachers. The study found: • Teachers who achieve National Board Certification do a measurable better job in the classroom • Students of NBCTs improved an average of seven percent more on their year-end math and reading tests than students whose teachers attempted but failed to gain certification ~ Another study demonstrates that National Board Certified Teachers increase student achievement. The study drew upon more than 200,000 students from 14 Arizona school districts with gain scores calculated from each student over four consecutive years • The gains made by students of NBCTs were more than one month greater than the gains made by the students of non-Board certified peer teachers. STATE IMPACT Total Number of Oklahoma National Board Certified Teachers as of Fall 2004 - 858 Oklahoma ranks eighth in the Nation • 355 Candidates were funded by the Oklahoma Legislature • A federal subsidy secured by the OCTP in the amount of $88,000 provided additional candidate support • 211 First Time Candidates completed the process • 179 Advanced Candidates participated in the retake process • 230 Candidates currently await their scores • 75 of Oklahoma's 77 Counties have either a National Board Certified Teacher (NBCT) or a candidate • 168 of the School Districts in Oklahoma currently have a NBCT • An additional 48 districts have candidates • The Legislature continues to award a $5,000 annual stipend to NBCTs who are fulltime classroom teachers in an Oklahoma school ELO CANDIDATE SUPPORTfTECHNOLOGY • ''Trainer of Trainers" for Durant ELO Training • Two two-day summer orientation programs in Durant for new candidates (in conjunction with Southeastern State University) • Distance Learning 2-Day Workshop - Eight regional Sites Statewide • Tapes of Distance Learning Workshop sent out to candidates who could not attend • Eight regional coordinators who coordinate and facilitate support and mentoring • Partnerships with Career Tech, Oklahoma Universities and Regional Professional Development Centers • A pre-candidacy video funded by a recruitment grant from the State Farm Foundation was distributed • Assessment Center Workshops - Two regional sites ELO GOALS • Recruit and attract teachers in under-represented regions and low-periorming schools • Promote benefits of using NBCTs to ensure high quality teaching such as: leaders on school, district, or state improvements committees • Determine the effectiveness of NBCTs in terms of student achievement • Have a National Board Certified Teacher in every school district in Oklahoma • Secure funding for 500 annual scholarships for National Board Candidates • Expand the mentoring program for National Board Certified Teachers • Provide the guidance and support needed by Candidates with specific feedback and in-depth study on standards and how they are incorporated • Provide leadership training for NBCTs RESULTS OF SEOL EVALUATION In April 2002, the Southwest Educational Development Laboratory (SEDL) was contracted to evaluate the ELO program. The purpose of the evaluation is to examine how National Board Certification of teachers in Oklahoma addresses quality teaching and advances school improvement efforts. What was the impact of becoming certified? Survey respondents overwhelmingly indicated that NBC is an excellent professional development experience and that they are much better teachers as a result. This was especially true for teachers with 4-10 years experience compared to those with 21+ years of experience . Teachers with fewer years of teaching experience and those teaching in elementary/middle schools and low periorming schools reported experiencing more personal outcomes as a result of certification. In comparison, more teachers in urban, larger schools reported experiencing professional outcomes. Survey respondents generally reported that certification improves their pedagogical skills, their knowledge of curriculum and subject area, and their ability to enhance and assess student learning. However, responses also indicate that the benefits of certification are still largely localized to NBCT's own classrooms. 100% -,-----------------------,,,.----......-,,..-----, 4-10 .11-20 021+ 20% -t------'------- 0% 1% 2% O%+---~--c=~-~- Did not impact your practice Made you a somewhat better Made you a much better teacher teacher Figure 1 Impact of Certification by Years of Teaching Experience Overall, 70% of NBCTs who responded to this survey indicated that certification made them a much better teacher. Thirty percent reported that the certification process made them a somewhat better teacher and only 1% said it did not impact their practice. While the responses generally were similar for teachers in different types of schools and those teaching different subject areas, the responses did vary by reported years of teaching experience. As illustrated in Figure 1, more teachers with 4-10 years of experience believe they are much better teachers as a result of certification compared to those with 21 + years of teaching. SEDL also asked NBCTs to rate the helpfulness of the certification process regarding 17 aspects of teaching (l.e., not at all helpful, somewhat helpful, very helpful). A summary of responses to these items is illustrated in Figure 2. Improving my self confidence 0/0 1'7 67% " ! 6:~% I f,( 0/0 -j -, a - j ,)RO/o -j '" I ')70/0 -j I ')')0/0 II ,),Ofn ". ,., I ') a I 490 I1 47% I " :.i, I 46% I I~' I 430/0 II 4 % II ,C 0/0 " <:> I 010 I 0/0 I Developing a stronger curriculum Recognizing individual differences in students and adjusting accordingly Using student assessments effectively Articulating learning goals for students Making multiple paths to learning available Mentoring other teachers Working with colleagues on instruction USing research to improve my practice Understanding how students learn Understanding of links to other subject areas Increasing knowledge of my subject area Working with parents Knowing how to impart subject area to students Trreating my students equitably Connecting to state standards Taking advantage of community resources 0% 20% 40% 60% 80% 100% Not At All Helpful • Somewhat Helpful oVery Helpful Figure 2 Helpfulness of NBC for Improving Teaching Responses indicate that certification is most important for improving teachers' self-confidence and developing a stronger curriculum for students. Certification is least helpful for taking advantage of community resources and connecting to the state standards. NBCTs overwhelmingly indicated that NBC is an excellent professional development experience and that they are much better teachers as a result. This was especially true for teachers with 4-10 years experience compared to those with 21+ years of experience. The findings also suggest that NBCTs experience different outcomes of National Board Certification depending on their years of experience, the schools in which they teach, and the subjects they teach. Teachers with fewer years of teaching experience and those teaching in elementary/middle schools and low performing schools reported experiencing more personal outcomes as a result of certification. In comparison, more teachers in urban, larger schools reported experiencing professional outcomes. Oklahoma National Board Certified Teachers and Education Leadership Oklahoma Candidatesl Applicants Through 2003 Cimarron [ Texas r Beaver I Harper Woods Grant Ka y / warT r IOttawa Alfalfa 20 Nowata Craig 3 1 1 22 3 I 1 1 73 222 I 2 762 1 1 Osage 10151 449 Woodward - "- 5 18 Garfield Noble ;II~S\ 1594 Major 7126 221 I 241 I Dewey 1 5 Blaine Roger Mills \ 23 Lincoln Custer 1383 15101 Be2c2k2ham I Washita l - Oklahoma National Board Certified Teachers Education Leadership Oklahoma Candidates (2002-2004) Education Leadership Oklahoma Applicants (2005) McCurtain 43 Le Flore Comanche 5 3 r1, '----"-1 442918 Pushmataha I I I Stephens 851 Jefferson 333 MAY :t ~ 2011
Object Description
Description
Title | Professional Development AR2004 |
Authors | Literacy First. |
OkDocs Class# | T550.3 A615p 2003/04 |
Digital Format | PDF, Adobe Reader required |
ODL electronic copy | Deposited by agency in print; scanned by Oklahoma Department of Libraries 7/2011 |
Rights and Permissions | This Oklahoma state government publication is provided for educational purposes under U.S. copyright law. Other usage requires permission of copyright holders. |
Language | English |
Full text | T 550.3 A6l5p 2003/04 c.l Oklahoma Commission for TeacherPreparadon --PROFESSIONAL DEVELOPMENT ANNUAL REPORT 2003-2004 Student Success Through Quality Teaching LITERACY FIRST PROFESSIONAL DEVELOPMENT INSTITUTE EAST CENTRAL UNIVERSITY IMPACT • This past year 2,650 participants participated in Literacy First Training; with 1,557 in Phase I, 918 in Phase II, and 124 in Phase III Leadership training. This represents a total of 493 (90%) of the state's school districts participating and 19,261 participants since inception. • High quality content and learning documented by participants. • 65% said their greatest need is more practice and ?6% wanted additional training. • Prior to the training, most respondents stated they were familiar with the knowledge and skill areas, and a few reported having a solid understanding. Three months after the training, the majority of respondents moved from being familiar to having a solid understanding and the ability to apply the knowledge to their teaching practice. For example, 75% of Phase I & II teachers and 94% of Phase N teachers are using Literacy strategies in their classroom. • 60-63% of teachers reported that increased content knowledge in phonological awareness, phonics, vocabulary, fluency, and comprehension is helping them teach reading more effectively. REFLECTION • Many more principals attended the Literacy First, Phase III trainings. This was a result of a successful effort to recruit more administrators to attend the PDI. The principal is the key to creating the changes needed. With our Phase IV schools this became even more apparent as those schools in which there is strong leadership tend to have the greatest gains in student achievement. • Change takes time. The first level of change is bringing about an awareness of the learning content. Next is implementation of the new skills and strategies within the context of the present system or changing one, and finally the institutionalization of the practices into the culture ofthe school and system. • Phase IV schools data indicates gains in content and teaching strategies at a higher rate of implementation than non phase IV schools. Executive Summary Literacy First POI The purpose of this report is to provide the Oklahoma Commission for Teacher Preparation (OCTP) with evaluation findings pertaining to the quality and impact of the literacy Professional Development Institutes (PDls) offered across the state during the 2003-2004 school year. The goal of OCTP is to provide teachers with the training and support they need for students to achieve grade-level mastery of reading skills. OCTP partnered with Literacy First to provide literacy PDls to the teachers in state of Oklahoma. In Oklahoma, Literacy First offers four "phases" of services to elementary school teachers and administrators. These phases are: • Phase I and Phase" for kindergarten through sixth-grade teachers, • Phase III for school administrators, and • Phase IV, a whole-school literacy program. During the 2003-2004 school year, the Southwest Educational Development Laboratory's (SEDL) Evaluation Services (ES) unit evaluated Phase I and Phase" PDls, as well as the Phase IV program. Information was gathered on five levels of evaluating professional development (Guskey, 2000): participants' satisfaction with the PDls (level 1), participants' gain of new knowledge and skills (level 2), organizational support participants receive for implementation (level 3), participants' implementation of new knowledge and skills (level 4), and student outcomes (level 5). To evaluate the Phase I and Phase" PDls, follow-up surveys were administered to past participants. To evaluate the Phase IV program, current and past Phase IV participants were contacted. Current Phase IV participants completed surveys and interviews, and site visits were conducted. Past Phase IV participants completed follow-up surveys. Student test data were also examined to determine whether student achievement had increased at a rate different from a matched comparison group of non-Phase IV schools and the state average. Findings from the surveys, interviews, and site visits showed that most of the past Phase I and Phase" participants were satisfied with the PDls and gained new knowledge or skills through their participation in the PDls (levels 1 and 2). Teachers indicated that they received some degree of organizational support (level 3), implemented the techniques they learned (level 4), and saw improvements in student reading (level 5). Gains were evident at these last three levels from teachers and principals at Phase IV schools and also from teachers who completed the Phase IV school program last year (2002-2003 school year). Although most of the Phase I and Phase" Follow-up Survey participants rated content and quality of the PDls highly, participants that responded from the Phase IV schools indicated that content and quality of the Phase I and Phase" PDls varied widely. Further, most of the teachers in Phase IV schools attended Phase I and" PDls approximately two years ago, with the range of attendance spanning from 1 to 5 years. As revisions to the Literacy First program occur periodically, some of these teachers were not up-to-date on the changes. Teachers also noted that they received conflicting information on implementing Literacy First strategies. Finally, teachers expressed strong needs for further training and practice using Literacy First strategies, suggesting that the above factors may be affecting the successful implementation of the Phase IV program at their schools. Feedback from the Phase IV school administrators and teachers revealed that about one-third of the teachers in their schools were resistant to the Phase IV program, although almost all of the administrators supported the implementation of Literacy First Phase IV program 1. The reasons for teacher reluctance to implement the process included: I Guskey argued that organizational support is necessary for teachers to implement what they learned at professional development institutes. • The lack of a full understanding of the Phase IV process as a whole, • Perceptions of administrators as tending to push the implementation of all aspects of the program at once, and • A lack of clarity about the role and responsibilities of the Literacy Resource Specialists. This was echoed by LRSs as well. • Spread Phase I and" POI meeting days out to provide more practice days for teachers back at their schools and feedback sessions in the POls to reflect on and discuss difficulties experienced in using the strategies. • Offer an orientation session to Phase IV school staff at the beginning of their entry into the Phase IV program to clarify the objectives of the program, the roles and responsibilities of teachers, LRSs, and administrators, and a review of the content and major strategies and techniques they will be expected to implement. • Provide more focus on the strategies that administrators and LRSs report having difficulty with or using only occasionally (e.g., observation, feedback, modeling). • Provide more focus on strategies that teachers report having difficulty with (e.g., constructing Literacy Centers, conducting and using student assessments, classroom management issues). For student achievement, scores from third grade norm-referenced tests of reading ability in Oklahoma for 2002- 2003 school year were compared between Phase IV schools, matched comparison schools, and state average. Phase IV schools showed a higher gain than did comparison schools and the state average. Based on all of these findings, ES offers the following recommendations: Overview The Southwest Educational Oevelopment Laboratory's Evaluation Services (ES) recently completed its second year of the evaluation of multiple professional development institutes (POlS) that the Oklahoma Commission for Teacher Preparation (OCTP) offers to Oklahoma teachers. The purpose of this report is to provide OCTP with evaluation findings pertaining to the quality and impact of the literacy POls offered in Oklahoma during the 2003-2004 school year. LITERACY PDI GOAls OCTP provides professional development for Oklahoma teachers so that the teachers have an opportunity and the tools to grow as educators and professionals (Oklahoma Commission for Teacher Preparation, 2002). Emphasizing the importance of elementary school teachers' knowledge of what to teach as well as howto teach, OCTP partnered with Literacy First to provide professional development in reading instruction. The goal of Literacy First is to provide support and training to teachers to facilitate students' grade-level mastery of reading skills. Literacy First trainings focus on increasing educators' knowledge of reading processes, such as phonics and comprehension, and pedagogical strategies that use reading assessments to inform instructional decisions. THE LITERACY FIRST PROGRAM Literacy First, located in Mill Creek, Washington, offers a reading program for primary and secondary teachers nationwide (Literacy First, 1999-2001). Program goals are to have 85 to 90% of students reading at grade level by the completion of a three-year school wide implementation process. Literacy First's approach to increasing student reading achievement is to staff schools with competent teachers who explicitly teach reading in a diagnostic and prescriptive manner. Literacy First trains teachers in the skills needed to assess student reading ability and to use student assessment data for developing individualized reading instruction. They also train teachers in instructional competencies pertaining to the essential components of reading instruction. In Oklahoma, Literacy First offers four phases of literacy training. Phase I The Phase I POI is a five-day training, that focuses on instructing teachers in a comprehensive, balanced reading process and is offered to teachers across the state of Oklahoma. The training includes hands-on activities and opportunities for teachers to develop practical classroom assessments. The five days are divided into two sets of consecutive days. The break between the two sets of training days allows teachers to try the strategies in their classrooms that they learned during the first session. Phase II The Phase 1/POI is a three-day training that builds on the Phase I training. It instructs teachers in using flexible skill groups and developing literacy centers. It also addresses topics such as cooperative learning, collaborative planning, and guided reading. Ouring Phase II, teachers have opportunities to discuss problems or concerns with their implementation of Phase I strategies. Teachers who complete the Phase I POI can attend the Phase II POls. Both Phase I and Phase II POls provide participants with resources for developing meaningful and fun reading activities. Phase I and Phase II POls offer two levels of training each: (1) Kindergarten through second grade and (2) third through sixth grades. The Grades K-2 curriculum includes: • Phonological awareness • Phonics • Spelling • Comprehension The Grades 3-6 POI curriculum includes: • Pre-requisite information • Phonological awareness • Phonics • Spelling • Comprehension • Vocabulary Phase III The phase III POI is a two-day training for school administrators. Phase III training provides school administrators with strategies for facilitating teachers' implementation of Literacy First strategies. An overview of the Phase I and the Phase II POls that their teachers attend is also provided. Phase IV Phase IV is a three-year process that promotes reading instruction reform at the school level. A school's commitment to this process includes the school administrator's support of their teachers' implementation of Literacy First strategies, and that at least 80% of their teachers have attended Phase I and Phase II POls.2 The commitment from the schools allows Literacy First program staff to collaborate with the schools to expand the impact of the Phase I and Phase II trainings. Schools that agree to participate in the Phase IV process are assigned a Literacy First consultant and are asked to appoint one teacher at the school to become their Literacy Resource Specialist (LRS). Once becoming a Phase IV school, principals and LRSs attend leadership trainings that are held six times each year during the three-year process. At leadership trainings, principals and LRSs discuss their roles in faCilitating teachers' implementation of 2 The Phase IV program itself does not contain trainings for teachers. The Phase I PDI and the Phase II PDI are considered to be the teacher trainings for Phase IV schools. Literacy First strategies, and have opportunities to collaborate with others (Literacy First consultants and other Phase IV school administrators and LRSs) on strategies for increasing implementation. To initiate the Phase IV process, Literacy First consultants conduct a comprehensive school assessment. From this information, consultants develop a Strategic Reading Plan in consultation with the principal and LRS. Consultants facilitate and support the implementation of schools' Strategic Reading Plans by developing the capacity of the principal and LRS to maintain the infrastructure and culture necessary to accomplish the goal of helping students to read at grade level. Consultants visit their assigned schools six times a year, during which time they analyze reading instructional practices, demonstrate reading lessons, identify intervention strategies for under-achieving students, model teacher conferencing procedures for the principal, and serve as a catalyst and facilitator of the change process. The LRS assists in the implementation of the process at the school by collaborating with the consultant, the principal, and the teachers. The Phase I, Phase II, and Phase III POls have been offered in Oklahoma through East Central University's continuing education program since March 1998. The POls have been offered at major cities in Oklahoma, allowing teachers in all regions of the state to participate. To accommodate teachers with busy schedules, POls are offered on many different dates. During the 2003-2004 academic year, a total of 32 Phase I POls and 19 Phase II POls were offered. More than a total of one thousand teachers and administrators participated in the Phase I and Phase II POls during this academic year. However, only four Phase III POls were offered during the 2003-2004 academic year due to small enrollment. During the fall of 2003, 14 schools started the Phase IV program, and in the spring of 2004, six schools started the program. Of those schools who had already started the process before the 2003-2004 school year, five continued the program for a second year and 13 schools continued for a third year. Three schools left the program before they completed the three-year process due to their districts' lack of funding. The number of 2003-2004 Phase IV schools totaled 38 in 2003-2004 school year3. Summary of Evaluation Findings The following chart displays participant respondents in the increased knowledge in relation to instructional outcomes. Instructional Outcomes , Instructional Outcomes TPehaacsheerIsV INVoTne-Pachhaesres My increased content knowledge in phonological awareness, phonics, vocabulary, fluency, and/or comprehension is helping me teach reading 63% 60% more effectivelv. iInadmividbueattleizreadbliensthtraunctiboenf.ore to use student assessment data to provide 61% 49% I am able to assess students' reading skills more accurately than before. 59% 46% I am able to literacy centers more, more efficiently, and more effectively. 59% 38% sMtryatiengstierusctiIolnearhnaesd.not changed; I have not incorporated the Literacy First 4% 3% Total Number of Respondents " ., . 56 37 To document Literacy First participant content knowledge and instructional strategy improvement, the Literacy First staff constructed 20-item tests of the POI participants' knowledge of the content offered in these sessions, administering the tests at the beginning of the institute sessions and at their conclusion. 3 Of the 38 schools, 2 schools are counted twice as each school runs two programs for different grade levels. As shown in Table 1, during the fall 2003 POls, 270 grades K-2 teachers completed the pre-test and 253 completed the post-test. Of those, a total of 230 teachers completed both pre- and post-tests. For the grades 3-6 POI participants, 156 teachers completed the pre-test, 151completed post-tests, and a total of 136 completed both pre-and post-tests. Grade K·2 Sessions4 Figure 1 shows the percentage of teachers who correctly answered items on the pre- and post-tests for the fall 2003 and spring 2004 institute training sessions combined. Percent of Teachers Correctly Answering Test Items (Fall 2003 and Spring 2004 Combined) .. CII .c Q9 E; , Z Q10 I E .2.1. Qll t; Q12 .C.I.I K-2 Teacher Percentage Correct by Item Total ~--~--- ------~- ----- ~-~-----~-~-~------~-----~~------- - --~--~ I Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 ,bo I I I 82 ~ ", I <;4 j j 84 I " I <;7 I Ij -I , , t;LI I I I 85 j l~ , I I 'f ,ko f9 -t I 93 , I IdR I I 1 I 81 I, I I I ':I" I 1 , I I I I 80 I I , , I '7~1 j 92 I , 'Lln I I I J I I I 62 ! I , , I II ; , I I Iko I b9 I I 8f -I , I I ':It; l I I -I I I I I 57I 171 -t 82 " I <4 1 "j I I 93 -l , I' I I' , , 12 -f r I "'( '.,n , i ,60 I 41 60 i I .,~I I , , 85 I I 1 , ~1 , I II 86 1 I I ! ,ko I I I 88 III I, I I l.pre-te~1 I Post-te?!JI I I 100 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 o 10 20 30 40 50 60 70 80 90 __~ ~_ ___ ~~~:cen~___ I As can be seen in the above Figure, the teachers' knowledge on the topics was lowest at the time of the pre-test on items 15 and 16, and highest on items 8, 10, and 13. For all but one test item, the percentage of teachers correctly responding to the items increased from the pre- test to the post-test. Highest gains were seen on items 14 and 18 (59%), item 1 (53%), and item 7 (48%). Grade 3·6 Sessions The following graph shows the percentage of teachers who correctly answered items on the pre- and post-tests for the fall 2003 and spring 2004 institute training sessions combined. Percent of Teachers Correctly Answering Test Items (Fall 2003 and Spring 2004) Grades 3-6 Teacher Percentage Correct by Item Total Q1 Q2 Q3 Q4 Q5 Q6 Q7 .. Q8 Q) .Q Q9 E ~ Q10 E •~.. Qll .•.. r:-3 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 0 iiiiiiiiiiiiil 9 • Pre-test iii Post-test ••• _ •• _iiiil88 20 40 60 80 100 Percent THE PHASE IV PROCESS The emphasis of Phase IV of the Literacy First program is on implementing the knowledge and skills teachers gained through attending Phase I and Phase II institutes. The information gathered through surveys and interviews indicated that teachers had gained knowledge and skills from the Phase I and II trainings. They reported that they were primarily pleased with the support they were receiving from their administrators, Literacy First consultants, and the LRSs. They further noted that their instructional practices were changing, which was subsequently impacting their students' reading performance. While teachers were generally satisfied with the support they received, many also noted that they received "mixed messages" from their LRSs, administrators, and consultants as to what they were supposed to be doing, and how. Some also mentioned feeling that the program was too prescriptive or rigid, not allowing for flexibility related to individual student needs or teaching styles. Perhaps the most stated need by teachers was for further training and/or more modeling of Literacy First techniques. Even principals and LRSs voiced their need for further training or "refreshers" in these areas as well as in observation and feedback skills, and developing and providing resources to their teachers. Year 4 School Teachers. Teachers at the schools that completed the Phase IV program at the end of the 2002- 2003 academic year were also asked about their continued use of Literacy First strategies. As seen in Table 15, 58- 78% of teachers reported currently using the strategies "all the time." Only a small percent of teachers indicated that they were not using the strategies. Year 4 Teachers' Current Use of Strategies Literacy First strategies Not at Some of All the Total All the Time Time (N) Assessment 1% 21% 78% 90 Use assessment data to plan lessons 0% 36% 64% 90 Phonological awareness 4% 32% 64% 89 Phonics 6% 28% 66% 89 Vocabulary 1% 41% 58% 88 Comprehension 1% 24% 75% 90 Fluency 3% 30% 67% 91 Flexible skill groups 2% 31% 67% 91 Literacy centers 4% 22% 74% 91 Note: Total number of surveys returned = 91; response rate = 34% The vast majority of Year 4 teachers that responded to this item (96-99%) also reported that they intended to continue using Literacy First strategies in the future. Comments by these teachers indicate their value of these strategies, and included: • I have various Literacy Centers. I will never go back to the traditional method of teaching. • I let the assessment determine the instruction; it saves a lot of otherwise wasted time. • Flex group time gives the teacher and students the time necessary to concentrate on specific needs. • I will continue to use Literacy Centers. I have also added centers for math. They give me more small group and one-on-one time. • I use assessment to guide instruction and group my students accordingly. I will maintain this - it works! In summary, more teachers in all years of the Phase IV program implemented Literacy First strategies at a higher level this year than last year. Further, in the third year of the program, a larger percentage of the teachers maintained their level of strategy use than did first or second year teachers, and under five percent of all teachers indicated using the Literacy First strategies less this year than last. A large majority of Year 4 teachers also continued to use Literacy First strategies even though they completed the program last year. Because of the benefits they have experienced, many of them expect to continue using them in the future. These findings may also reflect that through the routine use of many of these strategies, teachers integrate them into their instructional practices to the extent that they remain usable and valued methods for reading instruction even after a school has completed the Phase IV program. Conclusions The findings presented in this evaluation report are a synthesis of data collected from Phase I and II Literacy First POls as well as from the Phase IV program. They are presented to our client, the Oklahoma Commission for Teacher Preparation, to assist in their understanding of the quality and impact of the literacy professional development currently provided to Oklahoma teachers. The following is a summary of the findings and recommendations based on these findings to assist Literacy First in considering next steps for increasing the effectiveness of the program for Oklahoma teachers. PHASE I AND /I POls • Phases I and" are multiple day workshops where groups of teachers from different schools and districts are presented with information and skills focused on reading instruction. • The content of Phase I is a basic introduction to literacy instruction. Phase" workshops focus on implementing knowledge and skills that teachers acquired from Phase I. • Survey findings indicated that most of the teachers who attended Phase I and II workshops during the 2002-2003 academic year valued the content presented and believed it to be useful in improving reading instruction. • They agreed that they had learned new knowledge and skills from attending the POls, and that they were implementing them to various degrees. • Teachers who were currently in a Phase IV school tended to implement the knowledge and skills to a higher degree than teachers in non-Phase IV schools. • The majority of non-Phase IV teachers also reported that they implemented a fair portion of the content and strategies they gained at the POls. • Teachers who had taken the Phase I and II POls from Phase IV schools had more consistent support than those from non-Phase IV schools. 91% of the teachers from Phase IV schools perceived their administrators as more highly involved. • Large majority of non-Phase IV schoolteachers reported their administrators to be "interested" and "supportive," even if not being actively involved in assisting these teachers. • Both Phase I and II past participants indicated that their training had led to instructional changes, in that they were more knowledgeable about the various literacy topics, were more able to assess students' reading abilities, and use the assessment information to better individualize their instruction. • Teachers noted that they were perceiving student impacts such as improvements in reading ability, more interest in reading, and their enjoyment of Literacy Center activities. RECOMMENDA TlONS Literacy First achieves breadth in offering a large number of Phase I and II institutes each year to teachers in Oklahoma. Depth is also attained through focusing on a wide variety of concepts and skills related to literacy learning. In meeting the Oklahoma requirements for professional development, Literacy First provides 30-45 hours of training, separated into two blocks (Phase I) to allow teachers to practice the knowledge and skills they are learning. In addition, the three days of Phase II provides teachers with a forum for further reflection on the results of their implementation of program content back at their schools as well as further training in Literacy First skills. Teachers, however, continue to express the need for more practice and training to better implement the Phase I and II knowledge and skills at their schools. • Literacy First staff might consider restructuring Phase I and Phase" institutes to allow teachers more practice time at their schools in between POI meeting days, instead of only one block of practice between the two sections of the Phase I institute and none in the Phase II institute. • Literacy First staff might consider incorporating a school-wide one-day orientation into the Phase IV program. This orientation would occur prior to getting the program underway, and focus on an overview of school staff roles in implementing the Phase IV program (e.g., administrator, LRS, and teachers), a review of highlights of content and major strategies and techniques, and time for questions and answers by school staff. • The role and responsibilities of the LAS, the Literacy First Consultant, and the administrator(s) should be explicitly laid out, so that all staff will have an understanding of the type of support they can expect from each. OEAMATH PROFESSIONAL DEVELOPMENT INSTITUTE • OEA focused on recruiting districts rather than individual teachers for the Professional Development Institute (PDI). As a result, participation remained at a high percentage with school wide change and improvement expected. • 83% of participants felt more knowledgeable about the standards and were better able to integrate them into their teaching practice. • The highest gains were in Mathematical Reasonings and Proof, and in Using Writing About Mathematics. • Participants report that the PDI was the best professional development of their careers, and indicated that the skills and strategies would be transferred to their classrooms. IMPACT REFLECTIONS • These trainings validated what research has been telling us: long-term professional development utilizing a quality curriculum will have significant impact on teaching practice. Initial training with multiple follow-up sessions shows the importance of allowing participants time to reflect on and practice new knowledge and strategies. • Keeping current in the teacher's content area is important to the knowledge level of teaching. Students benefit from teachers who understand their content. • Administrators who participated in the training indicated they felt more prepared to encourage their teachers to implement the new math strategies from the PDI. OSU/STILLWATER PUBLIC SCHOOLS CONSORTIUM MATHEMATICS PROFESSIONAL DEVELOPMENT INSTITUTE IMPACT • Teachers reported an increased confidence in their ability to teach for conceptual understanding and gains in useful teaching strategies. • Positive feedback from school administrators was received as they assessed the impact on teachers and on student learning - "As a principal and former classroom teacher of 20+ years, I am always open to new and improved ways of learning. I have witnessed students with whom we have had trouble in the past become excited about math class. They appear confident in their new found ways of learning. I was very impressed with the new attitudes that seem to be surfacing among the students toward learning ." • Videotaping lessons are powerful evidence of teacher practice, even though initially there was resistance - "It was a wonderful experience .. .! learned a lot from watching not only myself but the others as well. I enjoyed the feedback from the other teachers because it helped me realize the good things I have been doing." • The largest gains for participants occurred in the area of using mathematical investigation as a method for student learning. REFLECTIONS • Having quality trainers who have implemented the Connected Math Program curriculum was vital to the learning. • Developing the capacity for understanding conceptual learning will require extended time. • Respondents indicated that the instructional content was useful, the organization was excellent, the materials were very helpful, the presentation was stimulating, and the involvement of the participants was excellent. • Teachers need resources to teach problem-based mathematics. They also need a book for each student, which was limited during the PDI. Executive Summary OEAlOSU Math POI The purpose of this report is to present evaluation findings to the Oklahoma Commission for Teacher Preparation (OCTP) regarding the effectiveness of the Mathematics Professional Development Institutes (POls) for FY03-04. The Mathematics POls are designed to provide middle school mathematics teachers with instruction and practice in implementing classroom strategies, which is intended to improve standards-based mathematics performance of middle school students in Oklahoma. The POls used the Connected Mathematics Project (CMP) curriculum, which is student-centered, hands-on, and is organized around several major themes such as the "big ideas" of mathematical concepts, making connections among different mathematics topics in meaningful ways, and teaching for student understanding by guiding students to use discovery methods in practical problem-solving situations. Two institutions, the Oklahoma Education Association (OEA) and Oklahoma State University (OSU), conducted mathematics POI's for the state. OEA held POls in Chickasha, Woodward, and Ardmore, Oklahoma. OSU's POls were located in Ada, Stillwater, and Tulsa, Oklahoma. OEA completed two of their POls in the Spring of 2004, and OSU completed only one. Due to the low number of participants at the one institute that OSU completed during the spring, and the even lower response rate to the end-of-institute survey, findings focus primarily on the OEA POI's, although a discussion of the OSU POI participants' responses is also provided. OCTP contracted with the Southwest Educational Development Laboratory (SEOL) in 2002 to evaluate the mathematics POls. This report presents findings from the second of a three-year evaluation. The evaluation design for the second year consisted of a survey of mathematics POI participants and observations of several OEA and OSU training sessions. The evaluation sought to address the following questions: 1. Did participants perceive the mathematics institutes to be valuable and of high quality? (Participant Reactions) 2. What knowledge and skills did participants gain as a result of the mathematics institutes? (Participant Learning) 3. What aspects of organizational support do teachers perceive they need to facilitate the application of their new skills and knowledge? (Organizational Support) 4. How do participants expect to use their new knowledge and skills following the institute? (Participants' Use of New Knowledge and Skills) 5. What impacts to student learning do teachers expect to see based on the implementation of the knowledge and skills learned at the POls? (Impact on Student Learning) Key Findings Key findings were based on survey responses from 60 of 100 teachers who participated in the OEA and OSU 2003-04 Mathematics POls. Participant Reactions ~ Teachers found the institutes to be valuable and well organized. Respondents: • rated the institutes high on content covered, organization, presentation, materials, and participant involvement. • gave highest ratings to having opportunities to share ideas and learn from other participants and to the instructors' knowledge. Participant Learning ~ Teachers increased their knowledge and skills after participating in the institutes. Respondents: • reported more confidence in their ability to teach for conceptual understanding. • reported increased ability to use diagnostic assessment as a tool to improve student learning. Regarding the learning of math content, respondents' highest gains were in using mathematical investigations and student discovery as methods for student learning. Administrators' Learning Administrators received valuable information and skills from their institute training. • Administrators gave highest ratings to gaining valuable knowledge about their teachers' math training and to feeling more able to support their teachers' implementation of the math curriculum that they are learning at the POls. Participants' Use of New Knowledge and Skills )- Teachers increased the use of knowledge and skills learned at POls. Respondents: • reported increases in their use of new knowledge and skills. • reported that they were more prepared to teach geometry and spatial sense and to use writing about mathematics in their classroom instruction. Impact on Student Learning ~ Teachers' changes in instructional practices are expected to impact their students. Respondents: • expect positive impacts on their students' performance. • felt optimism for using their newly acquired strategies for improving student learning in mathematics. Key Recommendations Respondent Recommendations ~ Teachers expressed the desire for more training, more planning time, and more time to collaborate with other teachers. They also perceived a need for more direct connection of the curriculum with the OKJNCTM PASS standards during the institute. Evaluator Recommendations ~ Promote the use of content tests for assessing the effectiveness of the math training in improving teachers' content knowledge, and for helping the trainers to understand what concepts teachers need additional help with learning. )- Promote the importance of the administrators' participation in the POls as school leaders are often key in the successful implementation of new instructional strategies or approaches aimed at improving student performance. )- Extend the evaluation of the Mathematics POls to include a more specific examination of the effects of teachers' implementation of POI knowledge and skills. OEA Mathematics POI Evaluation Findings This section includes evaluation findings from the OEA Mathematics POI. Findings are organized according to the five evaluation questions and are based on the post-institute surveys, teachers' mathematics pre-and post-content assessments, administrator survey responses, and observations of a sample of institute training sessions. OEA Survey Respondents • Site 1: Woodward (n=22) • Site 2: Chickasha (n=38) • Site 3: Ardmore (n=32) Site Pa#rtiocfipPaOnIts i #CoofmSpulervteedys ResRpaotnese Ardmore 32 14 44% Chickasha 38 26 68% Woodward 22 16 73% Total 92 56 61% OEA Participants I Reactions This section discusses findings on participants' reactions to OEA PDls that were collected using two methods: (1) a summary of responses from the post-institute survey and (2) a summary of observations made during two days of training in January 2004. Post-Institute Survey Responses In a survey administered at the end of the institute, participants were asked to rate their perceptions of the quality of the institute they attended on a scale of 1 to 5 as to its content, organization, presentation, materials, and participant involvement. Table 2 displays their responses. Survey respondents' overall reactions to the institute were positive. The majority of the respondents indicated that the institute content was very useful (57%), the organization was excellent (70%), the materials were very helpful (63%), the presentation was stimulating (66%), and the involvement of the participants was excellent (73%). Percentages of Overall OEA Institute Ratings Of little use 2 3 4 Very useful Content 0% 0% 5% 38% 57% Poor 2 3 4 Excellent Organization 0% 0% 4% 27% 70% Boring 2 3 4 Stimulating Presentation 0% 0% 5% 29% 66% Ohf eliltptle 2 3 4 Very helpful Materials 0% 0% 7% 30% 63% Poor 2 3 4 Excellent InPvaorltviceipmaenntt 0% 0% 4% 23% 73% The Mathematics POI participants were also asked to indicate whether they agreed or disagreed, on a 6- point scale, to statements regarding specific characteristics of the institute at the end of the training. These characteristics were related to the structure and organization of the training (e.g., whether objectives of the institute were clear, trainers were experienced, and active learning opportunities were provided). The following table presents the percentages of responses for each category: Percentages of Specific OEA Institute Ratings c,r' .~ , '" Strongly ,,' ". Strongly Il" Disagree 2 3 4 5 Agree The objectives were clear. 0% 4% 2% 13% 34% 48% tThheeoibnjsetcruticvteosr.s adequately addressed 0% 0% 4% 9% 23% 64% tIhnestrcuocntotersnt.were knowledgeable about 0% 0% 2% 2% 21% 75% The instructors had experience implementing strategies being 0% 0% 2% 2% 23% 73% discussed. aTchteiveselsesaiornnisngi.nvolved participants in 0% 2% 2% 7% 18% 70% pTrhaectiscees.sions provided opportunities to 0% 2% 2% 5% 20% 71% iPdaeratiscipaanndtsleahrandfroopmpoorttuhneirtiepsartticoipsahnatrse. 0% 0% 0% 7% 13% 80% Itewaochueldrsr.ecommend this training to other 0% 2% 4% 7% 27% 60% OSU Mathematics POI Evaluation Findings This section includes evaluation findings from the OSU Mathematics POI. OSU provided four POls during the 2003-2004 academic year. The POI offered in Ada, Oklahoma is the only one that was completed by the time of this report writing. Therefore, only the Ada POI data are summarized. Findings are organized according to the five evaluation questions and are based on the post-institute survey, the mathematics content pre- and post-assessments, and observation of an Ada institute training session. The post-institute survey asked for participants' perceptions of the quality of the training, participant learning, organizational support for their implementation of participant learning, expected use of new knowledge and skills, and the expected impact on participants' students. Overview of OSU Survey Respondents Oemographics and background information were collected from the OSU POI participants using the Demographic Survey. These data are reported for the following OSU mathematics POI participants (n=45). • Site 1: Tulsa (n=8) • Site 2: Ada (n=8) • Site 3: Tulsa IV (n=10) • Site 4: Stillwater II (n=19) Summary of OSU (Ada PDI) Findings The four Ada POI respondents indicated that overall, the institute content was useful to them, its organization was excellent, the materials were very helpful, the presentation was stimulating, and the involvement of the participants was excellent. Positive ratings were also given for having clearly and adequately addressed training objectives, involved participants in learning activities, solicited feedback, and provided opportunities for participants to share. Comments and content test results indicated that these participants gained new knowledge and strategies for teaching mathematics, and expected that they would change their instructional practices based on what they had learned. They also expected to see positive changes in their students' attitudes about learning math, confidence in their math ability, and performance on math assessments. Indeed, these reports of increased confidence in their ability to teach for conceptual understanding and gains in useful teaching strategies, illustrates the value of modeling instructional practices that participants can use with their own students as observed by the evaluator during the third day of training in the Ada institute. Similarly, having POI participants work in groups and share their problem-solving methods, tangibly demonstrated how teachers could structure mathematical investigations for their own students. Given that these findings are based on only a small number of respondents, it is premature to make any definitive statements about the progress of OSU's POls toward attaining their goals. Conclusions This report presented evaluation findings regarding the quality and impact of the OEA and OSU Mathematics POls for Oklahoma middle school mathematics teachers offered during the 2003-2004 school year. These Mathematics POls represent the first in a three-year program. Findings from this first year of the Mathematics POls addressed five evaluation questions: (a) did participants perceive the mathematics institutes to be valuable and of high quality; (b) what knowledge and skills did participants gain as a result of the mathematics institutes; (c) what aspects of organizational support exist at their schools to help or hinder the application of teachers' new knowledge and skills; (d) how do participants expect to use their new knowledge and skills following the institutes; and (e) what impacts to student learning do teachers expect to see based on the knowledge and skills they learned at the POls? A summary of evaluation findings is presented below according to these five questions. Because only one of the OSU institutes had taken place at the time this report was written, the majority of findings are based on participant responses to the OEA institutes. Participants' Reactions Survey respondents' overall reactions to the institutes were positive. They indicated that the institute content was useful, the organization was excellent, the materials were very helpful, the presentation was stimulating, and the involvement of the participants was excellent. They also provided positive ratings in terms of participant involvement in the training, receiving instructive feedback, and having opportunities for sharing with other teachers. Sixty percent of respondents strongly agreed that they would recommend the training to other teachers. The evaluator's observations made during one of the OEA POls reflected the survey findings. It was noted that the trainers used practical experiences to demonstrate conceptual and/or abstract thinking. Participants were guided through ways to introduce, clarify, and review concepts, which connect them to real-world experiences. In addition, the trainers modeled cooperative methods of learning that allowed participants to collaborate in pairs or teams to develop problem-solving strategies and find solutions to the problems. In both OEA and OSU POls, trainers modeled the instructional strategy that POI participants were expected to use in teaching their students. Participants'Learning Findings from surveys, content tests, and observations indicate that POI participants appreciated and valued the institutes and expected that the instructional strategies and skills they learned will be useful for instructing middle school students. Just over half of survey respondents strongly agreed that they had gained useful knowledge and 55% strongly agreed that they had gained useful strategies for teaching mathematics. Respondents reported that after the training they were most prepared to teach for conceptual understanding but least prepared to assess student knowledge and create standards-based lesson plans. This reflects the POI focus on mathematics concepts in the first year and the future focus on assessments and lesson plans in the second and third years. In terms of how prepared teachers were to apply specific skills and content knowledge in the classroom prior to and at the end of the math training, respondents generally felt more prepared in all areas, with largest gains for using mathematical investigations as a method for student learning, and learning through student discovery. With respect to teaching specific topics, largest gains were seen in teaching mathematical reasoning and proofs, writing about mathematics, and geometry and spatial sense. For many respondents, their perceived gains in content knowledge were corroborated by assessed gains in their content knowledge through a mathematics pre- and post-content test taken at the beginning and the end of the training. However, the gains were generally small. The highest percentage of participants experienced only a slight gain of 1 point in pre- to post-test scores. Although just under half of the participants had gains of 2 or more points, 16% did not experience any gains and 10% had lower scores on the post-test than on their pre-tests. These findings were perplexing and somewhat disconcerting given the emphasis on improving teachers' content knowledge in the first year of the POI. However, the POI vendors, through informal communications with the evaluators, indicated that the importance or relevance of these contents tests were minimized when tests were administered to teachers. From this we might conclude that participants may not have performed as well as they could have if they were aware of the role the tests may play in determining the effectiveness of the program. Nevertheless, more investigation is needed to determine the extent to which teachers are truly learning about and improving their knowledge of various mathematical concepts. Organizational Support Regarding the kinds of additional support that might help teachers implement the knowledge and strategies they learned in the institutes, more computers, calculators, mathematics manipulatives, and Internet access were mentioned. Respondents also noted a desire for more opportunities to observe other teachers or trainers in classroom settings, and for networking structures to facilitate collaboration and support for sharing ideas and information with teachers they met at the institutes as well as teachers at their schools. Other comments included having more teachers at their schools trained in the math curriculum and attending follow-up trainings. Administrators reported that they felt more able to support their implementation of the math curriculum as a result of participating in the administrator training. After the training, they were stated that they most prepared to encourage their teachers to implement the new math strategies from the POI, to provide teachers with the resources necessary to effectively teach the math curriculum, and to promote the effort to improve student math performance in meetings. These administrators also perceived small changes in teachers' behaviors beginning to occur. They also mentioned that they were providing various methods of support, such as helping with curriculum alignment and ensuring that teachers have the necessary materials to implement the math strategies. Most responded that student impacts had not occurred yet because it is still too soon to tell or predict. Several challenges to implementation were cited by the administrators, including insufficient funds for textbooks, time for teachers to plan lessons and implement training strategies in the classroom, and time during the school day for teachers to share ideas with one another. Use of New Knowledge and Skills Findings indicated that many teachers are able and willing to actively try new strategies for engaging their students in mathematics learning activities. For example, teachers felt confident in their abilities to effectively implement what they learned in the institutes and stated that they planned to use more hands-on exercises in their mathematics instruction. Many expected to emphasize discovery and hands-on learning and use small learning groups to explore and solve mathematical problems collectively using manipulatives and/or learning games. In contrast, the teachers expected that they would use fewer worksheets and bookwork to teach math, and less lecturing or '1elling" during lessons. These are positive indicators that teachers are well prepared to implement what they learned in the first year of the POls. Expected Student Impacts The POI participants were asked what changes they expect to see in their students if they implement the new instructional strategies. Many teachers expect to see positive changes in students' attitudes, confidence, higher-order thinking skills, and performance on assessments. They also expect that these results would be partly due to students working more collaboratively with each other. Several teachers mentioned that they expected their students to have more "interest" in learning math. While these predictions of student impacts are positive, further evidence will be needed to determine students' responses to the new instructional approaches in terms of such factors as motivation, engagement, and mastery of standards. Ultimately, it will be important to explore whether students are able to perform better on high-stakes assessments as a result of teachers' implementation of the new approaches to teaching mathematics. Summary Overall, we conclude that the Mathematics POls are making progress toward their goals (although limited information was gathered from the OSU participants due to the timing of training sessions in relation to the writing of this report). These findings are particularly important for gauging the progress that the POls are making in this first of the three-year program. As described in the overview, in this first year of the OEA and OSU POls the focus was primarily on developing teachers' content knowledge and understanding of mathematics concepts. Indeed, survey respondents reported that they gained useful and valuable information from the training and that they are eager to implement what they learned. One concern was raised by the small gains in scores on the content tests from the beginning to the end of training. Although one might expect more teachers to have mastered the content tests by the end of training, it is uncertain whether teachers took these assessments seriously and performed according to their true knowledge and ability. Nevertheless, this is something for OCTP and the POI vendors to consider as they continue with the second and third years of the training. The ultimate value of professional development programs is the ability to help teachers transform the learning environment for their students. Evidence clearly suggests that participants are learning from, and find utility in, the Mathematics POI training. One limitation to these findings, however, is that these are self-reported perceptions and, therefore, they do not reflect how well the POI learnings actually transfer to the classroom. Furthermore, limited information has been gathered regarding the long-term support that teachers receive to implement what they learn. In line with OEA's and OSU's plans for the second and third years to focus more on implementation and support, this evaluation will also explore the extent to which participants of the POls, including administrators, implement what they have learned and whether adequate supports are in place to facilitate implementation. Recommendations Recommendations for program improvement are discussed below, beginning with a summary of participants' recommendations reported in the post-institute survey, followed by a set of evaluator recommendations, which are based on the overall findings from the 2003-2004 evaluation of the Mathematics POls. Participant Recommendations While progress is being made toward the goals and objectives of the POls, participants offered suggestions for furthering the effectiveness of the POI trainings. Suggestions included: • More time to collaborate with other teachers, • More alignment of the POI curriculum with the OKINCTM PASS standards, and • More opportunities to observe trainers teach a classroom lesson. Teachers also expressed the need to maintain contact with their POI colleagues. They suggested the use of a ListServ and/or an interactive web site. It may be beneficial to explore this possibility. As an example, electronic mailing groups on commercial Internet sites, such as Yahoo, are a free resource that may be used to create a networking community for the exchange of information and ideas. Other helpful sites containing resources that Middle School Mathematics teachers might find useful are: • Middle Web (http://www.middleweb.com/mw/listserv/MWLarchive.html) • The Math Forum at Drexell University (http://mathforum.org/) • Michigan State University Connected Mathematics Math Web Site (http://www.mth.msu.edu/cmp/) Evaluator Recommendations Based on the overall findings discussed above, SEOL presents the following recommendation for OCTP to consider in its ongoing efforts to enhance and improve the Mathematics POls. • Promote the use of content tests for (a) assessing the effectiveness of the training in improving teachers' content knowledge and (b) helping the trainers to understand what concepts teachers need additional help with learning during the training. Research has shown that more highly qualified teachers, particularly those who are knowledgeable and certified in their content areas, are more effective in helping students learn and perform better on assessments. Raising the awareness and importance of teachers' content knowledge through the pre- and post-tests may serve the dual function of reinforcing teachers' learning and enhancing accountability of the POls to deliver what they purport to do. • Promote the importance of the administrators' participation in the POls. School leaders are often key to successful implementation of new instructional strategies or approaches because they can often find time for teachers to engage in activities that improve teachers' learning and effectiveness with students, such as planning lessons, analyzing student work, sharing strategies, and observing each other's classes. The administrators who responded to SEOL's survey indicated that they valued their participation in the training because they became familiar with what teachers were learning, but some indicated that they would like additional strategies in their roles as administrators for helping teachers implement what they learned in the POls. OCTP might consider requiring principal participation, or at least encouraging principals to participate. Further information about the benefits of principal participation will be useful for determining more precisely how teachers benefit from this type of principal support. • Extend the evaluation of the Mathematics POls to include a more specific examination of the effects of teachers' implementation of POI knowledge and strategies. For example, student performance in specific learning activities might be structured to elicit differences in students' mathematics learning and understanding before and after teachers attend the POls. While POI staff expects student products to be submitted by teachers attending the POls, they are currently not a part of the evaluation. Student data would aid in moving the their evaluation to a stronger position in assessing the overall quality of the mathematics POls. SUCCESS IN SCIENCE PROFESSIONAL DEVELOPMENT INSTITUTE IMPACT • The Science PDI had 258 participants with an average of 15 years teaching. • The average number of years since the respondents had taken a college-level science content course was 12 years. • 82% of the respondents "Strongly Agreed" they had gained useful knowledge and strategies for teaching science. • Huge growth was seen in teachers' understanding of Teaching Learning Through Student Discovery (2.11 % to 3.5%) and Using Hands-On Science Modules (1.5% to 3.7%). • Respondents reported seeing students improve in multiple areas including more positive attitudes, an eagerness and excitement to learn, greater interest, better understanding and retention, and overall better performance. • 95% of the participants recommend this Science PDI as quality professional development REFLECTION • Teachers involved with Success in Science PDI are reporting positive student impacts. There is a need to increase the availability to more teachers to impact learning at a greater rate. • The limited number of professional development days is an obstacle in promoting quality learning for teachers. • Even though the funding is not available to expand this PDI, school districts have been willing to fund the cost of resources if Success in Science PDI will provide the training. The results speak loudly. Executive Summary Success in Science Professional Development Institute The purpose of this report is to present evaluation findings to OCTP regarding the effectiveness of the Success in Science Professional Development Institute (SSPDI) offered during 2003-2004. The OCTP's goal for the science professional development institutes is to provide professional development that enables teachers to make science more accessible, understandable, and engaging for students. Key Findings from FY 03-04 ~ Participants continued to rate the SSPDI as satisfactory, valuable, and well designed. Respondents consistently. • rated the institutes high on the content covered, organization, presentation, and materials; • gave overall highest ratings to the instructors' knowledge of the content and experiences implementing institute strategies; and • indicated they would recommend this training to other teachers. ~ Teachers reportedly increased their knowledge and skills after participating in the science institutes. Respondents consistently: • reported highest ratings in knowledge in their confidence teaching for conceptual understanding in science; and • reported highest gains in their preparedness to use science module materials and notebooks or journals for assessment. ~ More than half of the respondents (63%) strongly agreed that they felt confident in their ability to effectively implement what they learned at the SSPDI. ~ Teachers (former SSPDI participants) reportedly applied the knowledge and skills gained during the science institutes in their instruction. Respondents: • reported highest ratings in using the science module materials and teaching using student discovery and science investigation as a method for learning; and • reported highest gains in students' motivation and active involvement in science learning. ~ Teacher responses to the open-ended questions provided strong evidence that the teachers felt confident in the knowledge and skills they learned from the SSPDls and that their new knowledge has impacted their instruction and their students in very positive ways. Overview Science Institute Goals The OCTP's goal for the science professional development institutes is to provide professional development that enables teachers to make science more accessible, understandable, and engaging for students. The OCTP aims to provide professional development that: • adheres to professional standards for science teacher professional development; • increases teachers' level of content knowledge in science; • improves teachers' ability to use inquiry-based instruction methods; and • trains teachers to value the role of communication, both verbal and written, as a tool to aid student learning. Approach to Attaining Goals Three members of the Oklahoma Teacher Education Collaborative (Southwestern Oklahoma State University, University of Tulsa, and Oklahoma State University's Center for Science Literacy) submitted a bid and were awarded the contract to conduct the Success in Science Professional Development Institute. The SSPDI collaborative proposed to provide Oklahoma K-6 elementary and 6-8 grade science teachers with professional development to increase the science literacy of Oklahoma students by engaging teachers in inquiry methodology for instruction of grade-specific science modules. The SSPDls use Full Option Science System (FOSS) and Science and Technology for Children (STC) curricula which are standards-based, science units that employ inquiry strategies to help students learn science topics in meaningful ways. The professional development emphasizes a student-centered approach with instruction focused on guiding students to use discovery methods in practical problem-solving situations. Previous success with this model of science instruction led to expectations for significant increases in student achievement in science and related courses. The SSPDI participants learn to use grade-specific science modules during days of training that occur during the school year. Teachers receive hands-on experience with the modules that are intended to promote their reflection, planning, and implementation of inquiry-centered strategies in elementary and middle school classroom instruction. The SSPDls are structured so that participants receive an initial intense multi-day period of methodology and curriculum training with several single days of follow-up for the Module 1 and Module 2 materials during the academic year. The Module 3 training involved teachers in training in pedagogical and science instruction strategies to enrich their use of the module kits. The teachers received a kit with instruction manual and necessary materials and supplies for approximately 16 lessons they use with their students throughout the module. Evaluation Questions The POI evaluation questions related to the science institutes are: (1) Did participants perceive the science institutes to be valuable and high quality? (2) What knowledge and skill did participants gain as a result of the science institutes? (3) How do participants perceive that school leaders are supporting and/or hindering the application of their new skills and knowledge? (4) Do participants expect to be able to integrate training knowledge and skills into classroom instructional practices? (5) Are the science institutes producing gains in student learning? The 2002-2003 Science POI evaluation centered on three of the evaluation questions and determined that last year's Science POI participants found the science institutes provided training that was valuable, high-quality, and useful for developing their knowledge and skills in science instruction. The participants also indicated that they felt prepared and expected to use the knowledge and skills they gained in their classroom instruction. This SSPOI report is based on two primary sources of data: (1) an end-of-institute survey for this year's (2003-2004) SSPOI participants, and (2) a follow-up survey of last year's (2002-2003) SSPOI participants. The report will first address information obtained from the current SSPDI participants to verify information gathered in the previous year. Then, the information from previous participants will be presented. The evaluation questions, methods, and data sources are listed in the following table: SSPDI Year 2 Evaluation Evaluation Question Method Data Source (1) Are SSPDI of high value and • End-of-Session Survey • 2003-2004 SSPDI (Module 3) quality? • Observations Participants (2) Do participants increase • End-ot-Session Survey • 2003-2004 SSPDI (Module 3) knowledge and skill during • Observations Participants Science PDI? • Follow-up Survey • 2002-2003 SSPDI Participants (3) Do participants have school • Follow-up Survey • 2002-2003 SSPDI Participants leadership support tor SSPDI instruction? (4) Are participants able to • End-ot-Session Survey • 2003-2004 SSPDI (Module 3) integrate SSPDI knowledge and • Observations Participants skills into classroom instruction? • Follow-up Survey • 2002-2003 SSPDI Participants (5) Are there gains in student • Follow-up Survey • 2002-2003 SSPDI Participants learning? Did participants perceive the science institutes to be valuable and high quality? At the end of the training, respondents were asked to rate the SSPDI on a scale of 1 to 5 whether (a) the content of the institute was useful, (b) the institute was organized, (c) the presentations were stimulating, (d) the materials were helpful, and (e) they were actively involved in learning. the following table indicates that the SSPDI respondents average ratings for the institute was high (between 4.5 and 4.8) for all five categories. 2003·2004 SSPDI Respondents' (N:56) Percentages of Overall Institute Ratings I," o Ofuslitetle 2 3 4 ." Veruyseful M(SeOan) a. Content -- -- -- 10 (18%) 46 (82%) 4.8 (.39) Poor 2 3 4 Mean Excellent (SO) b. Organization -- -- -- 14 (25%) 42 (75%) 4.8 (.44) Boring 2 3 4 Stimulating Mean (SO) c. Presentation -- -- 2 (3%) 15 (27%) 39 (70%) 4.7 (.55) Of little , Very Mean help 2 3 4 helpful (SO) d. Materials .. 1 (2%) 2 (3%) 23(41%) 30 (54%) 4.5 (.66) Poor 2 I~ 3 4 Excellent Mean (SD) e. Participant .. .. 1 (2%) 8 (14%) 47 (84%) 4.8 (.43) Involvement Note: Ratings on a 5-point Semantic Differential Scale. The scale for each item ranged from: (a) "of little use" to "very useful," (b) "poor" to "excellent," (c) "boring" to "stimulating," (d) "of little help" to ''very helpful," and (e) "poor" to "excellent." N=number of participant responses. SD=Standard Deviation What knowledge and skills did participants gain as a result of the science institutes? After responding to items on the survey about their reaction to the SSPDI sessions, the participants had an opportunity to indicate what knowledge and competencies they had obtained from the institute sessions. Participants were asked to respond to eight items upon which they could indicate how prepared they thought they were to use the instructional strategies that had been discussed in the SSPDI sessions. These strategies included instruction on how to use science investigation, cooperative groups, or problem solving strategies as a method for learning science. Participants rated these instructional strategy items (4-point scale, 1=not well prepared to 4=very well prepared) twice: the first time to represent how prepared they currently thought they were to use these instructional strategies now that they had participated in the SSPDI, and the second time to represent how prepared they thought they had been to use these strategies before they participated in the SSPDI. Average Ratings of SSPDI Module 3 Participant Preparedness Perceptions of Skills and Knowledge Gains Learning through student discovery ~ 12.1 T 13.6 1 I,", n' ., 12.2 .s. 13.6 I I 12.5 13.5 I 12.2 1 13,4 I 12.1 1- 13.5 I 11.6 j I 3.5 I 11.5 1 13,7 j I 12.1 I 13.4 . , -",,' ! losefore I ~ow Science investigation as a method for learning Cooperative student groups Problem-solving strategies Question strategies to promote reflection Notebooks or journals for assessment Science module materials Connecting content to real world and careers 234 Four-Point Scale: Not well prepared (1) to Very well prepared (4) N=112 Do participants expect to be able to integrate training knowledge and skills into classroom instructional practices? Another goal of the SSPDI is to determine how the institutes impact the participants' instructional actions after they complete the sessions 2003·2004 SSPDI Respondents' Expectations for Implementation DSitsroanggrelye' 2 3 4 5 SAtrgornegely M(SeDan) N tie. aI cfehelfomr ocroencceopntfuidaelntunindemrsytaanbdiilnityg to -- -- -- 2 17 37 5.6 (4%) (30%) (66%) (.56) 56 than I was prior to this training. oim. pIlfeemelencot nwfidheant tI hthaavteIlceaanrneefdfecattivtheilsy -- -- -- (42%) (3149%) (6335%) (5.5.67) 56 Science POI. . . Note: N=Number of respondents. SD=Standard Devianon. Scale range from Strongly Disagree (1) to Strongly Agree (6). Are the SSPDls producing gains in student learning? Although we have no direct measures of student impact from the survey, the participant responses to open-ended items (italicized questions in paragraphs below) describe an anticipated level of student impact as a result of what participants had experienced while implementing the science modules during the time period spanning the Science PDI. What Student Impact Have You Seen? In an open-ended item on the SSPDI survey, teachers (n=56) reported seeing improvements in their students during this year when they have begun to use the science modules in their instruction in multiple areas. These included seeing more positive student attitudes, an eagerness and excitement to learn, greater interest, better understanding and retention of what students learned, and overall better student performance. Parents, teachers from other subjects, and the students themselves were among those who reported to SSPDI participants that their science students "love to come to class" and had a new enthusiasm for learning. As one participant mentioned, students were more active in taking the initiative to learn, read, or create new problems to investigate. Students were also developing as cooperative learners and becoming "more respectful of others work," as one teacher reported. There were very few comments by teachers that showed a negative impact on students. One commented that her students "didn't like to write in notebooks," but that they were "still working" on using that learning method. The above teacher responses to the open-ended questions provide evidence that the teachers feel confident in the knowledge and skills they learned from the Science PDls and that their new knowledge has impacted their instruction and their students in very positive ways. The next section details information from the follow-up survey about teachers' perceptions about how the Science PDls have impacted their instruction, affected student performance in science, and been supported by administrators and other teachers at their schools. Are the science institutes prOducing gains in student learning? Most of the former SSPDI participants indicated that their students are more engaged in classroom activities (94% agreed or strongly agreed, Item f). Former SSPDI Respondents' Ratings ASfctieerncpeartPicDipIa..t.ing in the Success in DSitsroanggrelye Disaegre Agere, SAtrognregely M(SeDan) N f. cMlayssstruodoemntsacatirveitiemso.re engaged in (43%) (21%) (181%2 ) (7560%) (3.7.73) 66 Note: N=Number of respondents. SD=Standard Deviation. Scale range from Strongly Disagree (1) to Strongly Agree (4). Respondents also had the opportunity to respond to six items about the impact of implementing SSPOI instructional strategies with their students. The following table lists those responses: ormer espon en s a Ings 0 e mea 0 ns ruc Ion Noatll at vNeoryt Some mVuecryh M(SeOan) N much a. Mimyprsotuvedde.nts' science performance has (21%) -- (3232%) (6453%) (3.5.68) 66 b. Motyhesrtubdeetntetsr inintsecriaecntc/ceoocplaesrastees. with each (21%) -- (4371%) (5314%) (3.5.53) 66 c. Mscyiesntcued.ents are more motivated to study 7 59 3.9 66 -- -- (11%) (89%) (.31) d. Mleayrnstinugdensctsienacree. more actively involved in -- -- (64%) (9641%) (3.2.94) 65 e. sMcyiesntcuedecnltasssaessk. questions more often in 19 47 3.7 66 -- -- (29%) (71%) (.46) f. I learned from indirect sources (for example, the parents, other teachers, 4 1 27 34 3.4 students' friends) that my students' (6%) (2%) (41%) (51%) (.80) 66 science motivation has improved. F SSPDIR d t' R f f th I ct f SSPDII t tl Conclusions • This report provided confirmation that these three professional development levels continued to be fulfilled by the SSPOI (the value and quality of the institute, the knowledge and skill gained, and the expectations for integrating into classroom instruction). • Provides evidence that according to the perspectives of former SSPOI participants who responded, the final two levels of professional development (administrative support and gains in student learning) are also present as a result of the quality of professional development to the K-8 teachers who participated in the institute. • Respondents liked the sessions, believed the inquiry-based strategies and skills are useful for their classroom instruction, and expected to see changes in their instruction and in student learning in their classrooms. • Respondents reported that they thought they were more prepared now than they were before attending the science training to teach using key instructional strategies discussed during the institute. • Success in Science POI Module 3 respondents reported gains in their ability to use inquiry-based instructional practices that are student-centered, problem-based, and built around students' collaborative investigation of scientific phenomena. • Improved use of effective questioning strategies and methods for eliciting regular student verbal and written communication as an aid to learning and assessment were other areas of improvement that institute participants found especially promising for future instruction. • Former SSPOI participants reported continued use of the materials and instructional strategies they learned from the SSPOI with their students. The use of notebooking and journaling, assessment tools, and instructional strategies associated with their grade level modules were some of the specific SSPOI strategies mentioned by the teachers. • Respondents also indicated that they used cooperative groups, discovery learning, and scientific investigation techniques as a method for leading students in exploring the science ideas, principles, and information presented through the science modules. • The results indicated that generally, most respondents felt supported by their school leadership for using the Success in Science POI materials. • Overall, the teachers indicated that while they shared information and resources that they could from the Science POI with their colleagues, they wanted more teachers in their school to participate in the SSPOI so more students could benefit from these type of science learning experiences. • In general, respondents indicated that students benefited from instruction using the SSPOI modules and recommended strategies. Recommendations • Teachers want to participate in an additional SSPOI because they want to be prepared to teach another module for the same or a different grade level • The teachers also reported that they appreciated being able to spend more time and have more opportunity to work with different modules in preparation for instruction with students. • According to the former participants, participation in an additional SSPOI allowed them to come into the institute with an understanding of the process so that they could then concentrate on absorbing the content and information that would improve their ability to plan, prepare, and lead students in the science modules • It is recommended that, as much as is feasible, additional SSPOI sessions and materials should be provided for these teachers to continue their growth in inquiry-based science instruction. • Providing funds and opportunities for teachers to participate in follow-up sessions of the SSPOI may help solidify instruction, provide a way to deepen their content knowledge of the different science modules, and allow them to sharpen their skills at leading, assessing, and encouraging student learning of science. OEA MENTORING PROFESSIONAL DEVELOPMENT INSTITUTE • OEA developed partnerships with several school districts with the understanding that district support would increase successful implementation and outcomes. • Approximately 700 teachers have participated in a mentoring PDI since the inception. • Classroom management skills were enriched early in the training. • On average participants have formally mentored 1-3 teachers. • Three-year professional growth plans were written to enhance the teacher's practice. • 89% of participants agreed that beginning teachers and their own practice improved as a result of the institute. IMPACT REFLECTION • Aligning our evaluation system from pre-service to retirement is a result of the data received from this grant. A pre-service to retirement evaluation system is being proposed. Higher education institutes are being held accountable to the INTASC standards for their programs. These state adopted competencies will guide the development of a uniform system. • NCLB (The No Child Left Behind Act), OSRHE (Oklahoma State Regents for Higher Education), and NSDC (National Staff Development Council) concur that mentoring and/or coaching of teachers is the most vital and the most neglected aspect of assisting teachers to improve practice. The mentoring institutes must be expanded and supported to improve state wide student achievement. Executive Summary OEAlOSU Mentoring POI The Oklahoma Commission for Teacher Preparation (OCTP) is charged with creating a competency-based teacher preparation system to ensure that every Oklahoma classroom has a competent and qualified teacher. The OCTP contracts with independent vendors to provide professional development Institutes for teachers in the state of Oklahoma, including teacher mentoring. Since 1998, OCTP has provided two mentoring Institutes that provide training in essential skills necessary to support beginning teachers, which are applicable to supporting experienced colleagues as well. Two vendors were contracted to provide this training-the Oklahoma Education Association (OEA), using the Pathwise Induction model, and Oklahoma State University (OSU), using the Performance Learning System Coaching model. SEDL's Evaluation Services was contracted to conduct an external evaluation of the Mentoring Professional Development Institutes. This report presents findings from the second of a three-year evaluation contract, which began in April 2002. The purpose of this evaluation was to assess the progress that the Mentoring Institutes were making toward the goal of enhancing the induction process by providing consistent mentor training in the essential skills necessary to support beginning teachers. To do so, SEDL's evaluation worked from the following evaluation questions: 1. How many teachers participated in the training and how were they selected? 2. In what types of mentoring assignments and activities have trainees been engaged since participating in the Mentoring Institutes? 3. Have Mentoring Institute participants been supported in their use of the mentoring strategies? 4. What was the impact of the training on participants and beginning teachers? The evaluation findings are based on a survey that was administered to past participants of the OEA and OSU Mentoring Institute during the four-year period beginning with the 1999-2000 school year. The findings are discussed in terms of training participation and recruitment, mentoring assignments and activities, support for implementing mentoring strategies, and the impact of the training. Key Findings • Participation and Recruitment. An examination of participation rates and the recruitment process for the mentoring Institutes revealed that the purpose of training was effectively communicated although the training is reaching a relatively small number of teachers as a percentage of all teachers in the state. • Mentoring Assignments and Activities. The majority of survey respondents have mentored 1-3 beginning teachers although a number have still not had this opportunity. Survey respondents indicated that while most have mentored beginning teachers, the training did not result in being assigned to more Residency Teams. Based on reports of mentoring activities and assignments, the findings also suggest that the trained mentors are successfully implementing the mentoring strategies they learned in the Institutes and that they are providing substantial support to beginning teachers. • Support for Mentoring. Survey respondents reported that collaboration and follow-through are important to successful implementation of the mentoring strategies and that a number of participants reported having teams of teachers in their schools who were trained in the mentoring strategies. • Training Impact. Survey respondents suggested a variety of benefits for themselves and beginning teachers. In particular, respondents reported being more reflective about improving their own teaching practice and finding effective ways to help beginning teachers or other colleagues. The respondents' comments illustrate their appreciation of the training and the desire for continued support for implementing the strategies they learned. Key Recommendations Evaluation findings indicate that the Mentoring Institutes are beneficial to both the participants of the training and the beginning teachers (and colleagues) who they mentor. Findings also illuminate several areas where the Mentoring Institute or the implementation process in schools and districts could be strengthened. Based on these findings, several recommendations are suggested for ~CTP's consideration: • Increase the number of teams of trained mentors within districts or schools to promote collaboration and ongoing support for the use of mentoring strategies. • Promote principals' participation in the mentoring Institutes. • Promote the adoption of some or all of the mentoring strategies school-wide or district-wide. • Promote the benefits of assigning beginning teachers to trained mentors and encourage schools and districts to use trained mentors when possible. • Promote school and district policies that will ensure adequate numbers of trained mentors to serve as resources for beginning teachers as well as experienced colleagues throughout the state. Overview The Oklahoma Commission for Teacher Preparation (OCTP) is charged with creating a competency-based teacher preparation system to ensure that every Oklahoma classroom has a competent and qualified teacher. The OCTP's three-pronged approach toward attaining this goal includes the provision of competency-based teacher preparation and accreditation, candidate assessment, and ongoing growth and professional development of classroom teachers across the state. In support of federal legislation mandating that all states develop plans ensuring that all teachers of core academic subjects are highly qualified by the end of the 2005-2006 school year, the OCTP contracts with independent vendors to provide professional development Institutes for teachers in the state of Oklahoma. These Institutes emphasize both content and professional instructional practices in the areas of literacy, mathematics, science, and teacher mentoring. This report addresses this last emphasis, mentoring. Since 1998, the Oklahoma Commission for Teacher Preparation has provided two mentoring Professional Development Institutes that provide training in essential skills necessary to support beginning teachers, which are applicable to supporting experienced colleagues as well. Two vendors were contracted to provide this training-the Oklahoma Education Association (OEA), using the Pathwise Induction model, and Oklahoma State University (OSU), using the Performance Learning System Coaching model. This report presents findings from the second of a three-year evaluation contract, which began in April 2002. In Year 1 of the evaluation, SEDL's evaluation focused on the training provided by OSU during the 2002-2003 school year. Findings from the Year 1 evaluation are presented in the 2002-2003 Annual Report on Professional Development. Key findings regarding the first year evaluation of the OSU Mentoring Institute included the following: • Survey respondents reported being very satisfied with the training they received and that the OSU Mentor Institute was of high quality and contained useful content and materials that were presented effectively. • Three months following the end of training, survey respondents reported that they were applying their new skills and knowledge for mentoring beginning teachers and colleagues, indicating that they had increased their level of use of the new skills and strategies to the extent that the majority of respondents were now able to use the mentoring strategies most of the time. • Although survey respondents indicated that principals generally supported mentor strategies, information obtained from interviews indicated that implementation was significantly impacted only if the principals actively participated and promoted teachers' use of mentoring strategies. • Teachers reported that they still needed more time to practice and more involvement of other teachers at their school in order to increase their effectiveness as a mentor teacher. In the Year 2 evaluation, attention turned to both OEA and OSU Institutes and the overall quality and impact of the mentoring training over the four-year period of 1999-2000 and 2002-2003 school years. To do this, a survey was sent to all participants who took part in the OSU and OEA Mentoring Institutes during this time period. This report presents evaluation findings from the second year follow-up survey of the Mentoring Institute participants. The following sections of this report include a description of the two Mentoring Institutes, the evaluation approach, evaluation findings, conclusions, and recommendations. Mentor Professional Development Institute Goals and Objectives The goal of OCTP is to ensure that every student in Oklahoma has access to competent, caring, and qualified teachers. The Mentoring Institutes are expected to contribute to the OCTP goal by achieving the following: 1. Provide a systematic process for the support and development of mentor teachers for the Oklahoma Resident Teacher Program. 2. Provide quality mentor training that enables the mentor teacher to address the varied needs of the resident (beginning) teacher. 3. Create a framework for ensuring sustainability by developing and maintaining a cadre of mentor teachers throughout the state who are prepared to coach entry-level teachers and have the flexibility to coach colleagues interested in expanding instructional skills in the content area. OEA Mentoring Institute The OEA Mentor Institute uses the Pathwise model of mentor training developed by Education Testing Service (ETS). The institute aims to increase mentor teachers' ability to support beginning teachers in a variety of teaching practices. The training consists of the following components. • Classroom Management Skills • Introduction to Frameworks • An Assessment of Teacher Effectiveness • Performance-Based Work Products • Three-year Professional Growth Plan • The Colloquium The OEA Mentoring Institute consists of four days of training throughout the year followed by a celebration day in the spring, which is to recognize the accomplishments and professional growth of the beginning teachers who were mentored. OEA implemented a new approach to recruiting participants for the 2003-2004 school year. In this new approach, OEA partnered with several districts throughout the state to provide training to its mentor teachers with the understanding that district support for such a program would increase the likelihood of successful implementation and outcomes. In previous years, teachers from different districts participated in the training together. OSU Mentoring Institute The OSU Mentoring Institute is based on the Performance Learning System Coaching model. The project goals are to: • Provide a systematic process for support and development of mentor teachers for the Oklahoma Resident Teacher Program. • Provide quality mentor training that enables the mentor teacher to address the varied needs of the resident teacher. • Create a three-year framework for ensuring sustainability by developing and maintaining a cadre of mentor trainers who will be available throughout the state. • Provide a cadre of mentor teachers who are prepared to coach entry-level teachers and have the flexibility to coach colleagues interested in expanding instructional skills in the content area. OSU's Mentoring Institute consists of seven days of training held over a six-month period, with the first block of training days typically conducted in the summer or the beginning of the school year. The Mentoring Institute has been offered at various sites throughout the state. The 7 days of training focuses on a comprehensive set of mentoring strategies intended to help participants coach beginning teachers as well as experienced colleagues. In particular, the training consists of 10 coaching skills for successful teaching, which are presented over the course of the 7-day training and with which participants are given time to practice. The ten skills include the following: • Conducting Pre-observation Conferences for Effective Coaching • Using Open and Closed Questioning • Constructing Confirmatory Paraphrases • Using Positive Phrasing • Constructing Approval Statements • Conducting Post-observation Conferences • Selecting Modes of Questioning • Attending to Non-Verbal Communication • Using Pause Time • Selecting Data-Collection Formats The training involves formal presentation of materials as well as substantial opportunities to practice and receive feedback from peers and presenters. Participants are expected to videotape themselves during mentoring sessions at their schools in between training days. These videos are then shared and discussed with fellow participants during the following sessions. Participation According to OEA and OSU participation databases, a total of 695 teachers participated in both Mentoring Institutes over the four-year period beginning with the 1999-2000 school year. An average of 73 participants attended the OEA training each year and an average of 101 participants attended the OSU training each year. Table 2 shows the number of participants who attended the two Institutes each year. The total number of participants trained in both OEA and OSU mentoring programs represents roughly 2% of all Oklahoma public school teachers. Mentoring Institute Participation Rates 1999-00 2000-01 2001-02 2002-03 OEA participants 44 103 62 82 OSU participants 120 98 112 74 Mentoring activities To further explore the impact of the Mentoring Institute, SEDL asked training participants to indicate how often and in what types of activities they shared with their mentees. Additionally, mentor-training participants were asked to indicate how often they met with their mentee. The pattern was similar across all four years and across OSU and OEA training programs. Between 86% and 91% reported meeting with the beginning teacher at least once a week or more. Participants were also asked to report when they met with their mentees. Respondents indicated that they met most often after school (64%), but they also indicated meeting before school (38%), during a common planning time (38%), during lunch (31%), and other times such as during individual planning time or in workshops. (13%). These responses suggest that the mentors are providing a fairly substantial amount of support to beginning teachers on a regular basis throughout the school day. SEDL also asked the training participants to indicate what types of activities they have shared with their mentee. Table 6 shows the percentage of respondents who engage in the various mentoring activities. These activities, which respondents selected from a list of options, are presented in order of greatest to least engagement. As indicated in the table below, the most common type of activity reported was observing the mentee in the classroom and giving feedback. The response patterns were fairly similar between OEA and OSU training participants. The biggest difference in reports of mentoring activities between OSU and OEA (24 percentage points) related to planning lessons, units, and assessments. This difference may reflect the different approaches presented in the two Institutes. While this breakdown of activities may indicate that there is room for mentors to focus more on analyzing student work and test data, planning professional development, and co-teaching, it nevertheless suggests that there is still a fairly good balance of support activities provided to beginning teachers. en ortng C IVI ies OEA OSU (n=39) (n=41 ) Observing the mentee in the classroom and giving feedback 82% 90% Planning lessons, units, assessments, etc. 49% 73% Strategizing for communication with parents 64% 71% Referring the mentee to other people as resources 59% 71% Looking at student work 56% 54% Analyzing student test data 28% 44% Planning for professional development 33% 37% Co-teaching with the mentee 21% 29% IOphthileorso(peh.gy.,, deitscc.)ussions about classroom policies, teaching strategies or 5% 20% Mt' AfT Use of mentoring strategies Mentoring Institute participants were asked about the extent to which they have used a variety of mentoring strategies now that they have had time to implement what they learned in the training compared to the use of those strategies before training. A series of survey items focused on the implementation of specific mentoring strategies using a 4-point Lickert scale of 0 (not at all) to 3 (extensively). To assess the effect of the training, SEDL computed the difference in the ratings now and before training. The following table presents the mean differences on these items. The differences in reported use of the mentoring strategies now compared to before training was significant across both programs and for all strategies (p<.01). These responses provide a measure of the degree to which respondents perceive they have implemented what they learned in the mentoring training. It is notable that the mean differences between reported use of mentoring strategies now and before training were greater for OSU participants than OEA participants, although it is unclear why. Mean Differences in Mentoring Strategies Use Now and Before OEA OSU Mean N Mean N Observing classrooms 0.76 34 0.89 38 Providing feedback 0.86 35 1.21 38 Using questioning strategies 0.89 35 1.58 38 Conducting pre-observation conferences 0.91 34 1.26 38 Conducting post-observation conferences 0.79 34 1.16 38 Using data collection techniques 0.83 36 1.51 37 Using strategies for working with adults 0.62 34 1.21 38 Accepting feedback from colleagues 0.52 29 1 19 Being observed in my classroom 0.44 36 0.71 38 Reflecting on my own teachinc practice 0.86 36 1.03 38 Collaborating with my peers 0.63 35 0.89 38 The strategy for which OEA participants reported using more as a result of training (mean difference=.91) was conducting pre-observation conferences. The strategy for which OEA participants reported using least as a result of training (mean difference=.44) was being observed in the classroom. The strategy for which OSU participants reported using more as a result of training (mean difference=1.58) was using questioning strategies. The strategy for which OSU participants reported using least as a result of training (mean difference=.71) was being observed in the classroom. These findings may reflect the variations in training approaches used by OEA and OSU in their Mentoring Institutes. Nevertheless, they suggest that there is a significant increase in the reported use of mentoring strategies as a result of training, which provides evidence that trainees are indeed implementing the skills they learned. Impact of mentoring training on teaching practices SEDL asked participants to respond to a series of items about the ways in which the mentoring training has impacted beginning teachers' and their own teaching practice. Table 11 presents the percentages of respondents' ratings on this set of items. Overall, respondents generally agreed or strongly agreed that beginning teachers' and their own teaching practice improved as a result of the training. They also generally agreed that their working relationship with other teachers has improved and that they have learned more about supporting beginning teachers and supporting colleagues. Respondents were more divided regarding whether they learned more about district or state standards, student assessments, and planning for professional development. Finally, respondents generally disagreed or strongly disagreed that they already knew how to use the mentoring strategies prior to the institute training. The following table does not compare OSU and OEA responses because we found that the means were similar for OSU and OEA training participants for all items except ''The mentor institute has helped me learn more about supporting experienced COlleagues." The mean rating for OSU was higher (mean=2.54) than the mean rating for OEA (mean=2.06) on this item. This finding reflects the greater emphasis on working with colleagues in the OSU Mentoring Institute compared to the OEA training, which focuses almost exclusively on mentoring beginning teachers. Impact of Mentoring Training on Teaching Practices i.' , " , Strongly Strongly Disaaree Disagree Agree Aaree Mean N The teaching practice of the teacher I pmaerntictoipreadtionimpinrotvheedMaesnatorreTseualtcohfermy 1% 10% 65% 24% 2.1 72 Professional Development Institute. My working relationship with other colleagues has improved because of my participation in - 17 59 24 2.1 76 the Mentor Institute. My own teaching practice has improved as a result of my participation in the Mentor - 4 51 45 2.4 76 Institute. The Mentor Teacher Institute has helped me learn more about: Supporting beginning teachers - 3 42 55 2.5 74 Supporting experienced colleagues - 7 56 37 2.3 75 Instructional strategies · 10 53 37 2.3 73 The district or state standards · 30 55 15 1.9 74 Student assessment · 23 52 25 2.0 73 Planning for professional development 1 25 55 19 1.9 73 I already knew how to use the mentoring strategies we learned in the Institute 25 55 18 3 1.0 73 beforehand. Conclusions In this report we shared findings from the second of a three-year evaluation of the Mentoring Professional Development Institutes. The evaluation findings were based on the survey that was administered to participants of the OEA and OSU Mentoring Institute during the four-year period beginning with the 1999- 2000 school year. The findings were discussed in terms of training participation and recruitment, mentoring assignments and activities, support for implementing mentoring strategies, and the impact of the training. A brief summary of these findings is presented below, followed by a set of recommendations for OCTP's consideration. • Principal and/or district administration support and commitment at times seemed weak. • The majority of survey respondents have mentored 1-3 beginning teachers although a number have still not had this opportunity. • Respondents also indicated that while most have mentored beginning teachers, the majority has not been assigned to more Residency Teams as a result of training. • Respondents also reported engaging in a variety of mentoring activities, the most common were observing their mentee during a lesson and giving feedback, planning lessons, units, and assessments, strategizing for communication with parents, and referring the mentee to other resources. • The large majority of respondents reported meeting with their mentee at least once a week or more; this happens most often after school. • The trained mentors are using the mentoring strategies they learned in the Institutes and that they are providing substantial support to beginning teachers. • Collaboration and follow-through are important to successful implementation of the mentoring strategies and that a number of participants reported having teams of teachers in their schools who were trained in the mentoring strategies. • Respondents generally rated the support they receive from others to implement the mentoring strategies fairly high. The actions that helped them implement mentoring strategies the most focused on carving out time and opportunities during teachers regular schedule to observe each other and meet to discuss teaching practices. • Important actions also included clear expectations about the use of mentoring strategies and the importance of mentoring for school improvement efforts and the need to "scale-up" the mentor training. Respondents indicated that they would benefit from more staff participating in the Mentoring Institutes, including principals. • Respondents want more support for mentoring either with time to collaborate with other teachers or for more compensation to conduct these activities. • Respondents perceive they are much better teachers and better prepared to support beginning teachers. Recommendations Evaluation findings indicate that the Mentoring Institutes are beneficial to both the participants of the training and the beginning teachers (and colleagues) who they mentor. Findings also illuminate several areas where the Mentoring Institute or the implementation process in schools and districts could be strengthened. Based on these findings, several recommendations are suggested for OCTP's consideration. • Increase the number of teams of trained mentors within districts or schools to promote collaboration and ongoing support for the use of mentoring strategies. Findings indicated that teachers benefited from having other teachers in their schools go through the mentor training because they were able to help each other implement the mentoring strategies. • Promote principals' participation in the Mentoring Institute. Respondents indicated that the implementation of mentoring strategies would be enhanced even further by more support from principals and even their participation in the mentoring training. • Promote the adoption of some or all of the mentoring strategies school- or district-wide. • Promoting the use of mentoring/coaching strategies school- or district-wide would allow teachers to collaborate and support each other as they continue using the mentoring strategies with new teachers and experienced colleagues as well. • Promote adequate time to engage in mentoring activities, such as observing classrooms and meeting to discuss observations and other teaching practices. • Promote the benefits of assigning beginning teachers to trained mentors. • Promote school and district policies that will ensure adequate numbers of trained mentors to serve as resources for beginning teachers as well as experienced colleagues throughout the state. • Promote the assignment of trained mentors to beginning teachers. In conclusion, the evaluation findings indicate that the mentor training positively impacted participants of the training and that the new teachers whom they mentored were positively impacted as well. There is generally support for the use of these mentoring strategies but some respondents still reported not having the opportunity to work with beginning teachers. However, there are aspects of induction programs that we have not yet addressed in this evaluation and may be potential topics for the third year evaluation. For example, how well are Mentoring Institute trainees prepared to provide advice to beginning teachers about serving diverse student populations and students with a considerable learning needs (e.g., struggling readers). Many beginning teachers leave because they are overwhelmed by the challenging circumstances in which they find themselves, such as in high-poverty, low-performing schools, or in schools where administrators do not support teachers. We also know little about how stipends or other motivational factors influence teachers to participate in the training or on the Residency teams. These are potential questions for future consideration. EDUCATION LEADERSHIP OKLAHOMA HIGHLIGHTS FROM 2003·2004 NATIONAL IMPACT Independent Studies Confirms Effectiveness of National Board Certification ~ An independent study of 600,000 student records from students in North Carolina schools has found that children learn more from National Board Certified Teachers. The study found: • Teachers who achieve National Board Certification do a measurable better job in the classroom • Students of NBCTs improved an average of seven percent more on their year-end math and reading tests than students whose teachers attempted but failed to gain certification ~ Another study demonstrates that National Board Certified Teachers increase student achievement. The study drew upon more than 200,000 students from 14 Arizona school districts with gain scores calculated from each student over four consecutive years • The gains made by students of NBCTs were more than one month greater than the gains made by the students of non-Board certified peer teachers. STATE IMPACT Total Number of Oklahoma National Board Certified Teachers as of Fall 2004 - 858 Oklahoma ranks eighth in the Nation • 355 Candidates were funded by the Oklahoma Legislature • A federal subsidy secured by the OCTP in the amount of $88,000 provided additional candidate support • 211 First Time Candidates completed the process • 179 Advanced Candidates participated in the retake process • 230 Candidates currently await their scores • 75 of Oklahoma's 77 Counties have either a National Board Certified Teacher (NBCT) or a candidate • 168 of the School Districts in Oklahoma currently have a NBCT • An additional 48 districts have candidates • The Legislature continues to award a $5,000 annual stipend to NBCTs who are fulltime classroom teachers in an Oklahoma school ELO CANDIDATE SUPPORTfTECHNOLOGY • ''Trainer of Trainers" for Durant ELO Training • Two two-day summer orientation programs in Durant for new candidates (in conjunction with Southeastern State University) • Distance Learning 2-Day Workshop - Eight regional Sites Statewide • Tapes of Distance Learning Workshop sent out to candidates who could not attend • Eight regional coordinators who coordinate and facilitate support and mentoring • Partnerships with Career Tech, Oklahoma Universities and Regional Professional Development Centers • A pre-candidacy video funded by a recruitment grant from the State Farm Foundation was distributed • Assessment Center Workshops - Two regional sites ELO GOALS • Recruit and attract teachers in under-represented regions and low-periorming schools • Promote benefits of using NBCTs to ensure high quality teaching such as: leaders on school, district, or state improvements committees • Determine the effectiveness of NBCTs in terms of student achievement • Have a National Board Certified Teacher in every school district in Oklahoma • Secure funding for 500 annual scholarships for National Board Candidates • Expand the mentoring program for National Board Certified Teachers • Provide the guidance and support needed by Candidates with specific feedback and in-depth study on standards and how they are incorporated • Provide leadership training for NBCTs RESULTS OF SEOL EVALUATION In April 2002, the Southwest Educational Development Laboratory (SEDL) was contracted to evaluate the ELO program. The purpose of the evaluation is to examine how National Board Certification of teachers in Oklahoma addresses quality teaching and advances school improvement efforts. What was the impact of becoming certified? Survey respondents overwhelmingly indicated that NBC is an excellent professional development experience and that they are much better teachers as a result. This was especially true for teachers with 4-10 years experience compared to those with 21+ years of experience . Teachers with fewer years of teaching experience and those teaching in elementary/middle schools and low periorming schools reported experiencing more personal outcomes as a result of certification. In comparison, more teachers in urban, larger schools reported experiencing professional outcomes. Survey respondents generally reported that certification improves their pedagogical skills, their knowledge of curriculum and subject area, and their ability to enhance and assess student learning. However, responses also indicate that the benefits of certification are still largely localized to NBCT's own classrooms. 100% -,-----------------------,,,.----......-,,..-----, 4-10 .11-20 021+ 20% -t------'------- 0% 1% 2% O%+---~--c=~-~- Did not impact your practice Made you a somewhat better Made you a much better teacher teacher Figure 1 Impact of Certification by Years of Teaching Experience Overall, 70% of NBCTs who responded to this survey indicated that certification made them a much better teacher. Thirty percent reported that the certification process made them a somewhat better teacher and only 1% said it did not impact their practice. While the responses generally were similar for teachers in different types of schools and those teaching different subject areas, the responses did vary by reported years of teaching experience. As illustrated in Figure 1, more teachers with 4-10 years of experience believe they are much better teachers as a result of certification compared to those with 21 + years of teaching. SEDL also asked NBCTs to rate the helpfulness of the certification process regarding 17 aspects of teaching (l.e., not at all helpful, somewhat helpful, very helpful). A summary of responses to these items is illustrated in Figure 2. Improving my self confidence 0/0 1'7 67% " ! 6:~% I f,( 0/0 -j -, a - j ,)RO/o -j '" I ')70/0 -j I ')')0/0 II ,),Ofn ". ,., I ') a I 490 I1 47% I " :.i, I 46% I I~' I 430/0 II 4 % II ,C 0/0 " <:> I 010 I 0/0 I Developing a stronger curriculum Recognizing individual differences in students and adjusting accordingly Using student assessments effectively Articulating learning goals for students Making multiple paths to learning available Mentoring other teachers Working with colleagues on instruction USing research to improve my practice Understanding how students learn Understanding of links to other subject areas Increasing knowledge of my subject area Working with parents Knowing how to impart subject area to students Trreating my students equitably Connecting to state standards Taking advantage of community resources 0% 20% 40% 60% 80% 100% Not At All Helpful • Somewhat Helpful oVery Helpful Figure 2 Helpfulness of NBC for Improving Teaching Responses indicate that certification is most important for improving teachers' self-confidence and developing a stronger curriculum for students. Certification is least helpful for taking advantage of community resources and connecting to the state standards. NBCTs overwhelmingly indicated that NBC is an excellent professional development experience and that they are much better teachers as a result. This was especially true for teachers with 4-10 years experience compared to those with 21+ years of experience. The findings also suggest that NBCTs experience different outcomes of National Board Certification depending on their years of experience, the schools in which they teach, and the subjects they teach. Teachers with fewer years of teaching experience and those teaching in elementary/middle schools and low performing schools reported experiencing more personal outcomes as a result of certification. In comparison, more teachers in urban, larger schools reported experiencing professional outcomes. Oklahoma National Board Certified Teachers and Education Leadership Oklahoma Candidatesl Applicants Through 2003 Cimarron [ Texas r Beaver I Harper Woods Grant Ka y / warT r IOttawa Alfalfa 20 Nowata Craig 3 1 1 22 3 I 1 1 73 222 I 2 762 1 1 Osage 10151 449 Woodward - "- 5 18 Garfield Noble ;II~S\ 1594 Major 7126 221 I 241 I Dewey 1 5 Blaine Roger Mills \ 23 Lincoln Custer 1383 15101 Be2c2k2ham I Washita l - Oklahoma National Board Certified Teachers Education Leadership Oklahoma Candidates (2002-2004) Education Leadership Oklahoma Applicants (2005) McCurtain 43 Le Flore Comanche 5 3 r1, '----"-1 442918 Pushmataha I I I Stephens 851 Jefferson 333 MAY :t ~ 2011 |
Date created | 2011-07-11 |
Date modified | 2011-10-28 |