Professional development annual report

              T 550.3 A6l5p 2003/04 c.l
Oklahoma Commission
for
TeacherPreparadon
--PROFESSIONAL DEVELOPMENT
ANNUAL REPORT
2003-2004
Student Success Through Quality Teaching
LITERACY FIRST
PROFESSIONAL DEVELOPMENT INSTITUTE
EAST CENTRAL UNIVERSITY
IMPACT
• This past year 2,650 participants participated in Literacy First Training; with 1,557 in Phase I,
918 in Phase II, and 124 in Phase III Leadership training. This represents a total of 493 (90%) of
the state's school districts participating and 19,261 participants since inception.
• High quality content and learning documented by participants.
• 65% said their greatest need is more practice and ?6% wanted additional training.
• Prior to the training, most respondents stated they were familiar with the knowledge and skill
areas, and a few reported having a solid understanding. Three months after the training, the
majority of respondents moved from being familiar to having a solid understanding and the
ability to apply the knowledge to their teaching practice. For example, 75% of Phase I & II
teachers and 94% of Phase N teachers are using Literacy strategies in their classroom.
• 60-63% of teachers reported that increased content knowledge in phonological awareness,
phonics, vocabulary, fluency, and comprehension is helping them teach reading more
effectively.
REFLECTION
• Many more principals attended the Literacy First, Phase III trainings. This was a result of a
successful effort to recruit more administrators to attend the PDI. The principal is the key to
creating the changes needed. With our Phase IV schools this became even more apparent as
those schools in which there is strong leadership tend to have the greatest gains in student
achievement.
• Change takes time. The first level of change is bringing about an awareness of the learning
content. Next is implementation of the new skills and strategies within the context of the
present system or changing one, and finally the institutionalization of the practices into the
culture ofthe school and system.
• Phase IV schools data indicates gains in content and teaching strategies at a higher rate of
implementation than non phase IV schools.
Executive Summary
Literacy First POI
The purpose of this report is to provide the Oklahoma Commission for Teacher Preparation (OCTP) with evaluation
findings pertaining to the quality and impact of the literacy Professional Development Institutes (PDls) offered
across the state during the 2003-2004 school year. The goal of OCTP is to provide teachers with the training and
support they need for students to achieve grade-level mastery of reading skills. OCTP partnered with Literacy First
to provide literacy PDls to the teachers in state of Oklahoma.
In Oklahoma, Literacy First offers four "phases" of services to elementary school teachers and administrators.
These phases are:
• Phase I and Phase" for kindergarten through sixth-grade teachers,
• Phase III for school administrators, and
• Phase IV, a whole-school literacy program.
During the 2003-2004 school year, the Southwest Educational Development Laboratory's (SEDL) Evaluation
Services (ES) unit evaluated Phase I and Phase" PDls, as well as the Phase IV program. Information was
gathered on five levels of evaluating professional development (Guskey, 2000): participants' satisfaction with the
PDls (level 1), participants' gain of new knowledge and skills (level 2), organizational support participants receive
for implementation (level 3), participants' implementation of new knowledge and skills (level 4), and student
outcomes (level 5).
To evaluate the Phase I and Phase" PDls, follow-up surveys were administered to past participants. To evaluate
the Phase IV program, current and past Phase IV participants were contacted. Current Phase IV participants
completed surveys and interviews, and site visits were conducted. Past Phase IV participants completed follow-up
surveys. Student test data were also examined to determine whether student achievement had increased at a rate
different from a matched comparison group of non-Phase IV schools and the state average.
Findings from the surveys, interviews, and site visits showed that most of the past Phase I and Phase" participants
were satisfied with the PDls and gained new knowledge or skills through their participation in the PDls (levels 1 and
2). Teachers indicated that they received some degree of organizational support (level 3), implemented the
techniques they learned (level 4), and saw improvements in student reading (level 5). Gains were evident at these
last three levels from teachers and principals at Phase IV schools and also from teachers who completed the Phase
IV school program last year (2002-2003 school year).
Although most of the Phase I and Phase" Follow-up Survey participants rated content and quality of the PDls
highly, participants that responded from the Phase IV schools indicated that content and quality of the Phase I and
Phase" PDls varied widely. Further, most of the teachers in Phase IV schools attended Phase I and" PDls
approximately two years ago, with the range of attendance spanning from 1 to 5 years. As revisions to the Literacy
First program occur periodically, some of these teachers were not up-to-date on the changes. Teachers also noted
that they received conflicting information on implementing Literacy First strategies. Finally, teachers expressed
strong needs for further training and practice using Literacy First strategies, suggesting that the above factors may
be affecting the successful implementation of the Phase IV program at their schools.
Feedback from the Phase IV school administrators and teachers revealed that about one-third of the teachers in
their schools were resistant to the Phase IV program, although almost all of the administrators supported the
implementation of Literacy First Phase IV program 1. The reasons for teacher reluctance to implement the process
included:
I Guskey argued that organizational support is necessary for teachers to implement what they learned at professional
development institutes.
• The lack of a full understanding of the Phase IV process as a whole,
• Perceptions of administrators as tending to push the implementation of all aspects of the program at once,
and
• A lack of clarity about the role and responsibilities of the Literacy Resource Specialists. This was echoed by
LRSs as well.
• Spread Phase I and" POI meeting days out to provide more practice days for teachers back at their
schools and feedback sessions in the POls to reflect on and discuss difficulties experienced in using the
strategies.
• Offer an orientation session to Phase IV school staff at the beginning of their entry into the Phase IV
program to clarify the objectives of the program, the roles and responsibilities of teachers, LRSs, and
administrators, and a review of the content and major strategies and techniques they will be expected to
implement.
• Provide more focus on the strategies that administrators and LRSs report having difficulty with or using only
occasionally (e.g., observation, feedback, modeling).
• Provide more focus on strategies that teachers report having difficulty with (e.g., constructing Literacy
Centers, conducting and using student assessments, classroom management issues).
For student achievement, scores from third grade norm-referenced tests of reading ability in Oklahoma for 2002-
2003 school year were compared between Phase IV schools, matched comparison schools, and state average.
Phase IV schools showed a higher gain than did comparison schools and the state average.
Based on all of these findings, ES offers the following recommendations:
Overview
The Southwest Educational Oevelopment Laboratory's Evaluation Services (ES) recently completed its second year
of the evaluation of multiple professional development institutes (POlS) that the Oklahoma Commission for Teacher
Preparation (OCTP) offers to Oklahoma teachers. The purpose of this report is to provide OCTP with evaluation
findings pertaining to the quality and impact of the literacy POls offered in Oklahoma during the 2003-2004 school
year.
LITERACY PDI GOAls
OCTP provides professional development for Oklahoma teachers so that the teachers have an opportunity and the
tools to grow as educators and professionals (Oklahoma Commission for Teacher Preparation, 2002). Emphasizing
the importance of elementary school teachers' knowledge of what to teach as well as howto teach, OCTP
partnered with Literacy First to provide professional development in reading instruction. The goal of Literacy First is
to provide support and training to teachers to facilitate students' grade-level mastery of reading skills. Literacy First
trainings focus on increasing educators' knowledge of reading processes, such as phonics and comprehension, and
pedagogical strategies that use reading assessments to inform instructional decisions.
THE LITERACY FIRST PROGRAM
Literacy First, located in Mill Creek, Washington, offers a reading program for primary and secondary teachers
nationwide (Literacy First, 1999-2001). Program goals are to have 85 to 90% of students reading at grade level by
the completion of a three-year school wide implementation process. Literacy First's approach to increasing student
reading achievement is to staff schools with competent teachers who explicitly teach reading in a diagnostic and
prescriptive manner. Literacy First trains teachers in the skills needed to assess student reading ability and to use
student assessment data for developing individualized reading instruction. They also train teachers in instructional
competencies pertaining to the essential components of reading instruction. In Oklahoma, Literacy First offers four
phases of literacy training.
Phase I
The Phase I POI is a five-day training, that focuses on instructing teachers in a comprehensive, balanced reading
process and is offered to teachers across the state of Oklahoma. The training includes hands-on activities and
opportunities for teachers to develop practical classroom assessments. The five days are divided into two sets of
consecutive days. The break between the two sets of training days allows teachers to try the strategies in their
classrooms that they learned during the first session.
Phase II
The Phase 1/POI is a three-day training that builds on the Phase I training. It instructs teachers in using flexible skill
groups and developing literacy centers. It also addresses topics such as cooperative learning, collaborative
planning, and guided reading. Ouring Phase II, teachers have opportunities to discuss problems or concerns with
their implementation of Phase I strategies.
Teachers who complete the Phase I POI can attend the Phase II POls. Both Phase I and Phase II POls provide
participants with resources for developing meaningful and fun reading activities.
Phase I and Phase II POls offer two levels of training each: (1) Kindergarten through second grade and (2) third
through sixth grades. The Grades K-2 curriculum includes:
• Phonological awareness
• Phonics
• Spelling
• Comprehension
The Grades 3-6 POI curriculum includes:
• Pre-requisite information
• Phonological awareness
• Phonics
• Spelling
• Comprehension
• Vocabulary
Phase III
The phase III POI is a two-day training for school administrators. Phase III training provides school administrators
with strategies for facilitating teachers' implementation of Literacy First strategies. An overview of the Phase I and
the Phase II POls that their teachers attend is also provided.
Phase IV
Phase IV is a three-year process that promotes reading instruction reform at the school level. A school's
commitment to this process includes the school administrator's support of their teachers' implementation of Literacy
First strategies, and that at least 80% of their teachers have attended Phase I and Phase II POls.2 The
commitment from the schools allows Literacy First program staff to collaborate with the schools to expand the
impact of the Phase I and Phase II trainings.
Schools that agree to participate in the Phase IV process are assigned a Literacy First consultant and are asked to
appoint one teacher at the school to become their Literacy Resource Specialist (LRS). Once becoming a Phase IV
school, principals and LRSs attend leadership trainings that are held six times each year during the three-year
process. At leadership trainings, principals and LRSs discuss their roles in faCilitating teachers' implementation of
2 The Phase IV program itself does not contain trainings for teachers. The Phase I PDI and the Phase II PDI are considered to
be the teacher trainings for Phase IV schools.
Literacy First strategies, and have opportunities to collaborate with others (Literacy First consultants and other
Phase IV school administrators and LRSs) on strategies for increasing implementation.
To initiate the Phase IV process, Literacy First consultants conduct a comprehensive school assessment. From this
information, consultants develop a Strategic Reading Plan in consultation with the principal and LRS. Consultants
facilitate and support the implementation of schools' Strategic Reading Plans by developing the capacity of the
principal and LRS to maintain the infrastructure and culture necessary to accomplish the goal of helping students to
read at grade level. Consultants visit their assigned schools six times a year, during which time they analyze
reading instructional practices, demonstrate reading lessons, identify intervention strategies for under-achieving
students, model teacher conferencing procedures for the principal, and serve as a catalyst and facilitator of the
change process. The LRS assists in the implementation of the process at the school by collaborating with the
consultant, the principal, and the teachers.
The Phase I, Phase II, and Phase III POls have been offered in Oklahoma through East Central University's
continuing education program since March 1998. The POls have been offered at major cities in Oklahoma, allowing
teachers in all regions of the state to participate. To accommodate teachers with busy schedules, POls are offered
on many different dates. During the 2003-2004 academic year, a total of 32 Phase I POls and 19 Phase II POls
were offered. More than a total of one thousand teachers and administrators participated in the Phase I and Phase
II POls during this academic year. However, only four Phase III POls were offered during the 2003-2004 academic
year due to small enrollment.
During the fall of 2003, 14 schools started the Phase IV program, and in the spring of 2004, six schools started the
program. Of those schools who had already started the process before the 2003-2004 school year, five continued
the program for a second year and 13 schools continued for a third year. Three schools left the program before they
completed the three-year process due to their districts' lack of funding. The number of 2003-2004 Phase IV schools
totaled 38 in 2003-2004 school year3.
Summary of Evaluation Findings
The following chart displays participant respondents in the increased knowledge in relation to instructional
outcomes.
Instructional Outcomes
, Instructional Outcomes TPehaacsheerIsV INVoTne-Pachhaesres
My increased content knowledge in phonological awareness, phonics,
vocabulary, fluency, and/or comprehension is helping me teach reading 63% 60%
more effectivelv.
iInadmividbueattleizreadbliensthtraunctiboenf.ore to use student assessment data to provide 61% 49%
I am able to assess students' reading skills more accurately than before. 59% 46%
I am able to literacy centers more, more efficiently, and more effectively. 59% 38%
sMtryatiengstierusctiIolnearhnaesd.not changed; I have not incorporated the Literacy First 4% 3%
Total Number of Respondents " ., . 56 37
To document Literacy First participant content knowledge and instructional strategy improvement, the Literacy First
staff constructed 20-item tests of the POI participants' knowledge of the content offered in these sessions,
administering the tests at the beginning of the institute sessions and at their conclusion.
3 Of the 38 schools, 2 schools are counted twice as each school runs two programs for different grade levels.
As shown in Table 1, during the fall 2003 POls, 270 grades K-2 teachers completed the pre-test and 253 completed
the post-test. Of those, a total of 230 teachers completed both pre- and post-tests. For the grades 3-6 POI
participants, 156 teachers completed the pre-test, 151completed post-tests, and a total of 136 completed both pre-and
post-tests.
Grade K·2 Sessions4
Figure 1 shows the percentage of teachers who correctly answered items on the pre- and post-tests for the fall
2003 and spring 2004 institute training sessions combined.
Percent of Teachers Correctly Answering Test Items
(Fall 2003 and Spring 2004 Combined)
.. CII .c Q9 E;
,
Z Q10
I E
.2.1. Qll
t; Q12
.C.I.I
K-2 Teacher Percentage Correct by Item
Total
~--~--- ------~- ----- ~-~-----~-~-~------~-----~~------- - --~--~
I
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
,bo I I I
82
~
", I <;4 j j
84 I " I <;7 I Ij
-I , ,
t;LI I I I 85 j
l~ , I I 'f ,ko f9
-t I 93 , I IdR I I 1 I
81 I,
I I I ':I" I 1 , I I I I
80 I I , , I
'7~1 j
92 I , 'Lln
I I I J I I I
62 ! I , , I II
; , I I Iko I
b9
I I 8f
-I , I I
':It; l I I
-I I I I I 57I
171
-t 82
" I
<4 1 "j I I
93 -l ,
I' I
I' , ,
12
-f r I "'(
'.,n
, i ,60 I 41
60 i I .,~I I ,
, 85 I I 1 , ~1 , I II 86 1 I I ! ,ko I I I
88
III
I,
I
I
l.pre-te~1
I Post-te?!JI
I
I
100
Q13
Q14
Q15
Q16
Q17
Q18
Q19
Q20
o 10 20 30 40 50 60 70 80 90 __~ ~_ ___ ~~~:cen~___ I
As can be seen in the above Figure, the teachers' knowledge on the topics was lowest at the time of the pre-test on
items 15 and 16, and highest on items 8, 10, and 13. For all but one test item, the percentage of teachers correctly
responding to the items increased from the pre- test to the post-test. Highest gains were seen on items 14 and 18
(59%), item 1 (53%), and item 7 (48%).
Grade 3·6 Sessions
The following graph shows the percentage of teachers who correctly answered items on the pre- and post-tests for
the fall 2003 and spring 2004 institute training sessions combined.
Percent of Teachers Correctly Answering Test Items
(Fall 2003 and Spring 2004)
Grades 3-6 Teacher Percentage Correct by Item
Total
Q1
Q2
Q3
Q4
Q5
Q6
Q7
.. Q8 Q) .Q Q9
E
~ Q10
E
•~.. Qll .•..
r:-3 Q12
Q13
Q14
Q15
Q16
Q17
Q18
Q19
Q20
0
iiiiiiiiiiiiil 9 • Pre-test
iii Post-test
••• _ •• _iiiil88
20 40 60 80 100
Percent
THE PHASE IV PROCESS
The emphasis of Phase IV of the Literacy First program is on implementing the knowledge and skills teachers
gained through attending Phase I and Phase II institutes. The information gathered through surveys and interviews
indicated that teachers had gained knowledge and skills from the Phase I and II trainings. They reported that they
were primarily pleased with the support they were receiving from their administrators, Literacy First consultants, and
the LRSs. They further noted that their instructional practices were changing, which was subsequently impacting
their students' reading performance.
While teachers were generally satisfied with the support they received, many also noted that they received "mixed
messages" from their LRSs, administrators, and consultants as to what they were supposed to be doing, and how.
Some also mentioned feeling that the program was too prescriptive or rigid, not allowing for flexibility related to
individual student needs or teaching styles. Perhaps the most stated need by teachers was for further training
and/or more modeling of Literacy First techniques. Even principals and LRSs voiced their need for further training or
"refreshers" in these areas as well as in observation and feedback skills, and developing and providing resources to
their teachers.
Year 4 School Teachers. Teachers at the schools that completed the Phase IV program at the end of the 2002-
2003 academic year were also asked about their continued use of Literacy First strategies. As seen in Table 15, 58-
78% of teachers reported currently using the strategies "all the time." Only a small percent of teachers indicated that
they were not using the strategies.
Year 4 Teachers' Current Use of Strategies
Literacy First strategies Not at Some of All the Total
All the Time Time (N)
Assessment 1% 21% 78% 90
Use assessment data to plan lessons 0% 36% 64% 90
Phonological awareness 4% 32% 64% 89
Phonics 6% 28% 66% 89
Vocabulary 1% 41% 58% 88
Comprehension 1% 24% 75% 90
Fluency 3% 30% 67% 91
Flexible skill groups 2% 31% 67% 91
Literacy centers 4% 22% 74% 91
Note: Total number of surveys returned = 91; response rate = 34%
The vast majority of Year 4 teachers that responded to this item (96-99%) also reported that they intended to
continue using Literacy First strategies in the future. Comments by these teachers indicate their value of these
strategies, and included:
• I have various Literacy Centers. I will never go back to the traditional method of teaching.
• I let the assessment determine the instruction; it saves a lot of otherwise wasted time.
• Flex group time gives the teacher and students the time necessary to concentrate on specific needs.
• I will continue to use Literacy Centers. I have also added centers for math. They give me more small group
and one-on-one time.
• I use assessment to guide instruction and group my students accordingly. I will maintain this - it works!
In summary, more teachers in all years of the Phase IV program implemented Literacy First strategies at a higher
level this year than last year. Further, in the third year of the program, a larger percentage of the teachers
maintained their level of strategy use than did first or second year teachers, and under five percent of all teachers
indicated using the Literacy First strategies less this year than last. A large majority of Year 4 teachers also
continued to use Literacy First strategies even though they completed the program last year. Because of the
benefits they have experienced, many of them expect to continue using them in the future. These findings may also
reflect that through the routine use of many of these strategies, teachers integrate them into their instructional
practices to the extent that they remain usable and valued methods for reading instruction even after a school has
completed the Phase IV program.
Conclusions
The findings presented in this evaluation report are a synthesis of data collected from Phase I and II Literacy First
POls as well as from the Phase IV program. They are presented to our client, the Oklahoma Commission for
Teacher Preparation, to assist in their understanding of the quality and impact of the literacy professional
development currently provided to Oklahoma teachers. The following is a summary of the findings and
recommendations based on these findings to assist Literacy First in considering next steps for increasing the
effectiveness of the program for Oklahoma teachers.
PHASE I AND /I POls
• Phases I and" are multiple day workshops where groups of teachers from different schools and districts
are presented with information and skills focused on reading instruction.
• The content of Phase I is a basic introduction to literacy instruction. Phase" workshops focus on
implementing knowledge and skills that teachers acquired from Phase I.
• Survey findings indicated that most of the teachers who attended Phase I and II workshops during the
2002-2003 academic year valued the content presented and believed it to be useful in improving reading
instruction.
• They agreed that they had learned new knowledge and skills from attending the POls, and that they were
implementing them to various degrees.
• Teachers who were currently in a Phase IV school tended to implement the knowledge and skills to a
higher degree than teachers in non-Phase IV schools.
• The majority of non-Phase IV teachers also reported that they implemented a fair portion of the content and
strategies they gained at the POls.
• Teachers who had taken the Phase I and II POls from Phase IV schools had more consistent support than
those from non-Phase IV schools. 91% of the teachers from Phase IV schools perceived their
administrators as more highly involved.
• Large majority of non-Phase IV schoolteachers reported their administrators to be "interested" and
"supportive," even if not being actively involved in assisting these teachers.
• Both Phase I and II past participants indicated that their training had led to instructional changes, in that
they were more knowledgeable about the various literacy topics, were more able to assess students'
reading abilities, and use the assessment information to better individualize their instruction.
• Teachers noted that they were perceiving student impacts such as improvements in reading ability, more
interest in reading, and their enjoyment of Literacy Center activities.
RECOMMENDA TlONS
Literacy First achieves breadth in offering a large number of Phase I and II institutes each year to teachers in
Oklahoma. Depth is also attained through focusing on a wide variety of concepts and skills related to literacy
learning. In meeting the Oklahoma requirements for professional development, Literacy First provides 30-45 hours
of training, separated into two blocks (Phase I) to allow teachers to practice the knowledge and skills they are
learning. In addition, the three days of Phase II provides teachers with a forum for further reflection on the results of
their implementation of program content back at their schools as well as further training in Literacy First skills.
Teachers, however, continue to express the need for more practice and training to better implement the Phase I
and II knowledge and skills at their schools.
• Literacy First staff might consider restructuring Phase I and Phase" institutes to allow teachers more
practice time at their schools in between POI meeting days, instead of only one block of practice between
the two sections of the Phase I institute and none in the Phase II institute.
• Literacy First staff might consider incorporating a school-wide one-day orientation into the Phase IV
program. This orientation would occur prior to getting the program underway, and focus on an overview of
school staff roles in implementing the Phase IV program (e.g., administrator, LRS, and teachers), a review
of highlights of content and major strategies and techniques, and time for questions and answers by school
staff.
• The role and responsibilities of the LAS, the Literacy First Consultant, and the administrator(s) should be
explicitly laid out, so that all staff will have an understanding of the type of support they can expect from
each.
OEAMATH
PROFESSIONAL DEVELOPMENT INSTITUTE
• OEA focused on recruiting districts rather than individual teachers for the Professional
Development Institute (PDI). As a result, participation remained at a high percentage with
school wide change and improvement expected.
• 83% of participants felt more knowledgeable about the standards and were better able to
integrate them into their teaching practice.
• The highest gains were in Mathematical Reasonings and Proof, and in Using Writing About
Mathematics.
• Participants report that the PDI was the best professional development of their careers, and
indicated that the skills and strategies would be transferred to their classrooms.
IMPACT
REFLECTIONS
• These trainings validated what research has been telling us: long-term professional development
utilizing a quality curriculum will have significant impact on teaching practice. Initial training
with multiple follow-up sessions shows the importance of allowing participants time to reflect
on and practice new knowledge and strategies.
• Keeping current in the teacher's content area is important to the knowledge level of teaching.
Students benefit from teachers who understand their content.
• Administrators who participated in the training indicated they felt more prepared to encourage
their teachers to implement the new math strategies from the PDI.
OSU/STILLWATER PUBLIC SCHOOLS CONSORTIUM
MATHEMATICS
PROFESSIONAL DEVELOPMENT INSTITUTE
IMPACT
• Teachers reported an increased confidence in their ability to teach for conceptual understanding
and gains in useful teaching strategies.
• Positive feedback from school administrators was received as they assessed the impact on
teachers and on student learning - "As a principal and former classroom teacher of 20+ years, I am always
open to new and improved ways of learning. I have witnessed students with whom we have had trouble in the past
become excited about math class. They appear confident in their new found ways of learning. I was very impressed
with the new attitudes that seem to be surfacing among the students toward learning ."
• Videotaping lessons are powerful evidence of teacher practice, even though initially there was
resistance - "It was a wonderful experience .. .! learned a lot from watching not only myself but the others as well.
I enjoyed the feedback from the other teachers because it helped me realize the good things I have been doing."
• The largest gains for participants occurred in the area of using mathematical investigation as a
method for student learning.
REFLECTIONS
• Having quality trainers who have implemented the Connected Math Program curriculum was
vital to the learning.
• Developing the capacity for understanding conceptual learning will require extended time.
• Respondents indicated that the instructional content was useful, the organization was excellent,
the materials were very helpful, the presentation was stimulating, and the involvement of the
participants was excellent.
• Teachers need resources to teach problem-based mathematics. They also need a book for each
student, which was limited during the PDI.
Executive Summary
OEAlOSU Math POI
The purpose of this report is to present evaluation findings to the Oklahoma Commission for Teacher
Preparation (OCTP) regarding the effectiveness of the Mathematics Professional Development Institutes
(POls) for FY03-04. The Mathematics POls are designed to provide middle school mathematics teachers
with instruction and practice in implementing classroom strategies, which is intended to improve standards-based
mathematics performance of middle school students in Oklahoma. The POls used the Connected
Mathematics Project (CMP) curriculum, which is student-centered, hands-on, and is organized around
several major themes such as the "big ideas" of mathematical concepts, making connections among
different mathematics topics in meaningful ways, and teaching for student understanding by guiding
students to use discovery methods in practical problem-solving situations.
Two institutions, the Oklahoma Education Association (OEA) and Oklahoma State University (OSU),
conducted mathematics POI's for the state. OEA held POls in Chickasha, Woodward, and Ardmore,
Oklahoma. OSU's POls were located in Ada, Stillwater, and Tulsa, Oklahoma. OEA completed two of their
POls in the Spring of 2004, and OSU completed only one. Due to the low number of participants at the one
institute that OSU completed during the spring, and the even lower response rate to the end-of-institute
survey, findings focus primarily on the OEA POI's, although a discussion of the OSU POI participants'
responses is also provided.
OCTP contracted with the Southwest Educational Development Laboratory (SEOL) in 2002 to evaluate the
mathematics POls. This report presents findings from the second of a three-year evaluation. The evaluation
design for the second year consisted of a survey of mathematics POI participants and observations of
several OEA and OSU training sessions. The evaluation sought to address the following questions:
1. Did participants perceive the mathematics institutes to be valuable and of high quality? (Participant
Reactions)
2. What knowledge and skills did participants gain as a result of the mathematics institutes?
(Participant Learning)
3. What aspects of organizational support do teachers perceive they need to facilitate the application
of their new skills and knowledge? (Organizational Support)
4. How do participants expect to use their new knowledge and skills following the institute?
(Participants' Use of New Knowledge and Skills)
5. What impacts to student learning do teachers expect to see based on the implementation of the
knowledge and skills learned at the POls? (Impact on Student Learning)
Key Findings
Key findings were based on survey responses from 60 of 100 teachers who participated in the OEA and
OSU 2003-04 Mathematics POls.
Participant Reactions
~ Teachers found the institutes to be valuable and well organized. Respondents:
• rated the institutes high on content covered, organization, presentation, materials, and
participant involvement.
• gave highest ratings to having opportunities to share ideas and learn from other
participants and to the instructors' knowledge.
Participant Learning
~ Teachers increased their knowledge and skills after participating in the institutes. Respondents:
• reported more confidence in their ability to teach for conceptual understanding.
• reported increased ability to use diagnostic assessment as a tool to improve student
learning.
Regarding the learning of math content, respondents' highest gains were in using mathematical
investigations and student discovery as methods for student learning.
Administrators' Learning
Administrators received valuable information and skills from their institute training.
• Administrators gave highest ratings to gaining valuable knowledge about their teachers'
math training and to feeling more able to support their teachers' implementation of the
math curriculum that they are learning at the POls.
Participants' Use of New Knowledge and Skills
)- Teachers increased the use of knowledge and skills learned at POls. Respondents:
• reported increases in their use of new knowledge and skills.
• reported that they were more prepared to teach geometry and spatial sense and to use
writing about mathematics in their classroom instruction.
Impact on Student Learning
~ Teachers' changes in instructional practices are expected to impact their students. Respondents:
• expect positive impacts on their students' performance.
• felt optimism for using their newly acquired strategies for improving student learning in
mathematics.
Key Recommendations
Respondent Recommendations
~ Teachers expressed the desire for more training, more planning time, and more time to collaborate
with other teachers. They also perceived a need for more direct connection of the curriculum with
the OKJNCTM PASS standards during the institute.
Evaluator Recommendations
~ Promote the use of content tests for assessing the effectiveness of the math training in improving
teachers' content knowledge, and for helping the trainers to understand what concepts teachers
need additional help with learning.
)- Promote the importance of the administrators' participation in the POls as school leaders are often
key in the successful implementation of new instructional strategies or approaches aimed at
improving student performance.
)- Extend the evaluation of the Mathematics POls to include a more specific examination of the
effects of teachers' implementation of POI knowledge and skills.
OEA Mathematics POI Evaluation Findings
This section includes evaluation findings from the OEA Mathematics POI. Findings are organized according
to the five evaluation questions and are based on the post-institute surveys, teachers' mathematics pre-and
post-content assessments, administrator survey responses, and observations of a sample of institute
training sessions.
OEA Survey Respondents
• Site 1: Woodward (n=22)
• Site 2: Chickasha (n=38)
• Site 3: Ardmore (n=32)
Site Pa#rtiocfipPaOnIts i #CoofmSpulervteedys ResRpaotnese
Ardmore 32 14 44%
Chickasha 38 26 68%
Woodward 22 16 73%
Total 92 56 61%
OEA Participants I Reactions
This section discusses findings on participants' reactions to OEA PDls that were collected using two
methods: (1) a summary of responses from the post-institute survey and (2) a summary of observations
made during two days of training in January 2004.
Post-Institute Survey Responses
In a survey administered at the end of the institute, participants were asked to rate their perceptions of the
quality of the institute they attended on a scale of 1 to 5 as to its content, organization, presentation,
materials, and participant involvement. Table 2 displays their responses.
Survey respondents' overall reactions to the institute were positive. The majority of the respondents
indicated that the institute content was very useful (57%), the organization was excellent (70%), the
materials were very helpful (63%), the presentation was stimulating (66%), and the involvement of the
participants was excellent (73%).
Percentages of Overall OEA Institute Ratings
Of little use 2 3 4 Very useful
Content 0% 0% 5% 38% 57%
Poor 2 3 4 Excellent
Organization 0% 0% 4% 27% 70%
Boring 2 3 4 Stimulating
Presentation 0% 0% 5% 29% 66%
Ohf eliltptle 2 3 4 Very helpful
Materials 0% 0% 7% 30% 63%
Poor 2 3 4 Excellent
InPvaorltviceipmaenntt 0% 0% 4% 23% 73%
The Mathematics POI participants were also asked to indicate whether they agreed or disagreed, on a 6-
point scale, to statements regarding specific characteristics of the institute at the end of the training. These
characteristics were related to the structure and organization of the training (e.g., whether objectives of the
institute were clear, trainers were experienced, and active learning opportunities were provided). The
following table presents the percentages of responses for each category:
Percentages of Specific OEA Institute Ratings
c,r' .~ , '" Strongly ,,' ". Strongly
Il" Disagree 2 3 4 5 Agree
The objectives were clear. 0% 4% 2% 13% 34% 48%
tThheeoibnjsetcruticvteosr.s adequately addressed 0% 0% 4% 9% 23% 64%
tIhnestrcuocntotersnt.were knowledgeable about 0% 0% 2% 2% 21% 75%
The instructors had experience
implementing strategies being 0% 0% 2% 2% 23% 73%
discussed.
aTchteiveselsesaiornnisngi.nvolved participants in 0% 2% 2% 7% 18% 70%
pTrhaectiscees.sions provided opportunities to 0% 2% 2% 5% 20% 71%
iPdaeratiscipaanndtsleahrandfroopmpoorttuhneirtiepsartticoipsahnatrse. 0% 0% 0% 7% 13% 80%
Itewaochueldrsr.ecommend this training to other 0% 2% 4% 7% 27% 60%
OSU Mathematics POI Evaluation Findings
This section includes evaluation findings from the OSU Mathematics POI. OSU provided four POls during
the 2003-2004 academic year. The POI offered in Ada, Oklahoma is the only one that was completed by
the time of this report writing. Therefore, only the Ada POI data are summarized. Findings are organized
according to the five evaluation questions and are based on the post-institute survey, the mathematics
content pre- and post-assessments, and observation of an Ada institute training session. The post-institute
survey asked for participants' perceptions of the quality of the training, participant learning, organizational
support for their implementation of participant learning, expected use of new knowledge and skills, and the
expected impact on participants' students.
Overview of OSU Survey Respondents
Oemographics and background information were collected from the OSU POI participants using the
Demographic Survey. These data are reported for the following OSU mathematics POI participants (n=45).
• Site 1: Tulsa (n=8)
• Site 2: Ada (n=8)
• Site 3: Tulsa IV (n=10)
• Site 4: Stillwater II (n=19)
Summary of OSU (Ada PDI) Findings
The four Ada POI respondents indicated that overall, the institute content was useful to them, its
organization was excellent, the materials were very helpful, the presentation was stimulating, and the
involvement of the participants was excellent. Positive ratings were also given for having clearly and
adequately addressed training objectives, involved participants in learning activities, solicited feedback, and
provided opportunities for participants to share.
Comments and content test results indicated that these participants gained new knowledge and strategies
for teaching mathematics, and expected that they would change their instructional practices based on what
they had learned. They also expected to see positive changes in their students' attitudes about learning
math, confidence in their math ability, and performance on math assessments. Indeed, these reports of
increased confidence in their ability to teach for conceptual understanding and gains in useful teaching
strategies, illustrates the value of modeling instructional practices that participants can use with their own
students as observed by the evaluator during the third day of training in the Ada institute. Similarly, having
POI participants work in groups and share their problem-solving methods, tangibly demonstrated how
teachers could structure mathematical investigations for their own students.
Given that these findings are based on only a small number of respondents, it is premature to make any
definitive statements about the progress of OSU's POls toward attaining their goals.
Conclusions
This report presented evaluation findings regarding the quality and impact of the OEA and OSU
Mathematics POls for Oklahoma middle school mathematics teachers offered during the 2003-2004 school
year. These Mathematics POls represent the first in a three-year program. Findings from this first year of
the Mathematics POls addressed five evaluation questions: (a) did participants perceive the mathematics
institutes to be valuable and of high quality; (b) what knowledge and skills did participants gain as a result
of the mathematics institutes; (c) what aspects of organizational support exist at their schools to help or
hinder the application of teachers' new knowledge and skills; (d) how do participants expect to use their
new knowledge and skills following the institutes; and (e) what impacts to student learning do teachers
expect to see based on the knowledge and skills they learned at the POls?
A summary of evaluation findings is presented below according to these five questions. Because only one
of the OSU institutes had taken place at the time this report was written, the majority of findings are based
on participant responses to the OEA institutes.
Participants' Reactions
Survey respondents' overall reactions to the institutes were positive. They indicated that the institute
content was useful, the organization was excellent, the materials were very helpful, the presentation was
stimulating, and the involvement of the participants was excellent. They also provided positive ratings in
terms of participant involvement in the training, receiving instructive feedback, and having opportunities for
sharing with other teachers. Sixty percent of respondents strongly agreed that they would recommend the
training to other teachers.
The evaluator's observations made during one of the OEA POls reflected the survey findings. It was noted
that the trainers used practical experiences to demonstrate conceptual and/or abstract thinking.
Participants were guided through ways to introduce, clarify, and review concepts, which connect them to
real-world experiences. In addition, the trainers modeled cooperative methods of learning that allowed
participants to collaborate in pairs or teams to develop problem-solving strategies and find solutions to the
problems. In both OEA and OSU POls, trainers modeled the instructional strategy that POI participants
were expected to use in teaching their students.
Participants'Learning
Findings from surveys, content tests, and observations indicate that POI participants appreciated and
valued the institutes and expected that the instructional strategies and skills they learned will be useful for
instructing middle school students. Just over half of survey respondents strongly agreed that they had
gained useful knowledge and 55% strongly agreed that they had gained useful strategies for teaching
mathematics. Respondents reported that after the training they were most prepared to teach for conceptual
understanding but least prepared to assess student knowledge and create standards-based lesson plans.
This reflects the POI focus on mathematics concepts in the first year and the future focus on assessments
and lesson plans in the second and third years.
In terms of how prepared teachers were to apply specific skills and content knowledge in the classroom
prior to and at the end of the math training, respondents generally felt more prepared in all areas, with
largest gains for using mathematical investigations as a method for student learning, and learning through
student discovery. With respect to teaching specific topics, largest gains were seen in teaching
mathematical reasoning and proofs, writing about mathematics, and geometry and spatial sense.
For many respondents, their perceived gains in content knowledge were corroborated by assessed gains in
their content knowledge through a mathematics pre- and post-content test taken at the beginning and the
end of the training. However, the gains were generally small. The highest percentage of participants
experienced only a slight gain of 1 point in pre- to post-test scores. Although just under half of the
participants had gains of 2 or more points, 16% did not experience any gains and 10% had lower scores on
the post-test than on their pre-tests.
These findings were perplexing and somewhat disconcerting given the emphasis on improving teachers'
content knowledge in the first year of the POI. However, the POI vendors, through informal communications
with the evaluators, indicated that the importance or relevance of these contents tests were minimized
when tests were administered to teachers. From this we might conclude that participants may not have
performed as well as they could have if they were aware of the role the tests may play in determining the
effectiveness of the program. Nevertheless, more investigation is needed to determine the extent to which
teachers are truly learning about and improving their knowledge of various mathematical concepts.
Organizational Support
Regarding the kinds of additional support that might help teachers implement the knowledge and strategies
they learned in the institutes, more computers, calculators, mathematics manipulatives, and Internet access
were mentioned. Respondents also noted a desire for more opportunities to observe other teachers or
trainers in classroom settings, and for networking structures to facilitate collaboration and support for
sharing ideas and information with teachers they met at the institutes as well as teachers at their schools.
Other comments included having more teachers at their schools trained in the math curriculum and
attending follow-up trainings.
Administrators reported that they felt more able to support their implementation of the math curriculum as a
result of participating in the administrator training. After the training, they were stated that they most
prepared to encourage their teachers to implement the new math strategies from the POI, to provide
teachers with the resources necessary to effectively teach the math curriculum, and to promote the effort to
improve student math performance in meetings.
These administrators also perceived small changes in teachers' behaviors beginning to occur. They also
mentioned that they were providing various methods of support, such as helping with curriculum alignment
and ensuring that teachers have the necessary materials to implement the math strategies. Most
responded that student impacts had not occurred yet because it is still too soon to tell or predict. Several
challenges to implementation were cited by the administrators, including insufficient funds for textbooks,
time for teachers to plan lessons and implement training strategies in the classroom, and time during the
school day for teachers to share ideas with one another.
Use of New Knowledge and Skills
Findings indicated that many teachers are able and willing to actively try new strategies for engaging their
students in mathematics learning activities. For example, teachers felt confident in their abilities to
effectively implement what they learned in the institutes and stated that they planned to use more hands-on
exercises in their mathematics instruction. Many expected to emphasize discovery and hands-on learning
and use small learning groups to explore and solve mathematical problems collectively using manipulatives
and/or learning games. In contrast, the teachers expected that they would use fewer worksheets and
bookwork to teach math, and less lecturing or '1elling" during lessons. These are positive indicators that
teachers are well prepared to implement what they learned in the first year of the POls.
Expected Student Impacts
The POI participants were asked what changes they expect to see in their students if they implement the
new instructional strategies. Many teachers expect to see positive changes in students' attitudes,
confidence, higher-order thinking skills, and performance on assessments. They also expect that these
results would be partly due to students working more collaboratively with each other. Several teachers
mentioned that they expected their students to have more "interest" in learning math. While these
predictions of student impacts are positive, further evidence will be needed to determine students'
responses to the new instructional approaches in terms of such factors as motivation, engagement, and
mastery of standards. Ultimately, it will be important to explore whether students are able to perform better
on high-stakes assessments as a result of teachers' implementation of the new approaches to teaching
mathematics.
Summary
Overall, we conclude that the Mathematics POls are making progress toward their goals (although limited
information was gathered from the OSU participants due to the timing of training sessions in relation to the
writing of this report). These findings are particularly important for gauging the progress that the POls are
making in this first of the three-year program. As described in the overview, in this first year of the OEA and
OSU POls the focus was primarily on developing teachers' content knowledge and understanding of
mathematics concepts. Indeed, survey respondents reported that they gained useful and valuable
information from the training and that they are eager to implement what they learned. One concern was
raised by the small gains in scores on the content tests from the beginning to the end of training. Although
one might expect more teachers to have mastered the content tests by the end of training, it is uncertain
whether teachers took these assessments seriously and performed according to their true knowledge and
ability. Nevertheless, this is something for OCTP and the POI vendors to consider as they continue with the
second and third years of the training.
The ultimate value of professional development programs is the ability to help teachers transform the
learning environment for their students. Evidence clearly suggests that participants are learning from, and
find utility in, the Mathematics POI training. One limitation to these findings, however, is that these are self-reported
perceptions and, therefore, they do not reflect how well the POI learnings actually transfer to the
classroom. Furthermore, limited information has been gathered regarding the long-term support that
teachers receive to implement what they learn.
In line with OEA's and OSU's plans for the second and third years to focus more on implementation and
support, this evaluation will also explore the extent to which participants of the POls, including
administrators, implement what they have learned and whether adequate supports are in place to facilitate
implementation.
Recommendations
Recommendations for program improvement are discussed below, beginning with a summary of
participants' recommendations reported in the post-institute survey, followed by a set of evaluator
recommendations, which are based on the overall findings from the 2003-2004 evaluation of the
Mathematics POls.
Participant Recommendations
While progress is being made toward the goals and objectives of the POls, participants offered suggestions
for furthering the effectiveness of the POI trainings. Suggestions included:
• More time to collaborate with other teachers,
• More alignment of the POI curriculum with the OKINCTM PASS standards, and
• More opportunities to observe trainers teach a classroom lesson.
Teachers also expressed the need to maintain contact with their POI colleagues. They suggested the use
of a ListServ and/or an interactive web site. It may be beneficial to explore this possibility. As an example,
electronic mailing groups on commercial Internet sites, such as Yahoo, are a free resource that may be
used to create a networking community for the exchange of information and ideas. Other helpful sites
containing resources that Middle School Mathematics teachers might find useful are:
• Middle Web (http://www.middleweb.com/mw/listserv/MWLarchive.html)
• The Math Forum at Drexell University (http://mathforum.org/)
• Michigan State University Connected Mathematics Math Web Site (http://www.mth.msu.edu/cmp/)
Evaluator Recommendations
Based on the overall findings discussed above, SEOL presents the following recommendation for OCTP to
consider in its ongoing efforts to enhance and improve the Mathematics POls.
• Promote the use of content tests for (a) assessing the effectiveness of the training in improving
teachers' content knowledge and (b) helping the trainers to understand what concepts teachers
need additional help with learning during the training. Research has shown that more highly
qualified teachers, particularly those who are knowledgeable and certified in their content areas,
are more effective in helping students learn and perform better on assessments. Raising the
awareness and importance of teachers' content knowledge through the pre- and post-tests may
serve the dual function of reinforcing teachers' learning and enhancing accountability of the POls to
deliver what they purport to do.
• Promote the importance of the administrators' participation in the POls. School leaders are often
key to successful implementation of new instructional strategies or approaches because they can
often find time for teachers to engage in activities that improve teachers' learning and effectiveness
with students, such as planning lessons, analyzing student work, sharing strategies, and observing
each other's classes. The administrators who responded to SEOL's survey indicated that they
valued their participation in the training because they became familiar with what teachers were
learning, but some indicated that they would like additional strategies in their roles as
administrators for helping teachers implement what they learned in the POls. OCTP might consider
requiring principal participation, or at least encouraging principals to participate. Further information
about the benefits of principal participation will be useful for determining more precisely how
teachers benefit from this type of principal support.
• Extend the evaluation of the Mathematics POls to include a more specific examination of the
effects of teachers' implementation of POI knowledge and strategies. For example, student
performance in specific learning activities might be structured to elicit differences in students'
mathematics learning and understanding before and after teachers attend the POls. While POI staff
expects student products to be submitted by teachers attending the POls, they are currently not a
part of the evaluation. Student data would aid in moving the their evaluation to a stronger position
in assessing the overall quality of the mathematics POls.
SUCCESS IN SCIENCE
PROFESSIONAL DEVELOPMENT INSTITUTE
IMPACT
• The Science PDI had 258 participants with an average of 15 years teaching.
• The average number of years since the respondents had taken a college-level science content
course was 12 years.
• 82% of the respondents "Strongly Agreed" they had gained useful knowledge and strategies for
teaching science.
• Huge growth was seen in teachers' understanding of Teaching Learning Through Student
Discovery (2.11 % to 3.5%) and Using Hands-On Science Modules (1.5% to 3.7%).
• Respondents reported seeing students improve in multiple areas including more positive
attitudes, an eagerness and excitement to learn, greater interest, better understanding and
retention, and overall better performance.
• 95% of the participants recommend this Science PDI as quality professional development
REFLECTION
• Teachers involved with Success in Science PDI are reporting positive student impacts. There is
a need to increase the availability to more teachers to impact learning at a greater rate.
• The limited number of professional development days is an obstacle in promoting quality
learning for teachers.
• Even though the funding is not available to expand this PDI, school districts have been willing to
fund the cost of resources if Success in Science PDI will provide the training. The results speak
loudly.
Executive Summary
Success in Science Professional Development Institute
The purpose of this report is to present evaluation findings to OCTP regarding the effectiveness of the
Success in Science Professional Development Institute (SSPDI) offered during 2003-2004. The OCTP's
goal for the science professional development institutes is to provide professional development that
enables teachers to make science more accessible, understandable, and engaging for students.
Key Findings from FY 03-04
~ Participants continued to rate the SSPDI as satisfactory, valuable, and well designed. Respondents
consistently.
• rated the institutes high on the content covered, organization, presentation, and materials;
• gave overall highest ratings to the instructors' knowledge of the content and experiences
implementing institute strategies; and
• indicated they would recommend this training to other teachers.
~ Teachers reportedly increased their knowledge and skills after participating in the science
institutes. Respondents consistently:
• reported highest ratings in knowledge in their confidence teaching for conceptual
understanding in science; and
• reported highest gains in their preparedness to use science module materials and notebooks
or journals for assessment.
~ More than half of the respondents (63%) strongly agreed that they felt confident in their ability to
effectively implement what they learned at the SSPDI.
~ Teachers (former SSPDI participants) reportedly applied the knowledge and skills gained during
the science institutes in their instruction. Respondents:
• reported highest ratings in using the science module materials and teaching using student
discovery and science investigation as a method for learning; and
• reported highest gains in students' motivation and active involvement in science learning.
~ Teacher responses to the open-ended questions provided strong evidence that the teachers felt
confident in the knowledge and skills they learned from the SSPDls and that their new knowledge
has impacted their instruction and their students in very positive ways.
Overview
Science Institute Goals
The OCTP's goal for the science professional development institutes is to provide professional
development that enables teachers to make science more accessible, understandable, and
engaging for students. The OCTP aims to provide professional development that:
• adheres to professional standards for science teacher professional development;
• increases teachers' level of content knowledge in science;
• improves teachers' ability to use inquiry-based instruction methods; and
• trains teachers to value the role of communication, both verbal and written, as a tool to aid student
learning.
Approach to Attaining Goals
Three members of the Oklahoma Teacher Education Collaborative (Southwestern Oklahoma State
University, University of Tulsa, and Oklahoma State University's Center for Science Literacy) submitted a
bid and were awarded the contract to conduct the Success in Science Professional Development Institute.
The SSPDI collaborative proposed to provide Oklahoma K-6 elementary and 6-8 grade science teachers
with professional development to increase the science literacy of Oklahoma students by engaging teachers
in inquiry methodology for instruction of grade-specific science modules.
The SSPDls use Full Option Science System (FOSS) and Science and Technology for Children (STC)
curricula which are standards-based, science units that employ inquiry strategies to help students learn
science topics in meaningful ways. The professional development emphasizes a student-centered
approach with instruction focused on guiding students to use discovery methods in practical problem-solving
situations. Previous success with this model of science instruction led to expectations for significant
increases in student achievement in science and related courses.
The SSPDI participants learn to use grade-specific science modules during days of training that occur
during the school year. Teachers receive hands-on experience with the modules that are intended to
promote their reflection, planning, and implementation of inquiry-centered strategies in elementary and
middle school classroom instruction. The SSPDls are structured so that participants receive an initial
intense multi-day period of methodology and curriculum training with several single days of follow-up for the
Module 1 and Module 2 materials during the academic year. The Module 3 training involved teachers in
training in pedagogical and science instruction strategies to enrich their use of the module kits. The
teachers received a kit with instruction manual and necessary materials and supplies for approximately 16
lessons they use with their students throughout the module.
Evaluation Questions
The POI evaluation questions related to the science institutes are:
(1) Did participants perceive the science institutes to be valuable and high quality?
(2) What knowledge and skill did participants gain as a result of the science institutes?
(3) How do participants perceive that school leaders are supporting and/or hindering the application of
their new skills and knowledge?
(4) Do participants expect to be able to integrate training knowledge and skills into classroom
instructional practices?
(5) Are the science institutes producing gains in student learning?
The 2002-2003 Science POI evaluation centered on three of the evaluation questions and determined that
last year's Science POI participants found the science institutes provided training that was valuable, high-quality,
and useful for developing their knowledge and skills in science instruction. The participants also
indicated that they felt prepared and expected to use the knowledge and skills they gained in their
classroom instruction.
This SSPOI report is based on two primary sources of data: (1) an end-of-institute survey for this year's
(2003-2004) SSPOI participants, and (2) a follow-up survey of last year's (2002-2003) SSPOI participants.
The report will first address information obtained from the current SSPDI participants to verify information
gathered in the previous year. Then, the information from previous participants will be presented. The
evaluation questions, methods, and data sources are listed in the following table:
SSPDI Year 2 Evaluation
Evaluation Question Method Data Source
(1) Are SSPDI of high value and • End-of-Session Survey • 2003-2004 SSPDI (Module 3)
quality? • Observations Participants
(2) Do participants increase • End-ot-Session Survey • 2003-2004 SSPDI (Module 3)
knowledge and skill during • Observations Participants
Science PDI? • Follow-up Survey • 2002-2003 SSPDI Participants
(3) Do participants have school • Follow-up Survey • 2002-2003 SSPDI Participants
leadership support tor SSPDI
instruction?
(4) Are participants able to • End-ot-Session Survey • 2003-2004 SSPDI (Module 3)
integrate SSPDI knowledge and • Observations Participants
skills into classroom instruction? • Follow-up Survey • 2002-2003 SSPDI Participants
(5) Are there gains in student • Follow-up Survey • 2002-2003 SSPDI Participants
learning?
Did participants perceive the science institutes to be valuable and high quality?
At the end of the training, respondents were asked to rate the SSPDI on a scale of 1 to 5 whether (a) the
content of the institute was useful, (b) the institute was organized, (c) the presentations were stimulating,
(d) the materials were helpful, and (e) they were actively involved in learning. the following table indicates
that the SSPDI respondents average ratings for the institute was high (between 4.5 and 4.8) for all five
categories.
2003·2004 SSPDI Respondents' (N:56)
Percentages of Overall Institute Ratings
I,"
o Ofuslitetle 2 3 4 ." Veruyseful M(SeOan)
a. Content -- -- -- 10 (18%) 46 (82%) 4.8 (.39)
Poor 2 3 4 Mean
Excellent (SO)
b. Organization -- -- -- 14 (25%) 42 (75%) 4.8 (.44)
Boring 2 3 4 Stimulating Mean
(SO)
c. Presentation -- -- 2 (3%) 15 (27%) 39 (70%) 4.7 (.55)
Of little , Very Mean
help 2 3 4 helpful (SO)
d. Materials .. 1 (2%) 2 (3%) 23(41%) 30 (54%) 4.5 (.66)
Poor 2 I~ 3 4 Excellent Mean
(SD)
e. Participant .. .. 1 (2%) 8 (14%) 47 (84%) 4.8 (.43)
Involvement
Note: Ratings on a 5-point Semantic Differential Scale. The scale for each item ranged from: (a) "of little use" to
"very useful," (b) "poor" to "excellent," (c) "boring" to "stimulating," (d) "of little help" to ''very helpful," and (e) "poor"
to "excellent." N=number of participant responses. SD=Standard Deviation
What knowledge and skills did participants gain as a result of the science institutes?
After responding to items on the survey about their reaction to the SSPDI sessions, the participants had an
opportunity to indicate what knowledge and competencies they had obtained from the institute sessions.
Participants were asked to respond to eight items upon which they could indicate how prepared they thought they
were to use the instructional strategies that had been discussed in the SSPDI sessions. These strategies included
instruction on how to use science investigation, cooperative groups, or problem solving strategies as a method for
learning science. Participants rated these instructional strategy items (4-point scale, 1=not well prepared to 4=very
well prepared) twice: the first time to represent how prepared they currently thought they were to use these
instructional strategies now that they had participated in the SSPDI, and the second time to represent how prepared
they thought they had been to use these strategies before they participated in the SSPDI.
Average Ratings of SSPDI Module 3 Participant Preparedness
Perceptions of Skills and Knowledge Gains
Learning through student
discovery
~ 12.1 T
13.6
1 I,", n'
.,
12.2 .s.
13.6
I I 12.5
13.5
I
12.2 1
13,4
I 12.1 1-
13.5
I 11.6 j I
3.5 I 11.5 1
13,7
j I
12.1 I 13.4
. , -",,' !
losefore I
~ow
Science investigation as a
method for learning
Cooperative student groups
Problem-solving strategies
Question strategies to
promote reflection
Notebooks or journals for
assessment
Science module materials
Connecting content to real
world and careers
234
Four-Point Scale: Not well prepared (1) to Very well prepared (4) N=112
Do participants expect to be able to integrate training knowledge and skills into classroom instructional
practices?
Another goal of the SSPDI is to determine how the institutes impact the participants' instructional actions after they
complete the sessions
2003·2004 SSPDI Respondents' Expectations for Implementation
DSitsroanggrelye' 2 3 4 5 SAtrgornegely M(SeDan) N
tie. aI cfehelfomr ocroencceopntfuidaelntunindemrsytaanbdiilnityg to -- -- -- 2 17 37 5.6 (4%) (30%) (66%) (.56) 56
than I was prior to this training.
oim. pIlfeemelencot nwfidheant tI hthaavteIlceaanrneefdfecattivtheilsy -- -- -- (42%) (3149%) (6335%) (5.5.67) 56
Science POI. . . Note: N=Number of respondents. SD=Standard Devianon. Scale range from Strongly Disagree (1) to Strongly Agree (6).
Are the SSPDls producing gains in student learning?
Although we have no direct measures of student impact from the survey, the participant responses to open-ended
items (italicized questions in paragraphs below) describe an anticipated level of student impact as a
result of what participants had experienced while implementing the science modules during the time period
spanning the Science PDI.
What Student Impact Have You Seen? In an open-ended item on the SSPDI survey, teachers (n=56)
reported seeing improvements in their students during this year when they have begun to use the science
modules in their instruction in multiple areas. These included seeing more positive student attitudes, an
eagerness and excitement to learn, greater interest, better understanding and retention of what students
learned, and overall better student performance. Parents, teachers from other subjects, and the students
themselves were among those who reported to SSPDI participants that their science students "love to
come to class" and had a new enthusiasm for learning. As one participant mentioned, students were more
active in taking the initiative to learn, read, or create new problems to investigate. Students were also
developing as cooperative learners and becoming "more respectful of others work," as one teacher
reported. There were very few comments by teachers that showed a negative impact on students. One
commented that her students "didn't like to write in notebooks," but that they were "still working" on using
that learning method.
The above teacher responses to the open-ended questions provide evidence that the teachers feel
confident in the knowledge and skills they learned from the Science PDls and that their new knowledge has
impacted their instruction and their students in very positive ways. The next section details information from
the follow-up survey about teachers' perceptions about how the Science PDls have impacted their
instruction, affected student performance in science, and been supported by administrators and other
teachers at their schools.
Are the science institutes prOducing gains in student learning?
Most of the former SSPDI participants indicated that their students are more engaged in classroom
activities (94% agreed or strongly agreed, Item f).
Former SSPDI Respondents' Ratings
ASfctieerncpeartPicDipIa..t.ing in the Success in DSitsroanggrelye Disaegre Agere, SAtrognregely M(SeDan) N
f. cMlayssstruodoemntsacatirveitiemso.re engaged in (43%) (21%) (181%2 ) (7560%) (3.7.73) 66
Note: N=Number of respondents. SD=Standard Deviation. Scale range from Strongly Disagree (1) to
Strongly Agree (4).
Respondents also had the opportunity to respond to six items about the impact of implementing SSPOI
instructional strategies with their students. The following table lists those responses:
ormer espon en s a Ings 0 e mea 0 ns ruc Ion
Noatll at vNeoryt Some mVuecryh M(SeOan) N much
a. Mimyprsotuvedde.nts' science performance has (21%) -- (3232%) (6453%) (3.5.68) 66
b. Motyhesrtubdeetntetsr inintsecriaecntc/ceoocplaesrastees. with each (21%) -- (4371%) (5314%) (3.5.53) 66
c. Mscyiesntcued.ents are more motivated to study 7 59 3.9 66 -- -- (11%) (89%) (.31)
d. Mleayrnstinugdensctsienacree. more actively involved in -- -- (64%) (9641%) (3.2.94) 65
e. sMcyiesntcuedecnltasssaessk. questions more often in 19 47 3.7 66 -- -- (29%) (71%) (.46)
f. I learned from indirect sources (for
example, the parents, other teachers, 4 1 27 34 3.4
students' friends) that my students' (6%) (2%) (41%) (51%) (.80) 66
science motivation has improved.
F SSPDIR d t' R f f th I ct f SSPDII t tl
Conclusions
• This report provided confirmation that these three professional development levels continued to be
fulfilled by the SSPOI (the value and quality of the institute, the knowledge and skill gained, and the
expectations for integrating into classroom instruction).
• Provides evidence that according to the perspectives of former SSPOI participants who responded,
the final two levels of professional development (administrative support and gains in student
learning) are also present as a result of the quality of professional development to the K-8 teachers
who participated in the institute.
• Respondents liked the sessions, believed the inquiry-based strategies and skills are useful for their
classroom instruction, and expected to see changes in their instruction and in student learning in
their classrooms.
• Respondents reported that they thought they were more prepared now than they were before
attending the science training to teach using key instructional strategies discussed during the
institute.
• Success in Science POI Module 3 respondents reported gains in their ability to use inquiry-based
instructional practices that are student-centered, problem-based, and built around students'
collaborative investigation of scientific phenomena.
• Improved use of effective questioning strategies and methods for eliciting regular student verbal
and written communication as an aid to learning and assessment were other areas of improvement
that institute participants found especially promising for future instruction.
• Former SSPOI participants reported continued use of the materials and instructional strategies they
learned from the SSPOI with their students. The use of notebooking and journaling, assessment
tools, and instructional strategies associated with their grade level modules were some of the
specific SSPOI strategies mentioned by the teachers.
• Respondents also indicated that they used cooperative groups, discovery learning, and scientific
investigation techniques as a method for leading students in exploring the science ideas,
principles, and information presented through the science modules.
• The results indicated that generally, most respondents felt supported by their school leadership for
using the Success in Science POI materials.
• Overall, the teachers indicated that while they shared information and resources that they could
from the Science POI with their colleagues, they wanted more teachers in their school to participate
in the SSPOI so more students could benefit from these type of science learning experiences.
• In general, respondents indicated that students benefited from instruction using the SSPOI
modules and recommended strategies.
Recommendations
• Teachers want to participate in an additional SSPOI because they want to be prepared to teach
another module for the same or a different grade level
• The teachers also reported that they appreciated being able to spend more time and have more
opportunity to work with different modules in preparation for instruction with students.
• According to the former participants, participation in an additional SSPOI allowed them to come
into the institute with an understanding of the process so that they could then concentrate on
absorbing the content and information that would improve their ability to plan, prepare, and lead
students in the science modules
• It is recommended that, as much as is feasible, additional SSPOI sessions and materials should be
provided for these teachers to continue their growth in inquiry-based science instruction.
• Providing funds and opportunities for teachers to participate in follow-up sessions of the SSPOI
may help solidify instruction, provide a way to deepen their content knowledge of the different
science modules, and allow them to sharpen their skills at leading, assessing, and encouraging
student learning of science.
OEA MENTORING
PROFESSIONAL DEVELOPMENT INSTITUTE
• OEA developed partnerships with several school districts with the understanding that district
support would increase successful implementation and outcomes.
• Approximately 700 teachers have participated in a mentoring PDI since the inception.
• Classroom management skills were enriched early in the training.
• On average participants have formally mentored 1-3 teachers.
• Three-year professional growth plans were written to enhance the teacher's practice.
• 89% of participants agreed that beginning teachers and their own practice improved as a result
of the institute.
IMPACT
REFLECTION
• Aligning our evaluation system from pre-service to retirement is a result of the data received
from this grant. A pre-service to retirement evaluation system is being proposed. Higher
education institutes are being held accountable to the INTASC standards for their programs.
These state adopted competencies will guide the development of a uniform system.
• NCLB (The No Child Left Behind Act), OSRHE (Oklahoma State Regents for Higher
Education), and NSDC (National Staff Development Council) concur that mentoring and/or
coaching of teachers is the most vital and the most neglected aspect of assisting teachers to
improve practice. The mentoring institutes must be expanded and supported to improve state
wide student achievement.
Executive Summary
OEAlOSU Mentoring POI
The Oklahoma Commission for Teacher Preparation (OCTP) is charged with creating a competency-based
teacher preparation system to ensure that every Oklahoma classroom has a competent and qualified
teacher. The OCTP contracts with independent vendors to provide professional development Institutes for
teachers in the state of Oklahoma, including teacher mentoring. Since 1998, OCTP has provided two
mentoring Institutes that provide training in essential skills necessary to support beginning teachers, which
are applicable to supporting experienced colleagues as well. Two vendors were contracted to provide this
training-the Oklahoma Education Association (OEA), using the Pathwise Induction model, and Oklahoma
State University (OSU), using the Performance Learning System Coaching model.
SEDL's Evaluation Services was contracted to conduct an external evaluation of the Mentoring
Professional Development Institutes. This report presents findings from the second of a three-year
evaluation contract, which began in April 2002. The purpose of this evaluation was to assess the progress
that the Mentoring Institutes were making toward the goal of enhancing the induction process by providing
consistent mentor training in the essential skills necessary to support beginning teachers. To do so, SEDL's
evaluation worked from the following evaluation questions:
1. How many teachers participated in the training and how were they selected?
2. In what types of mentoring assignments and activities have trainees been engaged since
participating in the Mentoring Institutes?
3. Have Mentoring Institute participants been supported in their use of the mentoring strategies?
4. What was the impact of the training on participants and beginning teachers?
The evaluation findings are based on a survey that was administered to past participants of the OEA and
OSU Mentoring Institute during the four-year period beginning with the 1999-2000 school year. The findings
are discussed in terms of training participation and recruitment, mentoring assignments and activities,
support for implementing mentoring strategies, and the impact of the training.
Key Findings
• Participation and Recruitment. An examination of participation rates and the recruitment process
for the mentoring Institutes revealed that the purpose of training was effectively communicated
although the training is reaching a relatively small number of teachers as a percentage of all
teachers in the state.
• Mentoring Assignments and Activities. The majority of survey respondents have mentored 1-3
beginning teachers although a number have still not had this opportunity. Survey respondents
indicated that while most have mentored beginning teachers, the training did not result in being
assigned to more Residency Teams. Based on reports of mentoring activities and assignments, the
findings also suggest that the trained mentors are successfully implementing the mentoring
strategies they learned in the Institutes and that they are providing substantial support to beginning
teachers.
• Support for Mentoring. Survey respondents reported that collaboration and follow-through are
important to successful implementation of the mentoring strategies and that a number of
participants reported having teams of teachers in their schools who were trained in the mentoring
strategies.
• Training Impact. Survey respondents suggested a variety of benefits for themselves and beginning
teachers. In particular, respondents reported being more reflective about improving their own
teaching practice and finding effective ways to help beginning teachers or other colleagues. The
respondents' comments illustrate their appreciation of the training and the desire for continued
support for implementing the strategies they learned.
Key Recommendations
Evaluation findings indicate that the Mentoring Institutes are beneficial to both the participants of the
training and the beginning teachers (and colleagues) who they mentor. Findings also illuminate several
areas where the Mentoring Institute or the implementation process in schools and districts could be
strengthened. Based on these findings, several recommendations are suggested for ~CTP's consideration:
• Increase the number of teams of trained mentors within districts or schools to promote
collaboration and ongoing support for the use of mentoring strategies.
• Promote principals' participation in the mentoring Institutes.
• Promote the adoption of some or all of the mentoring strategies school-wide or district-wide.
• Promote the benefits of assigning beginning teachers to trained mentors and encourage schools
and districts to use trained mentors when possible.
• Promote school and district policies that will ensure adequate numbers of trained mentors to serve
as resources for beginning teachers as well as experienced colleagues throughout the state.
Overview
The Oklahoma Commission for Teacher Preparation (OCTP) is charged with creating a competency-based
teacher preparation system to ensure that every Oklahoma classroom has a competent and qualified
teacher. The OCTP's three-pronged approach toward attaining this goal includes the provision of
competency-based teacher preparation and accreditation, candidate assessment, and ongoing growth and
professional development of classroom teachers across the state. In support of federal legislation
mandating that all states develop plans ensuring that all teachers of core academic subjects are highly
qualified by the end of the 2005-2006 school year, the OCTP contracts with independent vendors to
provide professional development Institutes for teachers in the state of Oklahoma. These Institutes
emphasize both content and professional instructional practices in the areas of literacy, mathematics,
science, and teacher mentoring. This report addresses this last emphasis, mentoring.
Since 1998, the Oklahoma Commission for Teacher Preparation has provided two mentoring Professional
Development Institutes that provide training in essential skills necessary to support beginning teachers,
which are applicable to supporting experienced colleagues as well. Two vendors were contracted to
provide this training-the Oklahoma Education Association (OEA), using the Pathwise Induction model,
and Oklahoma State University (OSU), using the Performance Learning System Coaching model.
This report presents findings from the second of a three-year evaluation contract, which began in April
2002. In Year 1 of the evaluation, SEDL's evaluation focused on the training provided by OSU during the
2002-2003 school year. Findings from the Year 1 evaluation are presented in the 2002-2003 Annual Report
on Professional Development. Key findings regarding the first year evaluation of the OSU Mentoring
Institute included the following:
• Survey respondents reported being very satisfied with the training they received and that the OSU
Mentor Institute was of high quality and contained useful content and materials that were presented
effectively.
• Three months following the end of training, survey respondents reported that they were applying
their new skills and knowledge for mentoring beginning teachers and colleagues, indicating that
they had increased their level of use of the new skills and strategies to the extent that the majority
of respondents were now able to use the mentoring strategies most of the time.
• Although survey respondents indicated that principals generally supported mentor strategies,
information obtained from interviews indicated that implementation was significantly impacted only
if the principals actively participated and promoted teachers' use of mentoring strategies.
• Teachers reported that they still needed more time to practice and more involvement of other
teachers at their school in order to increase their effectiveness as a mentor teacher.
In the Year 2 evaluation, attention turned to both OEA and OSU Institutes and the overall quality and
impact of the mentoring training over the four-year period of 1999-2000 and 2002-2003 school years. To
do this, a survey was sent to all participants who took part in the OSU and OEA Mentoring Institutes during
this time period.
This report presents evaluation findings from the second year follow-up survey of the Mentoring Institute
participants. The following sections of this report include a description of the two Mentoring Institutes, the
evaluation approach, evaluation findings, conclusions, and recommendations.
Mentor Professional Development Institute Goals and Objectives
The goal of OCTP is to ensure that every student in Oklahoma has access to competent, caring, and
qualified teachers. The Mentoring Institutes are expected to contribute to the OCTP goal by achieving the
following:
1. Provide a systematic process for the support and development of mentor teachers for the
Oklahoma Resident Teacher Program.
2. Provide quality mentor training that enables the mentor teacher to address the varied needs of the
resident (beginning) teacher.
3. Create a framework for ensuring sustainability by developing and maintaining a cadre of mentor
teachers throughout the state who are prepared to coach entry-level teachers and have the
flexibility to coach colleagues interested in expanding instructional skills in the content area.
OEA Mentoring Institute
The OEA Mentor Institute uses the Pathwise model of mentor training developed by Education Testing
Service (ETS). The institute aims to increase mentor teachers' ability to support beginning teachers in a
variety of teaching practices. The training consists of the following components.
• Classroom Management Skills
• Introduction to Frameworks
• An Assessment of Teacher Effectiveness
• Performance-Based Work Products
• Three-year Professional Growth Plan
• The Colloquium
The OEA Mentoring Institute consists of four days of training throughout the year followed by a celebration
day in the spring, which is to recognize the accomplishments and professional growth of the beginning
teachers who were mentored.
OEA implemented a new approach to recruiting participants for the 2003-2004 school year. In this new
approach, OEA partnered with several districts throughout the state to provide training to its mentor
teachers with the understanding that district support for such a program would increase the likelihood of
successful implementation and outcomes. In previous years, teachers from different districts participated in
the training together.
OSU Mentoring Institute
The OSU Mentoring Institute is based on the Performance Learning System Coaching model. The project
goals are to:
• Provide a systematic process for support and development of mentor teachers for the Oklahoma
Resident Teacher Program.
• Provide quality mentor training that enables the mentor teacher to address the varied needs of the
resident teacher.
• Create a three-year framework for ensuring sustainability by developing and maintaining a cadre of
mentor trainers who will be available throughout the state.
• Provide a cadre of mentor teachers who are prepared to coach entry-level teachers and have the
flexibility to coach colleagues interested in expanding instructional skills in the content area.
OSU's Mentoring Institute consists of seven days of training held over a six-month period, with the first
block of training days typically conducted in the summer or the beginning of the school year. The Mentoring
Institute has been offered at various sites throughout the state. The 7 days of training focuses on a
comprehensive set of mentoring strategies intended to help participants coach beginning teachers as well
as experienced colleagues. In particular, the training consists of 10 coaching skills for successful teaching,
which are presented over the course of the 7-day training and with which participants are given time to
practice. The ten skills include the following:
• Conducting Pre-observation Conferences for Effective Coaching
• Using Open and Closed Questioning
• Constructing Confirmatory Paraphrases
• Using Positive Phrasing
• Constructing Approval Statements
• Conducting Post-observation Conferences
• Selecting Modes of Questioning
• Attending to Non-Verbal Communication
• Using Pause Time
• Selecting Data-Collection Formats
The training involves formal presentation of materials as well as substantial opportunities to practice and
receive feedback from peers and presenters. Participants are expected to videotape themselves during
mentoring sessions at their schools in between training days. These videos are then shared and discussed
with fellow participants during the following sessions.
Participation
According to OEA and OSU participation databases, a total of 695 teachers participated in both Mentoring
Institutes over the four-year period beginning with the 1999-2000 school year. An average of 73
participants attended the OEA training each year and an average of 101 participants attended the OSU
training each year. Table 2 shows the number of participants who attended the two Institutes each year.
The total number of participants trained in both OEA and OSU mentoring programs represents roughly 2%
of all Oklahoma public school teachers.
Mentoring Institute Participation Rates
1999-00 2000-01 2001-02 2002-03
OEA participants 44 103 62 82
OSU participants 120 98 112 74
Mentoring activities
To further explore the impact of the Mentoring Institute, SEDL asked training participants to indicate how
often and in what types of activities they shared with their mentees. Additionally, mentor-training
participants were asked to indicate how often they met with their mentee. The pattern was similar across all
four years and across OSU and OEA training programs. Between 86% and 91% reported meeting with the
beginning teacher at least once a week or more. Participants were also asked to report when they met with
their mentees. Respondents indicated that they met most often after school (64%), but they also indicated
meeting before school (38%), during a common planning time (38%), during lunch (31%), and other times
such as during individual planning time or in workshops. (13%). These responses suggest that the mentors
are providing a fairly substantial amount of support to beginning teachers on a regular basis throughout the
school day.
SEDL also asked the training participants to indicate what types of activities they have shared with their
mentee. Table 6 shows the percentage of respondents who engage in the various mentoring activities.
These activities, which respondents selected from a list of options, are presented in order of greatest to
least engagement.
As indicated in the table below, the most common type of activity reported was observing the mentee in the
classroom and giving feedback. The response patterns were fairly similar between OEA and OSU training
participants. The biggest difference in reports of mentoring activities between OSU and OEA (24
percentage points) related to planning lessons, units, and assessments. This difference may reflect the
different approaches presented in the two Institutes. While this breakdown of activities may indicate that
there is room for mentors to focus more on analyzing student work and test data, planning professional
development, and co-teaching, it nevertheless suggests that there is still a fairly good balance of support
activities provided to beginning teachers.
en ortng C IVI ies
OEA OSU
(n=39) (n=41 )
Observing the mentee in the classroom and giving feedback 82% 90%
Planning lessons, units, assessments, etc. 49% 73%
Strategizing for communication with parents 64% 71%
Referring the mentee to other people as resources 59% 71%
Looking at student work 56% 54%
Analyzing student test data 28% 44%
Planning for professional development 33% 37%
Co-teaching with the mentee 21% 29%
IOphthileorso(peh.gy.,, deitscc.)ussions about classroom policies, teaching strategies or 5% 20%
Mt' AfT
Use of mentoring strategies
Mentoring Institute participants were asked about the extent to which they have used a variety of mentoring
strategies now that they have had time to implement what they learned in the training compared to the use
of those strategies before training. A series of survey items focused on the implementation of specific
mentoring strategies using a 4-point Lickert scale of 0 (not at all) to 3 (extensively). To assess the effect of
the training, SEDL computed the difference in the ratings now and before training. The following table
presents the mean differences on these items.
The differences in reported use of the mentoring strategies now compared to before training was significant
across both programs and for all strategies (p<.01). These responses provide a measure of the degree to
which respondents perceive they have implemented what they learned in the mentoring training. It is
notable that the mean differences between reported use of mentoring strategies now and before training
were greater for OSU participants than OEA participants, although it is unclear why.
Mean Differences in Mentoring Strategies Use Now and Before
OEA OSU
Mean N Mean N
Observing classrooms 0.76 34 0.89 38
Providing feedback 0.86 35 1.21 38
Using questioning strategies 0.89 35 1.58 38
Conducting pre-observation conferences 0.91 34 1.26 38
Conducting post-observation conferences 0.79 34 1.16 38
Using data collection techniques 0.83 36 1.51 37
Using strategies for working with adults 0.62 34 1.21 38
Accepting feedback from colleagues 0.52 29 1 19
Being observed in my classroom 0.44 36 0.71 38
Reflecting on my own teachinc practice 0.86 36 1.03 38
Collaborating with my peers 0.63 35 0.89 38
The strategy for which OEA participants reported using more as a result of training (mean difference=.91)
was conducting pre-observation conferences. The strategy for which OEA participants reported using least
as a result of training (mean difference=.44) was being observed in the classroom.
The strategy for which OSU participants reported using more as a result of training (mean difference=1.58)
was using questioning strategies. The strategy for which OSU participants reported using least as a result
of training (mean difference=.71) was being observed in the classroom.
These findings may reflect the variations in training approaches used by OEA and OSU in their Mentoring
Institutes. Nevertheless, they suggest that there is a significant increase in the reported use of mentoring
strategies as a result of training, which provides evidence that trainees are indeed implementing the skills
they learned.
Impact of mentoring training on teaching practices
SEDL asked participants to respond to a series of items about the ways in which the mentoring training has
impacted beginning teachers' and their own teaching practice. Table 11 presents the percentages of
respondents' ratings on this set of items.
Overall, respondents generally agreed or strongly agreed that beginning teachers' and their own teaching
practice improved as a result of the training. They also generally agreed that their working relationship with
other teachers has improved and that they have learned more about supporting beginning teachers and
supporting colleagues. Respondents were more divided regarding whether they learned more about district
or state standards, student assessments, and planning for professional development. Finally, respondents
generally disagreed or strongly disagreed that they already knew how to use the mentoring strategies prior
to the institute training.
The following table does not compare OSU and OEA responses because we found that the means were
similar for OSU and OEA training participants for all items except ''The mentor institute has helped me learn
more about supporting experienced COlleagues." The mean rating for OSU was higher (mean=2.54) than
the mean rating for OEA (mean=2.06) on this item. This finding reflects the greater emphasis on working
with colleagues in the OSU Mentoring Institute compared to the OEA training, which focuses almost
exclusively on mentoring beginning teachers.
Impact of Mentoring Training on Teaching Practices
i.' , " , Strongly Strongly Disaaree Disagree Agree Aaree Mean N
The teaching practice of the teacher I
pmaerntictoipreadtionimpinrotvheedMaesnatorreTseualtcohfermy 1% 10% 65% 24% 2.1 72
Professional Development Institute.
My working relationship with other colleagues
has improved because of my participation in - 17 59 24 2.1 76
the Mentor Institute.
My own teaching practice has improved as a
result of my participation in the Mentor - 4 51 45 2.4 76
Institute.
The Mentor Teacher Institute has helped me
learn more about:
Supporting beginning teachers - 3 42 55 2.5 74
Supporting experienced colleagues - 7 56 37 2.3 75
Instructional strategies · 10 53 37 2.3 73
The district or state standards · 30 55 15 1.9 74
Student assessment · 23 52 25 2.0 73
Planning for professional development 1 25 55 19 1.9 73
I already knew how to use the mentoring
strategies we learned in the Institute 25 55 18 3 1.0 73
beforehand.
Conclusions
In this report we shared findings from the second of a three-year evaluation of the Mentoring Professional
Development Institutes. The evaluation findings were based on the survey that was administered to
participants of the OEA and OSU Mentoring Institute during the four-year period beginning with the 1999-
2000 school year. The findings were discussed in terms of training participation and recruitment, mentoring
assignments and activities, support for implementing mentoring strategies, and the impact of the training. A
brief summary of these findings is presented below, followed by a set of recommendations for OCTP's
consideration.
• Principal and/or district administration support and commitment at times seemed weak.
• The majority of survey respondents have mentored 1-3 beginning teachers although a number
have still not had this opportunity.
• Respondents also indicated that while most have mentored beginning teachers, the majority has
not been assigned to more Residency Teams as a result of training.
• Respondents also reported engaging in a variety of mentoring activities, the most common were
observing their mentee during a lesson and giving feedback, planning lessons, units, and
assessments, strategizing for communication with parents, and referring the mentee to other
resources.
• The large majority of respondents reported meeting with their mentee at least once a week or
more; this happens most often after school.
• The trained mentors are using the mentoring strategies they learned in the Institutes and that they
are providing substantial support to beginning teachers.
• Collaboration and follow-through are important to successful implementation of the mentoring
strategies and that a number of participants reported having teams of teachers in their schools who
were trained in the mentoring strategies.
• Respondents generally rated the support they receive from others to implement the mentoring
strategies fairly high. The actions that helped them implement mentoring strategies the most
focused on carving out time and opportunities during teachers regular schedule to observe each
other and meet to discuss teaching practices.
• Important actions also included clear expectations about the use of mentoring strategies and the
importance of mentoring for school improvement efforts and the need to "scale-up" the mentor
training. Respondents indicated that they would benefit from more staff participating in the
Mentoring Institutes, including principals.
• Respondents want more support for mentoring either with time to collaborate with other teachers or
for more compensation to conduct these activities.
• Respondents perceive they are much better teachers and better prepared to support beginning
teachers.
Recommendations
Evaluation findings indicate that the Mentoring Institutes are beneficial to both the participants of the
training and the beginning teachers (and colleagues) who they mentor. Findings also illuminate several
areas where the Mentoring Institute or the implementation process in schools and districts could be
strengthened. Based on these findings, several recommendations are suggested for OCTP's consideration.
• Increase the number of teams of trained mentors within districts or schools to promote
collaboration and ongoing support for the use of mentoring strategies. Findings indicated that
teachers benefited from having other teachers in their schools go through the mentor training
because they were able to help each other implement the mentoring strategies.
• Promote principals' participation in the Mentoring Institute. Respondents indicated that the
implementation of mentoring strategies would be enhanced even further by more support from
principals and even their participation in the mentoring training.
• Promote the adoption of some or all of the mentoring strategies school- or district-wide.
• Promoting the use of mentoring/coaching strategies school- or district-wide would allow teachers to
collaborate and support each other as they continue using the mentoring strategies with new
teachers and experienced colleagues as well.
• Promote adequate time to engage in mentoring activities, such as observing classrooms and
meeting to discuss observations and other teaching practices.
• Promote the benefits of assigning beginning teachers to trained mentors.
• Promote school and district policies that will ensure adequate numbers of trained mentors to serve
as resources for beginning teachers as well as experienced colleagues throughout the state.
• Promote the assignment of trained mentors to beginning teachers.
In conclusion, the evaluation findings indicate that the mentor training positively impacted participants of the
training and that the new teachers whom they mentored were positively impacted as well. There is
generally support for the use of these mentoring strategies but some respondents still reported not having
the opportunity to work with beginning teachers. However, there are aspects of induction programs that we
have not yet addressed in this evaluation and may be potential topics for the third year evaluation. For
example, how well are Mentoring Institute trainees prepared to provide advice to beginning teachers about
serving diverse student populations and students with a considerable learning needs (e.g., struggling
readers). Many beginning teachers leave because they are overwhelmed by the challenging circumstances
in which they find themselves, such as in high-poverty, low-performing schools, or in schools where
administrators do not support teachers. We also know little about how stipends or other motivational factors
influence teachers to participate in the training or on the Residency teams. These are potential questions
for future consideration.
EDUCATION LEADERSHIP OKLAHOMA
HIGHLIGHTS FROM 2003·2004
NATIONAL IMPACT
Independent Studies Confirms Effectiveness of National Board Certification
~ An independent study of 600,000 student records from students in North Carolina schools has found that
children learn more from National Board Certified Teachers. The study found:
• Teachers who achieve National Board Certification do a measurable better job in
the classroom
• Students of NBCTs improved an average of seven percent more on their year-end
math and reading tests than students whose teachers attempted but failed to gain
certification
~ Another study demonstrates that National Board Certified Teachers increase student achievement. The
study drew upon more than 200,000 students from 14 Arizona school districts with gain scores calculated
from each student over four consecutive years
• The gains made by students of NBCTs were more than one month greater than
the gains made by the students of non-Board certified peer teachers.
STATE IMPACT Total Number of Oklahoma National Board Certified Teachers as of
Fall 2004 - 858
Oklahoma ranks eighth in the Nation
• 355 Candidates were funded by the Oklahoma Legislature
• A federal subsidy secured by the OCTP in the amount of $88,000 provided additional candidate support
• 211 First Time Candidates completed the process
• 179 Advanced Candidates participated in the retake process
• 230 Candidates currently await their scores
• 75 of Oklahoma's 77 Counties have either a National Board Certified Teacher (NBCT) or a candidate
• 168 of the School Districts in Oklahoma currently have a NBCT
• An additional 48 districts have candidates
• The Legislature continues to award a $5,000 annual stipend to NBCTs who are fulltime classroom teachers
in an Oklahoma school
ELO CANDIDATE SUPPORTfTECHNOLOGY
• ''Trainer of Trainers" for Durant ELO Training
• Two two-day summer orientation programs in Durant for new candidates (in conjunction with Southeastern
State University)
• Distance Learning 2-Day Workshop - Eight regional Sites Statewide
• Tapes of Distance Learning Workshop sent out to candidates who could not attend
• Eight regional coordinators who coordinate and facilitate support and mentoring
• Partnerships with Career Tech, Oklahoma Universities and Regional Professional Development Centers
• A pre-candidacy video funded by a recruitment grant from the State Farm Foundation was distributed
• Assessment Center Workshops - Two regional sites
ELO GOALS
• Recruit and attract teachers in under-represented regions and low-periorming schools
• Promote benefits of using NBCTs to ensure high quality teaching such as: leaders on school, district, or
state improvements committees
• Determine the effectiveness of NBCTs in terms of student achievement
• Have a National Board Certified Teacher in every school district in Oklahoma
• Secure funding for 500 annual scholarships for National Board Candidates
• Expand the mentoring program for National Board Certified Teachers
• Provide the guidance and support needed by Candidates with specific feedback and in-depth study on
standards and how they are incorporated
• Provide leadership training for NBCTs
RESULTS OF SEOL EVALUATION
In April 2002, the Southwest Educational Development Laboratory (SEDL) was contracted to evaluate the ELO
program. The purpose of the evaluation is to examine how National Board Certification of teachers in Oklahoma
addresses quality teaching and advances school improvement efforts.
What was the impact of becoming certified?
Survey respondents overwhelmingly indicated that NBC is an excellent professional development experience and
that they are much better teachers as a result. This was especially true for teachers with 4-10 years experience
compared to those with 21+ years of experience . Teachers with fewer years of teaching experience and those
teaching in elementary/middle schools and low periorming schools reported experiencing more personal outcomes
as a result of certification. In comparison, more teachers in urban, larger schools reported experiencing professional
outcomes. Survey respondents generally reported that certification improves their pedagogical skills, their
knowledge of curriculum and subject area, and their ability to enhance and assess student learning. However,
responses also indicate that the benefits of certification are still largely localized to NBCT's own classrooms.
100% -,-----------------------,,,.----......-,,..-----,
4-10
.11-20
021+
20% -t------'-------
0% 1% 2%
O%+---~--c=~-~-
Did not impact your practice Made you a somewhat better Made you a much better
teacher teacher
Figure 1 Impact of Certification by Years of Teaching Experience
Overall, 70% of NBCTs who responded to this survey indicated that certification made them a much better teacher.
Thirty percent reported that the certification process made them a somewhat better teacher and only 1% said it did
not impact their practice. While the responses generally were similar for teachers in different types of schools and
those teaching different subject areas, the responses did vary by reported years of teaching experience. As
illustrated in Figure 1, more teachers with 4-10 years of experience believe they are much better teachers as a
result of certification compared to those with 21 + years of teaching.
SEDL also asked NBCTs to rate the helpfulness of the certification process regarding 17 aspects of teaching (l.e.,
not at all helpful, somewhat helpful, very helpful). A summary of responses to these items is illustrated in Figure 2.
Improving my self confidence 0/0
1'7
67%
" !
6:~%
I
f,( 0/0
-j -, a - j
,)RO/o
-j '" I
')70/0
-j I
')')0/0
II
,),Ofn
".
,., I ') a
I
490 I1
47% I
" :.i, I
46% I
I~' I
430/0 II
4 % II
,C 0/0
" <:> I
010
I 0/0
I
Developing a stronger curriculum
Recognizing individual differences in
students and adjusting accordingly
Using student assessments effectively
Articulating learning goals for students
Making multiple paths to learning
available
Mentoring other teachers
Working with colleagues on instruction
USing research to improve my practice
Understanding how students learn
Understanding of links to other subject
areas
Increasing knowledge of my subject area
Working with parents
Knowing how to impart subject area to
students
Trreating my students equitably
Connecting to state standards
Taking advantage of community resources
0% 20% 40% 60% 80% 100%
Not At All Helpful
• Somewhat Helpful
oVery Helpful
Figure 2 Helpfulness of NBC for Improving Teaching
Responses indicate that certification is most important for improving teachers' self-confidence and developing a
stronger curriculum for students. Certification is least helpful for taking advantage of community resources and
connecting to the state standards.
NBCTs overwhelmingly indicated that NBC is an excellent professional development experience and that they are
much better teachers as a result. This was especially true for teachers with 4-10 years experience compared to
those with 21+ years of experience. The findings also suggest that NBCTs experience different outcomes of
National Board Certification depending on their years of experience, the schools in which they teach, and the
subjects they teach. Teachers with fewer years of teaching experience and those teaching in elementary/middle
schools and low performing schools reported experiencing more personal outcomes as a result of certification. In
comparison, more teachers in urban, larger schools reported experiencing professional outcomes.
Oklahoma National Board Certified Teachers and Education
Leadership Oklahoma Candidatesl Applicants
Through 2003
Cimarron [ Texas
r
Beaver
I
Harper Woods Grant Ka
y
/ warT r IOttawa Alfalfa 20 Nowata Craig 3 1 1
22 3 I 1
1 73 222 I 2 762
1 1 Osage 10151 449
Woodward - "- 5 18
Garfield Noble
;II~S\
1594 Major 7126 221
I 241
I
Dewey
1 5 Blaine
Roger Mills \
23 Lincoln
Custer 1383
15101
Be2c2k2ham I Washita l
- Oklahoma National Board
Certified Teachers
Education Leadership Oklahoma
Candidates (2002-2004)
Education Leadership Oklahoma
Applicants (2005)
McCurtain
43
Le Flore
Comanche 5 3 r1, '----"-1
442918
Pushmataha I I
I
Stephens
851
Jefferson
333
MAY :t ~ 2011