CAEP Accountability Measures

There were 90 completers (both licensure programs) between years 2019 and 2022. There were a total of 30 males and 60 female completers.  Of the 90 completers, 19 (21%) identified as American Indian /Hawaiian/Other Pacific Islander.  The remaining 79% identified as white/Caucasian.

Six of the American Indian/Hawaiian/Other Pacific Islander completers were from the SUPT program with the remaining 13 gaining licensure as principals.

A pilot survey was sent March 14 to seven former MSU graduates who completed the EDLD SUPT and Principal programs in 2018.  No wording changes were recommended except for introductory email paragraph to make the greeting more personal and relational.  Six of 7 responded to the pilot. After reviewing pilot results, data were scrubbed and survey was sent out during the window of March 24, 2023 through April 14,2023 to completers of both SUPT and Principal programs for a total of 90 emails sent requesting survey completion.

Eight emails were returned as undeliverable; a total of 82 emails were successfully sent to MSU completers (2019 - 2022). Twenty were sent to SUPT program completers and 62 were sent to principal program completers.  A total of 39 of the 82 surveys were returned and completed for a total response rate of 48%.   Five of the 20 SUPT completers responded to the survey (25% response rate); 34 of the 62 principal completers also responded (54%).  Thirty- four respondents identified as white; five respondents identified as American Indian / Hawaiian/Other Pacific Islander.

39 respondents out of 82 program completers (principal and SUPT) from (2019, 2020, 2021, 2022)
Response rate =48%
Principal Completers (34 of 62= 54%) Superintendent Completers (5 of 20= 25%)
5 of respondents were American Indian out of 19 American Indian completers from 2019-22  Data were not disaggregated by race due to the low number of survey completers identifying as American Indian/Hawaiian/Other Pacific Islander
Survey has an alpha reliability of .942
Section A Curriculum /Experiences – Grand Mean = 7.61, SD=1.21
Section B Field Experience – Grand Mean = 6.88, SD= 2.25
Section C Self Efficacy – Grand Mean = 7.43, SD=1.66

Completers are generally very satisfied at how the MSU EDLD programs prepared them for work as principals and superintendents in schools.  Most ratings fall between 7 and 8 on the nine-point scale as completers.  Completers also generally believe in their skills and abilities to perform tasks related to their leadership positions as principals and superintendents as revealed in the aggregate ratings on the SELF-EFFICACY portion of the completer survey.  The self-efficacy measure was included because there are studies linking preparation programs to leader self-efficacy (Furrow, 2022; Tschannen-Moran, et.al., 2007; Versland, 2009).

The section with the lowest mean scores was the FIELD EXPERIENCE section.  While we know that some candidates get a broad and deep experience performing leadership tasks and developing relationships with colleagues, there are some candidates (typically because of their field experience placement) who may get few opportunities to participate in relevant and varied leadership experiences.  Some of those candidates live in remote, rural communities with inexperienced leaders who may be reluctant to “share too much information/ leadership” for fear that the candidate will appear more skilled than the leader. As faculty, we need to look into the tasks and responsibilities that were rated lower from the FIELD EXPERIENCE section and determine how we can include some of those in our coursework or during face-to-face meetings to ensure that all candidates gain exposure to relevant and important experiences.  We are also discussing adding course assignments that require students to more actively engage with external stakeholder groups (outside of parents).

We plan to review these survey results again in Fall of 2023 when the MSU EDLD EMPLOYER SURVEY results are back.  Those survey results, together with our Spring 2023 Exit Ticket Exercise results from our 2023 graduates will provide excellent triangulation for faculty and AAA committee to discuss program effectiveness.

MSU Educational Leadership faculty seek input from a variety of stakeholders (current candidates, recent graduates, and state agency partners), to determine what they perceive to be the strengths and weaknesses of the educational leadership program and also to hear what they think are current needs of schools and school leaders across the state. MSU benefits as we utilize their input to make our program relevant to meet the needs of schools in our state. Schools and candidates benefit when new school leaders experience contemporary and content rich courses and are able to transfer their learning to new leadership roles.

We also seek stakeholder input during the principal licensure field experience. MSU faculty and site supervisors (building leaders) evaluate the candidates together and discuss strengths/challenges of the candidates that can also provide information about program quality. Site supervisor feedback has been utilized in revising some of the assignments we initially required for candidates several years ago.

Dear School Superintendent/ School Board Member,
Greetings from Montana State University Educational Leadership Program! We are contacting you today to request your participation in a program evaluation survey for our program that trains and prepares principals and superintendents. Your insight and input are important to us as we strive for continual improvement in the MSU Educational Leadership Program. The purpose of this survey is to collect perceptions from employers (superintendents and board members) about their views of how the MSU EDLD graduates who are employed as principals, curriculum directors and superintendents were prepared by the MSU Educational Leadership Program. This survey IS NOT intended to be an evaluation of the school leaders in your district or their individual performances. Instead, this survey seeks information on how well you believe the MSU EDLD Program has prepared them in specific areas associated with leading a school/district.

Your participation in this survey is voluntary. Even if you begin the survey, you may stop taking it at any time. Your responses will be kept anonymous and will not be linked to any identifying information.

I agree to participate in the EDLD Program Graduate Survey YES ______ NO_______

Again, thank you for agreeing to complete this survey.

Demographic Information

Current Position: Superintendent_____ Board Member____ Clerk/Human Resources_________

Number of Years in Your Present Position: ______

Size of District: AA_____A_____B_____C_____ Independent Elementary District______

Leadership Preparation Experiences

This survey is based on the National Educational Leadership Program (NELP) Standards and the Montana Professional Educator Preparation Program (PEPP) Standards and Council for Accreditation of Educator Preparation (CAEP) standards. These standards guide the curriculum and experiences that the MSU Educational Leadership Program puts in place for candidates seeking principal and superintendent licensure. The 20 survey questions below ask you to select a response that best describes your perceptions of how the MSU EDLD Program prepared school leader(s) for their positions of leadership in your school. For each item, please choose a response that matches your perceptions of your school leader(s). The response scale ranges from (1) “Not at All” to (9) “A Great Deal” with (5) “To Some Degree” representing the mid-point between extremes.” You may choose any of the nine since each represents a degree on the continuum.

The Educational Leadership program at MSU effectively prepares leadership candidates to…

1. Establish and implement a shared vision and culture of high expectations for all stakeholders (students, staff, parents, and community). (NELP 1; ARM -705.1a.i, 1b, 1d; CAEP RA1.1) 

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

2. Collaboratively determine goals and implement strategies aligned with the school vision that support positive student outcomes. (NELP 6; ARM-705.1a.ii,1b.iii,1b.iv,1b.v;CAEP-RA1.1) 

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

3. Develop and support productive and trusting relationships among students, staff, and families. (NELP 5,2, ARM-705.1a.i, 1d,1d.ii,1f; CAEP-RA1.1)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

4. Understand the role relationships play in the development of a healthy and effective school culture. (NELP 7.2; ARM-705.1b;CAEP-RA1.1)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

5. Work with diverse stakeholders (staff, families, and community) as key partners to support student learning. (NELP 5; ARM-705.1a. i,1b,1d,1f)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

6. Reflect on their leadership performance and strive to grow professionally. (NELP 2; ARM -705.1a.ii, 1b, 1e.ii; )

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

7. Demonstrate and model professional norms such as integrity, fairness, transparency, trust, equity, democracy, digital citizenship, diversity, inclusiveness, and the belief that each child can learn. (NELP 2; ARM-705.1a.ii; 1d)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

8. Provide ethical leadership by advocating for children and ensuring student access to effective educators, programs, and services. (NELP 2, 3, ARM -705.1a.ii, 1b, 1e.iv, 1d.)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

9. Collaboratively develop and implement high-quality instruction and curricular programs. (NELP 4; ARM 705.1a.iii, 1b.i, 1b.ii, 1b.iii; CAEP- RA.1.1)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

10. Monitor and assess classroom instruction to promote teacher effectiveness and student achievement. (NELP 4; ARM-705 1a.iii,1b, 1b.ii, 1b.iv, 1b.v, CAEP-RA1.1)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

11. Provide feedback, coaching, and professional development to all staff through effective evaluation and supervision. (NELP 7; ARM-705.1b.ii,1b.iii, 1c.i;CAEP -RA1.1)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

12. Focus on improving student outcomes through organizational collaboration, resiliency, and change management. (ARM-705.1a.ii, 1b.i, 1b.ii,1e.iCAEP-RA1.1)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

13. Demonstrate knowledge of school-based budgeting to evaluate, acquire, and sustain programs that create equity among students. (NELP 6.2; ARM 705.1c.ii; CAEP RA1.1)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

14. Support and encourage responsible and ethical technology use (websites, emails, social media, learning platforms, etc.) to communicate with others. (NELP 7.3; CAEP-RA 1.1)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

15. Develop and practice strategies for cultivating leadership among staff. (NELP 7.3; ARM-705.1c.i; )

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

16. Participate in meetings that collaboratively engage families, diverse community members, and other constituencies to influence social, political, and economic decisions affecting communities and schools. (NELP 5.2; ARM-705. 1d, 1f;CAEP-RA1.1)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

17. Collect and analyze multiple sources of data in support of decision-making for a variety of circumstances (NELP 3, 4.2; ARM-705.1a.ii, 1a.iii, 1b.iv, 1d.iii; CAEP-RA1.1)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

18. Conduct an analysis of school processes and operations to determine impact on resource allocation, personnel decisions, and student experiences and outcomes. (NELP 6.1; ARM 705.1c.ii; CAEP-RA1.1)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

19. Utilize knowledge of state and federal laws as they apply to parental rights, student discipline, and personnel management regarding policy development and enactment. (ARM-705.1c.iii, 1e.iii, 1e.iv;CAEP-RA1.1)

1=Not at All 2 3=Very Little 4 5= To Some Degree 6 7= Quite a Bit 8 9= A Great Deal

20. What else would you like to tell us about the preparation of your school leaders and/or the MSU Educational Leadership Program that we haven’t already asked?

---

Thank you again for your participation!

---

	Initial Cycle 1	Initial Cycle 2	Initial Cycle 3	Initial Cycle 4	Advanced Cycle 1	Advanced Cycle 2	Advanced Cycle 3
2017-2018	Initial Employer/ Completer Survey 14-15 15-16 16-17
2018-2019
2019-2020	Initial Case Study Program Discretion for years
2020-2021	Initial Case Study Continued Covid Extension	Initial Employer Survey 16-17 17-18 18-19 Sp21 - Low Response Rate			Advanced Employer Survey Validity and Reliability Initial Work Au20 Sp21
2021-2022		Initial Completer Survey 17-18 18-19 19-20 Spring 2022			Advanced Completer Survey Ongoing validity and reliability work Sp22
2022-2023		Initial Case Study Program Discretion from 16-17 to 20-21			Advanced Case Study Revist CAEP revised standards
2023-2024			Initial Employer Survey 19-20 20-21 21-22			Advanced Employer Survey 19-20 20-21 21-22
2024-2025			Initial Completer Survey 20-21 21-22 22-23			Advanced Completer Survey 20-21 21-22 22-23
2025-2026			Initial Case Study Program Discretion from 19-20 to 23-24			Advanced Case Study pending
2026-2027				Initial Employer Survey 22-23 23-24 24-25			Advanced Employer Survey 22-23 23-24 24-25
2027-2028				Initial Completer Survey 23-24 24-25 25-26			Advanced Completer Survey 23-24 24-25 25-26
2028-2029				Initial Case Study 22-23 to 26-27			Advanced Case Study pending

 

Call for Comments - The Department of Education at Montana State University—Bozeman hosted an accreditation visit by the Council for the Accreditation of Educator Preparation (CAEP) on September 23, 2023. Programs under review were the undergraduate Teacher Education Program, the graduate Master of Arts in Teaching, the graduate Master of Education in Educational Leadership/Principal Preparation and the graduate Superintendent Certificate. Interested parties were invited to submit third-party comments to the CAEP evaluation team on or before August 12, 2023. Laudatory, critical, and merely observational comments were all welcome. Details at this link.

Mean	165.8696
Standard Error	1.631101
Median	166.5
Mode	165
Standard Deviation	11.06267
Sample Variance	122.3826
Kurtosis	3.17801
Skewness	-0.9992
Range	65
Minimum	125
Maximum	190
Sum	7630
Count	46

 

GPAs in low-N categories cannot be reported for reasons of student privacy as well as statistical validity.

Year Completed	GPA   at Admission	GPA at Completion	American Indian GPA at Admission	American Indian GPA at Completion
2019	Mean = 3.47 SD = .35 N= 36	Mean = 3.88 SD = .14 N= 36	Mean = 3.81 SD =.12 N= 13	Mean = 3.81 SD =.12 N= 13
2020	Mean = 3.46 SD =.36 N= 21	Mean = 3.95 SD =.09 N= 21	N too small to report	N too small to report
2021	Mean = 3.75 SD =.26 N= 19	Mean = 3.94 SD =.06 N= 19	N too small to report	N too small to report
Total	Mean =3.51 SD =.34 N= 76	Mean = 3.90 SD = .11 N =76	Mean = 3.71 SD = .32 N =18	Mean = 3.89 SD =.14 N =18

Data comparing candidates’ admission and completion GPA, indicate that candidates show growth in GPA from program beginning to end.  There is no significant difference in GPA when disaggregated by race. American Indian/Hawaiian/Other Pacific Islander candidates listed above are mostly recipient of the 2016 I LEAD grant designed to recruit and train American Indians for school leadership positions in or near reservation schools.

On-Campus Experience Agenda

8:00 – 8:30 Check in, set up for class 
8:30 – 9:30 Dr. Tena Versland 2020-2021 school leadership wrap-up 
9:30 – 9:40 Break – bring your own snacks and water 
9:40 - noon First 6 student presentations* Order will be random selected. 
12:00 – 12:20 Lunch – bring a sack lunch 
12:20 – 3:00 Last 7 student presentations* 
3:00 – 3:30 Closing comments/presentation by Dr. Daniel Farr (on any topics you would like to have addressed) and borrowing from Verne Beffert – Are you tough enough and well enough trained to be a school principal/school leader? 
This day will focus on the following PSEL Standards¹ 
Standard 4 Effective educational leaders develop and support intellectually rigorous and coherent systems of curriculum, instruction, and assessment to promote each student’s academic success and well-being. 
Standard 5 Effective educational leaders cultivate an inclusive, caring, and supportive school community that promotes the academic success and well-being of each student. 
Standard 6 Effective educational leaders develop the professional capacity and practice of school personnel to promote each student’s academic success and well-being. 
Standard 9 Effective educational leaders manage school operations and resources to promote each student’s academic success and well-being. 
Standard 10 Effective educational leaders act as agents of continuous improvement to promote each student’s academic success and well-being. 
Presentations: 
*Students will have 15 minutes to present (to the class), 1 to 3 reflective narrative(s) regarding their field experience, the purpose of which is to have you spend a reasonable amount of time reflecting on what you have learned through your internship. You may use hard copy handouts, PowerPoint slide stacks, or other means to support your presentation. 
Following are some ideas, however you are encouraged to develop your own topic that reflects your unique circumstance. An additional reason for this assignment is to share, with your cohort, a view of the principalship that they may not otherwise have an opportunity to experience. An additional 5 minutes will be provided for questions and discussion. Total presentation with Q/A time is 20 minutes/student. 
• • Explain to the class how you would address the year of student learning that has seen continued impacts on learning lost due to the Covid pandemic. Your approach should assume that you will be a principal next fall—what was done this academic year (what worked and how do you know) and what are the unmet needs going forward. 
• • While serving as the administrator for a day, share with the class how you resolved a situation based on what you learned in a specific MSU class, or what you learned from observing your supervising administrator. 
• • Share with the class a challenging situation you observed that was either resolved very well, or was not resolved well, from which you learned a lesson that you will apply in the future. As always, keep names anonymous. 
• • What advice would you give to anyone planning to pursue a career as a school principal. Not as it relates to your MSU preparation, but as it relates to what you perceive to be the job of a principal. 
• • Share with the class your thoughts about the areas of responsibility you feel you are now well trained to step into the role of principal, as well as those areas you do not feel you are well prepared to assume responsibility for—Send topics on any areas that you would like more info on (i.e., school investigations, Title IX, etc. ) to Dr. Farr by April 1, 2021. 
• • How much time do you feel a principal has to devote to any of the following: general leadership, instructional leadership, curriculum development, teacher observation and evaluation, etc. Explain to the class your thoughts about how you can ensure you provide a high level of instructional leadership that will positively effect student learning. 

I realize that 15 minutes is a very short time for some of the above topics. Recognize that you have a limited amount of time and spend enough time designing your presentation to cover all critical elements succinctly. As a principal you will have a limited amount of time to present to the superintendent, or to the Board, so this assignment is practice for using well the time you have.

---

¹In 2015, the National Policy Board for Educational Administration (NPBEA), a national alliance of major membership organizations committed to the advancement of school and school-system leadership, adopted the Professional Standards for Educational Leaders (PSEL), an update of their 2008 standards former known as ISLLC (Interstate School Leaders Licensure Consortium) standards. 

It is important to note that candidates do not always seek licensure recommendation immediately upon graduation and that some recommendations may still be in process.

Principal (Masters in Educational Leadership) Data as of July 18, 2023
Academic Year	Number of Completers	Number Recommended by Field Placement	Recommended in Montana	Recommended Out of State
2022-23	0	0	0	0
2021-22	10	3	3	0
2020-21	15	12	12	0

 

Superintendent Certificate Data as of July 18, 2023
Academic Year	Number of Completers	Number Recommended by Field Placement	Recommended in Montana	Recommended Out of State
2022-23	2	2	2	0
2021-22	3	0	0	0
2020-21	4	3	2	1