Please update your web browser or disable Compatibility View.

Institutional Report for NCATE

III. Evidence

Standard 2: Program Assessment and Evaluation

The unit has an assessment system that collects and analyzes data on applicant qualifications, candidate and graduate performance, and unit operations to evaluate and improve the unit and its programs.

Element 1: The Unit’s Assessment System.

Overview. Although a major component of the units’ assessment system was first used as early as 1996, the design of the unit’s present assessment system began with conversations and consultation during an August 1999 conference in Philadelphia sponsored by the National Council for the Accreditation of Teacher Education (NCATE). At that conference, focused on the use of that organization’s emerging performance-based accreditation standards (Professional Standards, 2001), members of the unit who attended that gathering explored plausible strategies and techniques for describing their candidates’ performance.

That conference encouraged initial sketches of how the emerging system’s elements might appear during the fall of 1999. As those sketches took a more definite form they were informally critiqued by the unit’s faculty and by experienced elementary and secondary teachers. Work on one of those elements, a standards-based performance profile for candidates completing student teaching, began in January of 2000. The development of procedures for assuring candidates’ opportunities to know, apply, and be assessed on content, professional, and pedagogical knowledge and skills for each licensure program provided further insight into the design of a viable, standards-focused performance assessment system.

In October of 2000, the unit contributed to the development of a set of assessment questions that would guide the collection and analysis of candidates’ performance information. The following month colleagues confirmed the significant features of a system that could respond to those questions. The design of the units’ system was set in December. That design is described in a plan shared with the unit’s faculty members for their review in January of 2001 (Unit Assessment System). The plan will be revised after its review by the unit’s faculty, the colleges’ Academic Assessment Committee, and by the unit’s Teacher Education Advisory Committee. Allowing for anticipated revisions derived from these reviews, the plan is nearly complete.

Assessment Questions. The unit’s assessment system reflects the joint mission of The College of Saint Benedict and Saint John’s University as residential colleges pursuing the liberal arts within the Catholic university tradition. These two rural Minnesota institutions work together to offer their students “a unified liberal arts curriculum which focuses on questions important for the human condition, demands clear thinking and communicating, and calls forth new knowledge for the betterment of humankind.” Through this shared curriculum these two colleges strive as one to offer their students “an integrative environment for learning which stresses intellectual challenge, open inquiry, collaborative scholarship, and artistic creativity.” They recall their monastic founders by celebrating learning within Benedictine settings that “foster attentive listening to the voice of God, awareness of the meaning of one’s existence, and the formation of community built on respect for individual persons” (Academic Catalog, p. 4).

The Department of Education jointly sponsored by these two liberal arts colleges takes as its aim the preparation of “exemplary teachers who have a strong liberal arts background, exemplify Benedictine values, and make professional decisions which can help all students achieve their full potential as persons and as responsible world citizens in a democratic society” (Education Department Conceptual Model, p. 5).

Focused by this aim, and consistent with the efforts of the two colleges, the Education Department’s mission is to offer those prospective teachers “a rich and diverse background of coursework and experiences that stress intellectual challenge, open inquiry, collaborative scholarship, and that promote clear thinking.” This unit’s mission encourages the preparation of “teachers who make their informed and ethical classroom decisions based on a firm knowledge of content, pedagogy, and the needs of their students” (Conceptual Model, p. 6).

Students enrolled by the colleges and prepared to be licensed as teachers by Minnesota’s Board of Teaching reflect this mission and aim as they work toward the Education Department’s program goals. The knowledge, skills, and values that are acquired and affirmed through candidates’ pursuit of these goals strengthen the decisions they make as they plan, implement, and evaluate their practice (Conceptual Model, p. 2). The department’s goals are guided by the Standards of Effective Practice for Teachers, licensure standards set by Minnesota’s Board of Teaching (1999). The 10 terminal and 120 enabling standards that form this collection were derived from guidelines developed by the Interstate New Teacher Assessment and Support Consortium (INTASC).

This sense of institutional and unit mission, the later nested within the former, guides the design of the assessment system through the specification of assessment questions. Taken together, these questions reflect the unit’s program theory (I. Institutional Overview) and reflect its aim, mission, and goals. They help guide review of candidates’ performances as well as an examination of the program of study and practice that prepares them for their roles as professional educators. Such questions focus the specification, development, collection, and analysis of assessment information that can support judgments of the unit’s effectiveness.

  • Do candidates prepared for licensure as teachers possess the basic academic skills that will sustain their learning while enrolled in this program?

  • Do these candidates possess an integrated body of knowledge, skills, and values drawn from one or more disciplines central to their area of licensure?

  • Do candidates possess pedagogical knowledge, skills, and values appropriate for their areas of licensure?

  • Can these candidates teach knowledge and skills from their areas of licensure to others?

Standards-Based Assessment. The unit’s assessment system is designed to respond to these questions from the performance contexts created by unit, state, and professional standards. The unit’s expectations are revealed in its conceptual framework and goals. The State of Minnesota’s program approval and candidate licensure standards together define another performance context for all of teacher preparation programs and the candidates they present for licensure. The professional standards that guide NCATE’s review of a unit offer a third context to help shape responses to the unit’s assessment questions. These nested standards in turn suggest candidate proficiencies to be examined and described.

Assessment Question 1. Do candidates we prepare for licensure as teachers possess the basic academic skills and values that will sustain their learning while enrolled in our program?

The first of these five assessment questions guides our search for information affirming that our students possess the basic academic skills and values that will sustain their learning in our program of study and practice when accepted as candidates preparing for licensure as teachers. Such skills include the ability to write well, to draw inferences from reading complex information, to use mathematics to solve everyday problems, and to share ideas with others through formal speech events.

Embedded within our belief in the necessity of a broad liberal arts education is an emphasis on the basic skills of reading, critical thinking, and writing. Therefore, we seek to provide experiences throughout our program that enhance the development of these skills. Further, we attempt to emphasize the Benedictine values of openness to change and lifelong learning as essential to continued teaching effectiveness” (Conceptual Model, p. 4).

Our expectation that candidates for teacher licensure possess academic skills and values consistent with the opportunities revealed through their liberal arts education is also consistent with related state and professional standards. The Minnesota Board of Teaching’s rules for the approval of teacher preparation programs require that “the institution recruits, admits, and retains candidates who demonstrate potential for professional success in schools (Minnesota Rules: Institutional Program Approval, 1999, 8700.7600.5.D.1). Furthermore, those approved programs use “multiple criteria and assessments…to identify candidates for admission who have the potential to become successful teachers” (5.D.2).

While such skills are not explicitly noted in the Professional Standards developed by the National Council for Accreditation of Teacher Education, we might safely presume that they would be needed by all who seek to “know and demonstrate the content, pedagogical, and professional knowledge, skills, and dispositions necessary to help all students learn” (Professional Standards, 2001, p.10). Toward that end, our review and affirmation of prospective and accepted candidates’ academic skills is consistent with institutional, state, and professional standards. The matrix for this first assessment question integrates sources of information available at each phase in candidates’ progress through our program with unit, state, and professional standards. Detailed review of information sources selected for this and subsequent assessment questions appears in the “Unit Assessment System” (2001).

Assessment Question 1: Do candidates possess the basic academic skills and values that will sustain their learning while enrolled in our program of study and practice?

(See table in pdf format)

Assessment Question 2. Do candidates possess an integrated body of knowledge, skills, and values drawn from one or more disciplines central to their area of licensure?

The conceptual framework that guides our candidates’ preparation for teaching encourages them to practice “humane educational decision-making based on appropriate professional knowledge, grounded in Benedictine values, and focused on the essential goals of meeting the needs and enhancing the lives of all students” (Conceptual Model, p.1). Within the realm of that body of professional knowledge we include “not only factual knowledge, but also organizing principles, central concepts,” and the epistemology practiced in the disciplines they will share with their students (Knowledge Base, p.2). An integrated understanding of a field of study from which such facts, concepts, principles, and ways of knowing are drawn contributes to the effectiveness of the “planning decisions” described by James Cooper (1999) and Carl Smith (1992) as central to teachers’ selections of what they will explore with their students. We thus require that our candidates for licensure “understand the central concepts, tools of inquiry, and structure of the disciplines they are preparing to teach so that they will be able to make this subject matter meaningful for their students” (Program Goal 1, Conceptual Model, p. 5).

This first of ten program goals reflects The Minnesota Board of Teaching’s rules guiding approval of teacher preparation programs. Candidates for licensure prepared in an approved program “complete a program of general studies in the liberal arts and sciences” that is “equivalent” to that required of all students enrolled in that institution (Institutional Program Approval Rules, 8700.7600.5.B.1). That general education curriculum must incorporate “multicultural and global perspectives” (5.B.3). Further, approved programs must “require candidates in teacher preparation programs to attain academic competence in the content that they plan to teach” (5.B.2). This goal is also congruent with Minnesota’s “Standards of Effective Practice for Teachers” (Minnesota Board of Teaching Licensure Rules, 8710.2000.2), a core of knowledge and skills guiding the preparation and practice of all who would teach Minnesota’s children.

Guided by these general rules, approved programs provide their candidates with opportunities to acquire knowledge and skills defined by content standards included in licensure rules set by Minnesota’s Board of Teaching. These rules incorporate the advice offered by nationally recognized professional associations as well as standards that reflect the unique needs of Minnesota’s children.

Our program offers preparation for Minnesota licensure in eight areas. These include…

Elementary Education with a Specialty (Minnesota Board of Teaching Licensure Rules, 8710.3200)
Communication Arts and Literature (8710.4250)
Mathematics (8710.4600)
Vocal Music and Instrumental Music (8710.4650)
Science (8710.4750)
Social Studies (8710.4750)
Visual Arts (8710.4900), and
World Languages and Cultures (8710. 4950).

The internal and external program approval process used for each licensure area identifies the opportunities afforded candidates to know, to apply, and to be assessed on relevant state content standards. Documents developed for that approval process describe instruction and formative review of students’ performance. While we share responsibility for such opportunities with our colleagues in academic departments, that formative review is embedded in courses and learning experiences planned and offered by them. The matrix for this second assessment question describes our summative assessment of candidates’ performance.

Providing candidates with opportunities to acquire, integrate, and use the subject matter they will teach is also consistent with the National Council for Accreditation of Teacher Education’s standards. That organization urges candidates for initial licensure to “know the subject matter they plan to teach as shown by their ability to explain important principles and concepts delineated in professional, state, and institutional standards” (Professional Standards, p. 14). Our first program goal is similar to this first element in the set defining NCATE’s conceptualization of “Candidate Performance.”

Driven by this first program goal and supported by relevant state and professional standards, we thus pursue our second assessment question. Do candidates recommended for licensure possess an integrated body of knowledge, skills, and values drawn from one or more disciplines central to their area of licensure? Performance indicators supporting this question include evidence of candidates’ understanding of their subject, their use of alternative views or theories drawn from that body of knowledge and skill, and candidates’ success connecting their content knowledge with other subject areas.

Assessment Question 2: Do candidates possess an integrated body of knowledge, skills, and values drawn from one or more disciplines central to their area of licensure.

(See table in pdf format)

3. Do candidates possess pedagogical knowledge, skills, and values appropriate for their areas of licensure?

The unit’s philosophy calls for the preparation of teachers who believe that “all students can learn” even if in different ways, at different rates, and at different levels. Those who accept this premise must therefore “not only be knowledgeable about the content they teach, but must also know about and be committed to making decisions that involve the use of a variety of instructional strategies and approaches appropriate for the diverse learning needs of their students” (Conceptual Model, p. 3). The decision-making model that is at the core of this unit’s preparation of professional educators calls upon candidates for that role to acquire and use a

“body of professional knowledge” that includes “foundational knowledge” (knowledge of learning, development, and human exceptionalities) and an understanding of the principles of effective practice (knowledge of pedagogy, instructional technologies, motivational strategies, management techniques, and assessment methods). This body of knowledge forms the basis of the information from which available alternative for the decisions questions are formulated (Conceptual Model, p. 2).

Elements of all ten of the unit’s program or learning goals respond to this need to anchor decisions in a foundation of professional knowledge. Explorations guided by each of these goals offer candidates’ the opportunity to realize the context within which their practice as professional educators will take place. Further, together they offer a “knowledge base” upon which to found that practice. This sense of a conceptual foundation for teaching practice is consistent with the performance-focused accreditation standards offered by the National Council for Accreditation of Teacher Education (NCATE). That organization finds that candidates for licensure should use “professional and pedagogical knowledge and skills delineated in professional, state, and institutional standards” to create experiences that will help all their students learn (Professional Standards, p. 14). In doing so, candidates enrich their understanding of the context in which they will practice through work directed by these five program goals. They are described in detail in the unit’s conceptual model as well as in the second section of this report (II. Conceptual Framework, pp.1-2). These five goals include…

As this foundation grows, it can begin to support the development of pedagogical content knowledge” representing a synthesis of content (the subject to be taught) with a growing knowledge of teaching methods adapted to that content (“how to teach this subject”). This interaction of subject matter knowledge and foundational knowledge informs, and is informed by, one’s evolving sense of a content specific pedagogy. Knowing “what to teach” tempers the selection and refinement of “how to teach.” Candidates thus adapt “a broad knowledge of instructional strategies” to the unique demands of their subjects, the needs and talents of their students, and the influences of the settings in which they will teach to offer the “multiple explanations” that will help all their students learn (Professional Standards, p. 15).

Two performance dimensions drawn from analysis of the first program goal reveal indicators of the candidate’s integration of subject matter and teaching methods. Four related program goals contribute an analysis of candidates’ pedagogical content knowledge, including dimensions of…

The unit’s perspective on the role of such pedagogical knowledge, skills, and values might play in candidates’ teaching of their disciplines or praxiologies is reinforced by the standards for the approval of teacher preparation programs by Minnesota’s Board of Teaching. That agency requires candidates for licensure to be prepared in “high quality education programs that are cohesive, comprehensive, and based on research, theory, and accepted practice (Minnesota Rules 8700.7600.5.A.1). Further, approved programs must require that its candidates “complete a professional sequence of courses” that provide opportunities to know, to apply, and to be assessed on each of the Minnesota Standards of Effective Practice for Teachers (8700.7600.5.A.2). In doing so, the Board expects that candidates “can integrate general, content, professional, and pedagogical studies as measured by teacher performance and the performance of the students they teach” (8700.7600.5.B.4). This integrative outcome process could be encouraged by faculty who “encourage the candidate’s development of reflection, critical thinking, problem solving, and professional dispositions” (8700.7600.5.G.10).

We should thus expect those who are prepared for licensure as teachers in Minnesota to have acquired a pedagogy appropriate for the content area and grade level they are prepared to teach. This expectation leads to the third assessment question. Do candidates possess these pedagogical knowledge, skills, and values appropriate for their areas of licensure? Reflecting the distinction between foundational and functional knowledge, the question is divided to address these two types of teaching knowledge. Matrices for these two sub-questions reveal the linkage between standards, proficiencies, and performance indicators. A review of the “Unit Assessment System” (2001) offers a detailed description of information sources selected for these questions.

Assessment Question 3a. Do candidates possess the professional and pedagogical knowledge and skills appropriate for their areas of licensure?

(See table in pdf format)

Assessment Question 3b. Do candidates possess pedagogical content knowledge appropriate for their areas of licensure?

(See table in pdf format)

Assessment Question 4: Can candidates teach knowledge and skills to others while modeling values appropriate for their areas of licensure?

Candidates prepared for licensure possess the academic skills to acquire and integrate a body of content, professional, and pedagogical knowledge, skills, and values. If, through the units’ program of study and practice, they should reach that end, can these candidates draw upon this fund of integrated knowledge and experience to teach others? To expect that they can do so is consistent with the unit’s philosophy and mission. If indeed “all students can learn…in different ways and at different rates,” then candidates for licensure must “use their content knowledge, pedagogical skills, and understanding of their students to make informed and ethical classroom decisions that foster their students’ learning (Conceptual Model, p.3, 4).

Such an outcome is expected of teacher preparation programs approved by Minnesota’s Board of Teaching. That agency anticipates that “candidates integrate general, content, professional, and pedagogical studies as measured by teacher performance and performance of the students they teach” (Minnesota Rules: Institutional Program Approval: 8700.7600.B.4 1999). The influence of candidates’ work on their students’ learning is implied in standards advanced by the National Council for Accreditation of Teacher Education, anticipating that a program’s “teacher candidates…have a positive effect on learning for all students (Professional Standards, p.16). Such expectations seem to reflect “common sense,” as the folk maxim that asserts “if the student has not learned, the teacher has not taught” would suggest.

It is also in the classroom where candidates’ dispositions to behave as educational professionals might be most clearly documented. Drawing on a foundation of values that tested during 1,500 years of monastic life guided by the Rule of Saint Benedict, the unit expects that candidates prepared for their roles as educators will reveal a “commitment to service” directed toward enhancing the lives of their students (Conceptual Model, p. 4). Further, candidates’ practice will reveal their disposition to advance their “concern for community” as they “extend genuine caring and respect for all students” (Unit Assessment System, Appendix C: Goal 5, Performance Dimension 5.a.1, “Distinguished “ Performance Level). Candidates’ should reveal behaviors that affirm their fundamental “respect for all persons” as they work “to develop a learning community in which individual differences are respected” (Unit Assessment System, Appendix C: Goal 3, Performance Dimension 3.b.1). Their “passion for learning” should be evident in self-reflection that sustains renewal by “eagerly seeking out opportunities for growth” (Unit Assessment System, Appendix C: Goal 9, Dimension 9.B.1, Distinguished performance level). Further, the unit expects that the candidates it prepares will “demonstrate dependability” (Unit Assessment System, Goal 9, Dimension 9.d.1), “initiative” (9.d.2), “enthusiasm” (9.d.3), a “commitment to professionalism” (9.d.4) and “flexibility and open-mindedness” (9.d.5) as they perform in field and most certainly clinical settings.

Assessment Question 4: Can candidates teach knowledge and skills to others while modeling values appropriate for their areas of licensure?

(See table in pdf format)

Element 2: Data Collection, Analysis, and Evaluation

Parts of the unit’s assessment system are now in use. The prototype for a comprehensive student database that can capture information they provide has been developed and is in use. The following table summarizes sources now in field testing or fully implemented as well as those scheduled for pilot or field testing in the future.

Assessment Questions

Sources Implemented

Sources To Be Implemented

1. Academic
Skills

PPST
Academic Profile
Speech Proficiency
College GPA
Unit GPA
Student Teaching Portfolio
(writing, math, speech)
Admissions Test Scores
High School GPA and Rank

Flag Papers (embedded writing) 1SEP01
Methods Course Work Sample
(writing, math, speech) 1SEP01

2. Content
Knowledge

Student Teaching Performance Profile
(Goal 1)
Major GPA

Praxis II (content test) 1SEP01
Integrative Experience 1SEP02
Field Experience Performance Profile 1SEP01
(derived from Methods work sample)
Student Teaching Work Sample (content) 1SEP01

3A.
Professional /
Pedagogical Knowledge

Student Teaching Performance Profile (Goals 2,3,5,9,10)

 

Piagetian Task Analysis 1SEP01
Educational Philosophy 1SEP01
Classroom Management Plan 1SEP01
Human Relations Project 1SEP01
Field Experience Performance Profile 1SEP01
(derived from Methods work sample)
Praxis II (pedagogical knowledge) 1SEP01

3. B Pedagogical Content Knowledge

Student Teaching Performance Profile (Goals 1,4,6,7,8)
Resource Portfolios (selected methods courses)

Praxis II (subject specific pedagogical knowledge) 1SEP01
Field Experience Performance Profile 1SEP01 (derived from work sample)
Methods Course Resource Portfolios 1SEP01

4. Teaching
Others

 

Student Teaching Performance Profile (Goals 3,5,9)

Field Experience Performance Profile 1SEP01
Methods Work Sample 1SEP01
Student Teaching Work Sample 1SEP01

Element 3: Use of Data for Program Improvement

Collecting and analyzing candidates’ performance data without using that information to answer evaluation questions about the program’s merit and worth provides little incentive for the design and maintenance of complex assessment systems such as the unit is developing. While intentions with respect to program evaluation based on assessment data are revealed in greater depth in the “Unit Assessment Plan” (2001), highlights included here suggest the range and depth of such studies as this rich pool of information would support.

The academic skills students seeking acceptance as candidates have long been a concern of the unit’s faculty. As early as 1996 the unit commissioned the colleges’ Instructional Developer to explore the feasibility of using fixed and free response testing to provide a more accurate estimate of those skills. Initial results suggested significant deficiencies. The unit revised its acceptance policies and provided a wider range of formal remedial opportunities for those prospective candidates whose reading comprehension, knowledge of writing “mechanics,” use of mathematics, and ability to write a persuasive essay fell below the unit’s basic standards. Remedial services responded to increased demand with new forms of assistance. Testing procedures were subsequently refined and validated to improve the accuracy of skills assessments.

With the benefit of some maturity in the use of these assessments, the unit’s faculty members now seek to confirm that successfully remediated candidates continue to use their refined skills in the work they complete for advanced foundation and methods courses. Writing assessments of “flag” papers embedded in selected courses (assessment question 1) will offer a preliminary response to this concern, as will review of work samples prepared during field and clinical settings.

As the unit gains experience in the use of these indicators, analysis of the relationship between Academic Profile scores and college entrance examination scores may confirm that students with ACT or SAT scores above a cut-point may be exempt from completing the fixed response Profile. A preliminary study suggested the viability of this approach based on a small sample of prospective candidates. The development of the unit’s performance database will enhance the feasibility of such evaluative work, and thus lead to program improvement.

Other evaluation questions are emerging from preliminary analyses of available performance data and from review of the unit’s knowledge base?

  • Do all candidates complete our program with a fund of content knowledge, skills, and values that will sustain effective instruction in the first years of their practice? Analyses of opportunities to know and apply content knowledge suggest that some areas may provide too little knowledge covering too many disciplines.
  • Do candidates form a useful “model” of how their students learn that reflects those students’ needs and talents, the content of instruction, and candidates’ talents What decision-making processes do methods or student teaching candidates use when selecting methods of instruction or creating instructional materials? How do the adapt their model when student’s learning differs from their expectations?
  • What performance patterns are evident in the foundations, methods, and content coursework of candidates whose student teaching performance profiles suggest “strong” as opposed to “weak” performance? What guidance might those patterns offer for candidate selection and preparation for practice?
  • What patterns are evident in student teachers’ use of assessment or instructional techniques? What do they draw upon to support their decisions to use these techniques? What reflection to they offer on the effectiveness of their choices?
  • With what success have the unit’s efforts to prepare candidates for practice in multicultural settings been effective? How do those candidates in such settings judge the effectiveness of their preparation? In what ways might they encourage the unit to improve its program?

References

Academic catalog, 2000-20001. (2000). Saint Joseph, MN: College of Saint Benedict and Saint John’s University.

Education department conceptual model: Teacher as decision-maker. (2000). Saint Joseph, MN: College of Saint Benedict and Saint John’s University.

Knowledge base for beginning teachers. (2000). Saint Joseph, MN: College of Saint Benedict and Saint John’s University, Education Department.

Minnesota Rules 8700.7600.B.4. (1999). Adopted permanent rules relating to institution and teacher preparation program approval. Roseville, MN: Department of Children, Families, and Learning, Minnesota Board of Teaching.

Minnesota Rules 8710.2000. (1999). Standards of effective practice for teachers. Roseville, MN: Department of Children, Families, and Learning, Minnesota Board of Teaching.

Professional standards for the accreditation of schools, colleges, and departments of education. (2001). Washington, D.C. National Council for Accreditation of Teacher Education.

Unit Assessment Plan. (2001). Saint Joseph, MN.: College of Saint Benedict and Saint John’s University, Education Department.