Standard 2: Program Assessment and Unit Evaluation
The unit has an assessment system that collects and analyzes data on applicant qualifications, candidate and graduate performance, and unit operations to evaluate and improve the unit and its programs.
Element 1: The Unit’s Assessment System.
Overview. Although one component of the units’ assessment of candidates was first used in 1996, the design of the unit’s present assessment system began with conversations and consultation during an August 1999 conference in Philadelphia sponsored by the National Council for the Accreditation of Teacher Education (NCATE). Members of the unit attending that conference choose to use the “NCATE 2000” performance-based accreditation standards as they prepared for an accreditation visit in April of 2001. These piloted performance-based standards would later appear as NCATE’s Professional Standards for the Accreditation of Schools, Colleges, and Departments of Education.
That conference encouraged initial sketches during the fall of 1999 of the emerging system’s primary elements. As those sketches took more definite form they were informally critiqued by the unit’s faculty and by experienced elementary and secondary teachers. Work on one of those elements, a standards-based “performance profile” for candidates completing student teaching, began in the summer of 2000. Preparation for a successful review of the unit’s licensure programs during the 2000-2001 academic year by Minnesota’s Board of Teaching offered further experience in the design of an assessment system driven by the need to assure all candidates of opportunities to know, apply, and to be assessed on their performance on licensure standards.
In October of 2000, the unit contributed a set of assessment questions that would guide the collection and analysis of candidates’ performance information. The following month colleagues confirmed the significant features of a system that could respond to those questions. The system’s design was completed in late December. That design was described in a comprehensive plan shared with the unit’s members in January of 2001 and approved by the Teacher Education Council in April of 2001 (Unit Assessment System).
The unit’s Assessment Committee contributed to the emerging system and monitored its use as the elements of the plan evolved into a functional mechanism for gathering, analyzing, and reporting candidate information. Since the unit approval of the system, some methods of gathering information have been deleted, many altered, and some introduced in response to curricular changes or the need for useful information ( Assessment Update). The core of the system, however, remains as planned. Challenges in storing and retrieving information, while vexing, appear to have been met. Access to candidate performance information has contributed to changes in the unit’s policies, procedures, and curriculum. The core of the system, however, remains as planned.
Assessment Questions. The unit’s assessment system reflects the joint mission of the College of Saint Benedict and Saint John’s University as residential colleges pursuing the liberal arts within the Catholic university tradition. These two rural Minnesota institutions work together to offer their students “a unified liberal arts curriculum which focuses on questions important for the human condition, demands clear thinking and communicating, and calls forth new knowledge for the betterment of humankind.” Through this shared curriculum these two colleges strive as one to offer their students “an integrative environment for learning which stresses intellectual challenge, open inquiry, collaborative scholarship, and artistic creativity.” They recall their monastic founders by celebrating learning within Benedictine settings that “foster attentive listening to the voice of God, awareness of the meaning of one’s existence, and the formation of community built on respect for individual persons” (Academic Catalog 2003-2005; Coordinate Mission, p. 4).
The Department of Education jointly sponsored by these two liberal arts colleges takes as its aim the preparation of “exemplary teachers who have a strong liberal arts background, exemplify Benedictine values, and make professional decisions which can help all students achieve their full potential as persons and as responsible world citizens in a democratic society” (Education Department Conceptual Framework; Mission, p. 5).
Focused by this aim, and consistent with the efforts of the two colleges, the Education Department’s mission is to offer those prospective teachers “a rich and diverse background of coursework and experiences that stress intellectual challenge, open inquiry, collaborative scholarship, and that promote clear thinking.” This unit’s mission encourages the preparation of “teachers who make their informed and ethical classroom decisions based on a firm knowledge of content, pedagogy, and the needs of their students” (Framework; Mission, p. 6).
Students enrolled by the colleges and prepared for licensure as teachers by Minnesota’s Board of Teaching reflect this mission and aim as they work toward the Education Department’s program goals. The knowledge, skills, and values that are acquired and affirmed through candidates’ pursuit of these goals strengthen the decisions they make as they plan, implement, and evaluate their practice. The department’s goals are guided by the Standards of Effective Practice for Teachers, a family of pedagogical standards set by Minnesota’s Board of Teaching (MSEPT, 1999). The 10 terminal and 120 enabling standards that form this collection were derived from guidelines developed by the Interstate New Teacher Assessment and Support Consortium (INTASC).
This sense of institutional and unit mission, the later nested within the former, guides the design of the assessment system through the specification of assessment questions. Taken together, these questions reflect the unit’s program theory in the context of its aim, mission, and goals. Such questions guide review of candidates’ performances in a program of study and practice that prepares them for their roles as professional educators.
- Do candidates prepared for licensure as teachers possess the academic skills that will sustain their learning while enrolled in this program?
- Do these candidates possess an integrated body of knowledge, skills, and values drawn from one or more disciplines central to their area of licensure?
- Do candidates possess pedagogical knowledge, skills, and values appropriate for their areas of licensure?
- Can these candidates teach knowledge and skills from their areas of licensure to others?
Standards-Based Assessment. The unit’s assessment system is designed to respond to these questions within the context of unit, state, and professional standards. The unit’s expectations are revealed in its conceptual framework and goals. The State of Minnesota’s program approval and candidate licensure standards together define another performance context for all of teacher preparation programs and the candidates they present for licensure. The professional standards that guide NCATE's review of a unit offer a third context to help shape responses to the unit’s assessment questions. These nested standards in turn suggest candidate proficiencies to be examined and described.
Assessment Question 1. Do candidates we prepare for licensure as teachers possess the academic skills and values that will sustain their learning while enrolled in our program?
The first of these five assessment questions guides our search for information affirming that our students possess the academic skills and values that will sustain their learning in our program of study and practice when accepted as candidates preparing for licensure as teachers. Such skills include the ability to write well, to draw inferences from reading complex information, to use mathematics to solve everyday problems, and to share ideas with others through formal speech and discussion.
Embedded within our belief in the necessity of a broad liberal arts education is an emphasis on the basic skills of reading, critical thinking, and writing. Therefore, we seek to provide experiences throughout our program that enhance the development of these skills. Further, we attempt to emphasize the Benedictine values of openness to change and lifelong learning as essential to continued teaching effectiveness” (Framework; Theme, p. 4).
Our expectation that candidates for teacher licensure possess academic skills and values consistent with the opportunities revealed through their liberal arts education is also consistent with related state and professional standards. The Minnesota Board of Teaching’s rules for the approval of teacher preparation programs require that “the institution recruits, admits, and retains candidates who demonstrate potential for professional success in schools (Minnesota Rules: Institutional Program Approval, 1999, 8700.7600.5.D.1). Furthermore, those approved programs use multiple criteria and assessments to identify candidates for admission who have the potential to become successful teachers (5.D.2).
While such skills are not explicitly noted in the Professional Standards developed by the National Council for Accreditation of Teacher Education, we might presume that they would be needed by all who seek to “know and demonstrate the content, pedagogical, and professional knowledge, skills, and dispositions necessary to help all students learn” (Professional Standards, 2002, p.10). Toward that end, our review and affirmation of prospective and accepted candidates’ academic skills is consistent with institutional, state, and professional standards. The matrix for this first assessment question integrates sources of information available at each phase in candidates’ progress through our program with unit, state, and professional standards. The plan for the unit’s assessment system includes a detailed review of information sources selected for this and subsequent assessment questions appears (Unit Assessment System).
Assessment Question 1: Do candidates possess the basic academic skills and values that will sustain their learning while enrolled in our program of study and practice?
Program Intentions: Program Phases: Formative Summative Assessment
Assessment Question 2. Do candidates possess an integrated body of knowledge, skills, and values drawn from one or more disciplines central to their area of licensure?
The conceptual framework that guides our candidates’ preparation for teaching encourages them to practice “humane educational decision-making based on appropriate professional knowledge, grounded in Benedictine values, and focused on the essential goals of meeting the needs and enhancing the lives of all students” (Framework; Model, p.1). Within that body of professional knowledge we include “not only factual knowledge, but also organizing principles, central concepts,” and the epistemology practiced in the disciplines they will share with their students (Knowledge Base; Theme, p.2). An integrated understanding of a field of study from which such facts, concepts, principles, and ways of knowing are drawn contributes to the effectiveness of the “planning decisions” described by James Cooper (1999) and Carl Smith (1992) as central to teachers’ selections of what they will explore with their students. We thus require that our candidates for licensure “understand the central concepts, tools of inquiry, and structure of the disciplines they are preparing to teach so that they will be able to make this subject matter meaningful for their students” (Framework; Model, Goal 1, p. 5).
This first of our ten program goals reflects The Minnesota Board of Teaching’s rules guiding approval of teacher preparation programs. Candidates for licensure prepared in an approved program “complete a program of general studies in the liberal arts and sciences” that is “equivalent” to that required of all students enrolled in that institution (Institutional Program Approval Rules, 8700.7600.5.B.1). That general education curriculum must incorporate “multicultural and global perspectives” (5.B.3). Further, approved programs must “require candidates in teacher preparation programs to attain academic competence in the content that they plan to teach” (5.B.2). This goal is also congruent with Minnesota’s “Standards of Effective Practice for Teachers” (Minnesota Board of Teaching Licensure Rules, 8710.2000), a core of knowledge and skills guiding the preparation and practice of all who would teach Minnesota’s children.
Approved programs provide their candidates with opportunities to acquire knowledge and skills defined by licensure standards included in licensure rules set by Minnesota’s Board of Teaching. These rules incorporate the advice offered by nationally recognized professional associations as well as standards that reflect the unique needs of Minnesota’s children.
Our program offers preparation for Minnesota licensure in eight areas. These include…
- Elementary Education with a Specialty
- (Minnesota Board of Teaching Licensure Rules, 8710.3200)
- Communication Arts and Literature (8710.4250)
- Mathematics (8710.4600)
- Vocal Music and Instrumental Music (8710.4650)
- Science (8710.4750)
- Social Studies (8710.4800)
- Visual Arts (8710.4900), and
- World Languages and Cultures (8710. 4950).
The unit’s preparation for the Board of Teaching’s program approval process helped us identify the opportunities we provide candidates to know, to apply, and to be assessed on all relevant content standards. Documents developed to support that approval process describe the formative review of candidates’ performance on those standards by faculty in supporting disciplines. The matrix for this second assessment question describes the unit’s summative assessment of candidates’ performance on those standards.
Providing candidates with opportunities to acquire, integrate, and use the subject matter they will teach is also consistent with the National Council for Accreditation of Teacher Education’s standards. That organization urges candidates for initial licensure to “know the subject matter they plan to teach as shown by their ability to explain important principles and concepts delineated in professional, state, and institutional standards” (Professional Standards, p. 14). Our first program goal is similar to this first element in NCATE’s conceptualization of “candidate performance.”
Driven by this first program goal and supported by relevant state and professional standards, we thus pursue our second assessment question. Do candidates recommended for licensure possess an integrated body of knowledge, skills, and values drawn from one or more disciplines central to their area of licensure? Performance indicators supporting this question include evidence of candidates’ understanding of their subject, their use of alternative views or theories drawn from that body of knowledge and skill, and candidates’ success connecting their content knowledge with other subject areas.
Assessment Question 2: Do candidates possess an integrated body of knowledge, skills, and values drawn from one or more disciplines central to their area of licensure?
Intentions: Program Phases: Summative Assessment
3. Do candidates possess pedagogical knowledge, skills, and values appropriate for their areas of licensure?
The unit’s philosophy calls for the preparation of teachers who believe that “all students can learn” even if in different ways, at different rates, and at different levels. Those who accept this premise must therefore “not only be knowledgeable about the content they teach, but must also know about and be committed to making decisions that involve the use of a variety of instructional strategies and approaches appropriate for the diverse learning needs of their students” (Framework, Philosophy, p. 3). The decision-making model that is at the core of this unit’s preparation of professional educators calls upon candidates for that role to acquire and use a “body of professional knowledge” that includes “foundational knowledge” (knowledge of learning, development, and human exceptionalities) and an understanding of the principles of effective practice (knowledge of pedagogy, instructional technologies, motivational strategies, management techniques, and assessment methods). This body of knowledge forms the basis of the information from which available alternative for the decisions questions are formulated (Framework; Theme, p. 2).
Elements of all ten of the unit’s program goals provide a foundation of professional knowledge in which candidates’ can anchor their decisions about teaching and learning. Course, field, and clinical experiences guided by these goals offer candidates the opportunity to realize the context within which their practice as professional educators will take place. Further, as they integrate these experiences, candidates begin to form a personal “knowledge base” to guide their decision-making.
A conceptual foundation for teaching practice is consistent with the performance-focused accreditation standards offered by the National Council for Accreditation of Teacher Education (NCATE). That organization finds that candidates for licensure should use “professional and pedagogical knowledge and skills delineated in professional, state, and institutional standards” to create experiences that will help all their students learn (Professional Standards, p. 14). In doing so, candidates enrich their understanding of the context in which they will practice through work directed by five of the unit’s program goals. They are described in detail in the unit’s knowledge base as well as in the second section of this report (II. Conceptual Framework, p.II.1-2). These five goals include…
- Goal 2. Student Learning and Development,
- Goal 3. Diverse Learners,
- Goal 5. Learning Environment,
- Goal 9. Reflection, and ,
- Goal 10. Collaboration.
As this foundation grows, it can begin to support the development of “pedagogical content knowledge” representing a synthesis of content (the subject to be taught) with a growing knowledge of teaching methods adapted to that content (how to teach the subject). This interaction of subject matter knowledge and foundational knowledge informs, and is informed by, one’s evolving sense of a content specific pedagogy. Knowing “what to teach” tempers the selection and refinement of “how to teach.” Candidates thus adapt “a broad knowledge of instructional strategies” to the unique demands of their subjects, the needs and talents of their students, and the influences of the settings in which they will teach to offer the “multiple explanations” that will help all their students learn (Professional Standards, p. 15).
Two performance dimensions drawn from analysis of the first program goal reveal indicators of the candidate’s integration of subject matter and teaching methods. Four related program goals contribute an analysis of candidates’ pedagogical content knowledge, including dimensions of…
- Goal 1. Subject Matter,
- Goal 4. Instructional Strategies,
- Goal 6. Communication,
- Goal 7. Planning, and,
- Goal 8. Assessment.
The unit’s perspective on the role that such pedagogical knowledge, skills, and values might play in candidates’ teaching is reinforced by the standards for the approval of teacher preparation programs by Minnesota’s Board of Teaching. That agency requires candidates for licensure to be prepared in “high quality education programs that are cohesive, comprehensive, and based on research, theory, and accepted practice (Minnesota Rules; 8700.7600.5.A.1). Further, approved programs must require that its candidates “complete a professional sequence of courses” that provide opportunities to know, to apply, and to be assessed on each of the Minnesota Standards of Effective Practice for Teachers (8700.7600.5.A.2). In doing so, the Board expects that candidates “can integrate general, content, professional, and pedagogical studies as measured by teacher performance and the performance of the students they teach” (8700.7600.5.B.4). This integrative outcome process could be encouraged by faculty who “encourage the candidate’s development of reflection, critical thinking, problem solving, and professional dispositions” (8700.7600.5.G.10).
We should thus expect those who are prepared for licensure as teachers in Minnesota to have acquired pedagogy appropriate for the content area and grade level they are prepared to teach. This expectation leads to the third assessment question. Do candidates possess the pedagogical knowledge, skills, and values appropriate for their areas of licensure? Reflecting the distinction between foundational and functional knowledge, the question is divided to address these two types of teaching knowledge. Matrices for these two sub-questions reveal the linkage between standards, proficiencies, and performance indicators. A review of the unit’s assessment system offers a detailed description of information sources selected for these questions.
Assessment Question 3a: Do candidates possess the professional and pedagogical knowledge and skills appropriate for their areas of licensure?
Intentions: Program Phases: Summative Assessment
Assessment Question 3b: Do candidates possess pedagogical content knowledge appropriate for their areas of licensure
Intentions: Program Phases: Summative Assessment
Assessment Question 4: Can candidates teach knowledge and skills to others while modeling values appropriate for their areas of licensure?
Candidates prepared for licensure possess the academic skills to acquire and integrate a body of subject matter, professional, and pedagogical knowledge, skills, and values. If, through the units’ program of study and practice, they should reach that end, can these candidates draw upon this fund of integrated knowledge and experience to teach what they know to others? To expect that they can do so is consistent with the unit’s philosophy and mission. If indeed “all students can learn…in different ways and at different rates,” then candidates for licensure must “use their content knowledge, pedagogical skills, and understanding of their students to make informed and ethical classroom decisions that foster their students’ learning (Framework; Mission, pp.3-4).
Such an outcome is expected of teacher preparation programs approved by Minnesota’s Board of Teaching. That agency anticipates that “candidates integrate general, content, professional, and pedagogical studies as measured by teacher performance and performance of the students they teach” (Minnesota Rules: Institutional Program Approval: 8700.7600.B.4 1999). Effects of candidates’ teaching on their students’ learning are implied in standards advanced by the National Council for Accreditation of Teacher Education, anticipating that a program’s “teacher candidates…have a positive effect on learning for all students (Professional Standards, p.16).
It is also in the classroom where candidates’ dispositions to behave as educational professionals might be most clearly documented. Drawing on a foundation of values elucidated in the Rule of Saint Benedict and tested by 1,500 years of monastic life, the unit expects that candidates prepared for their roles as educators will reveal a “commitment to service” directed toward enhancing the lives of their students ( Framework; Mission, p. 4). Further, candidates’ practice will reveal their disposition to advance their “concern for community” as they “extend genuine caring and respect for all students” (Unit Assessment System Appendix C1: Goal 5, Performance Dimension 5.1, and Appendix C2, indicator 5C, “Distinguished”). Candidates’ should reveal behaviors that affirm their fundamental “respect for all persons” as they work “to develop a learning community in which individual differences are respected” (Unit Assessment System, Appendix C1: Goal 3, Performance Dimension 3.3 and Appendix C2, Goal 3, indicator 3Q). Their “passion for learning” should be evident in self-reflection that sustains renewal by “eagerly seeking out opportunities for growth” (Unit Assessment System, Uunderline;">Appendix C1: Goal 9, Dimension 9.2, and Appendix C2, Indicator 9.E, “Distinguished” performance level). Further, the unit expects that the candidates it prepares will “demonstrate dependability” (Unit Assessment System, Appendix C1, Goal 9.4 and Appendix C2, Goal 9, indicator 9.E) as they perform in field and most certainly clinical settings.
Assessment Question 4: Can candidates teach knowledge and skills to others while modeling values appropriate for their areas of licensure?
Intentions: Program Phases:
Element 2: Data Collection, Analysis, and Evaluation
Implementation. The components of the unit’s assessment system that are now in use as planned in 2001 focus on key dimensions of candidates’ experiences in their licensure programs, offer useful information likely to strengthen those programs, and fit within the limits of time and energy available for managing the assessment system. Some planned methods for collecting information have not survived beyond their trial application, often because their development proved to be too difficult or their application demanded an investment greater than the value of the information they could provide. Development of performance profiles for candidates in methods courses, for example, were difficult to adapt to the characteristics of both two and four credit methods courses. Derived from the unit’s early success with such profiles for documenting student teaching performance, the methods profile may still have value if modified to reflect the shorter and more varied teaching experiences enjoyed by most candidates’ during their methods courses.
Methods work samples have fallen to similar constraints. While formal summative indicators of candidates’ emerging knowledge and skills could help monitoring their performance during the methods phase of their preparation, guiding candidates to prepare useful work samples presented a greater burden for instructors to carry than anticipated. The design of a work sample for use at the methods phase must respond to the limited teaching opportunities included in two credit courses and the varied teaching assignments that are often included in four credit courses. As more methods course form productive unions with faculty in our partner schools, teaching opportunities may support work sampling. We hope to reintroduce some variation on work sampling in selected four credit methods courses in the spring of 2007.
In place of a summative indicator, the formative assessments included in all methods courses of candidates’ emerging understanding of their licensure areas may continue to serve as a viable estimate for most candidates. All candidates are observed and critiqued as they teach K-12 students in area schools for their methods courses. Literacy profiles developed by elementary candidates working with K-6 students in area schools for Education 347, Reading, Writing and Language Growth, the teaching resource binders developed by elementary candidates in their methods courses, and the curriculum projects that all candidates prepare in Human Relations (Education 390) along with the classroom management plans all devise for Issues in Education (Education 359) together provide formative screens through which candidates deficient in knowledge and skills cannot pass.
While attractive as an additional indicator of candidates’ maturing knowledge of their subject matter, the future of a formal integrative experience as described in the 2001 plan remains uncertain. We now have assessments of candidates who design an instructional unit and related lessons as part of their student teaching work samples. This indicator can estimate candidates’ grasp of the “central concepts, tools of inquiry, and structures of the disciplines they are preparing to teach” (Framework; Program Goal 1). Arts and sciences faculty provide such experiences for many secondary level candidates. The development of an integrative experience remains an attractive option for affirming candidates’ knowledge in disjoined licensure areas such as social studies, but securing resources to design and assess such projects will prove challenging.
The unit has used a survey of graduate opinion for several years. Many weaknesses in the design and administration of this survey encouraged its revision in 2002. Despite a significant investment in identifying the location of licensed graduates in their first, second, or third year of teaching, few identified graduates completed and returned past surveys. Those teachers who did respond were almost always positive in their assessment of the program. On the few occasions when principals offered judgments of graduates’ performance, they were nearly always positive but rarely anchored in systematic, long-term observations. As confidence in the value of data provided by the unit’s graduate survey declined, the willingness to continue investing staff time its use waned.
The development of a goal-based candidate exit survey and interview, used with all elementary and secondary candidates since fall 2002 to gather their perspectives on areas of their licensure programs in need of review, stimulated the development of a new format and survey items that could be adapted to the graduate survey, thus “aligning” it with the unit’s conceptual framework. Offering more helpful information than the previous graduate survey at less cost, borrowing items from the exit survey and interview by converting them to a fixed response format proved to be more difficult than expected, slowing progress on the graduate survey. Managing other facets of the assessment system required more time than anticipated, further delaying the revised survey.
Meanwhile, two interim studies conducted by unit faculty offered attractive alternatives to reliance on self-selected respondent samples., A 2002 study of novice instructors nominated by their principals as “teachers of promise,” (Mullin), many of whom were our graduates, drew on interviews and survey data to identify characteristics that, if encouraged in prospective candidates, might predict professional success in K-12 schools. A second and longer study of how a sample of our graduates planned and taught science in their classrooms provided an additional model for review (Dickau). Using surveys, interviews, analysis of instructional materials, and observations of teaching, this study of “teacher efficacy” and the use of “constructivist lesson design” suggested new variables for consideration in the development of the unit’s graduate survey.
These more intensive studies of our candidates’ experiences offer attractive alternatives to the annual “temperature taking” survey formerly used to gather graduate opinion. They would also be difficult to adopt in view of the unit’s current resources. Focused studies of this kind, however, could be balanced with information provided by a more general survey to provide an optimal approach. Exploratory use of the internet to invite respondents to share their views suggests that this newer technology may increase efficiency of disseminating, collecting, and analyzing survey data. A new instrument that takes advantage of this emerging technology could be piloted as early as the spring semester of 2006.
The unit first used the Academic Profile to assess the academic skills of prospective candidates in 1986. This criterion-referenced test was developed by the Educational Testing Service and the College Board to describe students’ college level skills in reading, writing, and mathematics as well as their knowledge of the natural sciences, humanities, and social sciences. Mindful of the increased number and cost of licensure examinations required of our candidates, we explored relationships between performance levels on the Profile and norm-referenced sores on the ACT college entrance examination completed by nearly all our candidates. We found that skill sub-scores on that examination predicted the levels of performance derived from the Academic Profile. Subsequent investigation found similar relationships between Praxis 1 (PPST) scores and the Academic Profile. The unit discontinued the Academic Profile in 2003 and adopted use of candidates’ ACT and PPST scores as indicators of expected performance in reading, writing “mechanics,” and mathematics.
Each semester the unit offers prospective candidates an opportunity to affirm their writing skills by completing an essay on a current education topic. Essays are holistically scored by unit members to estimate writing performance. The evolution of the Education Department Writing Assessment provides a helpful estimate of a critically important academic skill not otherwise accurately confirmed by either the PPST or ACT. Embedded assessments of writing in foundations courses supplement information from the writing assessment to help us track writing skill as candidates move through their licensure programs. Results are reported to the unit’s members following each administration of the writing assessment and in an annual summary highlighting trends in candidates’ writing performance.
While the development of the Student Teaching Performance Profile to track elementary candidates’ performance on key unit and state standards was an early success in the assessment system, the departure of key personnel delayed the development of an equally useful profile to describe secondary candidates’ performance. Additional work on the secondary profile, completed by experienced secondary educators, resulted in the first systematic review of secondary candidates during the fall of 2003. While further refinements are likely, all candidates are now assessed on key standards using behaviorally anchored rating scales. Follow-up studies suggesting weaknesses in the rating of candidates by some college supervisors have encouraged additional training in their use of this important summative performance indicator to improve reliability of performance estimates.
While candidates have planned and completed units of instruction as part of their student teaching experience for many years, the systematic review of those units in the summer of 2000 encouraged our adaptation of the teacher work sample used in some teacher preparation programs. As the primary indicator of whether candidates can help all their students learn, the work sample is an important indicator in our assessment system. Following revision of the framework for such samples, including models of its elements for candidates to review and review of those elements for their college supervisors, the work sample is at the core of every candidates’ comprehensive student teaching portfolio.
While the findings of assessment research driven by the unit’s system were shared with the unit in the first year of its development, more extensive explorations of candidates’ performance were hampered by changes in the technology used to develop and maintain the performance database that could capture, store, and examine useful information.
Prototypes developed in several software formats were partially successful, but became obsolete as our colleges’ adopted new programs. Temporary “work around” solutions, while providing needed data, were both time consuming and unable to realize the full promise of the planned system. A final prototype for our performance database has been tested, refined, and is now in use.
The following table summarizes information sources now in field testing or fully implemented as well as those scheduled for pilot or field testing in the near future.
Element 3: Use of Data for Program Improvement
Collecting and analyzing candidates’ performance data without using that information to answer evaluation questions about the program’s merit and worth provides little incentive for the design and maintenance of complex assessment systems such as the unit is developing. The unit’s expectations for how the planned assessment system might contribute to improvement of its programs intentions were described in that 2001 plan for that system. Some of those anticipated applications have been realized, while others remain before us. While we have invested much of our energy in refining elements of the assessment system, we are finding ways to use that system to improve our licensure programs as our system matures.
The academic skills students seeking acceptance as candidates have long been a concern of the unit’s faculty. As early as 1996 the unit explored the feasibility of using fixed and free response testing to provide a more accurate estimate of those skills. Initial results suggested significant deficiencies. The unit revised its acceptance policies and provided a wider range of formal remedial opportunities for those prospective candidates whose reading comprehension, knowledge of writing “mechanics,” use of mathematics, and ability to write a persuasive essay fell below the unit’s basic standards. Developmental services responded to increased demand with new forms of assistance. Testing procedures were substantially refined and validated to improve the accuracy of skills assessments.
Five years later, we find fewer candidates entering our program with skill deficiencies in reading, writing, or mathematics. Those who do reveal weaknesses appear to profit from diagnostic review and developmental instruction. Despite the loss of Learning Plus, an inexpensive computer-based developmental program, guided tutoring and completion of formal courses are helping candidates acquire the academic skills they will need to succeed in our program. In writing performance, for example, an embedded assessment in candidates’ fourth year foundations course finds that fewer of those whose essay performance fell below the unit’s standard continuing to write at below standard after completing developmental work.
As the unit gained experience in the use of academic skills indicators, analysis of the relationship between Academic Profile scores and college entrance examination scores did confirm that students with ACT scores above 24 could be exempted from completing an academic skills examination in reading, writing mechanics, and mathematics. This finding lead to the elimination of the Academic Profile as a required examination for all candidates. We continue to use the performance levels developed for that test as our standard.
Do all candidates complete our program with a fund of subject matter Knowledge, skills, and values that will sustain effective instruction in the first years of their practice? Licensure test results and subsequent analyses of test performance provided by the Educational Testing Service confirm that nearly all candidates do so. That information, reviewed annually, also suggests areas of some licensure programs in need of further review. Looking at opportunities to know and apply content knowledge confirmed the need to strengthen the preparation of elementary candidates seeking a “specialty” in language arts. Additional course work in adolescent literature and the introduction of a more challenging writing course provided a stronger foundation for candidates seeking to teach middle level language arts Review of the effects of curricular changes will await licensure examination data describing the performance of candidates completing the revised program.
Nearly all candidates (98-100%) who complete Minnesota’s licensure examinations in academic skills (PPST), pedagogy, and subject matter knowledge (Praxis II) meet or exceed state licensure standards (Title II). Closer examination of the content examination for social studies found possible weaknesses in secondary candidates’ knowledge of world history. Elementary candidates pursuing a grade 5-8 specialty in social studies revealed weakness in their knowledge of world and U.S. history. The unit invited arts and sciences faculty and experienced secondary social studies teachers to review these findings in light of the licensure standards for both elementary and secondary candidates. Their preliminary recommendations will form the basis for more intensive review.
Assessment information that reveals strengths in candidate performance can also be beneficial. In the spring of 2005 we also met with members of our college’s mathematics faculty to review elementary and secondary candidates’ licensure test results in their discipline. We used this opportunity to reveal that nearly all candidates passed all required examinations. We encouraged our colleagues to continue their efforts as we reviewed licensure standards as they might relate to the design of Praxis tests in mathematics.
Analysis of our student teachers performance profiles and our analysis of their teacher work samples finds that nearly all candidates perform at the “proficient” level on key unit and state standards. While there are outstanding performers rated as “distinguished,” there are also very few are rated at the “basic” level on behaviorally anchored rubrics for those standards. The small number of weaker performers hampers our search for performance patterns that could help us better prepare weaker performers.
Have the unit’s efforts to prepare candidates for practice in multicultural settings been successful? While comments from graduates teaching in highly diverse settings suggest the helpfulness of the unit’s efforts to provide all candidates with opportunities to teach students from a different economic, cultural, ethnic, and racial backgrounds, much more remains to be learned about the demands our graduates face and the ways in which our licensure programs can help them meet those demands. Several semesters of formative review of our prospective elementary candidates’ experiences in their “urban plunge” uncovered facets of that experience that were used to inform the development of a similar urban experience for secondary candidates. Our diversity transcript, while not yet able to document all students ‘ experiences in diverse educational settings, suggests that more candidates enjoy a greater range of experiences than in past years. Student teachers’ experiences working with diverse learners, as revealed in their portfolios and work samples, have been positive. While we are only beginning to unlock the potential for further work in this area, initial results are promising.
Academic catalog, 2003-2005. (2003). Saint Joseph, MN: College of Saint Benedict and Saint John’s University.
Dickau, B. (2003) New Teachers’ Sense of Teacher Efficacy and Use of Constructivist Teaching Techniques in Science.
Education Department Conceptual Framework: Teacher as Decision Maker. (2000). Saint Joseph, MN: College of Saint Benedict and Saint John’s University.
Knowledge Base for Beginning teachers. (2004). Saint Joseph, MN: College of Saint Benedict and Saint John’s University, Education Department.
Minnesota Rules 8700.7600.B.4. (1999). Adopted permanent rules relating to institution and teacher preparation program approval. Roseville, MN: Minnesota Education Department, Minnesota Board of Teaching.
Minnesota Rules 8710.2000. (1999). Standards of effective practice for teachers. Roseville, MN: Minnesota Department of Education, Minnesota Board of Teaching.
Mullin, D. (2002) Teachers of Promise. Paper presented at AACTE Annual Conference
Professional standards for the accreditation of schools, colleges, and departments of education. (2002). Washington, D.C. National Council for Accreditation of Teacher Education.
Theisen, J. The Rule of Saint Benedict: Introduction. The Order of Saint Benedict, Collegeville, MN. (http://www.osb.org/gen/rule.html)
Unit Assessment System: 2001-2005. (2001). Saint Joseph, MN.: College of Saint Benedict and Saint John’s University Education Department.
Updated July 18, 2005