Element 3: Use of Data for Program Improvement
Collecting and analyzing candidates' performance data without using that information to answer evaluation questions about the program's merit and worth provides little incentive for the design and maintenance of complex assessment systems such as the unit has developed and now employs. The unit's expectations for how the planned assessment system might contribute to improvement of its programs intentions were described in that 2001 plan for that system. Some of those anticipated applications have been realized, while others remain before us. While we have invested much of our energy in refining elements of the assessment system, we are finding new ways to use that system to improve our licensure programs as our system matures.
- The academic skills students seeking acceptance as candidates have long been a concern of the unit's faculty. As early as 1996 the unit explored the feasibility of using fixed and free response testing to provide a more accurate estimate of those skills. Initial results suggested significant deficiencies. The unit revised its acceptance policies and provided a wider range of formal remedial opportunities for those prospective candidates whose reading comprehension, knowledge of writing "mechanics," use of mathematics, or ability to write a persuasive essay fell below the unit's basic standards. Developmental services responded to increased demand with new forms of assistance. Testing procedures were substantially refined and validated to improve the accuracy of skills assessments. On average, about 10 percent of the unit's prospective candidates were advised to seek developmental work before applying for acceptance in their second year of study.
Minnesota's new licensure examinations, adopted in the fall of 2010, include tests of academic skills in reading, writing, and mathematics that the test developer has set at or near the expected skills and knowledge of a second year college student. The former skills test developed by the Educational Testing Service, the Pre-Professional Skills Test (PPST), while never "normed," was described as at or below the performance expected of high school graduates. In the past two years since the MTLE Basic Skills test was specified by the Board of Teaching for all prospective teachers, about 20 percent of our prospective teachers who attempt the test in their first year of study, as requested, report scores at or below the passing score on reading, writing, or mathematics.
Using prospective candidates' scores on the units EDUC 111 embedded writing assessment as well as the MTLE and ACT subtests for writing, we found strong positive correlations between the MTLE for the small sample now available. We also found low correlations between those two fixed response tests, each of which includes a small writing sample on a topic of little interest to test takers, and our screening essays writing for course assignments on topics related to teaching and learning. As the candidate pool increases we may be able to set a threshold score for our screening exam and for the ACT that could predict a reasonable chance of passing the MTLE. In the past we found an ACT score of 24 or higher on the writing subtest predicted a passing score on the PPST for three out of four prospective candidates.
As our sample of test takers increases, additional exploration of the relationship between MTLE academic skills scores and ACT scores with success on MTLE licensure examinations in content or pedagogy. Relationships among these measures may help the unit and its prospective candidates decisions about needed developmental work, as recent studies from ETS suggest that candidates' low PPST scores as predictive of low scores on licensure content tests.
- One of our continuing concerns is the extent to which candidates who complete our licensure programs possess a fund of subject matter knowledge, skills, and values that will sustain effective instruction in the first years of their practice. The opinions of recent graduates, collected using our on-line survey, generally confirm our expectation that they have the knowledge they need to begin their practice, then learn to acquire what they may need as their practice matures. Previous licensure test results and subsequent analyses of test performance, provided by the Educational Testing Service, previously confirmed the views of our candidates. Our new MTLE from Pearson do not offer national comparisons of our unit or state candidates.
Preliminary licensure test results suggest that while our overall three year running average exceeds the expected 80% passing rate for program completers as confirmed by our Title II reports, two of our licensure programs counter that trend. Our K-6 candidates who seek to add a middle level mathematics license to their core elementary licenses have a low passing rate which approximates the state passing rate of less than 20% of examinees. Candidates who have attempted this test report a wide range of possible flaws in its design, from too little time to read test instructions, to an on-screen calculator that does not record keystrokes as do modern mathematics calculators. The Board of Teaching's staff members have shared these concerns with Pearson, and received in return promised improvements. We have examined our middle level mathematics curriculum in light of these test results, but without more detailed test data, our search for flaws in the courses and topics that form this licensure has not been successful.
A similar problem has emerged for with those seeking the middle level endorsement in Spanish so that they might teach that language to students in kindergarten through grade eight. Our unit passing rate for this test is below the 80% threshold, approaching the 30% state-wide pass rate experienced by middle level Spanish candidates from all institutions. Our first at why candidates might not pass suggests that their preparation in the content area may emphasize the culture and literature of Spain and other Spanish-speaking countries over competent speaking, listening, and writing in that language. Although all candidates must pass a screening examination conducted by the Spanish department's faculty, the proficiency level set by state standards may be too lower than required to pass the licensure test. Flaws in the design of this test are also evident from the shared experience of those who have completed it.
- In years past, nearly all candidates (98-100%) who completed Minnesota's licensure examinations in academic skills (PPST), pedagogy, and subject matter knowledge (Praxis II) met or exceeded state licensure standards (Title II). Closer examination of the content examination for social studies found possible weaknesses in secondary candidates' knowledge of world history. Elementary candidates pursuing a grade 5-8 specialty in social studies revealed weakness in their knowledge of world and U.S. history. The unit invited arts and sciences faculty and experienced secondary social studies teachers to review these findings in light of the licensure standards for both elementary and secondary candidates. Their preliminary recommendations will form the basis for more intensive review. We have asked pedagogy and content faculty to complete MTLE tests in their areas in the absence of meaningful test information and study guides from the vendor.
- Analysis of our student teachers performance profiles and our analysis of their teacher work samples finds that nearly all candidates perform at the "proficient" level on key unit and state standards. While there are outstanding performers rated as "distinguished," there are also very few are rated at the "basic" level on behaviorally anchored rubrics for those standards. Given the "mastery model" we advance in our student teaching experiences, the small number of candidates performing above or below the criterion level (proficient) hampers our search for performance patterns that could help us better prepare weaker performers or derive guidance from those who excel.
Have the unit's efforts to prepare candidates for practice in multicultural settings been successful? While comments from graduates teaching in highly diverse settings suggest the helpfulness of the unit's efforts to provide all candidates with opportunities to teach students from a different economic, cultural, ethnic, and racial backgrounds, much more remains to be learned about the demands our graduates face and the ways in which our licensure programs can help them meet those demands. Several semesters of formative review of our prospective elementary candidates' experiences in their "urban plunge" uncovered facets of that experience that were used to inform the development of a similar urban experience for secondary candidates. Our diversity transcript, while not yet able to document all students ' experiences in diverse educational settings, suggests that more candidates enjoy a greater range of experiences than in past years. Student teachers' experiences in working with diverse learners, as revealed in their portfolios and work samples, have been positive. Immigration will continue to bring new residents from other countries to Minnesota who will settle in urban areas, including Saint Cloud. Several Saint Cloud elementary schools in which our candidates complete field and clinical experiences are now more diverse that Twin Cities Metropolitan area schools. While we are only beginning to unlock the potential for further work in this area, initial results are promising. National and state conversations on the nature, causes, effects, and reduction of the "achievement gap" among Minnesota's students of color continue to look to teachers to fill that gap. While skilled, committed teachers can help close the gap, closing significant "opportunity" gaps related to employment and income will likely have a stronger effect.