University of Alberta Peer Review of Teaching

prepared by Dr. Susan Gibson, Associate Director CTL

Preamble
Increased emphasis is being placed on peer review of teaching in post secondary institutions (Blackmore, 2005; Byrne, Brown, & Challen, 2010; Chism, 2007; Hamilton, & Gaff, 2009); however, as Hira and Cohen (2011) note, "Teaching excellence is becoming a difficult summit to climb. The most precarious part of that climb is a missing support - the lack of a reliable system for accessing and rewarding teaching" (para. 2).

At the heart of the issue is the debate over the purpose of peer review as quality enhancement versus quality assurance (Byrne, Brown, & Challen, 2010; Kilfoil, 2014). The latter is often seen as 'managerial surveillance imposed by leaders" (Napier et al., 2014) that does little to improve or enhance teaching (the focus of the former) (Sachs, & Parsell, 2014).

Another issue is the reluctance on the part of academics to embrace peer review of their teaching even though there is wide acceptance of peer review of research (Hira, & Cohen, 2011; Iqbal, 2013). Assessment by peers can be daunting, but heavy reliance on quantitative student feedback for determining the effectiveness of teaching is also problematic (Bernstein, Burnett, Goodburn, & Savoy, 2006; Hira, & Cohen, 2011). While student feedback is important as one measure of instructor effectiveness, that of the day-to-day interaction in the classroom, Bernstein et al. (2006) argue that "peer review of teaching should refer to a much more complete examination of the intellectual work of teaching" (p. 6). These other aspects include: course content, academic rigor and appropriateness of objectives and topics (eg. preparation for advanced course work); subject matter expertise; instructional materials and methods; and, assessment and grading.

At the University of Alberta, the recent Renaissance Committee Final Report emphasizes the need for "a comprehensive, clear and transparent [evaluation] system …[which] has multiple sources of information collected through a variety of methods and assessed at multiple points in time" (p. 69). The U. of A. General Faculties Council policy states that, "Evaluation of teaching shall be multifaceted. Multifaceted evaluation shall include the Universal Student Ratings of Instruction set out in Section 111.3 and other methods of assessing teaching designed within the individual Faculties to respond to the particular conditions of that Faculty. Such assessments shall include one or more of the following: input from administrators, peers, self, undergraduate and graduate students, and alumni" (section 111.2). Peer review of teaching is one way in which to address multifaceted evaluation of teaching.

What is peer review of teaching?
Peer review of teaching is informed collegial assessment of faculty teaching for either fostering improvement or making personnel decisions. There are two main types of peer review: formative and summative. Both formative and summative are recognized as integral to a comprehensive evaluation of teaching.

Formative review
Formative review of teaching has as its purpose the development and improvement of teaching practice. At the University of Alberta, this form of peer review can range from engagement in a number of activities offered by the Centre for Teaching and Learning including: Teaching Squares in which the focus is on gaining new insight into one's own teaching by observing others in their classrooms; Mentoring Circles and Teaching Co-mentorships where colleagues collaborate informally as either a group or one-to-one in mutually beneficial conversation about teaching and learning; and, Peer Consultation in which a trained peer consultant works with an instructor at his/her request to gather information from both classroom observation and student feedback regarding specific instructor questions or concerns.

Summative review
"When teaching is viewed as professional knowledge there must be an accepted way to define characteristics of teaching excellence and to make judgments based on a stated set of criteria and standards that reflect the complexity of teaching" (Chism, 2007, p. 13). In that regard, summative review is informed collegial judgment about teaching intended for evaluative purposes. It is comparative in nature thereby enabling the evaluator to determine the quality of teaching performance with respect to performances of peers and to predetermined standards. Summative review of teaching is also multidimensional in that it includes multiple kinds of valid and reliable evidence (ratings, reflective statements, narrative appraisals, teaching artifacts such as syllabi, etc.) from multiple sources (administrators, students, peers and self).

"Performance appraisal is a complex and controversial human resource technique" (Blackmore, 2005, 219). Accordingly, summative review is a formal, systematic process that requires consistent standards and structures that are seen as fair and equitable across faculty. Having a clearly articulated plan for summative peer review of teaching is essential to the success of any program. Faculty need to be well informed about what will be expected of them throughout the process and to feel that the process is consistent, fair and equitable.

The following guidelines provide a step-by-step framework for initiating a summative peer review of teaching. This process will need to be adapted to address the unique teaching and learning contexts of the faculty, department and/or unit wishing to implement such a review process.

Six phases have been identified as being integral to the planning of any summative review:

establishing the purpose for peer review
determining what will be reviewed
choosing who will be reviewed
deciding on procedures for conducting a review
implementing the review process
revising, renewing and sustaining the review process

Phase 1: Establishing the purpose for summative peer review of teaching
The first step in designing a review protocol is to decide what you wish to achieve by implementing a summative peer review of teaching process. This phase begins with the clear articulation of goals and desired outcomes for the program (See Appendices A & B). Here too decisions will need to be made about what information should be collected to address the desired outcomes, how that information is to be used, and by whom. In order to determine readiness for such an undertaking and to encourage faculty involvement, a survey aimed at identifying their knowledge about summative peer review of teaching and any questions and concerns about it they may have is recommended. This feedback can then be issued to guide the development of the faculty/department/unit protocol.

Phase 2: Determining what will be reviewed
Once the goals and desired outcomes for the summative review of teaching program have been determined the next step is to decide what aspects of teaching are to be evaluated as part of the review. Initially the unit/department/faculty will need to identify what is considered to be effective teaching and valuable student learning (See Appendices C & D). These may vary for specific teaching contexts and academic disciplines. The elements of effective teaching and learning identified can then be defined in terms of standards or benchmarks to used in the actual evaluation. (In some cases these standards may already exist, i.e. those needing to be addressed for accreditation purposes). Here is where it is important that teaching is recognized as more than classroom performance. Other aspects of teaching to be considered as part of the review include course design, course materials used, contribution of the instructor to teaching in the faculty/department, and impact of the course/the instructor on student learning.

Phase 3: Choosing who will be reviewed
Thirdly, decisions are made about who will be reviewed (probationary and/or tenured faculty, sessionals, clinicians, etc.) and whether participation in the review process will be considered mandatory or optional.

Phase 4: Deciding on procedures for conducting the review
Phase 4 addresses the logistics of the review process. Once the decision has been made about who is to participate, the next step is to determine when and how often over a career a faculty member should engage in summative peer review of teaching. A second important aspect of this phase is to decide what success criteria would provide evidence of the standards identified in Phase 2 (See Appendix E & F), and what performance expectations are to be applied to these criteria (i.e. meets expectations, exceeds expectations, below expectations, etc.). Following that, what evidence for checking the criteria against will need to be collected (i.e. observation(s), syllabi, course materials, assignments, the instructor's philosophy of teaching statement, materials that communicate course policy and practices, assignments and assessment of student performance, samples of instructor feedback on student work, reflective statements, teaching dossiers etc.) and what kinds of discipline specific tools will be used to collect this evidence (templates, checklists, guiding questions, rating scales, etc). (See Appendices G, H, I, J, K, L, M, N, O for sample tools)

Another aspect of this phase requires thinking about who the reviewers will be, who selects them and on what basis, how many will be needed, and how they will be trained. It is recommended that more than one reviewer is used for each case in order to allow for comparison of notes and the writing of one consolidated letter of recommendation. One of the two reviewers could be drawn from outside the faculty in which the review is taking place to alleviate concerns around power and authority. This external person would focus mainly on pedagogy, with the internal reviewer bringing content expertise. Training reviewers will be a very important element in the success of the program; this is where CTL can be of assistance. Being a reviewer can take a significant amount of time so it will also be important to think about how their time will be recognized (acknowledged at FEC).

Lastly, decisions need to be made about who will be responsible for interpreting the evidence (solely the reviewers, the faculty member, chair or director, combination of these people), what format the reporting will take and to whom the report will be provided. Confidentiality is of primary concern here (See Appendices P and Q for Sample Report forms).

Phase 5: Implementing the review process
Once the first four phases are completed, the next critical step is to communicate the plan to the faculty. Faculty members will need to be clear on: the purpose of the review, the standards for teaching and learning to be applied, the areas to be assessed, the aspects of teaching to be reviewed, the review protocol, and the outcomes of review. To increase ownership in the review program, you may also wish to provide opportunities for some faculty control over aspects of the review such as allowing the person being reviewed to decide what class will be observed and what aspects of their teaching they would like the review to focus on, encouraging pre-review discussion for the instructor under review to have an opportunity to describe unique and specific teaching and disciplinary contexts, and post-discussion for them to address and expand on things noticed during the review (Donnelly, 2007; Harris, Farrell, Bell, Devlin, & James, 2008).

Phase 6: Revising, renewing and sustaining the review process
Once established, regular review of the protocol is highly recommended as a final phase of any summative peer review of teaching plan. It will be important to decide what type of program review will provide the desired information about its successes and glitches, and how specifically the plan will be monitored and assessed. Encouraging feedback from the reviewers, those whose work is being reviewed and from faculty members at large will be important to improving and sustaining the process. Ultimately the greatest outcome from engaging in the process (as with formative review) would be to use the data collected to improve the quality of teaching in the faculty/department/unit. The accompanying chart provides an overview of this protocol along with sample questions to guide each phase. The attached appendices include examples to accompany each phase as well.

References:
Bernstein, D., Burnett, A., Goodburn, A., & Savory, P. (2006). Making teaching and learning visible: Course portfolios and the peer review of teaching. Bolton, MA: Anker Pub. Co. Inc.

Blackmore, J. A. (2005). A critical evaluation of peer review via teaching observation within higher education. The International Journal of Educational Management, 19(2), 218-232.

Brent, R. & Felder, R. (2004). A protocol for peer review of teaching. Retrieved from http://www4.ncsu.edu/unity/lockers/users/f/felder/public/Papers/ASEE04(Peer-Review).pdf

Byrne, J., Brown, H., & Challen, D. (2010). Peer development as an alternative to peer observation: A tool to enhance professional development. International Journal of Academic Development, 15(3), 215-228.

Chism, N. V. (2007). Peer review of teaching: A sourcebook (2nd Ed.). Bolton MA: Anker Pub. Co. Inc.

Crisp, G. et al. (2009). Peer review of teaching for promotion purposes: Final report and appendices. Australian Learning & Teaching Council. Retrieved from http://www.adelaide.edu.au/teaching-projects/peerreview/

Donnelly, R. (2007). Perceived impact of peer observation of teaching in higher education. Retrieved from http://arrow.dit.ie/cgi/viewcontent.cgi?article=1023&context=ltcart

Fernandez, C., & Yu, J. (2007). Peer review of teaching. The Journal of Chiropractic Education, 21(2), 154-161.

Hamilton, N., & Gaff, J. (2009). Proactively justifying the academic profession's social contract. University of St. Thomas School of Law Legal Studies Research paper No. 09-17. Retrieved from papers.ssrn.com/sol3/papers.cfm?abstract_id=1444587

Harris, K-L, Farrell, K., Bell, M., Devlin, M., & James, R. (2008). Peer review of teaching in Australian Higher Education. Australian Learning & Teaching Council. Retrieved from http://www.cshe.unimelb.edu.au/research/teaching/docs/PeerReviewHandbook_eVersion.pdf

Hira, A., & Cohen, D. (2011). Peer review could improve teaching assessment. University Affairs. Retrieved from http://www.universityaffairs.ca/opinion/in-my-opinion/peer-review-could-improve-teaching-assessment/

Hubball, H., & Clarke, A. (2011). Scholarly approaches to peer-review of teaching: Emergent frameworks and outcomes in a research-intensive university. Transformative Dialogues: Teaching & Learning Journal, 4(3). Retrieved from http://www.viu.ca/integratedplanning/documents/TD437_HubbalClarke_Peer_Review.pdf

Iqbal, I. (2013). Academics' resistance to summative peer review of teaching: Questionable rewards and the importance of student evaluations. Teaching in higher education, 18(5), 557-569.

Kilfoil, W. (2014). Peer review as quality assurance. In J. Sachs, & M. Parsell, (Eds.), Peer review of learning and teaching in higher education: International perspectives (pp. 105 - 123), Dordrecht, Netherlands: Springer

Napier, J., Riazi, M., & Jacenyik-Trawoger, C. (2014). Leadership: A cultural perspective on review as quality assurance versus quality enhancement. In J. Sachs, & M. Parsell, (Eds.), Peer review of learning and teaching in higher education: International perspectives (pp. 53-66), Dordrecht, Netherlands: Springer

Sachs, J. & Parsell, M. (Eds.) (2014). Peer review of learning and teaching in higher education: International perspectives. Dordrecht, Netherlands: Springer

Shulman, L. (2004). Teaching as community property. San Francisco, CA: Jossey-Bass.

University of Alberta. GFC Policy Manual. Retrieved from http://www.gfcpolicymanual.ualberta.ca/en/111TeachingandLearningandTeach.aspx.

University of Alberta. Report of the Renaissance Committee (Nov. 22, 2013). http://www.renaissance.ualberta.ca/~/media/renaissance/RenaissanceCommitteeFinalReport.pdf