Combining educational psychology and occupational therapy leads to a career in improving research and practice in health-professions education and educational measurement

Mary Roduta Roberts

Mary Roduta Roberts is an associate professor in the Department of Occupational Therapy and an associate dean of graduate studies in the Faculty of Rehabilitation Medicine. She is also a registered occupational therapist (OT) in Alberta with clinical experience in geriatric psychiatry and community-based case management. 

With a PhD in measurement, evaluation and cognition, Roduta Roberts leads an active research program that focuses on assessment and evaluation methods within health-professions education. A collaborative researcher, Roduta Roberts works with instructors, curriculum groups and other educational and health researchers locally, nationally and internationally. She also advises students on projects involving quantitative methods, research design and applied measurement.

Roduta Roberts won the 2023 American Educational Research Association’s Established Investigator Award, Education in the Professions, for her work assessing the use of Casper (computer-based assessment for sampling personal characteristics) in occupational therapy program-admissions decisions.

We caught up with Roduta Roberts to learn more about her and her work.

Can you describe your research program?

Because of my background as an OT, with a PhD in measurement, I run two lines of research. One is educational measurement and the other is studying assessment within health-professions education, specifically within occupational therapy.

Oftentimes people might think of assessment as just the actual measure, but actually, my work spans educational assessment as both a process and an instrument.

How did you go from being an OT to assessing the learning habits of others?

I've always liked education. When I was a student clinician, I was curious and worried about being assessed by my preceptors. I would think, “How am I being assessed? If I were a preceptor, how would I do this?” So when I became a clinician, I was interested in injecting education into my career. 

My work in OT was largely focused on assessment — the process of gathering information using tools, observations, and interviews — to understand my clients. I was especially interested in cognitive assessments. I ended up wanting to go to grad school and learn more about this. 

That was in 2004, and I talked to different people to decide on a path forward. I talked to people in the Faculty of Rehabilitation Medicine, including Helen Madill and Sharon Warren, but it was my conversation with Lili Liu that changed things. I don’t think I would be on this path if not for this one question that she asked me, which was, “Why would you do a master’s in OT? Why don’t you do it in a different area and bring it back?” She was talking about the cross-pollination of ideas and why that's valuable. Because of that conversation, I decided to pursue a measurement degree here at the U of A in the Department of Educational Psychology. 

Can you say more about your work with Casper and the admissions process?

Casper is a situational judgment test where candidates respond to issues in hypothetical scenarios. The assumption is that their responses can uncover non-cognitive characteristics and behavioural tendencies that influence their judgment in those situations. Casper is becoming well known because health-professions programs are using it more frequently to inform admissions decisions.

Traditionally, the sources of information used to assess non-cognitive attributes are statements of intent, letters of reference and interviews. But these measures have been criticized due to the potential for bias in their interpretation and use. Casper may be a more objective and accessible way to assess non-cognitive attributes because it's administered by a third party, online, and uses an open-response format, which means it could be a more valid measure in reflecting what a person would do. 

In my paper, my colleagues and I used traditional and novel analytic methods to show how Casper scores at admission were related to student performance in our program OSCEs (objective structured clinical examinations) and fieldwork. We also showed the potential for Casper to inform selection decisions with the goal of increasing a cohort’s diversity.

Where do you want your career to be in 10 years?

I hope my work leads to a rethinking of the ‘what’ and ‘how’ in educational measurement and assessment practices, to support better decision-making about students and educational systems with positive impacts. When we're thinking about assessment and evaluation, from admissions to assessing competencies, how can we think about practices that are transparent, fair and equitable? I care a lot about working towards measurement and assessment practices and tools that provide students with information to support their growth, which can be motivating and empowering for them.

What one piece of wisdom would you offer your grad students?

I think one thing that we don't emphasize enough is the importance of maintaining or cultivating your other interests that are not academic, whether that's hobbies or engaging with your community, whatever that looks like. To see any sort of impact or effect from your research could take years, so you need something to help balance that, something that energizes you, helps you to keep going, and brings you joy.