Assessment - FAQ

From March 2023 clerkship students in their 3rd year will be assessed on Entrustable Professional Activities (EPAs) on Elentra.

View all student and assessor EPA resources 

https://uoft.me/Assessment-Security
Written Assessment Security Guidelines for Students

These Guidelines are to be followed in conjunction with the MD Program Assessment Rules and Regulations (https://md.calendar.utoronto.ca/academic-regulations)

Students are required to report absences for all activities that are scheduled assessments by completing and submitting a U of T Verification of Illness (VOI) form found on the school absences page. 

Examplify creates multiple encrypted copies of the assessment every 60 seconds on the assessment taker’s hard drive. Assessment takers should perform a hard reboot of their device if they run into any issues with Examplify, they can then re-enter the assessment where they encountered the issue.

To perform a hard reboot of your device and re-enter the assessment, do the following:

-       Turn off your device (press and hold the power button), then turn your device back on.

At invigilated assessments students should inform their invigilator before performing a hard reboot. If Examplify prompts for a resume code, an invigilator will input one for you.

If an assessment taker requires technical assistance when they are writing their weekly feedback quiz:

During regular office hours: Monday to Friday 9 a.m. to 5 p.m.

Contact MD Program Examplify Support

Outside of regular business hours: 

Contact ExamSoft Support 24/7 via e-mail OR call 866.429.8889 (US Toll-Free), 1.954.429.8889 (International)

There will be a confirmation screen on the students’ device congratulating them on successfully uploading their assessment. Ensure you see this green screen before exiting if completing a weekly feedback quiz or for invigilated assessments, show your invigilator your green screen; otherwise your assessment may not be uploaded correctly. See below:

Foundations

Students who are unsuccessful in completing any course requirement in Foundations will be reviewed by the Student Progress Committee, which then makes recommendations to the Board of Examiners regarding the student’s status in the course. For further details, please refer to the following policies on student progress for Foundations Curriculum students:

Standards for grading and promotion of MD students - Foundations (Years 1 and 2)
Guidelines for assessment of undergraduate medical trainees in academic difficulty - Foundations Curriculum

Clerkship

Students in Clerkship should review the following policies on the standards of grading, promotion and academic difficulty and speak to their course director or the clerkship director if they have any questions or concerns:

Standards for grading and promotion of MD students – Clerkship (Years 3 and 4)

Guidelines for the assessment of MD students in academic difficulty – Clerkship (Years 3 and 4)

All students should refer to the following policy on the assessment of student professionalism:

Guidelines for the Assessment of Student Professionalism

Professionalism does not appear on a student’s transcript. If a student is required to complete remediation in professionalism by the Board of Examiners they would need to self-report this when asked specifically about professionalism. If the student’s remediation in professionalism is deemed as unsuccessful, it may be indicated on the Medical Student Performance Record (MSPR) as not meeting professionalism expectations. However, this decision would be made on a case by case basis.    

Students will not be provided with correct answers; however students will receive feedback about the topics (learning objectives) which they did not answer correctly. This decision was made based on the MD program Test Committee’s research on the best practices in student assessment in medical education and its recommendation against providing students with actual questions and answers.

The rationale behind this decision is to support the role of assessment as a tool for self-appraisal and not as an end unto itself.  The results of the assessment are meant to help students identify whether they have mastered the material they covered.  If they haven't, the expectation is that they return to the source material until they have a better grasp of the content.  The MD Program has created a bank of very high quality questions that are directed at assessing not just knowledge but comprehension and application.  This requires the use of a secure and well-mapped exam bank which precludes disseminating the questions and answers.  Students are reminded that copying and disseminating questions is against university regulations.

No. Individual assessment component marks will never appear on transcripts or other documentation provided by the MD Program to authorized external individuals or organizations.

No. Final transcripts will only display Credit/No Credit (CR/NC).      

Currently, Borderline Regression (BLR) is used as the method to set a pass mark in Year 1 and Year 2 Clinical Skills OSCE as well as in the Clerkship OSCE.

BLR calculates individual station pass marks based on the results of a regression analysis, using the overall performance mark as the independent variable and the station score as the dependent variable (the combined mark of station checklist marks and global rating marks for Year 1 and Year 2 and the combined mark of competency scores and Key Feature Question (KFQ) marks for Clerkship). The regression line is calculated based on all of the interactions between examiners and students for a particular station. The station pass mark is then the point at which the regression line crosses the borderline category. The overall pass mark of OSCEs using the BLR method is calculated as the average of all of the station pass marks plus two* Standard Error of Measurement (SEM).

[*This number might change depending on the internal consistency of the OSCE stations].

Overall Assessment: 1=Unsatisfactory, 2=Borderline, 3=Meets Expectation, 4=Exceeds expectation and 5=Outstanding

Mastery exercises are mandatory invigilated written assessments for first, second and third year students. Students complete these assessments using Examplify on their own devices. These assessments may include multiple-choice questions, short-answer questions, essay questions or key feature questions. 

No, mastery exercises are weighted according to the amount of material (number of weeks of material) they cover in the first and second years.

Faculty will provide a debrief or ‘teaching point’ for questions answered poorly by the class. Neither the question nor answers will be released. This is to maintain the integrity of the exam bank, and to ensure that students focus on the learning objective addressed by the question or the learning gap identified as opposed to the question itself.

Students who performed poorly on a Mastery Exercise should reach out to the OHPSA academic coach or course director to discuss learning and studying strategies. Rules and regulations are in place to maintain the integrity of the MD Program’s question banks. Hence, viewing of assessments is limited to written assessments that include short answer questions (SAQs) and/or other narrative components, in order to request a re-mark of a written assessment.

Multiple choice questions (MCQs) on mastery exercises are written to assess one or more end-of-week learning objective(s). MCQs are developed by content experts who are usually directly involved in teaching. The question stems in the MCQs will include all of the information required to answer the question. There is one correct answer and three incorrect alternatives known as distractors.

Mastery exercise questions are designed to assess not just whether students can remember facts and understand concepts, but also if they can apply the content being taught to a real-life like clinical situation.

Any learning objective can serve as the basis of an assessment question on a mastery exercise, including those objectives that are covered by pre-week and self-learning.

Following each mastery exercise, and prior to approving the students’ results for publication, the course director reviews the performance statistics of each question and the mastery exercise as a whole in accordance with evidence-informed processes endorsed by the Test Committee.

Once the mastery exercise results are approved by the course director students will receive a report summarizing their performance that can be used to guide their further studying.

Based on the input and review of the the Student Assessment and Standards Committee (SASC), the Foundations Committee, and the Curriculum Committee, we currently use the Hofstee method for standard setting for Mastery Exercises (ME). This approach asked members from SASC and the Foundations Committee to examine ME data from the previous academic year. Using their understanding of the objectives and goals of each course as well as the historical ME data, each committee member was asked to estimate what the acceptable values would be for the minimum and maximum pass mark in each course as well as the minimum and maximum number of students who could possibly be below the pass mark. This does not mean that there will always be the minimum number of students below the pass mark every year. Instead, these judgements are just an approach to help judges communicate their sense of the expected difficulty of each course.  Combing all of this information from members of both committees, we plot the average values on to previous ME data for a particular course and identify which score satisfies the best estimates of the committee. This point becomes the new pass mark for the course. The new pass marks are deployed and re-evaluated again at end of the academic year. 

The pass mark can be found on the Elentra course page

Reports will be released to students approximately 1-2 weeks after the completion of mastery exercises.

Following the completion of each mastery exercise, the course director conducts an analysis of the assessment to ensure that all questions meet minimum standards for quality. This includes a review of item difficulty, reliability and clarity. Adjustments will be made to optimize the overall reliability of the assessment prior to the release of reports to students.  The 1-2 week timeframe also allows for the small proportion of students who need to write a make up assessment for various reasons (typically scheduled on the next self-learning day), so that the results can be released to the entire class at the same time.

The report you receive provides a breakdown by type of question and the specific learning objective assessed in the mastery exercise. You can use the report to determine how you did on certain types of questions (e.g. remembering, understanding, applying or evaluating) and which learning objectives you should revisit to ensure that you understand the material well enough. Each code corresponds to a learning objective which you can find in Elentra.

In Foundations, if a student falls below standard on an ME, they will receive a message from the Student Progress team providing information about support services available through the Office of Learner Affairs, the option to book a meeting to discuss course content with the relevant Course Director and/or book an appointment to meet with the Assistant Registrar, Registration and Student Success.

At the end of a course, if a student does not satisfactorily complete all mandatory components or has a weighted ME average that does not meet the course pass mark, they will be notified by the relevant Course or Curriculum Director about being presented to the Student Progress Committee (SPC).

In Clerkship, if a student falls below the standard of 60% on an ME or below expectations on a clinical assessment, they will receive notification from the relevant Course Director or Administrator. They will be offered to meet with the Faculty Lead, Academic Success or Clerkship Director and/or the Assistant Registrar, Registration and Student Success.

Failure of a mandatory component in Clerkship leads to a student being presented to the Board of Examiners (BOE): BOE_FAQ

The progress test is a comprehensive knowledge-based test designed to show students' growth and knowledge acquisition over the 4 years of medical school. Each test is comprised of 100 multiple-choice questions written by trained faculty and targeted at the level of the graduating medical student.

Students are required to take the scheduled progress tests each year, according to their year of study. Year 1, 2, and 3 students will have 3 progress tests each academic year; year 4 students will have 1 progress test during the year. Marks from Progress Tests are not used in progress decisions. The test is a stable measure that shows students where they are and where they need to be by the end of the program.

The blueprint of the progress test aligns with the Medical Council of Canada Qualifying Examination (MCCQE) Part I and the series of progress tests throughout the MD program have been shown to correlate with performance on the MCCQE Part 1.

The Director of Foundations, Clerkship Director, Director of Student Assessment, Course Directors, Student Progress Coordinator and Student Progress Committee members will have access to the Learner Chart of all students. Academy Scholars will have access to the Learner Chart profiles of students who are assigned to them. No external groups have access to a student’s Learner Chart. The Leaner Chart is purely for personal (student) and internal (MD Program) use.

Students will be assessed throughout the MD Program by a faculty member using the following form:

MP Program Professionalism Form.pdf

Written assessments will be available for download in the student's Examplify account.  

For more information about Examplify, refer to the MD Program assessment and evaluation technology page. 

Standard setting is the process by which pass marks are determined for assessments. This area of education has a great deal of research and evidence to support. There is no single best way to set pass marks for tests just as there is no objective or mathematical formula to decide on the pass mark. At the same time, pass marks are not determined arbitrarily. They are set considering the goals of the curriculum, the level of mastery expected of students, expert opinion, and statistical analysis combined through rigorous methodology. (The MD Program’s curriculum goals can be found here: https://md.utoronto.ca/mdprogramcompetencies). 

Standard setting is reviewed by three important committees in which students have a voice and representation: the Student Assessment and Standards Committee (SASC), the Foundations Committee, and the Curriculum Committee. 

The standards we use for the curriculum are not intended to increase the number of students who do not meet the standard; there is no norm referenced standard setting in the MD program (i.e., we do not aim to flag a certain portion of the class as below the standard after each Mastery Exercise (ME) or course). Instead, our pass marks are adjusted on a yearly basis based on the difficulty of the questions and ensuring that the assessments used in the program enable a determination if a student has met the required course objectives and competencies for MD training. It is possible all of the students will be above the pass mark for an ME despite the estimates of the judges. Additionally, quality control metrics are used to evaluate each ME and all assessments on a routine basis including item difficulties, appropriateness of questions for objectives, and overall validity. 

The Hofstee method for standard setting is used for Foundations Curriculum mastery exercises. The Hofstee method is well described in the literature on standard setting. You may also find out more from your student reps on SASC and Foundations. Mckinley DW, Norcini JJ. How to set standards on performance based examinations: AMEE Guide No 85. Med Teach. 2014;36:97–110. (https://www.ncbi.nlm.nih.gov/pubmed/24256050) 

You can find the information on the Assessment tab of the course’s Elentra page. 

Please refer to this document for information on the Board of Examiners (BOE), and answers to commonly asked student questions: BOE_FAQ
 

High standards are set for the quality of our mastery exercises by the MD Program and much effort and expertise is dedicated by faculty to write questions that adhere to best evidence for assessment. After each mastery exercise the course director, supported by the Office of Assessment and Evaluation, reviews the mastery exercise statistics including the overall performance of the assessment and a step-by-step review of each item’s psychometric properties. Items that do not comply with the high standards set, are considered for removal and either removed completely or edited for improvement and use in future mastery exercises, but not counted towards the mark for the current mastery exercise.

Biometric data is a term that refers to “biometric identifier” and “biometric information”. “Biometric identifier” means a retina or iris scan, fingerprint, voiceprint, or scan of hand or face geometry. “Biometric information” means any information, regardless of how it is captured, converted, stored, or shared, based on an individual’s biometric identifier used to identify an individual.

In relation to the ExamSoft document titled "Notice of Collection of Biometric Data and Consent", student biometric data will be used by ExamSoft for the purposes of proctoring applicable assessments.

The MD Program will instruct ExamSoft to destroy student biometric data when it is no longer required by the MD Program to confirm a student's identity for assessments, i.e., at the end of a student’s completion of the MD Program.

Information on the ExamSoft Privacy Policy and User Service Terms

From March 2023 clerkship students in their 3rd year will be assessed on Entrustable Professional Activities (EPAs) in Elentra.

Student and assessor guides on how to start an assessment in Elentra