Assessment - FAQ

You can find further details on the MD Program assessment and evaluation technology page.

Written Assessment Security Guidelines For Students.pdf
Written Assessment Security Guidelines for Students

These Guidelines are to be followed in conjunction with the MD Program Assessment Rules and Regulations (https://md.calendar.utoronto.ca/academic-regulations)

1. For computer-based assessments, students are responsible for bringing their own device (laptop or iPad). Students must ensure that their personal device meets the minimum system requirements for Examplify and ExamID (https://examsoft.com/resources/examplify-minimum-system-requirements/).

2. Students should only be directed to a back-up room and/or be provided with a back-up device if they have a technical issue that cannot be addressed in the main assessment room. In the case of technical issues, students should be allowed the full scheduled time to complete the assessment. Back-up rooms and/or devices are not a requirement for Clerkships.

3. The invigilator/examiner has the authority to assign seats to students in the assessment room. Only the students completing the assessment and those supervising the assessment will be allowed into an assessment room during an assessment.

4. Before students proceed to their assessment desk/table, bags and books are to be deposited in areas designated by the invigilator/examiner and are not to be taken to the assessment desk/table. All other electronic devices are to be turned off and must remain in the designated area. This includes cell phones, smart watches or other electronic devices. Turning off electronic devices is important so that they do not disrupt the class during the assessment.

5. Students must place their student photo ID card at the front of their desk for the invigilator/examiner to record details if suspicious behaviour is identified. Attendance and photo identification of each student will also be completed on Examplify via ExamID on each student’s personal device before the start of the assessment. If masks are required, the student will remove their mask for the time that the photo is being taken and put the mask back on after the photo is taken. If the student receives a message that requires a continuation code to proceed, the invigilator/examiner will ask the student to e-mail md.examsoft@utoronto.ca (for Foundations assessments) or the course administrator (for Clerkship assessments) to receive a code to proceed. In this situation the invigilator/examiner must check the students photo ID card and make a note of attendance for the MD Program.

6. Students who arrive late to an assessment will be permitted to enter the room. Students who arrive ten (10) or more minutes late for the assessment will be directed to a back-up room (Foundations and Progress Tests only, while Clerkship students will remain in the same room). Students who arrive late will have their names and the time they arrived recorded. Students who arrive late will not be allowed any additional time.

7. Students will agree to the following Statement of Acknowledgement for every written assessment.
“I will not share the content of this assessment with anyone other than the Course Director (if the need arises). I will not solicit or provide any information about items, however vague, through verbal or electronic communication including, but not limited to email, text, social media platforms, and chatrooms. I appreciate that doing so will place me in breach of the University of Toronto’s Code of Behaviour on Academic Matters. In such an instance, I acknowledge that I would be liable for academic penalties, under the Code.”

8. Examplify is the only application permitted to be open on the students device, from the start of the assessment until the student has finished and left the assessment room.

9. The use of paper during the assessment is not allowed. No other materials or aids can be used during the assessment unless explicitly authorized by the invigilator/examiner.

10. Irregularities/errors relating to item wording, spelling, punctuation, numbers or notations, and any questions about the content will not be dealt with during the assessment. Ambiguities and questions should be referred to the course director in writing within 24 hours of the assessment.

11. If leaving the assessment room for a washroom break, students are not permitted to remove their personal devices, cellphones, and bags from the room except by authority of the invigilator/examiner. Students will be escorted to the washroom by an invigilator.

12. Students will remain in their seats until an invigilator approaches their desk to check that they have uploaded their assessment by viewing the confirmation of upload screen. Only once this is complete will the student be able to exit the room. All electronic devices cannot be used until the students exists the assessment room.

13. Students may finish their assessment and leave the assessment room no earlier than thirty (30) minutes after the start of the assessment, and under the supervision of an invigilator. Students shall remain seated at their desks during the final ten (10) minutes of each assessment with their laptop closed, even if they have completed the assessment.

14. If a student finishes their assessment and is expected to stay in their seat they must do so with their laptop or iPad closed and not in use. No other electronic devices can be accessed until they exit the assessment room.

15. The invigilator/examiner instructions and decisions must be strictly adhered to. Students who are not in adherence will have their photo ID card recorded and the issue will be escalated to the course director. If a student has concerns in relation to the delivery of the assessment they should reach out to the course director as soon as possible after the completion of the assessment.

Students are required to report absences for all activities that are scheduled assessments by completing and submitting a U of T Verification of Illness (VOI) form found on the school absences page. 

Examplify creates multiple encrypted copies of the assessment every 60 seconds on the assessment taker’s hard drive. Assessment takers should perform a hard reboot of their device if they run into any issues with Examplify, they can then re-enter the assessment where they encountered the issue.

To perform a hard reboot of your device and re-enter the assessment, do the following:

-       Turn off your device (press and hold the power button), then turn your device back on.

At invigilated assessments students should inform their invigilator before performing a hard reboot. If Examplify prompts for a resume code, an invigilator will input one for you.

If an assessment taker requires technical assistance when they are writing their weekly feedback quiz:

During regular office hours: Monday to Friday 9 a.m. to 5 p.m.

Contact MD Program Examplify Support

Outside of regular business hours: 

Contact ExamSoft Support 24/7 via e-mail OR call 866.429.8889 (US Toll-Free), 1.954.429.8889 (International)

There will be a confirmation screen on the students’ device congratulating them on successfully uploading their assessment. Ensure you see this green screen before exiting if completing a weekly feedback quiz or for invigilated assessments, show your invigilator your green screen; otherwise your assessment may not be uploaded correctly. See below:

Foundations

Students who are unsuccessful in completing any course requirement in Foundations will be reviewed by the Student Progress Committee, which then makes recommendations to the Board of Examiners regarding the student’s status in the course. For further details, please refer to the following policies on student progress for Foundations Curriculum students:

Standards for grading and promotion of MD students - Foundations (Years 1 and 2)
Guidelines for assessment of undergraduate medical trainees in academic difficulty - Foundations Curriculum

Clerkship

Students in Clerkship should review the following policies on the standards of grading, promotion and academic difficulty and speak to their course director or the clerkship director if they have any questions or concerns:

Standards for grading and promotion of MD students – Clerkship (Years 3 and 4)

Guidelines for the assessment of MD students in academic difficulty – Clerkship (Years 3 and 4)

All students should refer to the following policy on the assessment of student professionalism:

Guidelines for the Assessment of Student Professionalism

If a Foundations Curriculum student fails any aspect of their assessment they should set up a meeting with their course director to review their performance on the assessment and determine next steps. Please review the Course Requirement document on Elentra.

At the end of each course, the student’s academic assessment data is reviewed by the Student Progress Committee, which makes recommendations to the Board of Examiners, who determine if the student has successfully completed the course “Satisfactory Progress” or has shown “Partial” or “Unsatisfactory Progress”.

To successfully complete each course, and attain a standing of “Satisfactory Progress” the student must: satisfactorily complete all marked assessments within the course; perform satisfactorily on any non-marked learning activities; and achieve satisfactory professionalism competency in every component. The Director of Foundations will contact the students who have not satisfactorily completed the anatomy assessment at the end of the course.

Professionalism does not appear on a student’s transcript. If a student is required to complete remediation in professionalism by the Board of Examiners they would need to self-report this when asked specifically about professionalism. If the student’s remediation in professionalism is deemed as unsuccessful, it may be indicated on the Medical Student Performance Record (MSPR) as not meeting professionalism expectations. However, this decision would be made on a case by case basis.    

Students will not be provided with correct answers; however students will receive feedback about the topics (learning objectives) which they did not answer correctly. This decision was made based on the MD program Test Committee’s research on the best practices in student assessment in medical education and its recommendation against providing students with actual questions and answers.

The rationale behind this decision is to support the role of assessment as a tool for self-appraisal and not as an end unto itself.  The results of the assessment are meant to help students identify whether they have mastered the material they covered.  If they haven't, the expectation is that they return to the source material until they have a better grasp of the content.  The MD Program has created a bank of very high quality questions that are directed at assessing not just knowledge but comprehension and application.  This requires the use of a secure and well-mapped exam bank which precludes disseminating the questions and answers.  Students are reminded that copying and disseminating questions is against university regulations.

No. Individual assessment component marks will never appear on transcripts or other documentation provided by the MD Program to authorized external individuals or organizations.

No. Final transcripts will only display Credit/No Credit (CR/NC).      

Currently, Borderline Regression (BLR) is used as the method to set a pass mark in Year 1 and Year 2 Clinical Skills OSCE as well as in the Clerkship OSCE.

BLR calculates individual station pass marks based on the results of a regression analysis, using the overall performance mark as the independent variable and the station score as the dependent variable (the combined mark of station checklist marks and global rating marks for Year 1 and Year 2 and the combined mark of competency scores and Key Feature Question (KFQ) marks for Clerkship). The regression line is calculated based on all of the interactions between examiners and students for a particular station. The station pass mark is then the point at which the regression line crosses the borderline category. The overall pass mark of OSCEs using the BLR method is calculated as the average of all of the station pass marks plus two* Standard Error of Measurement (SEM).

[*This number might change depending on the internal consistency of the OSCE stations].

Overall Assessment: 1=Unsatisfactory, 2=Borderline, 3=Meets Expectation, 4=Exceeds expectation and 5=Outstanding

Weekly feedback quizzes are short quizzes delivered through Examplify that first year and second year Foundations Curriculum students will complete at home each week. While low stakes, these will help students examine how well they have learned the week’s material. These quizzes will focus on key concepts that must be understood, but some higher-level questions related to comprehension and application of concepts will also be included.

There is no time "limit" to complete the quiz; however, it is best to do the quiz in one sitting. Students are expected to complete the quizzes before the deadline in order for their results to be uploaded to the Learner Chart.

Yes, you will be able to see your score as soon as you complete the quiz. Also, once your quiz has been uploaded you will receive a detailed feedback report in your ExamSoft account.

Students have access to two additional attempts of each weekly feedback quiz. These attempts are for learning purposes only and will not appear in the Learner Chart. Students will receive feedback reports in their ExamSoft accounts, showing the learning objectives attached to all questions and the question stems, after completing the quiz and after each subsequent attempt.

There is no make-up time or approvals of absence or deferrals if students miss a quiz. Missed weekly feedback quizzes will be indicated as incomplete in your Learner Chart. 

Mastery exercises are mandatory invigilated written assessments for first, second and third year students. Students complete these assessments using Examplify on their own devices. These assessments may include multiple-choice questions, short-answer questions, essay questions or key feature questions. 

No, mastery exercises are weighted according to the amount of material (number of weeks of material) they cover in the first and second years.

Faculty will provide a debrief or ‘teaching point’ for questions answered poorly by the class. Neither the question nor answers will be released. This is to maintain the integrity of the exam bank, and to ensure that students focus on the learning objective addressed by the question or the learning gap identified as opposed to the question itself.

Students who performed poorly on a Mastery Exercise should reach out to the OHPSA academic coach or course director to discuss learning and studying strategies. Rules and regulations are in place to maintain the integrity of the MD Program’s question banks. Hence, viewing of assessments is limited to written assessments that include short answer questions (SAQs) and/or other narrative components, in order to request a re-mark of a written assessment.

Multiple choice questions (MCQs) on mastery exercises are written to assess one or more end-of-week learning objective(s). MCQs are developed by content experts who are usually directly involved in teaching. The question stems in the MCQs will include all of the information required to answer the question. There is one correct answer and three incorrect alternatives known as distractors.

Mastery exercise questions are designed to assess not just whether students can remember facts and understand concepts, but also if they can apply the content being taught to a real-life like clinical situation.

Any learning objective can serve as the basis of an assessment question on a mastery exercise, including those objectives that are covered by pre-week and self-learning.

Following each mastery exercise, and prior to approving the students’ results for publication, the course director reviews the performance statistics of each question and the mastery exercise as a whole in accordance with evidence-informed processes endorsed by the Test Committee.

Once the mastery exercise results are approved by the course director students will receive a report summarizing their performance that can be used to guide their further studying.

Based on the input and review of the the Student Assessment and Standards Committee (SASC), the Foundations Committee, and the Curriculum Committee, we currently use the Hofstee method for standard setting for Mastery Exercises (ME). This approach asked members from SASC and the Foundations Committee to examine ME data from the previous academic year. Using their understanding of the objectives and goals of each course as well as the historical ME data, each committee member was asked to estimate what the acceptable values would be for the minimum and maximum pass mark in each course as well as the minimum and maximum number of students who could possibly be below the pass mark. This does not mean that there will always be the minimum number of students below the pass mark every year. Instead, these judgements are just an approach to help judges communicate their sense of the expected difficulty of each course.  Combing all of this information from members of both committees, we plot the average values on to previous ME data for a particular course and identify which score satisfies the best estimates of the committee. This point becomes the new pass mark for the course. The new pass marks are deployed and re-evaluated again at end of the academic year. 

The pass mark can be found on the Elentra course page

Reports will be released to students approximately 1-2 weeks after the completion of mastery exercises.

Following the completion of each mastery exercise, the course director conducts an analysis of the assessment to ensure that all questions meet minimum standards for quality. This includes a review of item difficulty, reliability and clarity. Adjustments will be made to optimize the overall reliability of the assessment prior to the release of reports to students.  The 1-2 week timeframe also allows for the small proportion of students who need to write a make up assessment for various reasons (typically scheduled on the next self-learning day), so that the results can be released to the entire class at the same time.

The report you receive provides a breakdown by type of question and the specific learning objective assessed in the mastery exercise. You can use the report to determine how you did on certain types of questions (e.g. remembering, understanding, applying or evaluating) and which learning objectives you should revisit to ensure that you understand the material well enough. Each code corresponds to a learning objective which you can find in Elentra.

Your mastery exercise will be rescheduled.

If a student fails a mastery exercise, they should contact their course director to review their performance on the assessment and determine next steps. 

The progress test is a comprehensive knowledge-based test designed to assess the targeted exit competency of students in the MD program. Progress tests are completed by Year 1, 2, 3 and 4 students. The best preparation for the test is to engage in the curriculum and stay up to date throughout the program.

Progress tests are designed to show ​students' growth and competency acquisition over time. It’s a stable measure that shows ​students where they are, and where ​they need to be by the end of the program.

Progress tests will be given to students at regular intervals throughout the four years of the MD program.  Year 1, 2 and 3 students will have three progress tests every academic year. Year 4 students will have two progress tests every academic year. The schedule can be found in MedSIS.

Question writers are experienced faculty who have been trained to write high quality assessment questions and much effort goes into ensuring they ​adhere to the highest ​standards. Our data confirms that the progress test questions perform very well psychometrically. As such, the progress test is an excellent tool for estimating whether our students are on track with their acquisition and application of medical knowledge and can be relied upon as a valuable means of self-assessment. Our research indicates that students’ longitudinal performance on the progress test has a moderate level of correlation with other types of assessments (i.e. MEs during Foundations and Clerkship).

The blueprint of the progress test aligns with the Medical Council Canada Qualifying Examination (MCCQE) Part I. A series of progress tests throughout the MD program have proven to be a reliable screening tool for MCCQE Part I with a high level of sensitivity (100%) and specificity (93.7%) (2T0 cohort N=227).

The Director of Foundations, Clerkship Director, Director of Student Assessment, Course Directors, Student Progress Coordinator and Student Progress Committee members will have access to the Learner Chart of all students. Academy Scholars will have access to the Learner Chart profiles of students who are assigned to them. No external groups have access to a student’s Learner Chart. The Leaner Chart is purely for personal (student) and internal (MD Program) use.

Students will be assessed throughout the MD Program by a faculty member using the following form:

MP Program Professionalism Form.pdf

Written assessments will be available for download in the student's Examplify account.  

For more information about Examplify, refer to the MD Program assessment and evaluation technology page. 

Standard setting is the process by which pass marks are determined for assessments. This area of education has a great deal of research and evidence to support. There is no single best way to set pass marks for tests just as there is no objective or mathematical formula to decide on the pass mark. At the same time, pass marks are not determined arbitrarily. They are set considering the goals of the curriculum, the level of mastery expected of students, expert opinion, and statistical analysis combined through rigorous methodology. (The MD Program’s curriculum goals can be found here: https://md.utoronto.ca/mdprogramcompetencies). 

Standard setting is reviewed by three important committees in which students have a voice and representation: the Student Assessment and Standards Committee (SASC), the Foundations Committee, and the Curriculum Committee. 

The standards we use for the curriculum are not intended to increase the number of students who do not meet the standard; there is no norm referenced standard setting in the MD program (i.e., we do not aim to flag a certain portion of the class as below the standard after each Mastery Exercise (ME) or course). Instead, our pass marks are adjusted on a yearly basis based on the difficulty of the questions and ensuring that the assessments used in the program enable a determination if a student has met the required course objectives and competencies for MD training. It is possible all of the students will be above the pass mark for an ME despite the estimates of the judges. Additionally, quality control metrics are used to evaluate each ME and all assessments on a routine basis including item difficulties, appropriateness of questions for objectives, and overall validity. 

The Hofstee method for standard setting is used for Foundations Curriculum mastery exercises. The Hofstee method is well described in the literature on standard setting. You may also find out more from your student reps on SASC and Foundations. Mckinley DW, Norcini JJ. How to set standards on performance based examinations: AMEE Guide No 85. Med Teach. 2014;36:97–110. (https://www.ncbi.nlm.nih.gov/pubmed/24256050) 

You can find the information on the Assessment tab of the course’s Elentra page. 

Please refer to this document for information on the Board of Examiners (BOE), and answers to commonly asked student questions: BOE_FAQ
 

High standards are set for the quality of our mastery exercises by the MD Program and much effort and expertise is dedicated by faculty to write questions that adhere to best evidence for assessment. After each mastery exercise the course director, supported by the Office of Assessment and Evaluation, reviews the mastery exercise statistics including the overall performance of the assessment and a step-by-step review of each item’s psychometric properties. Items that do not comply with the high standards set, are considered for removal and either removed completely or edited for improvement and use in future mastery exercises, but not counted towards the mark for the current mastery exercise.

Important: From April 2022 written assessments are in-person. E-proctoring will only be considered under pre-determined circumstances.

Biometric data is a term that refers to “biometric identifier” and “biometric information”. “Biometric identifier” means a retina or iris scan, fingerprint, voiceprint, or scan of hand or face geometry. “Biometric information” means any information, regardless of how it is captured, converted, stored, or shared, based on an individual’s biometric identifier used to identify an individual.

In relation to the ExamSoft document titled "NOTICE OF COLLECTION OF BIOMETRIC DATA AND CONSENT", student biometric data will be used by ExamSoft for the purposes of proctoring applicable assessments.

The MD Program will instruct ExamSoft to destroy student biometric data when it is no longer required by the MD Program to confirm a student's identity for assessments, i.e., at the end of a student’s completion of the MD Program.

Information on the ExamSoft Privacy Policy and Terms of Use can be found by visiting www.examsoft.com.

From March 2023 clerkship students in their 3rd year will be assessed on Entrustable Professional Activities (EPAs) in Elentra.

Student and assessor guides on how to start an assessment in Elentra 

From March 2023 clerkship students in their 3rd year will be assessed on Entrustable Professional Activities (EPAs) on Elentra.

View all student and assessor EPA resources