The use of Objective Structured Clinical Examination (OSCE) in medical education has picked up in the last few decades, with an aim to be fair and objective in the assessment of students’ performance. In India, the medical education system has undergone a major change in the last couple of years, with the revision of the undergraduate curriculum and shift from the earlier predominantly knowledge-based medical education to the newer competency-based medical education (CBME), with most expectations at “knows” and “knows how” level. OSCE as a means of assessment of undergraduate skills is already in place,; however, it is often used to assess the knowledge of “shows how” rather than actual “shows how.” With postgraduate curriculum and teaching methods likely to undergo a similar haul, developing more structured, objective ways of assessing the skills at Miller’s “shows how” and “does” levels is pertinent. In the UK, the Clinical Assessment of Skills and Competencies (CASC), introduced as part of the gateway exam to be a psychiatrist, is an example of the adaptation of OSCE for psychiatry-specific evaluation for aspiring consultants. Research on the use of OSCEs in psychiatry postgraduate examination is limited. While OSCEs as part of the examination is not something new, in India, it is still an upcoming area, specifically in psychiatry. During the recent pandemic, due to a lack of patients, a group of Indian psychiatry teachers recommended OSCE as a standby method for postgraduate evaluation.
In India, where students travel from all parts of the country with different cultural backgrounds and deal with people from equally different cultures, the outcome of the practical examination is often dependent on the patient allotted to the candidate. Therefore, OSCEs have an advantage in ensuring that these factors are minimized and the progress in the skills obtained during the training is assessed fairly and objectively. To ensure that the OSCE model remains truly impartial and unbiased, significant importance needs to be given to the process of setting up the OSCE at the postgraduate level. The Department of Psychiatry, Kasturba Medical College, Manipal, hence came up with this concept of introducing OSCEs in postgraduate examination in April 2017. This paper describes the model used by the department while conducting the OSCE and the shape it has taken over the years.
OSCE Model for Psychiatry Postgraduates at Kasturba Medical College, Manipal
Objectives
The department introduced OSCE during the university examination as part of an additional assessment beside the traditional long and short cases. This was done with the primary aim of introducing variability in the cases being assessed and widening the scope of assessment in an objective manner. A secondary objective was to familiarize the postgraduate residents with the assessment methods widely used worldwide. OSCEs also help assess some of the more important softer skills in the field of psychiatry, namely communication, rapport building, and the conduct of the interview. OSCE provides the necessary control over the extent of cooperation, language, and content, to specifically assess these without other patient-, student- or examiner-related factors influencing the outcome of the student’s performance. Depending on the task assigned, the OSCEs could have huge potential in psychiatry in assessing the competence level of “shows how.”
Methods
Conduct of the OSCE
The department sought permission from the college council to introduce the OSCE as part of the regular postgraduate clinical examination. The OSCE model at Kasturba Medical College, Manipal consists of a set of four OSCE stations designed to specifically assess four fairly different areas in clinical skills: (a) Assessment of phenomenon/diagnostic interview, (b) Demonstration of an assessment in mental status examination, administration of a relevant scale, or physical examination, (c) Handling a distress/crisis situation or breaking bad news, and (d) Psychoeducation of patient/ family about diagnosis/therapy methods. The tasks kept are ones that can be reasonably completed in ten minutes. At each station, a brief introduction about the clinical scenario (2-3 sentences) is given, and the task is specified. A warning bell is given at nine minutes to let the student meaningfully terminate the interview. A two-minute break is given before moving to the next station. The examiners rate the students based on the objective assessment template provided by the examination expert. Each OSCE station score is reduced to a score out of 5, and a cumulative score of up to 20 marks can be obtained through the four OSCE stations.
Training of the Students in OSCE
OSCEs are included as part of the academic program under the umbrella of periodic special case conferences. The purpose of these OSCE case conferences is to familiarize the students with the OSCE pattern and, at the same time, train them to do appropriate short interviews and assessments. These OSCE case presentations allow teachers to give the necessary feedback to fine-tune students’ knowledge, skills, and attitude. The chairperson prepares the scripts for the actors and checklists for the students to observe the process and content of the interview. OSCE is also introduced as part of periodic practical internal assessments in the initial days, to familiarize the students. Care is taken to introduce variability in the tasks during both the case conferences and the examination. A guideline for conducting OSCE case conferences was prepared by RKB and SKP and made available to all chairpersons.
Training of Actors for the OSCE
The department contacted a local drama/actors association in the region that participates in state-level drama competitions and regularly provides actors for stage performances across the state. The actors were initially briefed about the need for special acting skills, which is, for examination purposes, and the need to be flexible in the script while adhering to the expected content. The actors expressed several challenges, which were addressed during systematic training for the assigned tasks. Two faculty (RKB, RM ) were given this primary task of training the actors. The first step was to define the tasks to be completed and the scoring format for the OSCEs (RKB, PSVNS, SKP ). The clinical scenario needed from the actors was then prepared with a broader script to give a brief outline of the background of the concerned patient, including history, brief family information, current history, negative history, and descriptions of emotions experienced at different time points. RKB and RM provided the actors with the medical understanding of the illness, the symptoms, and other relevant information required to help the actors get into the mindset of the patients. Interview videos from the department repository were shown to the actors to help them understand the communication and behavioural oddities noted in the said conditions. The actors were also asked to watch certain movies and characters that fairly displayed the given psychiatry condition, albeit with a warning that some of it is more dramatic than what is seen in real-life settings. A demo run was conducted (RKB/RM) to provide feedback to the actors about the emotional expression, modulations in speech, pauses required, how much information to share and how much depending on the type of questions framed. As a final step, two more faculties not involved in the training served as mock students and performed the OSCE task. This exposed the actors to being prepared for different sets of questions while continuing to adhere to the content in the interview. The trainer and actors discussed some more questions/ways of questioning that can come up during the interview and obtained clarity on how to respond in such situations.
Orientation of Examiners to OSCE
Examiners are given a standard set of instructions by the exam coordinator 15 minutes before the conduct of the OSCE. One examiner is assigned to one OSCE station so that it is easy to get oriented to the expected answers and to avoid interrater variability in assessment as they get to rate all students on the same case scenario. Instructions are given to ensure not crossing the time allotted, periodic warning bells, and adherence to the marking guide. The examiner is instructed not to interrupt the interview till the time is out and only to observe the process. No clarifications are sought or questions raised even after the interview. The examiners are made aware that no physical examination is required during the OSCE unless the task specifically mentions so. The internal examiners, which include faculty from the department, get trained during the OSCE case presentations. All these measures are taken to enhance the reliability and validity of the assessments.
Evaluation
Marking Guide for OSCEs
The Marking Guide has seven areas that assess both the content and process of the interview during OSCE. Of these, 3 or 4 would be allocated for the interview process, including rapport building, empathy, summarizing, etc., and the remaining would be focused on interview content to ensure that the said task was performed completely. Each item is to be scored based on (a) did not perform, (b) performed but inadequate, and (c) performed adequately. Depending on the task, when adequately performed, some items would get a score of 1 while others would get 2. A total score of 10 is obtained on appropriate and adequate completion of the task.
Post-OSCE Process
Feedback is obtained from students, examiners, and actors to fine-tune the process and address any lacunae or challenges in the OSCE during subsequent administration. In the subsequent OSCEs, the actors could incorporate the changes suggested during the feedback. The OSCE expert would use the obtained feedback to refine the scenarios, tasks, and marking guide.
Evolution of the OSCE Model
As part of refining the process, the faculty involved in designing the OSCE had attended a workshop on OSCE in UG medical education, conducted by the University of Chicago in 2017, and a workshop on CBME-based curriculum in June 2019. A faculty from the UK who was involved in designing and conducting CASC in MRCPsych exams provided direct inputs during his tenure as a part-time faculty in the department. The revised model of OSCE now involves a 3-domain evaluation: (1) interview process (10 points), (2) interview content (10 points), and (3) interpretation of the interview (five points). The timing of OSCE has been revised to 10 minutes for the interview, 2 minutes for the questions, and 3 minutes for moving to the next station. The first two domains are rated on observation by the examiner, while the third domain allows the examiner to ask two relevant questions related to the interview scenario for assessing the appropriate interpretation. The training of the actors is the same. The actors are provided with a likely set of questions when preparing the script itself. The actors can practice among themselves with mock scenarios, and the actors more experienced in OSCE train the newcomers in the field through peer feedback. A trial run is conducted to ascertain whether the doubts related to the individual scenarios are cleared, and any inappropriate responses are rectified.
Advantages and Disadvantages of Our Model of OSCE
This model, like any OSCE model, provides an objective method of assessing clinical skills (like interview skills and communication skills), specific soft skills, as well as performance of specified tasks that are difficult to assess in routine examination patterns (which are more diagnosis- and management-oriented and assess skills to a very limited extent). The assessed skills cover not only the “psychomotor” domain (e.g., interview skills to elicit specific psychopathology or clinical sign) but also the “cognitive” domain (e.g., application of knowledge to “break the bad news” or “psychoeducation of a disorder”) and the “affective” domain (e.g., attitude towards the patient or caregivers), which are mandated by CBME postgraduate curriculum. The use of actors seems to be an alternative to the ethical concerns of repeated interviews on real patients used in OSCEs. The model also provides an opportunity to assess more clinical conditions, which may not usually be possible due to the assessment revolving around only two case presentations in psychiatry and one in neurology, which largely leaves the students at the mercy of the nature of the case and the unpredictability of the cooperativeness of the patients. Furthermore, possible unfairness related to the language barrier, need for interpreters, and other red herrings due to cultural differences are minimized with the use of OSCEs.
The disadvantage exists in terms of the evaluation still being restricted to controlled scenarios while the students need to be prepared for more varied and real-life scenarios where cooperativeness, language barriers and cultural difference exist. Similar findings are reflected in the literature reviews on OSCE.,– Although the “cognitive,” “psychomotor,” and “affective” domains are assessed during OSCE, limited assessment of knowledge is possible because of few stations, and the assessment of attitude can also be biased owing to the artificiality of the situation. Also, using OSCEs for examination requires more time, infrastructure, staff efforts, and additional expenses, which can be prohibitive. The intense effort required to plan and execute OSCE during examinations could potentially limit their generalizability and sustainability in other institutes, specifically in resource-limited settings. As a part of summative assessment, currently, only 20 marks are allotted to OSCE, as this mode of assessment is currently not mandated by the National Medical Commission in the postgraduate curriculum; this can be suitably increased once OSCE is widely adopted as a standard assessment method, maybe as an alternative to short cases.
Future Directions
Our model of using OSCE could further be refined with feedback received from the faculty, students, and actors, to make it more robust for summative assessment. This model can be tested in other centers to examine for suitability in summative assessments. Also, the potential for using this model for formative assessment is immense if done seriously, with feedback shared from not only the faculty but also the trained actors, especially regarding the skills, to enhance learning. Furthermore, OSCE stations can be developed as part of periodic “skill labs” that allow the trainees to learn in a controlled setting.
Conclusions
The Manipal model of OSCE for sychiatry postgraduate students provides a comprehensive model to evaluate students on a wide variety of psychiatry case scenarios on all the domains, knowledge, skills, and attitude, in a controlled objective way, without adding to the challenges related to subjecting patients to the discomfort and distress of repeated interviews.
Declaration of Conflicting Interests The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding The authors received no financial support for the research, authorship, and/or publication of this article.
References
- 1. Patrício MF, Julião M, Fareleira F, Carneiro AV. Is the OSCE a feasible tool to assess competencies in undergraduate medical education? Med Teach, 2013; 35: 503–514.
- 2. Jacob KS. Medical Council of India’s new competency-based curriculum for medical graduates: a critical appraisal. Indian J Psychol Med, 2019; 41: 203–209.
- 3. Bhatnagar KR, Saoji VA, Banerjee AA. Objective structured clinical examination for undergraduates: is it a feasible approach to standardized assessment in India? Indian J Ophthalmol, 2011; 59: 211–214.
- 4. Joshi MK, Srivastava AK, Ranjan P, Singhal M, Dhar A, Chumber S, . OSCE as a summative assessment tool for undergraduate students of surgery - our experience. Indian J Surg, 2017; 79: 534–538.
- 5. Miller GE. The assessment of clinical skills/competence/performance. Acad Med, 1990; 65: S63–S67.
- 6. Marwaha S. Objective structured clinical examinations (OSCEs), psychiatry and the clinical assessment of skills and competencies (CASC) same evidence, different judgement. BMC Psychiatry, 2011; 11: 85. DOI: 10.1186/1471-244X-11-85.
- 7. Plakiotis C. Objective Structured Clinical Examination (OSCE) in psychiatry education: a review of its role in competency-based assessment. Adv Exp Med Biol, 2017; 988: 159–180.
- 8. Kishor M, Shah H, Chandran S, Mysore AV, Kumar A, Menon V, . Psychiatry postgraduate examinations for 2020 in the middle of COVID19 crisis: Suggestions from Indian teachers of psychiatry. Indian J Psychiatry, 2020; 62: 431–434.
- 9. Pugh D, Touchie C, Wood TJ, Humphrey-Murto S. Progress testing: is there a role for the OSCE? Med Educ, 2014; 48: 623–631.
- 10. Hijazi M, Downing SM. Objective structured clinical examinations as an assessment method in residency training: practical considerations. Ann Saudi Med, 2008; 28: 192–199.
- 11. National Medical Commission Postgraduate Medical Education Board. Guidelines for Competency Based Postgraduate Training Programme for MD in Psychiatry, https://www.nmc.org.in/wp-content/uploads/2022/revised/MD_Psychiatry_(%20revised%20).pdf. (accessed , )
- 12. Smith V, Muldoon K, Biesty L. The objective structured clinical examination (OSCE) as a strategy for assessing clinical competence in midwifery education in Ireland: a critical review. Nurse Educ Pract, 2012; 12: 242–247.
- 13. Abu Bdair IA, Abuzaineh HF, Burqan HMR. Advantages and disadvantages of the objective structured clinical examination (OSCE) in nursing education: a literature review. Int J Trend Sci Res Dev, 2019; 3: 270–274.
- 14. Walters K, Osborn D, Raven P. The development, validity and reliability of a multimodality objective structured clinical examination in psychiatry. Med Educ, 2005; 39: 292–298.