Application to achievement: association between pre-admission factors, admission scores, and medical students’ performance

Abstract

Background

Students have long been admitted into medical colleges using standardized tests/interviews. However, limited evidence exists on their association with academic achievement during medical education. Moreover, the relationship between its individual components and subsequent academic achievement remains unexplored. This study aims to determine the association between medical students’ demographics and admission scores with their academic performance during medical college.

Methods

This cross-sectional study was conducted in 2023 at one of the oldest private medical colleges in a South Asian low- and middle-income country, where data for medical students graduating with a Bachelor of Medicine, Bachelor of Surgery (MBBS) degree in 2018–2022 were retrieved electronically. Through an extreme groups approach (EGA) sampling, factors leading to students falling within the top 20% scorers in preclinical, clinical, procedural, and non-procedural clerkships were identified. Binary logistic regression models assessed the association between demographics and admission scores with their academic performance in medical college.

Results

From data of 418 students, EGA sampling included a total of 168 and 169 students for comparison between the top and bottom 20% scorers in preclinical and clinical rotations, respectively. Female sex (adjusted OR:4.10, 95% CI:1.94,8.65) and higher biology, physics, and mathematical reasoning scores on the university’s admission test independently predicted preclinical academic achievement. Female sex and higher mathematical reasoning scores significantly predicted academic achievement in clinical, procedural, and non-procedural clerkships. Higher biology scores also independently predicted achievement in non-procedural clerkships (adjusted OR:1.03, CI:1.01,1.06). Prior schooling from the British education system was significantly associated with higher mean percentage for admission scores and clinical clerkships for applicants compared to the local education system.

Conclusion

Higher scores on medical college admission tests can predict medical students’ academic achievement during undergraduate studies. Understanding the specific test components associated with students’ success can refine the selection process, ultimately fostering efficient healthcare professionals.

Background

The ability to provide quality healthcare to patients is the epitome of competence for medical professionals [1, 2]. Hence, it becomes important to strengthen the process of admitting medical students, who will go on to become future healthcare providers [3].

Evidence suggests using a multi-tiered approach for effective applicant selection, incorporating knowledge-based and interpersonal attributes which are vital for healthcare professionals [4]. Traditional selection methods in medical colleges include evaluating previous academic performance, medical college admission tests, and interviews [5]. The American Medical Association (AMA) requires applicants to undergo standardized testing namely the Medical College Admission Test (MCAT) [6], which tests scientific knowledge and critical thinking [7]. Similarly, candidates applying to medical colleges in the United Kingdom (UK) and Australia are required to take the University Clinical Admission Test (UCAT) and Graduate Australian Medical School Admissions Test (GAMSAT), respectively [8, 9].

While multiple studies have shown the efficacy of admission tests in predicting overall medical students’ performance in medical college [10, 11], limited literature exists on the efficacy of individual admission components in predicting academic performance and clinical skills acquisition during medical college. Prior studies show an association between superior high-school grades and better academic performance in medical college [5, 12]. A study from Australia of over 700 medical students found pre-admission interview scores to be a better predictor of medical college academic performance than GAMSAT scores [13]. Even with nationally recognized admission tests in place, medical colleges often develop admission criteria based on their teaching approach for making their selection process more robust. Evaluating the components of such examinations in granularity is essential to predict their association with academic performance during medical college.

Approximately 85,000 medical students are currently enrolled in 117 medical colleges across Pakistan [14]. Most of these universities use the Medical and Dental Colleges Admission Test (MDCAT) for admitting students which assesses candidates in basic sciences, language usage, and logical reasoning. The Aga Khan University Medical College (AKU-MC), one of the country’s oldest private medical colleges, utilizes a self-administered admission test for all applicants, followed by structured faculty interviews of shortlisted candidates [15]. Continuous review of the admission process has enabled the university to make it more rigorous. However, its association with medical college academic performance remains unexplored. Therefore, this study aimed to determine the association between applicant-level (sex and prior system of education) and system-level (admission test and interview scores) factors with students’ success in scholastic and clinical skills performance during medical college.

Methods

Study setting

The AKU-MC was established in 1983 in Pakistan, a South Asian low- and middle-income country (LMIC), with the aim of developing quality healthcare professionals and academic leaders under the core principles of impact, quality, relevance, and access (IQRA) [16]. Offering a 5-year integrated curriculum towards a Bachelor of Medicine and Bachelor of Surgery (MBBS) degree, AKU-MC enrolls 100 students annually. With merit being the only qualifier, each class exhibits ethnic and geographical diversity within the students.

The Higher Secondary School Certificate (HSC), International General Certification of Secondary Education (IGCSE), and International Baccalaureate (IB) comprise the different educational systems offered throughout the country, with HSC and IGCSE being the most common. HSC is part of the local education system where students complete 10 years of Secondary School Certificate (SSC), followed by two years of HSC [17], whereas, IGCSE is the British education system where students complete 13 years of education through Ordinary (O-Level) and Advanced Level (A-Level). Applicants can also switch from O-Level to HSC for their secondary education.

The university’s admission process

AKU-MC’s admission test uses a multiple-choice question (MCQ) format, where applicants are assessed in five scientific domains (biology, physics, chemistry, mathematical reasoning, and scientific reasoning) and the English language. Applicants passing this test undergo two independent structured interviews conducted by AKU-MC’s faculty. Final student selection is based on an overall rank following a stakeholder committee review of each application.

MBBS curriculum in pakistan

The MBBS degree comprises two initial years of basic science knowledge (preclinical), followed by three years of clinical clerkships and electives. The preclinical curriculum utilizes a modular system and involves problem-based learning through case-based discussions in seven disciplines: Anatomy, Biochemistry, Physiology, Basic concepts of General Pathology, Pharmacology, Microbiology, and Community Health Sciences. The clinical curriculum composes of experiential learning with didactic and skills sessions, where medical students are integrated into healthcare teams during predetermined clerkships.

During their five years at AKU-MC, students are evaluated through multiple modalities. In the preclinical years, an end-of-module summative assessment evaluates all seven disciplines through MCQs and short answer questions (SAQs). These assessments also include a component of Alternative-To-Practical (ATP). Combined scores from each module contribute to the certifying examinations at the end of each academic year. For clinical years, faculty assess students based on their performance during curricular clerkships. The score from this continuous assessment, Objective Structured Clinical Examination (OSCE), and written examination scores are combined to obtain a summative score for each student during their clinical years.

Participants and data source

Our sample included all students completing the MBBS degree at AKU-MC between 2018 and 2022. We extracted data from the Registrar’s Office electronic records. Variables included student demographics (age, sex, country of residence), type of pre-admission educational system (British: O-Level/A-Level, local, and others: IB, O-Level/HSC etc.), admission test scores, interview scores, and individual module/rotation scores during undergraduate education. Student records with incomplete or unavailable data were excluded.

After anonymizing all data to ensure confidentiality using de-identified codes, we entered it into a private online database only accessible to the study investigators. The protocol was reviewed, and exempted by AKU’s Ethical Review Committee (Reference No. 2022-7825-23172).

Statistical analysis

We analyzed the data using StataCorp. 2015. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP. Participants’ undergraduate medical education scores were used as markers of their academic achievement. For scores obtained on the admission test and during medical college, percentages were calculated for each individual and cumulative component. Individual interview ratings of two interviewers, ranging from ‘Excellent’ to ‘Below average’ were coded numerically. A cumulative percentage was then calculated for analysis.

Categorical variables are presented as frequencies and percentages, while continuous data was checked for normality and have been represented using means and standard deviations accordingly. Since the students’ preclinical and clinical scores had an overall narrow range, we used extreme groups approach (EGA) sampling to analyze data based on low versus high scores [18, 19]. For our study, we used Kumwenda et al.’s approach of EGA and divided the extreme groups based on the top 20% and bottom 20% scorers during undergraduate education to analyze the odds of a student finishing in the highest ranking, against those ending in the lowest ranking of the group [20]. For this purpose, the scores of the students were ordered in an ascending manner, followed by removal of scores falling between the top and bottom 20%. This was done separately for all scores being assessed (preclinical, clinical, procedural, and non-procedural) since different students with varying characteristics excelled in each area. Because of this variation, our study aimed to identify the variables that are associated with each area separately on a granular level.

Binary logistic regression models were constructed to assess the independent relationship between students’ demographics, pre-admission, and admission factors with their academic performance. Academic performance was evaluated for preclinical and clinical years. Hypothesizing some students to perform better at procedural versus non-procedural clerkships within their clinical years, the clinical percentages were further categorized into these two areas to identify their respective achievement predictors. The cumulative score percentages for procedural rotations were calculated using end-of-clerkship scores from ENT (Ear, Nose, Throat), Ophthalmology, General Surgery, Orthopaedics, Anesthesia, Obstetrics and Gynecology. For non-procedural rotations, Medicine, Family Medicine, Psychiatry, Community Health Sciences, Radiology, Paediatrics, Infectious Diseases, and Maternal, Newborn and Child Health (MNCH) were considered for a cumulative score percentage. In the univariable analysis, variables with a p-value ≤ 0.25 were included in the multivariable model. For multivariable regression, statistical significance was attributed to variables with a p-value below 0.05.

We used Bonferroni pair wise comparisons with one way analysis of variance (ANOVA) to further evaluate the differences in mean percentages of scores obtained in the admission test and medical college based on prior educational systems, where a p-value of < 0.05 was considered statistically significant.

Results

A total sample of 418 participants was obtained after excluding 74 incomplete records. EGA sampling with a cut-off of top 20% and bottom 20% scores categorized 84 students in each group for cumulative preclinical score percentage, while for cumulative clinical percentage comparison, the bottom 20% had 85 and the top 20% had 84 medical students.

Of 418 students, most participants graduated between the ages of 26–28 years (n = 240, 57.4%). Males constituted the majority for both preclinical (51.2% vs. 48.8%) and clinical (57.4% vs. 42.6%) scores. Most participants resided in Pakistan (n = 381, 91.1%) and studied O-level/A-levels (n = 352, 84.2%). Nearly 10% of participants were accepted to AKU-MC on the second attempt. Table 1 illustrates student demographics, their pre-admission characteristics, and a classification of students included after EGA sampling for preclinical and clinical scores.

Table 1 Demographic and entry test characteristics stratified according to preclinical and clinical percentages

Figure 1 shows mean percentages for preclinical and clinical scores of the total cohort (n = 418) during undergraduate education when categorized according to admission test score percentages. The mean preclinical and clinical percentages for those who scored between 85 and 95% in their admission test were significantly higher than those who scored less than 65% (Preclinical: 80.27% vs. 70.05%, p-value: <0.001; Clinical: 74.09% vs. 70.81%, p-value: 0.02).

Fig. 1

Comparison of preclinical and clinical percentage as per admission test scores. *Significant difference between the categories for preclinical and clinical percentage (p-values: Preclinical: <0.001; Clinical: 0.02)

Table 2A highlights the predictors of being in the top 20% of scorers during preclinical years. With every percentage increase in the biology and mathematical reasoning component of the admission test, the odds of being within top 20% of preclinical examination scorers increased by 6% (95% CI 1.03, 1.09) and 7% (95% CI 1.03, 1.11), respectively, compared to the bottom 20% preclinical examination scorers. The odds of females being within the top 20% of preclinical examination scorers was higher compared to males (adjusted OR 4.10; 95% CI 1.94, 8.65).

Table 2 Predictors for Higher Preclinical and Clinical Percentage (For being in top 20% scorers)

Table 2B shows the predictors for being in the top 20% scorers for clinical years, which demonstrates that higher scientific reasoning component scores were negatively associated with higher clinical percentages, with the odds decreasing by 9%. The odds of females being within the top 20% of clinical examination scores was 3.81 times (95% CI 1.76, 8.27) compared to males. Similarly, the odds of students from O-Level/A-Level being within the top 20% for their cumulative clinical year score was 3.71 times compared to those from the local education system.

EGA sampling included 174 students (bottom 20%: 90; top 20%: 84) and 180 students (bottom 20%: 85; top 20%: 95) for procedural and non-procedural scores, respectively. Table 3A shows predictors for being in the top 20% scorers in procedural clerkships. The results showed that higher English and mathematical reasoning scores in the admission test were positively associated with scoring amongst top 20% in procedural clerkships compared to the bottom 20%. However, higher scientific reasoning scores at admission decreased procedural percentage by 3% (95% CI 0.94, 1.00). The odds of females being within the top 20% of procedural rotation scores was 4.91 times compared to males.

Table 3 Predictors for Higher Procedural and Non-Procedural Percentage (For being in top 20% scorers)

Table 3B shows the predictors for being in the top 20% scorers in non-procedural clerkships. Like procedural rotations, higher scientific reasoning scores also decreased the odds of being in the top 20% for non-procedural clerkships by 8%. However, higher biology and mathematical reasoning scores in the admission test were positively associated with being in the top 20% non-procedural scorers compared to the bottom 20%. The odds of females being within top 20% of non-procedural rotation scorers were higher compared to males (adjusted OR 2.54; 95% CI 1.22, 5.30). Similarly, the odds of students from O-Level/A-Level being within top 20% of non-procedural clerkship scorers was 4.89 times compared to those from the local education system. Interview scores were not significantly associated with academic achievement (for both preclinical and clinical cumulative score percentages).

Since the prior education system was seen to predict clinical performance, ANOVA was performed to identify differences within specific admission process components and annual scores for undergraduate medical education based on British/Local/Other educational systems. For this purpose, the original cohort (n = 418) was used to ascertain these differences in a larger sample.

Table 4 shows mean score percentages categorized according to the prior education system. Significantly higher mean biology and chemistry score percentages were observed in students from the local education system compared to the British, whereas a significantly higher scientific reasoning score percentage was seen in students from the British education system compared to the local system (76.00% vs. 65.42%). Similar results were also seen in mathematical reasoning, English, cumulative admission scores, interview ratings, cumulative admission scores, and cumulative scores during clinical years, where students from the British system had a significantly higher mean score than students from the local education system.

Table 4 Mean scores categorized according to the prior system of education

Discussion

This study identifies factors associated with academic achievement during medical college and explores correlations with admission tests and medical students’ performance in preclinical and clinical years. Our results demonstrate higher scores in the mathematical reasoning component of the admission test and female sex to be consistently associated with being in the top 20% scorers in both preclinical and clinical years independently. Having a British education background prior to medical college was associated with higher score percentages in individual admission components, and being within the top 20% scorers for clinical and non-procedural percentages.

In our cohort, female sex and obtaining higher scores in the biology, physics, and mathematical reasoning components of the university’s admission test was independently associated with being in the top 20% scorers for preclinical subjects. These findings can be attributed to the fact that concepts learned for the university’s admission test are carried forward during preclinical years, where new knowledge is built upon prior scientific notions. Therefore, students with strong scientific concepts score higher in admission tests and ultimately excel in preclinical years. A study from Australia demonstrated similar results where higher weighted GAMSAT scores predicted unimpeded progress in preclinical years (Effect size for year 1: 1.5, p-value < 0.05) [5]. Additionally, a Spanish study demonstrated higher biology scores in admission examinations to predict optimal performance during the first three years of medical college [21]. Similarly, the science test within the Arabian Gulf University (AGU)’s admission test was found to be the strongest predictor for academic performance during the first year of medical college [12, 22].

Our results indicated that female sex, having a British educational background, and scores on the mathematical reasoning component of the admission test positively predict being in the top 20% scorers for clinical clerkships. Interestingly, neither primary scientific components nor interview scores showed significant associations with clinical achievement in medical college. Continuous assessments during clerkships based on students’ clinical knowledge and performance are assigned a weightage of approximately 50% at AKU-MC. Since history-taking and clinical examinations are taught during medical college, it might decrease the association of clinical performance with admission test score percentages, as shown by our study. Previous studies have not reached a consensus regarding this finding. Similar to our results, Shen et al. demonstrated the predictive power of MCAT scores to decline significantly during the assessment of clinical performance [23]. Another study evaluating the association between aggregate academic achievement index (derived from MCAT scores and GPA) and three medical college clerkships (Family Medicine, Internal Medicine, and Pediatrics) concluded that the index only predicts performance for Family Medicine clerkship (standardized β = 0.18, p = 0.07) rotations [24]. This was attributed to Family Medicine being the only out-patient rotation among the three, where students worked solely with a single educator who evaluated them at the rotation’s end, compared to multiple evaluations conducted by different faculty in other rotations, leading to inter-rater subjectivity [24]. In contrast, a few studies indicated positive correlations between admission examinations and clinical achievement in medical college [25,26,27]. While one study showed interview scores to predict clinical performance [5], another study indicated all scientific components of the admission tests, including mathematics, to be significantly associated with achievement in clinical examinations [28].

Further analysis revealed higher English and biology scores in the admission test significantly predicted achievement in procedural and non-procedural rotations, respectively. While studies have assessed factors including the United States Medical Licensing Examination (USMLE) Step 1 score, learning styles, and performance in prior clerkships to predict achievement in specific clerkships (surgery, internal medicine etc.) [29, 30], their association with admission tests scores remain unexplored. Therefore, this finding is a novel addition to literature. Since English is the primary language of communication between medical students and their peers/faculty, competency within it can make students more confident in presenting their subject knowledge and its application efficiently [31], ultimately leading them to score higher during clerkships.

Similar to a previous study from AKU-MC [15], our results indicated that students from a British educational background had significantly higher admission and interview scores compared to students from the local or other systems of education. A multi-institutional study from the UK also highlighted A-level education to predict a higher overall medical college academic performance [32]. This difference can be attributed to the different teaching and learning styles in these education systems, where the local system allegedly promotes rote learning over analytical thinking, compared to the British education system [15, 33]. These findings necessitate refining national educational systems and improving the efficacy of learning methods to match critical thinking and knowledge attainment with students receiving their education per international standards.

Interestingly, interview score percentages were not associated with academic achievement in our study. While similar results were also obtained by Sladek et al., [5] studies from Australia and Malaysia report contrasting findings where high interview scores were associated with better clinical performance during medical college [2, 34]. While there is ambiguity regarding the impact of interviews on medical college performance, it remains vital to the admission process in medicine, owing to its value in understanding personality traits, emotional intelligence, and social competence [2]. These domains are not part of medical college examinations, yet are significant for a healthcare provider. Consideration should be given to implementing multiple-mini interviews (MMIs) which have shown to be an effective tool in evaluating candidates’ non-cognitive attributes [35,36,37].

To the best of our knowledge, this study is the first of its kind evaluating associations between individual components of admission tests and academic achievement in medical college, especially for procedural and non-procedural rotations which are part of the clinical years. Therefore, it provides an in-depth insight into the importance of admission tests from a career standpoint. Since the results consider both admission test components and interview scores, it provides a holistic view of the admission criteria used and their association with academic performance. Furthermore, EGA sampling accounts for the extreme values in a sample distribution, allowing for achieving greater statistical power in our study [18]. However, this study has limitations. While the EGA sampling allows for non-linear relationship between the top and bottom 20% scorers, it limits analysis of a linear relationship, given the removal of students who fell between these ranges. Furthermore, EGA reduces the reliability of the results, but has also been determined suitable for exploratory analysis in literature [19, 38]. Since data was extracted electronically from the medical college’s registrar’s office, sociodemographic variables including family income and scores obtained in secondary and higher school examinations etc. were unavailable, which could have provided additional findings. Moreover, country of residence was not reported for all the included participants. This could be considered for future studies to determine if a student’s residence is associated with their academic achievement during medical college. Given that the data is from a single medical college, its generalizability is potentially limited. However, the students enrolled each year are from various parts of the country and the world, depicting diversity within the cohort. Further studies from different regions of Pakistan and beyond assessing the same objective can strengthen these findings and provide evidence for required changes in the admission process of medical college.

Conclusion

Knowledge-based admission tests have the predictive power to determine academic achievement during medical college. Recognizing the association of individual scientific- and language-based components of these tests with achievement in medical education can enable continuous improvements in the selection process. Performing a rigorous review of the admission criterion and modifying the interview process to assess students’ competency beyond medical knowledge can further ensure a reliable and credible admission process, allowing a diverse cohort of medical students to develop into competent healthcare professionals.