Succeeding In The Biomedical Admissions Test (B... BETTER
Biomedical research graduate programs have grown in size significantly over the last ten years [16], and many of these programs emphasize GRE scores for admissions decisions [2]. Few studies focus specifically on the relationship between biomedical Ph.D. student success and GRE scores. A recent study of 57 Puerto Rican biomedical students at Ponce Health Sciences University revealed a shared variance between GRE and months to defense (r2 = .24), but no relationship between GRE score and degree completion or fellowship attainment [17]. Another small study of 52 University of California San Francisco (UCSF) biomedical graduate students attempted to show that general GRE scores are not predictive of student success [18], however, as UCSF students address in their critique, a vague definition of success and weak research methods confound the interpretation [19]. The UCSF students conclude that more rigorous studies, such as this one, are needed.
Succeeding in the Biomedical Admissions Test (B...
Larger studies of biomedical graduate students have shown shared variance between GRE scores and graduate GPA (r2 ranging from .05 to .25), as well as faculty ratings of student performance (r2 ranging from .05 to .25), but rely on data that are 19 or more years old and do not account for more recent changes to the exam [4,5,20]. ETS regularly updates questions and changed the GRE in 2002, replacing the Analytical Ability section with the Analytical Writing Assessment section. Newer GRE scores may show different predictions for biomedical Ph.D. student success than they did 19 years ago, and there appears to be no study that examines the individual contribution of the GRE Writing subtest on biomedical doctoral student outcomes. Furthermore, Ph.D. students have changed. Incoming cohorts of students are vastly more diverse [21] and with more robust research experience [22] than in previous years. The revision of the GRE in concert with changes in student populations prompt this focused and updated investigation.
This study investigates the predictive validity of GRE scores on various quantitative and qualitative measures of success for biomedical Ph.D. students including measures of progress in the program (passing the qualifying exam, graduation with a Ph.D., and time to defense), research productivity (presentation and first author publication rates and obtaining individual grants and fellowships), grades (first semester and overall GPAs), and faculty evaluations of students obtained at the time of thesis defense. Faculty evaluations, while being subjective measures of success, are important for the IGP given that most faculty do not directly select graduate students to enter their labs. Instead the admissions committee selects a cohort of biomedical students that they hope will meet the expectations of their faculty colleagues. Post-graduate career outcomes were excluded from the study, as we are hesitant to categorize one career as more or less successful than another. This, this study focuses solely on measures of success up to and including graduation.
We explore the importance of the GRE General Test in the biomedical field using a large and up to date dataset. This study covers hundreds of students from 11 departments and programs and looks at a wider range of outcomes and control variables than prior studies. Such an up-to-date, comprehensive evaluation of the use of the GRE in evaluating prospective biomedical graduate students is important to ensure that the admissions process aligns with the goals of the institution and to determine whether a GRE requirement for graduate school admission is worth the inherent biases that the test might bring into the admissions process.
Data were collected on 683 students who matriculated into the Vanderbilt University IGP from 2003 to 2011, a time period in which reliable GRE scores are available. Over 80% of students have had time to complete the program. GRE Quantitative, Verbal, and Analytical Writing scores were used to test the hypothesis that they could predict several measures of graduate school performance, including (1) graduation with a Ph.D., (2) passing the qualifying exam, (3) time to Ph.D. defense, (4) number of presentations at national or international meetings at time of defense, (5) number of first author peer-reviewed publications at time of defense, (6) obtaining an individual grant or fellowship, (7) performance in the first semester coursework, (8) cumulative graduate GPA, and (9) final assessment of the competence of the student as a scientist as evaluated by the research mentor. In order to determine the independent contributions of GRE scores on outcome measures, additional admissions criteria were included in the analyses as controls: undergraduate GPA, undergraduate institution selectivity, whether a student has a prior advanced degree, underrepresented minority status, international student status, and gender. Details on variables are described below. The research was approved by Vanderbilt University IRB (#151678). Consent was not given as data were analyzed anonymously.
Following a line of research that examines predictive validity of test scores, in order to evaluate the influences of each independent variable in the presence of the other admission criteria, linear regression analyses were used [24,25]. Admission cohort was included as a fixed effect to account for systematic changes that occur over time. We first looked at the influence of GRE scores and other admissions criteria on measures of progress in the program, defined as Passing the Qualifying Exam, Graduation with a Ph.D., and Time to Defense. We then investigated measures of productivity (Presentation Count, First Author Publication Count, and Obtaining an Individual Grant or Fellowship), grades (First Semester GPA and Graduate GPA), and faculty evaluations.
The results collected in Table 2 allowed us to see the effect of GRE scores upon an outcome variable collected during graduate training, in this case graduation with a Ph.D. Continuous independent variables (GRE Quantitative, GRE Verbal, GRE Writing, Undergraduate GPA, and Undergraduate Institution Selectivity) were standardized before entering the regression, whereas binary independent variables (Prior Advanced Degree, Underrepresented Minority, International, and Female) were not standardized and are shaded in gray to indicate that they are unstandardized regression coefficients. We used linear probability models for the binary dependent variables, so the coefficients should be interpreted as a change in the probability of the event happening (i.e. graduating). The table is constructed to display first the effect of a single variable, GRE Quantitative, on Graduation with a Ph.D. (Column (1)). The simple bivariate regression between GRE Quantitative and Graduation with a Ph.D. revealed no influence of GRE Quantitative scores. Moving rightward, when GRE Verbal and Writing scores were added to the model (Column (3)), none were shown to predict Graduation with a Ph.D. The rightmost column (Column (9)) is especially informative as it shows the independent contribution of each GRE subtest after controlling for all the other observed admissions variables. Again, none of the GRE subtests predicted graduating with a Ph.D. Undergraduate GPA significantly predicted Gradation with a Ph.D., such that one standard deviation increase in Undergraduate GPA was associated with a 0.05 increase in the probability of attaining a Ph.D. Note that one standard deviation of Undergraduate GPA in the sample was 0.32 on a 4 point scale (Table 1). Underrepresented Minority status also predicted Graduation with a Ph.D., such that Underrepresented Minority students had a 0.13 decrease in the probability of attaining a Ph.D relative to non-minority students. The full model accounted for 29% of the variance in Graduation with a Ph.D. (see Adjusted R2 in Table 2), most of which was driven by the inclusion of cohort fixed effects which control for a host of unobserved factors that were consistent with cohort. We chose to present linear probability models because their coefficients are directly interpretable as changes in the probability of graduating; however, logit models showed the same sign and significance and thus were qualitatively similar to the linear regression results.
Linear regression analyses were used to compare GRE scores to quantitative measures of research productivity. These measures include Presentation Count (Table C in S1 Supporting Information), First Author Publication Count (Table D in S1 Supporting Information), and obtaining an Individual Grant or Fellowship (Table E in S1 Supporting Information). Continuous independent and dependent variables were standardized before entering the regressions. When all admission criteria were included in the models, none of the GRE subtests predicted the above dependent variables. Minority Status was the only significant predictor of obtaining an Individual Grant or Fellowship, and no variables significantly predicted Presentation Count or First Author Publication Count. GRE scores and most standard objective measures for admissions did not predict measures of student productivity.
Linear regressions were run to examine student classroom performance, starting with First Semester Grades (Table 3). For this continuous outcome, the variable was standardized, as were all of the continuous independent variables such that regression coefficients can be interpreted as effect sizes. The shaded, binary independent variables remained unstandardized. GRE Quantitative scores moderately predicted First Semester GPA (Column (1)). A one standard deviation increase in GRE Quantitative was associated with a 0.29 standard deviation increase in First Semester Grades (Column (1)). When GRE Verbal scores were added (Column (2)), the model accounted for an additional 4% of the variance in First Semester Grades. GRE Writing scores did not predict First Semester Grades. GRE Quantitative and Verbal continued to predict First Semester Grades after controlling for other factors (Column (9)), although the magnitude of the relationship was attenuated with the inclusion of other predictors, given their overlapping influence on grades. Undergraduate GPA, Admission Rate, and Underrepresented Minority status also predicted First Semester Grades, with Undergraduate GPA having a higher coefficient than the GRE subtests. Undergraduate Institution Selectivity negatively contributed to First Semester Grades. Since competitive schools have lower selectivity scores, higher selectivity represents less competitive schools and predicted lower First Semester Grades. Underrepresented Minority status was associated with a 0.35 standard deviation decrease in grades. When all admissions variables were included, the model accounted for 40% of the variance in First Semester Grades, and was strongly driven by the cohort fixed effect. 041b061a72