How should we be selecting our graduate students?

ABSTRACT

We use many quantitative undergraduate metrics to help select our graduate students, but which of these usefully discriminate successful from underperforming students and which should be ignored? Almost everyone has his or her own pet theory of the most predictive criteria, but I hoped to address this question in a more unbiased manner. I conducted a retrospective analysis of the highest- and lowest-ranked graduate students over the past 20 years in the Tetrad program at the University of California at San Francisco to identify undergraduate metrics that significantly differed between these groups. Only the number of years of research experience and subject graduate record exams (GREs) were strong discriminators between the highest- and lowest-ranked students, whereas many other commonly used admissions metrics (analytical, verbal, and quantitative GREs, grade point average, and ranking of undergraduate institution) showed no correlation with graduate performance. These are not necessarily the same criteria that matter at other graduate programs, but I would urge faculty elsewhere to conduct similar analyses to improve the admissions process and to minimize the use of useless metrics in selecting our students.

INTRODUCTION

As cochair of graduate admissions of the Tetrad program at the University of California at San Francisco (UCSF), I conducted a retrospective analysis of our highest- and lowest-ranked graduate students over the past 20 years to identify undergraduate metrics that significantly differed between these groups. I interviewed 30 core faculty members with a significant history of Tetrad graduate students and asked them to identify the very best versus most underperforming students they have known over the past two decades from their own labs, thesis committees, rotations, and so on. Because different faculty members have different criteria for ranking students, I next whittled this list down to unanimous highest- and lowest-ranked groups by interviewing the students’ thesis committees and other core faculty members and removed students for whom there was any disagreement. From this I obtained a unanimous cohort of 31 highest-ranked and 21 lowest-ranked students, and I analyzed how various undergraduate metrics differed between the groups.

RESULTS

1. The single largest discriminator of the highest-ranked versus lowest-ranked group was the number of years of previous research experience (p < 0.001). More than half (52%) of our lowest-ranked group versus 0% of our highest-ranked group had performed <2 years of previous research by the time they interviewed. It is likely that students who have done <2 years of research do not know what they are getting into. Furthermore, the letters of recommendation (our dominant criteria for choosing students) are more meaningful when written by principal investigators who have interacted with the students for long enough to know them well. Beyond a point, more research did not necessarily correlate with higher performance. The same proportion of highest-ranked versus lowest-ranked students did more than 3 years of research, but too little research is bad. On an educational note, making sure that students who are bound for experimental work in graduate school get significant hands-on exposure to real, not just course-based, lab work should be a high priority component of their undergraduate training.
2. None of the standard graduate record exams (GREs) exhibited a highly significant difference between the highest- and lowest-ranked students. The means of the verbal, analytical, and quantitative tests all differed by fewer than 6 points for the highest-ranked versus lowest-ranked students (Figure 1A). Some of our highest-ranked students had percentiles below 30% on verbal and analytical GREs. Only the subject test yielded a more significant difference (15 points) between the highest- and lowest-ranked students (p = 0.015).FIGURE 1:Testing how various undergraduate metrics correlate with success in graduate school. (A) The six paired bar graphs show the mean and standard error of the mean for the aggregated highest-ranked students (dark bars) and lowest-ranked students (light bars) at UCSF for their undergraduate GREs (percentage is on y-axis), previous research experience (years is on y-axis) and GPA (GPA is on y-axis). Only the number of years of research conducted prior to entering graduate school and the subject GRE were highly significantly different (p < 0.01) for the highest-ranked vs. lowest-ranked students. Of course, differing means are not necessarily useful in making admission decisions, if the distributions are highly overlapping. Most helpful would be a lower threshold one could use for each metric that would minimally exclude the highest-ranked students but maximally exclude the lowest-ranked students. The far right panel shows this analysis. (B) For each metric, a minimum threshold was established that captured 90% of the highest-ranked students, and we calculated how many lowest-ranked students would be excluded if this threshold were applied to everyone. For previous research, a 2-year cutoff rejects <10% of the highest-ranked students but rejects 52% of the lowest-ranked students. Similarly, a threshold score of 77 for the subject GREs rejects <10% of the highest-ranked students but rejects 41% of the lowest-ranked students. For a simulation sampling variance, bootstrapped samples were generated for both the highest-ranked group and the lowest-ranked group (i.e., sampling with replacement from the population, with the same total number of people), the tenth percentile was identified for the highest-ranked group, and we determined what fraction of the lowest-ranked group would be excluded at the cutoff. This analysis was repeated 100 times to generate a box-and-whisker plot. The box includes the interquartile range 25–75% percentile. Only number of years of previous research and subject GREs significantly enrich for the highest-ranked students.
3. There were no significant differences in grade point averages (GPAs) between our highest- and lowest-ranked students, although it is important to note that we generally only accepted students with GPAs of 3.0 or above.
4. There were no significant differences between the proportion of the highest-ranked (45%) versus lowest-ranked (38%) students that came from top 10 life sciences universities, as ranked by U.S. News & World Report and the National Research Council.

Of course, you cannot simply select a class with these numbers alone. Other factors, such as the strength of the letters (particularly from the primary research advisor) and the quality of the applicant’s previous research and essays are massively important. These were strong for all of the students that we accepted and were (and continue to be) our primary criteria for selecting interviewees.

To the extent that we use universal metrics such as GPA, GRE, and the amount of research experience, it is worth noting that some of these are significantly more discriminating than the others. Figure 1B shows the percentage of lowest-ranked students that are excluded for thresholds that capture 90% of the highest-ranked students for each selection criteria. Only number of years of previous research experience and subject GREs significantly enrich for highest-ranked students. These are not necessarily the same criteria that will matter at other schools, but I would urge faculty elsewhere to conduct similar analyses to improve the admissions process and to minimize the use of useless metrics in selecting our students

ACKNOWLEDGMENTS

I thank the faculty, staff, and Tetrad admission committee at UCSF for their help in identifying highest- and lowest-ranked graduate students and Rahul Deo for thoughtful discussion of the results and for helping with the analysis used in Figure 1B. This work was supported by National Institutes of Health Grant R01 GM084040.Go to:

ABBREVIATIONS USED:

GPA	grade point average
GRE	graduate record exam
UCSF	University of California at San Francisco

FAQs

Is the Graduate Program in Neuroscience part of the UW medical school?

No. We are an interdisciplinary neuroscience program through the Graduate School. We do have a number of faculty who are in departments within the Medical School, but we are not limited to those faculty and our degree does not lead to an MD which is a clinical degree, but rather a PhD in a research training program.

Does your program use rolling admissions, or are all applications considered at the same time after the deadline?

We only accept applications during the open application period and we only review applications after the deadline of November 30.

Are there any prerequisites or requirements to apply to your program?

The Neuroscience program has a holistic approach to our admissions process but we do have certain characteristics that we look for. These include:

• A sustained academic record of basic sciences including biology, chemistry, physics, and mathematics.
• In depth research experience (at least 6 months) with an output of presentations, posters, or publications listed on your CV.
• Strong letters of recommendation.
• An ability to clearly communicate as exemplified through answers to short answer questions.
• A commitment to broader scientific impacts, such as community outreach, scientific communication, and improving diversity, equity, and inclusion in the sciences.

Applications are reviewed by an Admissions Committee consisting of faculty and students. All committee members use an evaluation rubric that rank the following criteria: Academic Preparation; Self-motivation, Perseverance, and Teamwork; Conscientiousness and Broader Impacts; Research Experience; Program Fit; Letters of Recommendation. 

What are your average test scores and interview/acceptance rates?

As an interdisciplinary program we do not have a “cut-off point” or minimums on GRE scores or GPA. We try to balance the application review by looking at the overall picture of the student’s application.

• We usually interview 40-45 applicants.
• We make offers to about 35 applicants.
• Usually 8-12 applicants accept.
• Average applicant GPA is 3.50
• Average applicant GREs are in the 80-90’s percentiles for verbal and quantitative

Can I get my application fee waived?

There is a fee-waiver process and it is means-tested. You will be able to request a fee waiver prior to application submission, at the point in the application when you are requested to enter your credit card information. You will also be requested to provide some financial information. The waiver can take 4-5 working days so you need to submit your application in advance of our deadline.

Do you share my applications with different departments?

Yes, we do share applications with other departments with your consent.

Do I need to take the GRE to complete my application?

No, GRE scores are optional and therefore not needed to apply to our program.

If I decide to take the GRE, do I need official scores to submit my application?

Self-reported GRE scores are accepted if your official scores have not yet been made available.

If I decide to take the GRE, do I need to re-send my GRE scores for a different cycle then the one I originally applied for?

Yes, you need to re-send the scores for the current application cycle. When the admissions committee reviews applications, they only look at the current applications; there is no way to link information from past applications to the current one.

The UW Neuroscience program requires three rotations. What are some things I should be considering when choosing a lab?

Choosing a lab requires a lot of consideration. It is helpful to attend our interviews with thoughts about possible labs you would consider working in. Our program contains both experimental and computational labs. Experimental labs often require students to work with animals, such as worms, flies, zebrafish, mice, and primates. Students working in computational labs may primarily interact with neuroscience data and mathematical models. While graduate students often come in with some idea of a research direction they want to pursue, many students switch their focus or field based on interests they acquire while rotating through different labs.

Can I qualify for the program without Neuroscience/Research (lab) experience?

Our admissions process values research experience because it provides students with a clear sense of what it means to do lab research prior to embarking upon a PhD. Research experience also helps students prepare for the challenges of a graduate research-training program. However, prior research experience is less about acquisition of specific lab skills than it is about the general practice of scientific inquiry. Students typically pursue different research directions than those they studied prior to joining the program.