‘Dysfunctional.’ NSF graduate fellowship review process draws criticism | Science

Michaela Horger was shocked last week when she saw the reviewer comments in response to her application for a US National Science Foundation (NSF) graduate research fellowship. As part of the “broader impacts,” where applicants describe how their activities will benefit society, she’d written about the challenges disabled scientists such as herself face and what she wants to do about it. One reviewer noted that information in the section meant to address her “intellectual merits,” questioning whether her physical disability would require special lab accommodations. “I thought that was wildly inappropriate,” says Horger, a structural biology Ph.D. student at the Scripps Research Institute.

Horger took to Twitter, where a chorus of voices had already been building. Some expressed outrage over reviewer comments they had received. Others called for changes in how NSF administers reviews for the program, which is considered one of the premier graduate fellowships in the United States — providing 3 years of financial support amounting to $ 138,000 per awardee. (Full disclosure: The author of this story received an NSF graduate research fellowship in 2006.)

Many details of the review process are a tightly held secret, and reviewers are asked not to speak about them. But in interviews with Science Careers, reviewers echoed calls for change based on their behind-the-scenes experiences. “This is a dysfunctional process that needs to be overhauled,” says Simon Mitra *, a chemistry professor who has served as a reviewer off and on over the past 15 years. (* Mitra is a pseudonym for a reviewer who spoke to ScienceCareers on the condition that he remain anonymous.)

In his view, the concerns raised on Twitter only scratch the surface because applicants aren’t privy to the full slate of feedback reviewers submit to NSF. Each application is typically reviewed by three reviewers, all of whom assign an overall score between 0 and 50. But applicants do not see those scores; they only receive their qualitative reviews.

NSF officials ultimately decide who is awarded fellowships, but reviewer scores help determine who is considered “meritorious” and they can make or break applications. Only 2000 fellowships are awarded each year — out of a pool of roughly 13,000 applications — and competition is so stiff that reviewers are often put in a position of “trying to decide between this ridiculously high qualified person and that ridiculously high qualifying person,” says Nicole Campione-Barr, a professor of psychology at the University of Missouri who has reviewed applications for NSF for 5 years.

Reviewers — who are selected based on their expertise and NSF’s desire to include diverse perspectives — receive training on implicit bias and are told to make their assessments using a holistic approach that takes into account each applicant’s unique set of achievements, skills, and experiences and that balances the intellectual merit and broader impacts components of the applications. “The idea was always to move away from scores and grades and all that to more a balanced consideration of the person,” says Gisèle Muller-Parker, who was the director of the graduate research fellowship program in 2010, when NSF switched to holistic review ; Muller-Parker left NSF in 2018.

The agency declined to offer a current representative to comment for this story. According to a statement from an NSF spokesperson, “Reviewers are asked to assess applicants on their potential to develop into outstanding scientists, engineers, and mathematicians based on the entirety of information in the application.”

But according to several reviewers, NSF does not provide sufficient concrete guidance on how to weigh different parts of the application, such as a student’s academic and research track record, the quality of their research proposal, and their potential to make a societal impact. Because of that, the resulting scores are sometimes highly variable among reviewers, they say.

“Some reviewers might look at a student who comes from a top-tier university and has done great in their classes and, you know, maybe done a little bit of tutoring or something like that and say, ‘OK, they checked the broader impacts box, ‘”says Ryan Gutenkunst, an associate professor of molecular and cellular biology at the University of Arizona, Tucson, who has served as a reviewer for 5 years. “Another reviewer might say, ‘Oh, they’ve hardly done anything'” on broader impacts and give the applicant a much lower score. “An explicit rubric would be really helpful.”

“Everybody’s told that they should be considering all these factors, but it’s not regulated,” Mitra says. “Some people review it in their own way.” During panel discussions, which are meant to debate the merits of borderline applications, he noticed how much reviewer approaches vary. For example, he uses research statements to gauge whether the student can identify a meaningful problem, explain it, and describe plausible solutions and experiments. “My understanding is that NSF would ultimately fund the applicant, not the research specifically,” he explains. But “some reviewers still just hammer the research statement as if it’s like a $ 10 million proposal.”

It can be challenging to compare applicants with different educational backgrounds, Campione-Barr adds. For instance, many of the applicants who have publications and presentations — which some reviewers view as indicators of strong research potential — come from well-resourced institutions. “It’s hard to say that that was all of them — look at all these things they did — when it probably has more to do with what was available to them,” she says. “I do not know if there’s a way to account for it,” she acknowledges, “because I want students to be able to talk about their presentations and publications and things like that.” But she also does not want to discount applicants who did not have access to research labs as undergraduate students.

Michelle Underhill *, an assistant professor in a biomedical field who has served as a reviewer for 3 years, thinks the process would be smoother if NSF provided more guidance on how to judge applicants who did not attend research-intensive institutions — if a “ student falls into this category, here’s some things to consider. ” The same goes for grades. “For some students, if they’ve got outside life happening and they’re still getting Bs, maybe that’s really impressive,” she says— “versus another student that has a different situation and [is] getting As; they’re still doing great, but with less challenges. ” (* Underhill is a pseudonym.)

Mitra notes that the problem of variable reviewer scores will not be easy to fix, given that NSF is charged with administering tens of thousands of reviews each year. “It’s just an enormous, colossal undertaking.” But he has suggested to NSF that it thinks about getting a fourth review for each application and then excluding the outlier.

“I like that plan quite a bit,” Campione-Barr says. “A big piece of that would be them needing a deeper reviewer pool. The fact that I keep getting asked every single year — I do not think it’s just because ‘Hey, you’ve done this and we’ll put you on.’ It’s literally a ‘we’re running out of people and if you know anybody who might be interested let us know’ kind of a scenario. ” (Qualified individuals can volunteer here.)

Campione-Barr would also like to see NSF provide more oversight before sending the reviewer comments to applicants, although she acknowledges doing so would be extremely labor intensive. “Some of the really disheartening things that I have heard from students… I hope that no human actually saw that and thought it was OK to go out that way.”

“It’s not all of the reviewers,” Underhill says. “Some of us are really trying.” But she agrees the process could be improved. “As a reviewer, it’s disheartening to see the comments that these students are getting.”

Leave a Reply

Your email address will not be published.

Back to top button