NIH Wants to Rethink How It Measures "Impact" and It's Asking You First
A bi-weekly newsletter where I share news, opinion, grant opportunities, success tools and tips on personal growth.
Oliver's Note
NIH Wants to Rethink How It Measures "Impact" and It's Asking You First
For as long as most of us have been writing grants, the scorecard has looked roughly the same: publication counts, citation rates, and dollars awarded. NIH now openly acknowledges what many researchers have argued for years — that these traditional measures may no longer capture what actually advances biomedical discovery.
In a June 18Extramural Nexus post, the agency makes the case that biomedical research has become increasingly collaborative, interdisciplinary, and data-intensive, with progress often depending on teams of investigators, shared datasets and software, rigorous validation of findings, and sustained mentorship and training. The metrics, in other words, haven't kept pace with the work.
So NIH is asking a pointed question: are we measuring and rewarding the activities that matter most for advancing biomedical discovery?
The vehicle is a Request for Information —NOT-OD-26-087— with comments accepted electronically throughAugust 19, 2026.
The agency frames this as part of its broader push on rigor and reproducibility, explicitly tying it to its Replication and Reproducibility Initiative and its Plan to Drive Gold Standard Science.
What's notable is the breadth of input being solicited. NIH lists a range of topics it wants to hear about, including
incentivizing collaborative research and recognizing contributions to team science;
strategies that support reproducing or replicating existing findings;
approaches that encourage data sharing and reuse;
spurring innovation and entrepreneurship;
and how to quantify the impact of training for those early in their careers.
It also flags a thornier methodological concern: rigor with regard to defining causal research; it asks commenters to weigh feasibility across career stages, disciplines, and institution type.
Crucially, NIH isn't limiting this to PIs: input is welcome from investigators, trainees, research administrators, institutional leaders, patient advocates, and others.
Here's why this one isworth your attentioneven if RFIs usually go in your "read later" pile.
How impact gets defined eventually filters down into how applications are scored, how biosketches are framed, and what counts as a competitive record at review.
If you've ever felt that your data-sharing work, your software contributions, your mentoring, or your replication efforts were invisible on paper while a publication count did all the talking, this is the rare moment the agency is asking for exactly that perspective, before the rules are written rather than after.
The most useful comments will likely be specific: a concrete indicator that worked, an incentive that backfired, an implementation lesson from your own program or institution.
One honest caveat worth naming: this RFI doesn't exist in a vacuum. It arrives alongside a charged national conversation about "Gold Standard Science" and the role of political priorities in federal research, and reasonable people in our community read that framing differently. The RFI itself, though, is a genuine and fairly open-ended request for input on metrics and incentives.
The takeaway: Whatever your view of the surrounding policy landscape, the practical move is the same: if you have a perspective on how scientific contribution should be measured, the comment window is open, and silence is rarely the strategy that gets you heard.
In Case You Missed It
AACR Sounds the Alarm on OMB's Proposed Grant Rule and Asks Members to Act
I wrote about thisOMB change on June 2nd- please heed AACR's call to action if you have not already.
The American Association for Cancer Research (AACR) hasissued a statementand "Call to Action" urging its members to weigh in on a sweeping Office of Management and Budget proposal that could reshape how federal research grants are reviewed, awarded, managed, and terminated.
AACR's central worry: the rule could make federal research funding more vulnerable to political influence, less predictable for investigators and institutions, and more difficult to use for the scientific activities that move discoveries toward patients.
Three provisions drive that concern.
First, agency heads would designate senior appointees to conduct pre-issuance review of every discretionary award, with peer review recommendations remaining advisory and not routinely deferred to.
Second, awards could be terminated if they no longer support program goals, agency priorities, or the national interest as they exist at the time of termination — a provision the rule's preamble expressly analogizes to "termination for convenience" in federal procurement.
Third, new applicant risk review, payment verification, and E-Verify obligations could add administrative load to a system already strained by funding delays.
AACR plans to submit formal comments and has issued the Call to Action so members can submit their own. Comments are due July 13, 2026, with the rule scheduled to take effect October 1, 2026.
The docket is OMB-2026-0034 onRegulations.gov, and AACR — like several peer societies — emphasizes that individualized comments explaining real-world impact carry more weight than form letters.
The takeaway: This is one of the most consequential changes to federal grants administration in over a decade, and the comment window is unusually short. If your work touches federal funding, this is a rare moment where a few hundred specific words could matter.
Growth Mindset
Anticipate the Objections Before Reviewers Do
Every grant has weaknesses — and reviewers will find them. The question is whether you find them first.
In my workshop I share the FRAME framework (Focus, Reduce cognitive load, Anticipate objections, Maximize narrative coherence, Emphasize alignment) which puts anticipating objections at its center.
For each high-risk element of your approach, include alternative strategies. For each potential methodological critique, offer a preemptive response.
A proposal that acknowledges its limitations and addresses them confidently reads as more credible than one that pretends they don't exist.
The takeaway: The best defense to a reviewer's objection is given before you submit.
Success Tools
Before You Trust an AI Agent With Your Data Retrieval, Read This
If you're starting to lean on AI agents to pull sequences, datasets, or records for your research, anew Anthropic studyoffers a sharp, practical caution.
The team tested leading scientific agents on retrieving viral sequence data from NCBI Virus and found that even the strongest models did not consistently achieve the level of accuracy and reproducibility required for reliable dataset construction.
On their benchmark, agent accuracy ranged from about 17% to 91% — and, troublingly, the same model often produced substantially different answers when asked the identical question three times. One Ebola query returned 266 expected sequences but yielded 106, then 15, then 5 across three identical runs.
The takeaway for your workflow: the answers could look plausible while still being wrong, especially dangerous because retrieval is usually the first step in a much longer analysis. A bad pull at step one quietly corrupts everything downstream. In the study, incomplete retrievals shifted an inferred outbreak origin date by months or, in one run, by decades.
Three things you can do now:
Don't treat agent retrieval as deterministic. Run the same query more than once. If you get different counts, that's your signal something is off.
Verify against a known ground truth e.g. a manual query or expert-curated set, before building anything on top of the data.
Prefer purpose-built tools over raw agent navigation.
The study found that adding a deterministic retrieval layer (the team's gget virus tool) pushed accuracy above 90% for every agent and largely eliminated run-to-run variability. Notably, the right tool mattered more than picking the newest or most expensive model. anthropicanthropic
The takeaway: AI agents are genuinely useful, but for the boring, high-stakes plumbing of data retrieval, reliability comes from deterministic tools not from trusting the model to click through a messy interface on its own.
