For organizations managing complex technology portfolios, a water power peer review offers a useful example of how independent technical evaluation can connect project-level progress with broader program strategy. The U.S. Department of Energy’s Water Power Technologies Office (WPTO) used its 2019 review to assess hydropower and marine and hydrokinetic (MHK) research and development, examine management practices, and gather feedback to guide future decisions.
Held in October 2019, the review covered 77 projects: 36 in hydropower and 41 in MHK. Twenty-four external experts from industry, academia, nonprofit organizations, and federal agencies evaluated the work. The exercise was WPTO’s first comprehensive review as an independent office and included projects representing about $230 million in executed funding, including prior-year appropriations and non-federal cost share.
Why independent evaluation matters
Technical programs need more than periodic status updates to determine whether research remains relevant and is being managed effectively. A formal peer review brings qualified, independent perspectives to questions of technical merit, expected results, productivity, and management. Its documented feedback can help program leaders decide whether to continue, adjust, or redirect work.
Peer review is distinct from other decision processes. A proposal merit review helps select projects for funding; a stage-gate or go/no-go review determines whether a project should advance to another phase. A portfolio peer review instead considers ongoing work and program direction, providing a broader view of how projects relate to objectives and stakeholder needs.
How the water power peer review was structured
WPTO organized projects and strategies into four review tracks spanning the Hydropower and MHK Programs. Project teams presented their work publicly, while office staff explained program strategies and priority initiatives. External panels were assembled around relevant technical areas, with reviewers required to confirm they had no conflicts of interest with the projects they assessed.
Reviewers considered program strategy and individual projects separately. At the program level, four equally weighted criteria were used:
- Strategy and objectives, including alignment with industry challenges.
- The program portfolio and its composition.
- The management approach.
- Stakeholder engagement, outreach, and dissemination.
For individual projects, reviewers considered objectives and impacts, alignment with program strategy, end-user engagement, technical and management approaches, accomplishments, and planned future work. The scoring approach also accounted for whether a project was new, ongoing, or nearing completion.
Program strategy and project execution are different tests
One of the review’s central findings was that reviewers scored both programs’ strategies above the average scores for their project portfolios. The Hydropower Program strategy received an average score of 4.17, compared with 3.96 for its reviewed projects. For the MHK Program, the corresponding averages were 4.38 and 3.81.
That gap is a reminder that a sound strategic direction does not automatically mean every project will execute it equally well. WPTO noted that some objectives had been updated after the earliest projects began. The higher strategy scores therefore indicated alignment with evolving industry needs, while project-level feedback could identify issues in implementation, timing, or engagement.
Reviewers were particularly supportive of two initiatives: HydroWIRES, focused on hydropower’s role in grid reliability, resilience, and storage, and Powering the Blue Economy, which explores marine energy applications. The distinction between program ambition and individual project delivery is relevant to any R&D portfolio: assess both, and do not use one score as a substitute for the other.
Four improvement areas for technical programs
Engage end users earlier
Feedback indicated that earlier and more frequent industry engagement could have benefited several projects, including new small hydropower work. WPTO committed to working with project teams to ensure stakeholder engagement is appropriate throughout the project cycle. For technical programs, early user input can help clarify the operational problem a project is intended to address and shape a relevant dissemination plan.
Define meaningful performance measures
Reviewers identified further work to do on performance metrics at both project and program levels. Project teams need clear definitions of success; program managers need ways to assess broader impact, return on investment, and commercialization outcomes. WPTO reported that it was developing marine energy device metrics through its TEAMER program and planned a pilot program-wide impact assessment in fiscal year 2020.
Make research data usable
The review recognized that WPTO collected substantial data but found that existing arrangements did not adequately ensure data quality or ease of access. Data collection alone is not enough: quality assurance, organization, access, and dissemination affect whether results can inform later research and decision-making.
Explain project decisions
Reviewers said they would have benefited from more information about go/no-go decisions and how funded projects progress. Clearer explanations of selection and continuation decisions can help participants understand how technical evidence informs portfolio management. WPTO planned to incorporate this feedback into future peer reviews.
Transparency requires participation and follow-through
WPTO designed the review to support transparency, gather feedback, share research results and tools, and provide objective information. Opportunities included anonymous comment boxes and an open town hall with office staff. The office reported that 84% of post-event survey respondents would consider attending a future review even if they were not asked to serve as reviewers or present project work.
The review also highlighted the value of using multiple funding mechanisms rather than relying only on traditional laboratory contracts and cooperative agreements. WPTO cited funding opportunity announcements, prize competitions, and technical assistance notices as examples of approaches it could use to address different R&D challenges. The practical lesson is to match the funding structure to the problem while maintaining clear objectives and accountability.
Practical implications for R&D leaders
The 2019 water power peer review demonstrates how a structured external assessment can test the alignment between strategy and project delivery. Its findings point to concrete management priorities: involve end users throughout the work, define success measures, improve data quality and access, and communicate how projects advance or stop.
For technical decision-makers, these practices help make a research portfolio easier to evaluate and its results easier to use. A rigorous review is most valuable when feedback is documented, translated into program changes, and revisited in future planning—not treated as a one-time scoring exercise.