The National Institutes of Health (NIH) funds training programs to increase the numbers and skills of scientists who obtain NIH research grants, but few programs have been rigorously evaluated. The sizeable recent NIH investment in developing programs to increase the diversity of the NIH-funded workforce, implemented through the Diversity Program Consortium (DPC), is unusual in that it also funds a Consortium-wide evaluation plan, which spans the activities of the 10 BUilding Infrastructure Leading to Diversity (BUILD) awardees and the National Research Mentoring Network (NRMN). The purpose of this article is to describe the evaluation design and innovations of the BUILD Program on students, faculty, and institutions of the 10 primarily undergraduate BUILD sites.
National reports call for improving America’s leadership in scientific research, accelerating degree attainments, and diversifying the scientific workforce to foster innovation. However, slow progress and persistent disparities across growing U.S. populations are evident on key science workforce indicators, from degree attainment to career achievements. The purpose of this article is to provide a conceptual basis and overview of a national effort funded by the National Institutes of Health (NIH) that advances inclusive science practice and systemic change. We introduce the context, features, and rationale that drive practice and evaluation in the Diversity Program Consortium (DPC) approach, which is an experimental program to implement and evaluate evidence-based and novel practices to expand and diversify the biomedical workforce.
The National Institutes of Health (NIH)-funded Diversity Program Consortium (DPC) includes a Coordination and Evaluation Center (CEC) to conduct a longitudinal evaluation of the two signature, national NIH initiatives - the Building Infrastructure Leading to Diversity (BUILD) and the National Research Mentoring Network (NRMN) programs - designed to promote diversity in the NIH-funded biomedical, behavioral, clinical, and social sciences research workforce. Evaluation is central to understanding the impact of the consortium activities. This article reviews the role and function of the CEC and the collaborative processes and achievements critical to establishing empirical evidence regarding the efficacy of federally-funded, quasi-experimental interventions across multiple sites. The integrated DPC evaluation is particularly significant because it is a collaboratively developed Consortium Wide Evaluation Plan and the first hypothesis-driven, large-scale systemic national longitudinal evaluation of training programs in the history of NIH/National Institute of General Medical Sciences.
The NIH-funded National Research Mentoring Network (NRMN) aims to increase the representation and success of underrepresented groups (URGs) in biomedical research by enhancing the training and career development of individuals from diverse backgrounds, communities, and cultures. The national scope of NRMN, its wide array of innovative programs in mentor and mentee matching and training across the career spectrum, requires a collaborative evaluation strategy that addresses both internal and external evaluation needs. Due to the variety of programs implemented for each target group, the NRMN program is responsible for its own process and short-term outcome evaluations and the national Coordination and Evaluation Center (CEC) is responsible for assessing the medium and long-term effectiveness of the implemented strategies and program sustainability. Using a collaborative, utilization-focused evaluation framework, both internal NRMN evaluators and the CEC are working to translate findings into information that can be used to make both short term and long-term decisions about the efficacy and reach of the NRMN model. This important information can then inform efforts to institutionalize the current programs and potentially replicate them elsewhere.
Science identity represents the extent to which students conceive of themselves as scientists and is one of the hallmarks of the NIH Diversity Program Consortium (DPC).a Data from the 2017 Student Annual Follow-up Survey (SAFS) indicates that BUILD students in biomedical majors have a higher average science identity score than non-BUILD biomedical students at the same institutions. The higher science identity among BUILD students occurs for most racial/ethnic groups and for all items in the science identity scale. Given the importance of science identity to continuing on the biomedical career pathway, these results suggest that BUILD students are more likely to persist in science majors and careers than other biomedical science students at their institutions.
A major NIH initiative is underway to determine the most effective ways to engage students from diverse backgrounds in biomedical research, and to prepare students to become potential future contributors to the NIH-funded research enterprise. The 10 grantees vary widely, from Historically Black Colleges and Universities (HBCUs), where most biomedical graduates are from underrepresented groups, to large public schools with moderate representation of underrepresented groups among their biomedical graduates. Graduation data from four years pregrant and two early years of the BUILD program show a general trend of constant proportions of underrepresented minority (URM) students graduating in biomedical majors, with a few institutions showing a rapid growth in the numbers that reflects overall increases in their number of students graduating with biomedical degrees over that time. Since few graduating students would have been involved in the new programs in the period shown, these trend data are best considered baseline or preintervention information that can be used in future years to help assess the institutional impact of the BUILD grants.
The National Research Mentoring Network (NRMN) provides researchers across all career stages with mentorship and professional development programming that emphasizes the benefits and challenges of diversity, inclusivity and culture (https://nrmnet.net). Individuals register on the NRMN online portal to use virtual mentoring, access online resources, or sign up for a wide variety of training events. When enrolling through the online portal, all registrants are required to identify as a mentor or mentee, regardless of how they use NRMN.
Within the biomedical disciplines, faculty-student mentorship is a key component for success. Mentors provide support, motivation, and advice to students that increase their social capital in research, as well as their skills and identities as scientists. Increasing mentoring self-efficacy is one of the hallmarks (FAC-3) that serves as an intermediate outcome for the NIH Diversity Program Consortium (DPC).1 Data from the 2016–2017 HERI Faculty Survey2 finds that biomedical faculty respondents at the 10 BUILD sites report having higher mentoring self-efficacy scores if they are involved with the BUILD program than if they are not. This suggests that BUILD students are likely to have access to higher quality mentoring than other biomedical students.