How a National Healthcare Training Program's Impacts Vary Locally

This brief summarizes the estimated impacts of the HPOG 2.0 grants, both overall and for each local grantee program. The impact is calculated as the difference between the treatment and control group outcomes (where the treatment group had access to HPOG and the control group did not.)

HPOG 2.0 local programs randomly assigned eligible applicants into a control group (whose members were not offered access to HPOG) or a treatment group (whose members were offered access to HPOG). Because these two groups are not systematically different, the difference between outcomes is a strong estimate of the “impact” of being offered access to HPOG’s training and supports.

For the evaluation’s overall impact analysis, the authors computed the impact as the difference between the treatment group’s and control group’s average outcomes (as is standard practice, adjusting for sample member characteristics at baseline).

Although the evaluation uses other data and measures, this brief is based on analyses for measures that use data from two sources:

• HPOG 2.0 Short-Term Follow-Up Survey was fielded to about one-fifth of the study sample and had a 76 percent response rate. The survey provided 15-month follow-up data on three measures reported in this brief: 1) educational progress, defined as completed or still in training; 2) months in training, defined as a count of months with any training; and 3) employment in healthcare defined as reporting being in a job in the healthcare field.
• National Directory of New Hires data is available for the full study sample, providing data on one measure reported in this brief: Quarterly earnings defined as earnings as of 15 months of follow-up, or the fifth quarter.

(This report is 7 pages long.)

The researchers identified the following findings:

• Collectively, the local HPOG 2.0 programs—and most of them individually—are improving educational progress in the short term.
• Collectively, the local HPOG 2.0 programs—and about half of them individually—increase the number of months in training.
• In the short term, the HPOG 2.0 programs modestly increase healthcare employment.
• In the short term, one of the local HPOG 2.0 programs increases earnings, one decreases earnings, and the evidence is ambiguous for the rest.

Certainly, local programs vary on the many potential factors that might contribute to program impacts, such as:

• Length of supported programs
• Screening of applicants or specific types of participants served
• Quantity and quality of counseling
• Instructional quality
• Proactive case management
• Availability and use of various support services
• Alignment of offered programs with local labor demand
• Availability of alternative training programs
• Local demand for healthcare workers
• Local wages

The researchers identified the following recommendations:

Information on the difference that each local program made on its participants (i.e., its “impacts”) can be used for at least three purposes:

• To guide funding decisions, both by the federal government and potentially by other funders, e.g., “we should be investing in this program over that program.”
• To guide individual applicants’/trainees’ enrollment decisions, e.g., “I should enroll in this program over that program.”
• To identify local programs that might be worthy of replication or further study, e.g., “the current version of these programs is worthy of further study and perhaps replication; those programs only after retooling and another round of evaluation to see whether the changes increase their effectiveness.”

• Last Updated:
• Created:
• Resource Publication Date: 2022
• Author(s): Peck, L.; Judkins, D.; Kolenikov, S.
• Funding Source: Administration for Children and Families (ACF), U.S. Department of Health and Human Services
• Resource Availability: Publicly available
• Posted by:
• Posted in: Workforce System Strategies