Process Study of the U.S. Department of Labor's "Pay for Success" Pilots in Two States: Final Report

The DOL PFS projects involved two state workforce agencies: the Massachusetts Executive Office of Labor and Workforce Development (EOLWD) and the New York State Department of Labor (NYSDOL), which would receive approximately $12 million each if the projects met key impact targets on recidivism and employment. The EOLWD targeted young male parolees and probationers, while the NYSDOL focused on high-risk adult parolees. The impacts would be determined by evaluations using an experimental design involving a randomized controlled trial.

This report is a process study of the two PFS pilots that documents project implementation and provides information on the PFS approach to policymakers and program administrators.

The PFS model was envisioned as an approach for potentially helping payers—often state and local governments—fund services and produce long-term benefits, such as reduced programmatic costs, more efficient spending, and better outcomes for people and communities. PFS potentially offers governments and other payors a way to finance services that might not otherwise be feasible due to limited budgets. Beyond this primary goal, PFS initiatives are thought to have the potential to influence how payers will operate services in the future by creating incentives for innovation, effective preventive or proactive services, flexible service delivery, and flexible management structures.

Selected key components of the two PFS pilots were:

• Both pilots opted to address the problem of recidivism experienced by individuals who had been formerly incarcerated.
• Under a PFS model, the sponsor (the government in this case) hires an intermediary to develop the PFS project, manage all aspects of it, coordinate the roles and responsibilities of all stakeholders, and secure the private sector and philanthropic financing.
• Both pilots opted to conduct an experimental evaluation involving a randomized control trial (RCT).
• Both projects selected service providers that had been in operation for many years and had solid reputations in their field and the capacity to manage data and participate in an RCT.
• Each pilot established the outcome measures that would be used to estimate program impacts and formulas for calculating the level of effectiveness of the interventions and the estimated public budget savings.

(This report is 78 pages long.)

(This study took place between 2017–2018.)

The researchers identified the following findings:

• Partners pointed to many positive aspects of the pilots and are optimistic about achieving outcome targets in the second phase despite not achieving target outcomes in the first phase.
• It was challenging to conduct the projects within the four-year timeframe of the DOL grants.
• The RCT evaluations were successfully implemented and approved by their respective validators, demonstrating the feasibility of their use in the PFS context. However, conducting the RCTs requires significant organizational expertise and experience.
• The PFS partnership structure is crucial to a viable PFS pilot.
• PFS projects require significant time and resource investment by the key partners.
• Partners had different perspectives on the costs of the PFS pilots.
• The PFS partnership structure was a crucial aspect of a viable PFS project.

The researchers proposed the following recommendations:

• If RCT methodology is required, the government sponsor should be specific about design requirements and specify how results should be presented.
• Implementing a high-quality, rigorous evaluation of the interventions proved to be the most challenging aspect of the two projects and may have contributed to the inability to detect positive impacts and pay investors under the DOL pilots. This indicates the importance of carefully planning and supporting this aspect of the PFS model in future efforts.

• Last Updated:
• Created:
• Resource Publication Date: 2020
• Author(s): Kean, E.; Schwartz, D.; Boland, B.; Sarna, M.; Martinson, K.
• Funding Source: U.S. Department of Labor, Employment and Training Administration
• Resource Availability: Publicly available
• Posted by: Wesley Peterson
• Posted in: Workforce System Strategies