A Conceptual Framework for Studying the Sources of Variation in Program Effects

“Evaluations of public programs in many fields reveal that (1) different types of programs (or different versions of the same program) vary in their effectiveness, (2) a program that is effective for one group of people might not be effective for other groups of people, and (3) a program that is effective in one set of circumstances may not be effective in other circumstances. This paper presented a conceptual framework for research on such variation in program effects and the sources of this variation. The framework was intended to help researchers—both those who focus mainly on studying program implementation and those who focus mainly on estimating program effects—see how their respective pieces fit together in a way that helps to identify factors that explain variation in program effects and thereby support more systematic data collection on these factors. The ultimate goal of the framework was to enable researchers to offer better guidance to policymakers and program operators on the conditions and practices that are associated with larger and more positive effects” (p.iii). (Abstractor: Author)

Researchers should be aware of and collect data on a program’s treatment plan, the implementation process, and the theory behind how the program is expected to produce its intended effects. “For a study of program efficacy (its potential effects under favorable conditions), researchers should collect the preceding information as well as data on the observed treatment contrast and observed program effects. For a study of program effectiveness (its actual effects in multiple locations under normal operating conditions), researchers ought to attempt to collect the preceding information, especially data on the program treatment contrast. This can be used to help explain any observed differences in program effects between the efficacy and effectiveness tests and any observed differences in program effects across sites, contexts, and client types in the effectiveness test” (p.41). (Abstractor: Author and Website Staff)

• Last Updated:
• Created:
• Resource Publication Date: 2013
• Author(s): Weiss, Michael J.; Bloom, Howard S.; Brock, Thomas
• Organizational Author(s): MDRC
• Funding Source: William T. Grant Foundation
• Resource Availability: Publicly available
• Posted by: Justin Venneri
• Posted in: Workforce System Strategies