Evaluation of Community-Wide Interventions

"Most community-wide initiatives are fundamentally incompatible with commonly used evaluation methods, and random assignment frequently represents a very poor evaluation design in this setting... [For example, these initiatives] are 1) designed to capitalize on spillover effects from participants in programs on those who never participate; 2) offer linked sets of services and case management rather than a single, specific treatment or intervention..; and 3) are tailored to specific local conditions and capacity in a way that makes each initiative unique in its particular mix of program components.... However, there are evaluation methods that can address these challenges.... Innovative methods, many of which use propensity score reweighting methods in novel ways, can address the fundamental challenges" of evaluating community-wide initiatives (p.1). (Abstractor: Author and Website Staff)

“A credible evaluation of community-wide initiatives uses an evaluation design that allows for spillovers, accounts for dynamic adjustments of treatment to intermediate outcomes, and correctly models the nature of the treatment being studied. Describing the treatment requires knowing details about implementation on the ground, and comparing the results of treatment in vastly different places is difficult at best. However, a synthetic control approach married to a g-estimation design can address a single site’s effect on outcomes, and a meta-analytic approach that uses propensity score methods to adjust for differences in population served holds out the hope of aggregating information across sites” (p.4). (Abstractor: Author)

• Last Updated:
• Created:
• Resource Publication Date: 2013
• Author(s): Nichols, Austin
• Organizational Author(s): Urban Institute
• Funding Source: Funding source not identified
• Resource Availability: Publicly available
• Posted by: Laura Aron
• Posted in: Workforce System Strategies