Machine Learning in Workforce Development Research: Lessons and Opportunities
About
This exploratory study was designed to explore how to use machine learning methods, including web scraping, supervised learning, and natural language processing, to collect and analyze a large volume of data on career pathways program implementation.
The study design included two phases: (1) an initial data collection phase to identify an approach and techniques for collecting data, and (2) an analysis phase to synthesize the data. While only the first phase was completed as part of this project, the experience yielded important lessons for future machine learning studies.
This brief summarizes lessons learned from utilizing machine learning to study the implementation of career pathways programs. First, this brief describes the research questions that guided the study and summarizes the machine learning methods designed for the data collection and analysis activities, including study limitations and challenges encountered. It then provides lessons learned on using machine learning methods for social science research. Finally, the brief discusses strategies for using these methods in future workforce development projects and other areas, particularly federally funded efforts.
(This resource is 19 pages in length.)
Major Findings and Recommendations
The researchers identified the following selected findings:
- Machine Learning can be a powerful tool in the right context. The term “machine learning” covers a wide array of analytic approaches and tools which must be appropriately tailored to the research questions of interest and the data that will be analyzed.
- Any given application of machine learning methods involves some risk, requiring an understanding of their limitations and expected results. Machine learning methods have been incorporated into the day-to-day work of fields such as information technology and medicine, but they have not been widely applied in workforce development or other social science research. Based on the experience of this project, work is ongoing to determine how these methods can address important questions in workforce development research.
- In the context of implementation research, machine learning methods may have difficulty replicating the detail and nuance capable of human researchers. While machine learning algorithms’ ability to process large quantities of data—known as big data—is valuable in some contexts, machine learning algorithms will struggle to match the nuance of analyses by human researchers.
- Some projects should expect to dedicate substantial time and resources to defining key terms, creating and applying a coding framework, and interpreting results. Lack of standardization around key terms in workforce development can present challenges for implementing certain machine learning tasks. Based on this project’s experience, projects applying machine learning methods designed to provide information on concepts without a set of standard definitions and rules may face similar issues.
The researchers identified the following recommendation:
- The experience of this project suggests that policymakers, government agencies, and other stakeholders interested in using machine learning may first want to execute a “discovery phase” to maximize the efficacy of future machine learning efforts. Specifically, during the discovery phase, data scientists and subject matter experts could collaborate to generate research questions of interest, explore and assess potential data sources, and create design options that could appropriately apply machine learning techniques to answer questions of interest.
Materials
Comments
Content Details
Methodology:
Topics:
Target Populations:
Programs:
Geographic Locations:
Industry Sectors:
- Last Updated:
- Created:
- Resource Publication Date: 2021
- Author(s): De La Rosa, S. M., Greenstein, N., Schwartz, D., Lloyd, C.
- Funding Source: U.S. Department of Labor
- Resource Availability: Publicly available
- Posted by: Wesley Peterson
- Posted in: Workforce System Strategies