Evidence-based Practices In Depth
Overview
Education-to-workforce (E-W) decision makers must use data to drive action. In many cases, the data may point to a need to address system conditions and distribute resources more equitably, such as increasing funding, hiring more advisors, or offering more early college classes in schools. However, system conditions are not the only levers for change. Often a new practice, program, policy, product, or intervention may be needed. For example, students who have not yet met grade level expectations may need individualized support, such as an academic intervention, tutoring, or summer programs.
To help E-W decision makers determine which practices are most likely to be effective, the framework offers summary guidance on how to vet and select practices that meet evidence standards and the unique needs of their communities. This guidance is accompanied by 26 illustrative examples of evidence-based practices that have been shown to move the needle on key individual outcomes and milestones and related system conditions. These include examples from the U.S. Department of Education’s What Works Clearinghouse (WWC), the Every Student Succeeds Act (ESSA), and the U.S. Department of Labor’s Clearinghouse for Labor Evaluation and Research (CLEAR).
What is an Evidence-based Practice?
We define an evidence-based practice as a practice, program, policy, product, or intervention that is informed and supported by strong evidence demonstrating consistent, positive impacts on individual outcomes.
Three types of evidence of impacts, from weakest to strongest:
Descriptive. Summarizes how the outcomes of individuals supported by the practice changed over time, but without comparing to outcomes of individuals not supported by the practice. Example: Evidence showing outcomes improved after the practice was implemented.
Correlational. Suggests a relationship between the practice and outcomes by comparing outcomes of individuals supported and not supported by the practice, but it does not rule out other potential causes of the difference. Example: Evidence showing individuals supported by the practice had better outcomes than those who were not supported by the practice over the same time period.
Causal. Compares “apples to apples” outcomes for individuals supported by the practice and those who were not, by ensuring the only difference between the two groups is the practice itself. Example: Evidence showing individuals randomly assigned to be supported by the practice had better outcomes than those randomly assigned to a comparison group.
Newer practices or those more difficult to evaluate may be promising but understudied. As a result, the strength of the available research is an important consideration for assessing the evidence for a practice, program, or intervention, but it is not the only thing to consider. It is also important to consider other aspects of the available evidence, such as the number of studies conducted; whether they demonstrate consistent, positive findings; and whether they were conducted in multiple, diverse settings. We recommend E-W decision makers consider several factors in assessing the strength of evidence available (see Chapter IV of the full report for more information).
How to Select an Evidence-based Practice
We recommend following a four-step process to select a practice:
- Diagnose the need by conducting a root cause analysis. Before assessing possible practices and their evidence base, decision makers should have a clear understanding of the need they are trying to address (for example, reducing disparities in early college coursework and credit completion). The E-W Framework’s data equity principles offer guidance and links to resources on how to disaggregate data to analyze disparities, examine social and historical contexts, engage community members in the decision-making process, and develop data-driven solutions.
Identify potential evidence-based practices. Once a clear need has been identified, decision makers should consider whether the root cause analysis suggests that need can be addressed by improving system conditions or supports (for example, improving course placement policies, increasing the number of dual-credit courses offered across subjects, and subsidizing exam fees), or if the need requires implementation of a new practice (such as after-school tutoring to help students manage coursework and prepare for exams). If a new practice is needed, decision makers should examine the quality, quantity, and relevance of studies that make up the evidence base.
We recommend developing a logic model for how the potential intervention is expected to improve outcomes, drawing on research to hypothesize cause and effect. Creating a logic model is a useful and recommended exercise for selected interventions regardless of the strength of the available evidence, as this process helps decision makers identify the necessary inputs and outputs that make an intervention “work.”
- Weigh the evidence base against the feasibility of implementation and select a practice. Decision makers should consider the list of potential practices they’ve identified and weigh the evidence base for each practice against the feasibility of successfully implementing the practice for in their context. Community members and practitioners should be a part of this process, as they may have the best understanding of how potential practices will play out in the local context. Key implementation factors to consider include whether funding is available to enact and sustain the practice; whether there are qualified staff to implement it as designed (including leadership to oversee and facilitate implementation and resources); what additional training or support staff may need to build their capacity; and whether there is buy-in from key community members, such as teachers, students, and families. It is also important to consider whether any adaptations to the practice selected will be needed, given the implementation context, and how those adaptations might affect effectiveness.
- Plan and monitor the implementation and outcomes of the practice. Once a practice has been identified, decision makers must plan how to implement the practice. This includes planning for resources, staffing, professional development, communication, and other facets of execution, as well as planning for how to monitor implementation and outcomes. Decision makers should track progress on the key indicator(s) they used to identify the need, as well as the leading outcome or related system indicators expected to improve if the practice is being implemented effectively. This step should occur continually to ensure the practice is consistently being implemented well, that there is progress on key indicator(s), and whether a new cycle of inquiry is needed.