Download a printable version of this document (.pdf, 2 pages)
The Top Tier Evidence initiative uses the following solicitation, review, and reporting processes to identify and validate interventions meeting the “Top Tier” or “Near Top Tier” evidence standards.
- We solicit nominations of interventions as candidates for the Top Tier or Near Top Tier in all areas of social policy. See current nomination instructions here. In addition to soliciting nominations, we proactively seek out promising candidate interventions from other sources, such as those rated on various websites of evidence-based programs.
- The standard we use to evaluate candidates for the Top Tier, based on the Congressional legislative language, is: “Interventions shown in well-conducted randomized controlled trials, preferably conducted in typical community settings, to produce sizable, sustained benefits to participants and/or society.”
- In applying this standard, we use the Checklist For Reviewing a Randomized Controlled Trial (linked here), which closely tracks guidance from the U.S. Office of Management and Budget, National Academy of Sciences, Institute of Education Sciences, and other respected research bodies, and reflects well-established principles on what constitutes a high-quality trial (e.g., adequate sample size, low sample attrition, valid outcome measures, intention to treat analysis, and so on). It also addresses the importance of replication in establishing strong evidence – namely, demonstration of effectiveness in at least two well-conducted trials or, alternatively, one large multi-site trial.
- Our main focus, for each candidate intervention, is on assessing whether there is strong evidence that the intervention’s effects are sizable and sustained. However, in some cases, we might also take into account such factors as the intervention’s cost and ease of implementation (e.g., cases where the cost is exceptionally low).
- Over time, we develop short case summaries illustrating the reasoning we use in applying the above standard and guidance to particular studies, thus building a body of additional guidance for reviewers and applicants that is grounded in case-by-case decisions. (This approach – using actual case decisions to grow the body of guidance over time – has been long used by the Food and Drug Administration in its well-established procedures for reviewing randomized controlled trials of pharmaceutical drugs.) These summaries are shown here.
- The standard we use to evaluate candidates for Near Top Tier is: Interventions shown to meet almost all elements of the Top Tier standard, and which only need one additional step to qualify. This category includes, for example, interventions that meet all elements of the standard in a single site, and just need a replication trial to confirm the initial findings and establish that they generalize to other sites. The purpose of this category is to help increase the number of Top Tier interventions, by enabling policy officials and others to identify particularly strong candidates for replication trials whose results, if positive, would provide the final element need for the Top Tier.
- For each viable candidate, we search the literature and contact experts to identify all other well-conducted randomized trials of the intervention (in addition to those initially brought to our attention). Also, for interventions being considered for Top Tier or Near Top Tier on the basis of a limited number of well-conducted trials, we check the literature of high-quality non-randomized studies of the intervention, to look for any patterns of effects that differ from those in the trials (possibly suggesting problems in generalizability) or for any adverse intervention effects.
- The initiative’s Advisory Panel of nationally-recognized, evidence-based researchers and former public officials, decides which interventions to identify as Top Tier or Near Top Tier. A list of Panel members and their titles/affiliations, is shown here. The Panel meets twice per year.
- Interventions that the Panel has identified as Top Tier or Near Top Tier are posted here. Each posting concisely summarizes in plain, jargon-free language: (a) the intervention, and how it differed from what the control group(s) received; (b) the populations and settings in which the intervention was evaluated; (c) the design and implementation of each well-conducted randomized controlled trial of the intervention (tracking the items in the Checklist); and (d) the effects found in each trial on the main outcomes of interest (including any findings of no effect). The posting also notes any relevant limitations of each trial summarized (e.g., small sample size). Prior to posting, we ask the intervention provider and/or researchers who conducted the trials to review each draft write-up for any inaccuracies or items we may have missed.
- For each nominated intervention not approved as Top Tier or Near Top Tier, we contact the nominator informally to convey the result and briefly explain the reason (e.g., the studies did not use random assignment, or the Panel awaits longer-term follow-up to see if effects are sustained).
- We also post on the initiative’s website a complete list of all interventions that the initiative has reviewed, to enable readers to identify and alert us to any viable candidates for Top Tier or Near Top Tier that we may have missed. The list is posted here.