Guidelines for Operationalizing ACL Criteria for Evidence-Based Programs
7 min read
The purpose of the Evidence-Based Program Review Process is to identify new community programs that meet the criteria established by the Administration for Community Living (ACL) for evidence-based programs funded through the Older Americans Act (OAA) Title III-D, and other discretionary funding.
Learn more about the review process.
What are the evidence-based program review process criteria?
Health promotion and disease prevention programs are evaluated through the Evidence-Based Program Review Process based on the following criteria established by the Administration for Community Living:
- Demonstrated through evaluation to be effective for improving the health and well-being or reducing disease, disability and/or injury among older adults; and
- Proven effective with older adult population, using Experimental or Quasi-Experimental Design;* and
- Research results published in a peer-review journal; and
- Fully translated** in one or more community site(s); and
- Includes developed dissemination products that are available to the public.
The Review Council has further defined these criteria below.
Criteria 1: Demonstrated through evaluation to be effective for improving the health and well-being or reducing disease, disability and/or injury among older adults. |
---|
1a. Intervention targets at least one primary behavioral, psychosocial, physical and/or physiological outcome(s) relevant to improving the health and well-being, or reducing disease, disability or injury among older adults (age 60+). Being “relevant” depends on what intervention is being studied, e.g. for physical activity, may include strength or function; for depression, includes depressive symptoms. Changes to knowledge or attitudes is not sufficient. |
1b. Meaningful improvement is demonstrated in at least one relevant primary outcome at least 6 months following the end of the intervention. “Meaningful improvement” is indicated by effect size or other clinically or statistically significant change in outcome using a valid and reliable measure. |
1c. Outcomes are reported as effect sizes or provide data to be able to calculate effect sizes (e.g. mean, SD, N). |
1d. Study provides eligibility criteria and descriptive statistics (demographics, representatives) on study participants to describe the study population (at least half of which are older adults).* *We recommend that descriptive statistics include age; gender; education or income; other chronic conditions; disease severity; recruitment source/setting; enrollment rate. While not a minimum threshold, ACL supports the development of evidence-based programs that a) are broadly applicable and b) reduce or at a minimum do not exacerbate health disparities experienced by under-served populations (e.g. tribal communities, people who speak languages other than English) and fill gaps in health areas (e.g. oral health, nutrition, hypertension). |
1e. Evidence is provided for the safety and tolerability of the intervention as indicated by: (a) minimal/no adverse events directly associated with intervention delivery; and (b) dropout rate is reported for the intervention group and is comparable (or better) than the study’s control group or for similar interventions with similar populations. |
Criteria #2: Proven effective with older adult population*, using Experimental or Quasi-Experimental Design.** |
---|
2a. Intervention is evaluated using an appropriate experimental or quasi-experimental design that includes an appropriate control group. |
2b. The sample size provides sufficient power to determine an effect. |
2c. If more than one study is published, there are consistent trends in study findings (direction and magnitude). |
2d. Information is provided on the implementation of the intervention during the study (e.g., planned and actual frequency, intensity and duration; participation rates). |
2e. Methods are reported in sufficient detail for replication and are appropriate given study design. |
*Title III-D criteria specifically addresses older adults; however, programs submitted through this review process may address older adults. **Experimental designs use random assignment and a control group. Quasi-experimental designs do not use random assignment. See appropriate designs from the US Preventive Services Task Force here on page 74 of the article (page 31 of the PDF). |
Criteria #3. Research results published in a peer-review journal. |
---|
3a. The published study article(s) has gone through a journal’s independent, external peer-review. |
3b. Journal has a published Impact Factor or other published measure of quality. |
3c. Journal is indexed in a national scientific indexing database such as PubMed or Web of Science. |
Criteria #4: Fully translated** in one or more community site(s). |
---|
4a. The program has been delivered with fidelity AND achieved positive outcomes in at least one community site that was not part of the original research study. |
4b. The program developer and/or replication sites can be contacted to learn about program implementation and maintenance. |
4c. The program’s forms can be adapted for local context using appropriate standards (e.g. changes to program setting, population or modality) without removing or significantly altering core functions. Appropriate standards include EBCCP (formerly RTIPs) and HHS/ACF. Forms are “modes of delivery, who delivers, materials/tools, dose, frequency/intensity,” that can be tailored to local literacy, language, culture and learning styles. |
**For purposes of the Title III-D definitions, being “fully translated in one or more community sites” means that the evidence-based program in question has been carried out at the community level (with fidelity to the published research) at least once before. Sites should only consider programs that have been shown to be effective within a real-world community setting. |
Criteria #5. Developed dissemination products that are available to the public. |
---|
5a: The program training is standardized and available on a regular basis, so sites that adopt the program can be trained within 6 months of selecting the program. |
5b: There is a reliable way to contact the program developer or national office to obtain training, manuals, and dissemination materials; to discuss implementation; and to receive timely technical assistance regarding implementation on an ongoing basis. |
5c: Supports and guidelines for implementing the program are readily available, including implementation manual, quality assurance/fidelity guidelines, data collection protocol, anticipated costs for implementing the program, and overall technical assistance. |
5d: Supports for implementing the program are updated on a regular basis. |