Ryan White HIV/AIDS Program Part A Manual

Outcomes evaluation looks at the effectiveness of a service or program in achieving its intended results. It can help Ryan White programs determine if they are making a difference in the lives of people living with HIV/AIDS (PLWHA). Documentation of outcomes can be used in multiple ways, including:

• Ensuring and improving service quality

• Helping guide program planning

• Setting priorities and allocating resources, and

• Securing funding from public and private resources.

Policy and funding decisions at the Federal level are increasingly being determined by outcomes. This includes the Ryan White HIV/AIDS Program, where outcomes evaluation is used to document changes in HIV-related health disparities for racial/ethnic minority populations served.

In their work with providers, Eligible Metropolitan Areas/Transitional Grant Areas (EMAs/TGAs) should include outcomes data requirements in their Requests for Proposals (RFPs) and provider contracts so they can document results. Use of standardized outcomes data—such as HRSA/HAB's core clincial performance measures—can show how standards of care are being met. This approach demonstrates how outcomes evaluation is related to Clinical Quality Management, which programs use to identify how service delivery impacts health status outcomes and to make needed program and service changes.

In addition to outcomes evaluation requirements in the Ryan White legislation, outcomes evaluation is driven by other considerations:

• The Government Performance and Results Act (GPRA) directs all Federal programs to document progress towards specific measurable objectives

• The Inspector General has recommended establishment of evaluation systems at the national, State, and local levels to support outcomes evaluation, and

• Municipalities often require documentation of program impacts, whether they are supported solely through Ryan White HIV/AIDS Program resources or by a combination of public and private funding sources.

Section 2602(b)(4)(C) of the Ryan White legislation requires Part A planning councils to "establish priorities for the allocation of funds within the eligible area, including how best to meet each such priority and additional factors that a grantee should consider in allocating funds under a grant" based on factors that include:

"(ii) demonstrated (or probable) cost effectiveness and outcome effectiveness of proposed strategies and interventions, to the extent that data are reasonably available."

Section 2602(b)(4)(E) states that the planning council may, at its discretion, "assess the effectiveness, either directly or through contractual arrangements, of the services offered [in the EMA/TGA] in meeting the identified needs."

Section 2604(d) requires that support services funded under Part A be "needed for individuals with HIV/AIDS to achieve their medical outcomes," which are defined as "those outcomes affecting the HIV-related clinical status of an individual with HIV/AIDS."

The HIV/AIDS Bureau's Division of Service Systems (HAB/DSS) believes that EMAs/TGAs need to document the impact of Ryan White HIV/AIDS Program funds on improving access to quality care including medical treatment. EMAs/TGAs also need to ensure that they have in place clinical quality management and evaluation mechanisms to assess the impact of Ryan White HIV/AIDS Program resources on improving health-status outcomes.

HAB does not require a specific type of outcomes evaluation. Every planning council and grantee may determine for itself what outcomes indicators and approaches it will use in evaluating program results. However, many Ryan White programs have indicated a need for guidance from HAB on outcomes evaluation. Several outcomes evaluation guides and evaluation monographs have been developed to address this need. Materials from the Outcomes Evaluation Technical Assistance Guide: Getting Started are used in this chapter.

Outcomes evaluation is often a part of Clinical Quality Management. HAB has developed a set of recommeded performance measures. Many include some outcomes measures .

In their applications, as part of their Clinical Quality Management discussion, Part A grantees are expected to describe "ongoing activities and/or specific concrete plans to use data to show how Part A-funded services, including support services, are improving HIV-related clinical health outcomes of PLWHA in the EMA/TGA."

Outcomes evaluation requires human and financial resources. Prior to the Ryan White CARE Act Amendments of 2000, administrative caps could make it difficult for grantees and planning councils—especially those in small EMAs/TGAs—to carry out program evaluation and clinical quality management activities with Ryan White HIV/AIDS Program funds. The 2000 Amendments specifically permit the use of grant funds for evaluation as part of the required Clinical Quality Management programs, and the 2006 Amendments continue this provision under Clinical Quality Management (CQM). Part A grantees may spend up to 5 percent of grant funds or $3 million, whichever is less, on such programs.

Obtaining Clinical Data on Client Health Status and Links to Primary Care

HIV/AIDS care is now based largely on a medical model of service delivery designed to reduce morbidity and mortality. Determining the effectiveness of Ryan White HIV/AIDS Program services therefore requires understanding whether, overall, such services are helping clients to access and remain in primary care and realize improved health status. This means that outcomes evaluation for almost any Ryan White HIV/AIDS Program-supported service category, from case management to transportation, needs to include an indication of whether program participation can demonstrate linkages to primary care, since it is medical care that is most directly linked to improved clinical outcomes. Provider access to data on client health status is improving. In addition, providers can document their ability to link clients to primary care using other data measures (e.g., helping them enter primary care, keep appointments, adhere to medications).

Because the Ryan White HIV/AIDS Program is the "payer of last resort," many Ryan White clients also get care through other payers like Medicaid, Medicare, Veterans Affairs,and private insurance. Ryan White pays for care not otherwise covered by these other payers. Obtaining client-based clinical data from such sources is a particular challenge, and providers of other services need to identify other types of outcomes data that are more accessible.

Ryan White HIV/AIDS Programs are collecting client-level data as of January 2009. This system will capture information necessary to demonstrate program performance and accountability, including data on client health status, and will greatly facilitate data availability for evaluation of outcomes related to client health status.

Evaluating Systems of Care

Part A programs typically help to support a system of HIV/AIDS services, and grantees and planning councils want evaluation data that can guide decision making about program priorities and resource allocations. Ideally, this means understanding the outcomes associated with not just one category of service (e.g., primary medical care or case management) but rather a combination of primary care and support services-or an entire system of care.

Evaluation linking supportive or enabling services to health outcomes often requires some form of system-level evaluation. This has been extremely challenging, in part because of the lack of client-level data.  Once Ryan White agencies have fully implemented their client-level data systems, this type of evaluation should become more feasible.

See Outcomes Evaluation TA Guide and see other TA resources in the TARGET Center .