Abstract

The use of combination antiretroviral therapy has dramatically reduced morbidity and mortality attributed to human immunodeficiency virus type-1 (HIV) infection and the acquired immunodeficiency syndrome (AIDS). Current literature suggests that adherence rates of 95% or better are necessary for optimal therapeutic outcomes; however, adherence is difficult to maintain. Few evaluations of adherence support programs have been conducted, and most have focused on short-term improvements in adherence. We describe our experiences and insights regarding the use of a cross-site evaluation methodology to determine the effectiveness of ART adherence support interventions implemented in 12 centers in the U.S. These adherence support interventions were targeted toward underserved populations diagnosed with HIV in the United States. We also delineate the benefits and challenges of this approach, and suggest implications for clinical practice.

TOP

Acknowledgments

This research was supported by funds from the Health Resources and Services Administration, HIV AIDS Bureau, Special Projects of National Significance, Grant #6 H97 HA 00128-03 05, CFDA # 93.928. The authors gratefully acknowledge the assistance of the adherence support program clients and staff at the Health Services Center, Hobson City, AL; Chase-Brexton Health Services, Baltimore, MD; Dimock Community Health Center, Roxbury, MA; Harlem Hospital Center, New York, NY; Johns Hopkins University School of Medicine, Baltimore, MD; Joseph Mailman School of Public Health/Columbia University, New York, NY; Mission Neighborhood Health Center, San Francisco, CA; Multnomah County Department of Health, Portland, OR; North Broward Hospital District, Ft. Lauderdale, FL; NY State AIDS Institute, New York, NY; State University of New York-Downstate Medical Center, Brooklyn, NY; Action Point Center, San Francisco Department of Public Health, San Francisco, CA; Helena Hatch Special Care Center, Washington University School of Medicine, St. Louis, MO.

TOP

Introduction

The analysis of cross-site data is a major challenge, requiring the careful weighing of the advantages and disadvantages of different analytic approaches. The analysis is not straightforward and there is no cookie cutter approach. There is usually more than one analytic approach to answer a particular evaluation question, and the tradeoffs in selecting the analysis must be laid out. The complexity of pooling data from multiple sites, with variations in setting, provider, and target population characteristics, intensifies the problems encountered in single-site, single intervention study. This paper discusses the analytical and statistical approach to cross-site data, and provides exemplary analyses, with an eye to providing practical guidance to researchers about how to approach such data.

TOP

Evaluation Strategy Overview

We aimed to evaluate the effectiveness of adherence programs taking into account three major differences in programs: Core adherence support intervention components; target population characteristics; intervention setting and context (See Table 1).
Taxonomy of Adherence Support Interventions. The adherence support interventions typically were multifaceted and incorporated components of: (1) HIV and adherence-related education (100% of sites; education about HIV disease and medications, reminder strategies and tools, identification of promoters and barriers to adherence, problem-solving strategies); (2) case management/social services (83% of sites; coordination of mental health, substance abuse, psychosocial, entitlements, housing, and transportation services); (3) readiness training (42% of sites; preparatory training and education, psychosocial support, participant-provider relationship-building, mock trials that allow rehearsal of drug-taking regimen); (4) peer-based counseling (25% of sites; provision of adherence support, education, outreach, referral services, advocacy, and AIDS counseling one-to-one or in groups, by individuals living with HIV); (5) pharmacist assistance (17% of sites; regimen review, management of side effects and toxicity); and (6) modified directly observed therapy (DOT) (8% of sites; one site conducted on-site dispensing of ART with clinician observation of pill ingestion, monitoring, support, problem-solving) to give participants practice in taking ART.
Populations Served. The target populations in this evaluation study reflect the face of the US AIDS epidemic. All of the programs served highly vulnerable populations with HIV who historically have difficulty in accessing care. Approximately two-thirds (67%) of the programs explicitly targeted participants with substance abuse problems and 58% targeted participants with psychiatric co-morbidities, both which add to their medication-taking difficulties. One-third (33%) targeted homeless populations. Half of the programs were in urban, freestanding community-based clinics in eight states.
Adherence Support Staff and Context. At all sites, adherence support services were delivered by a multidisciplinary health care provider team, including various constellations of physicians, nurses, nurse practitioners, pharmacists, psychologists, case managers, social workers, health educators and peers. The primary institutional bases for the program differed across sites: five of the programs were hospital-based, six were in community health centers, and one was in a freestanding non-clinic-based community center. Programs were located in practically every region of the US, with a domination of sites in the Northeast.

Table 1. Characteristics of the 12 Adherence Support Programs

Program Characteristic	Number of Programs	Percent of Programs**
Core intervention components*
HIV- and Adherence-Related Education	12	100
Case Management/Social Services	10	83
Readiness Training	5	42
Peer-Based Counseling	3	25
Pharmacist-Based	2	17
Modified Directly Observed Therapy	1	8
Target populations
Substance Users	8	67
Psychiatric History	7	58
Homeless	4	33
Predominantly MSM	1	8
Predominantly Women	1	8
Children/Adolescents	1	8
Caregivers	1	8
Settings
Community Health Centers	6	50
Hospital-Based	5	42
Community-Based Organization	1	8
Geographic regions
Northeast	4	33
Mid-Atlantic	2	17
West Coast	2	17
South	1	8
Southeast	1	8
Midwest	1	8
Pacific Northwest	1	8

* Program core; these components may be part of the interventions in other sites, but not the primary thrust.
** Percentages do not total to 100 in all cells due to overlapping categories and to rounding

TOP

Background

The use of combination antiretroviral therapy (ART) has dramatically reduced morbidity and mortality related to the acquired immunodeficiency syndrome (AIDS). [Ref. 1-6] It also has highlighted the crucial role that adherence plays in effective treatment of human immunodeficiency virus type-1 (HIV) disease because the potential benefits of ART depend on sustaining a high level of adherence.[Ref. 7-9] Recent findings suggest that adherence rates of 95% or better are necessary for optimal therapeutic outcomes, [Ref. 10] yet most studies of antiretroviral regimen adherence find that most clients report taking between 56% and 88% of their doses. [Ref. 11-13] People on ART regimens may perceive side effects [Ref. 14] and rigorous dosing schedules as impediments to their quality of life more so than HIV infection itself, [Ref. 15] particularly among persons who contend with mental illness, substance abuse and the complexities of poverty or homelessness. Some studies of populations with complex problems indicate that adherence to ART decreases with increasing length of time on ART. [Ref. 16]

Although the complexity of the treatment regimen and its potential side effects are major challenges to adherence, other factors impact adherence. [Ref. 17] Individual patient barriers to adherence include active substance use; [Ref. 19-24] depression; [Ref. 13, 19, 25-30] and other psychosocial factors such as lower levels of perceived social support, [Ref. 27, 31-34] lower treatment adherence self-efficacy; [Ref. 15, 29, 31, 34-36] active psychiatric illness; [Ref. 10] psychological stress; [Ref. 13] "HIV burnout"; [Ref. 37] a history of conflicted social interactions and abuse; [Ref. 23, 29] and poor coping skills. [Ref. 13, 28-29] Relevant dimensions of the relations between the patient and the provider include trust or its absence, [Ref. 11, 38-40] communication, and provider experience in caring for HIV patients. Systemic characteristics of the health care setting include interruptions in health insurance coverage, barriers to prescription refills, difficulty with transportation to the clinic, inconvenient hours, and length of waiting time, and unstructured psycho-educational approaches to readiness training education.[Ref. 41-45]

To address the challenges patients face with their HIV treatment regimens, many health agencies and community-based organizations in the United States (US) have initiated programs to support antiretroviral medication adherence, even though program implementation has forged ahead of information on intervention efficacy and effectiveness. Interventions have included (1) patient-oriented strategies, involving patient education and support; [Ref. 46] (2) treatment-related strategies, such as directly observed therapy (DOT), reminder systems and regimen simplification; [Ref. 47-48] and (3) multifaceted interventions ranging from counseling by a pharmacist, written schedules and pillboxes to integrated mental health, substance abuse treatment and HIV treatment education.[Ref. 49] Additional strategies have included incentives and enablers [Ref. 43] and readiness assessment.

Few evaluations of antiretroviral adherence support programs have been conducted to date, and those that have been accomplished have focused on short-term improvements in adherence.[Ref. 46-48] These studies have been characterized by small sample sizes and a limited range of intervention types and settings [Ref. 35, 50-54] resulting in a limited ability to generalize across clinical settings and populations. Evaluations also have failed to assess the separate effects of each component of complex programs and the synergistic contributions of participants, program, and setting features. Many evaluations have not assessed the feasibility of support interventions or adherence after completion of the intervention. Studies that systematically evaluate and compare adherence support programs used in clinical settings are needed.

Increasingly, cross-site evaluations, in which several clinical programs are evaluated using common methods have been used to enhance the generalizability and utility of the findings. Recent examples include the evaluation of follow-up care in soon-to-be-released HIV-positive inmates in correctional facilities, palliative care models for people with end-stage HIV disease, and HIV-positive persons in hospital and community-based health care programs.[Ref. 55-57] In this paper, we describe a cross-site approach to assess the effectiveness of a diverse, non-standardized set of clinical HIV antiretroviral medication adherence support interventions at sites across the US, its advantages and limitations, and the implications for evaluation and intervention practice in clinical programs.

TOP

The Context of the Cross-Site Evaluation

In 1999, the Health Resources and Services Administration (HRSA), HIV/AIDS Bureau (HAB), Special Projects of National Significance (SPNS) funded a national demonstration project to evaluate the effectiveness of adherence support programs in the US. The project mission was to evaluate innovative service models targeted to improving adherence to ART among underserved, disenfranchised HIV-positive populations. At the onset of the project, all HRSA SPNS grantees agreed to participate in a national cross-site evaluation in addition to each site's local evaluation. The national evaluation stipulated the measurement of a common set of variables using a common assessment instrument administered in a standardized manner. In addition, HRSA funded The New York Academy of Medicine (NYAM) as an evaluation center. NYAM created the Center for Adherence Support and Evaluation (CASE) to conduct the cross-site evaluation of these programs and to provide technical assistance to participating grantees in evaluation. The SPNS grantees were funded for a total of four years and the CASE Evaluation Center, for five years.

Twelve sites were funded to implement new or evaluate existing adherence support programs; two had started data collection more than a year prior to the commencement of the cross-site evaluation as part of a New York State AIDS Institute (NYSAI)-funded adherence support initiative and used NYSAI rather than CASE measures.[1]

Both cross-site and local evaluations were conducted by SPNS grantees at each demonstration site. The local evaluations were designed to address site-specific intervention goals and objectives. Sites differed in their target populations and in the design of their adherence support interventions; local evaluations were tailored to evaluate how well their intervention worked for their target population. In contrast, the primary goal of the cross-site evaluation was to pool the findings across sites to identify characteristics of effective adherence support interventions and determine which worked best in improving ART adherence as well as improving biological outcomes, e.g., CD4 and HIV viral load, for different target populations. The variability in local program design allows assessment across sites of several intervention features that included: appropriate levels of service utilization, different types of providers, the duration of effectiveness of support programs, and the effectiveness of different programs in supporting adherence of populations facing different barriers to their adherence. The cross-site evaluation is still underway.

The selection of cross-site variables and assessment procedures occurred through a collaborative process among SPNS grantees and CASE staff. This process allowed each site to collect a core set of quantitative and qualitative data that contributed to the pooled CASE dataset using identical procedures, and also collect project-specific data important for addressing local evaluation questions. The evaluation methods consisted of individual and group interviews with staff, individual interviews with participants, baseline and quarterly abstraction of data from participants' medical records, including regular assessments of adherence support encounters and services, and documentation of program features for cost analyses conducted at a later time. Evaluation Center staff conducted training of interviewers at all sites to ensure standardization of the data collection process.

Mutually agreed uniform measures of participant outcomes (e.g., adherence rates, viral load levels, barriers to adherence), support intervention process, (e.g., number and types of services), and program context (e.g., setting, theoretical background of intervention) were used. In addition, subsets of sites collected data on particular domains of interest (e.g., social support, trust in primary HIV care providers, and ART self-efficacy) through repeated structured interviews. Data subsets were pooled for detailed analysis of subsets of the entire dataset.

TOP

Cross-Site Evaluation Design Methods

The impact of individual and program-level factors on participants' adherence is assessed using a mixed-methods design in both data collection and analysis.[Ref. 58-59] We outline here the specific methodology used in our cross-site evaluation to illustrate the depth obtainable with this approach.

Table 2. Participant-Level Variables for Analysis in the Cross-Site Evaluation of HIV Antiretroviral Therapy Adherence Support Interventions

Participant-level Data. The evaluation assessed changes in participants' adherence to ART as well as in their knowledge, beliefs and attitudes in response to ART adherence support services. (For a summary of participant-level variables, see Table 2).

1. Repeated structured interviews with participants. Structured interviews were administered prior to the receipt of adherence support (baseline) and repeated quarterly over an 18-month period at all sites.[2] The primary outcome variable for assessing intervention effectiveness was a self-report about the number of doses prescribed and the number of doses missed per antiretroviral drug in the three days prior to the interview.[Ref. 60] Other questions included self-reported difficulty taking medications on time, average number of days per week at least one dose was missed, last time dose missed, whether doses were missed on weekends, and ease of following special instructions regarding the taking of the medications.

All sites collected data on knowledge of HIV infection, ART, health status, recent hospitalization, recent substance use, disclosure of HIV status, support for taking medications, comfort in taking medications in front of others, and social characteristics of the participant. In addition, some sites included one or more optional "supplemental" questions or scales on the side effects of ART, depression, substance use, participants' trust in primary HIV medical provider, adherence self-efficacy, and measures of beliefs about ART, enabling more detailed analyses for the subset of sites using a particular supplemental instrument.

2. Medical or clinical chart abstractions. Documentation of HIV/AIDS status, HIV-1 viral load levels, CD4 cell counts, clinical psychiatric diagnosis, adherence to medical visits, and demographics were obtained from medical records by SPNS project staff using a standardized form and criteria.[3] HIV-1 viral load and CD4 counts are indicators of disease severity and are believed to reflect the level of adherence. The chart abstractions were conducted within a 45-day window period before or after each quarterly interview over an 18-month period.

3. Qualitative participant interviews. We conducted qualitative interviews using a standardized interview guide with a convenience sample of 103 participants from 10 of the 12 sites at nine to 11 months after program start-up. These interviews explored participants' narratives about the adherence services they received, their adherence support providers, and barriers to ART adherence, therefore providing a context for understanding the circumstances of nonadherence.

4. Process documentation. The frequency of participants' HIV service utilization, the types of services received, characteristics of the service providers, and recently missed medical appointments were tracked each time a participant received adherence support services. For every participant visit, we collected data on the (1) type of services delivered (e.g., clinical, adherence support, addictions treatment, HIV education, case management); (2) service delivery setting; (3) provider characteristics (licensure, training); and (4) intervention delivery mode (individual/group). These data were linked to individual participant data through a unique identifier system.

Program-level Data. The measurement of program-variables in the cross-site evaluation enables capture of the different contexts, content and implementation of each adherence support program.

1. Group interviews. Program-level data collection included 11 semi-structured qualitative group interviews with the adherence support team, local clinical providers and evaluators, and individual interviews with convenience samples of adherence support staff, clinical providers and participants at each of the sites. Providers for the group interviews were selected based on their experience with the adherence support program at their institution, while client participants represented a self-selected sample to reflect varying lengths of participation in the adherence support program. These interviews provided information about program and service delivery features at each site, descriptions of staff characteristics and roles, intervention implementation process, and problems encountered in program implementation. These data were used to examine the association between program-variables and participant adherence as well as provide key information for a cost-effectiveness analysis.

The group interviews, at nine to 11 months after start-up, allowed time for program stabilization. Questions were specifically asked about changes in the intervention since its inception, permitting assessment of the extent to which sites implemented their programs as planned.

2. Provider interviews. Assessment of providers' perceptions of their roles and responsibilities in the adherence support program was crucial since some studies indicate that the providers' relationships with participants (e.g., rapport, respect, communication, sensitivity to participants' culture, and experience in treating persons with HIV) can influence participants' ability to adhere to ART.[Ref. 39, 50, 61] In addition, we collected sociodemographic and professional information on the providers to investigate beliefs that demographic matching of providers and participants by gender and race/ethnicity may enhance quality of care.

3. HIV care environment. We constructed three variables using secondary data sources to rate the HIV care environment in each program: (1) adequacy of the AIDS Drug Assistance Program (ADAP) which provides HIV treatment to low-income uninsured and underinsured HIV-positive individuals (based on the level of expenditure in dollars per participant served in June 2002, eligibility as a percent of the Federal Poverty Line, number of drugs for opportunistic infection prophylaxis covered and number of other drugs covered); (2) adequacy of the Medicaid program (based on the average spending per SSI recipient, AIDS-specific Medicaid rates, special provisions for AIDS coverage, and existence of HIV Medicaid waiver regarding the eligibility and coverage for people living with HIV/AIDS); and (3) poverty index (based on the federal share for each state's Medicaid program).

These observations on the intervention program context were used to generate a standard set of program variables (Table 3). Key variables included the adequacy of the HIV care system, site characteristics (e.g., hospital based or clinic setting), intervention structure and features (e.g., participant assigned to treatment conditions based on assessment; intervention delivery mode); core services (e.g., peer program, readiness training, pharmacy home delivery, pick-up of medications where adherence support services are delivered, case management and other social services), provider type and roles, staffing levels, service delivery characteristics (home visits, integration of adherence support and other HIV medical care services; whether mental health, case management, dental, and other services were co-located with adherence support services); and distribution of adherence reminder tools (e.g., beepers, pill boxes).

Table 3. Program Level-Variables for Analysis in the Cross-Site Evaluation of HIV Antiretroviral Adherence Support Interventions

TOP

Applications of the Cross-Site Evaluation Model

The multi-level and multi-site dataset generated through the cross-site evaluation model dramatically extends the scope and type of evaluation questions that could be addressed.

Testing the effectiveness of different intervention models. Because cross-site evaluation has generated observations on diverse interventions and program settings, it is optimal for evaluating the effectiveness of adherence support interventions. Different intervention models in the cross-site evaluation (e.g., services offered and intervention modalities) permit analysis of intervention effectiveness by intensity, duration, and types of adherence support providers. Moreover, we assessed whether specific program components were more likely to be associated with positive changes in ART adherence (e.g., whether participants starting on a new medication regimen who participated in a readiness intervention reported better adherence than those who did not receive a readiness intervention before the start of ART).

The pooled data also facilitates the comparison of different program outcomes and their long-term durability for specific populations with particular barriers to adherence. The expanded power of the pooled data allows a more detailed analysis of effectiveness of adherence support for specific subpopulations, such as mothers with children, substance users, the mentally ill, the homeless, specific racial and ethnic groups, and people with language barriers. The dataset also allows assessment of the effects of different barriers to health care utilization. Unlike clinical trials that seek to evaluate the same intervention at multiple sites, cross-site evaluations can assess the effectiveness of different adherence support interventions that can be compared to each other. The scope of research questions addressed by this cross-site evaluation methodology is shown in Table 4.

Table 4. Questions Addressed in a Cross-Site Evaluation

Testing for program factors and context. Another contribution of the cross-site evaluation has been the ability to assess the impact of program structure, the context of the health care setting, and the adequacy of the HIV care environment on participants' ART adherence. Assessing these program and contextual effects provides important insights about whether and how the type of provider, service configuration, and service location, contribute to adherence. For example, are adherence-specific project staff more or less effective than those who delivered adherence support in addition to other services (e.g., case management, medical care)? The pooled data from diverse provider and participant groups also enables us to test the association effect of participant demographic characteristics with adherence. Salient institutional differences, such as whether the program was delivered in a community health center or hospital setting and whether it was integrated or independent of HIV medical care, can be explored to determine their association with intervention effectiveness.

If the pooled sample is large enough, as in our cross-site evaluation, one can conduct post-hoc analysis of programs with a similar cluster of components to estimate effect magnitudes.[Ref. 62-63] This will provide a classification algorithm to group interventions with common elements, and then compare their effects on participants' adherence to HIV antiretroviral therapy. For example, we are able to examine the effect of individual and aggregate sets of program components on adherence.

The enhanced power of the cross-site evaluation dramatically expanded the multi-level modeling of adherence support program effectiveness. Hierarchical linear modelling in which individuals were categorized within sites and sites nested within program types is one analytic approach that was used to determine which interventions have elements that are effective in enhancing adherence.64-65 Only a handful of studies have previously investigated the role of system-level factors in predicting participants' adherence.[Ref. 61, 66-67] Our cross-site study examined system factors, including the adequacy of the HIV care environment, that might influence participants' adherence to HIV antiretroviral medications.

Intervention fidelity and other process indicators. The documentation of actual versus planned program interventions at the different sites rendered an opportunity to assess program fidelity including the thoroughness of intervention implementation and whether there was intervention drift, that is, did the intervention deviate from the original plan, and, if so, how the interventions were adapted over time.

The linkage of participants' service delivery data and program variables also allows us independently to check the accuracy of intervention program descriptions. For example, if a program described itself as peer-based, we can look at the encounter data to determine the proportion of participants who actually had peer-based visits.
Medication regimens. Depending on the number of participants on a given regimen, the relationships between different ART combinations and adherence can be more effectively assessed with the greater power of the pooled cross-site data. Thus, we can move beyond the examination of simple associations between "pill burden" or number of pills taken daily and adherence, to more complex characteristics of regimens, including numbers of doses (or pill-taking episodes) required daily; the size and taste of specific pills; side effect profiles; and food and water restrictions.

Improving evaluation capacity. At the outset of our cross-site evaluation project, we identified that variations in project staff skills and experience in data collection, data management and evaluation might affect the accuracy, timeliness, consistency and quality of data collection across sites. Regular conference calls, technical assistance from CASE for training site-specific staff in the data collection protocol, the frequent data quality checks, and continuous monitoring of the data imported for cross-site analysis were conducted to maintain the quality of the pooled data.

Potential replicability. A cross-site evaluation yields a synthesis of "best intervention practices" that can be disseminated in the field, adapted and replicated with other populations on ART regimens, and ultimately utilized within entire health systems. The approach, systems, and structures developed for this multi-site evaluation also can serve as a prototype and be readily applied in the evaluation of HIV prevention, care and support models and other health issues.

TOP

Key Methodological Challenges

The CASE cross-site evaluation has numerous strengths; it also poses methodological challenges. As we have proceeded with the cross-site evaluation of ART adherence support programs, funded to provide care to vulnerable populations living with HIV infection, we have encountered several significant methodological challenges. An overarching challenge of the cross-site evaluation has been to maintain the balance between the rigor of standardization versus flexible pragmatism. We learned that flexibility is a cardinal principle in the implementation of a cross-site evaluation in real-world service delivery settings, and that one must take care not to overburden participants with assessments.

Aggregating data across sites. A key challenge has been to adjust for the diversity of intervention features and settings, target populations, eligibility criteria, and participant retention. Our decision to aggregate populations or interventions was determined by the questions to be addressed, relevant variables and adequacy of the sample size for each specific analysis. For some evaluation questions and analyses, data from all 12 sites were included because all data were relevant to the question. For more narrow questions, subsets of similar participants were selected from the sample. The pooling of data provides a synopsis of intervention effects across all sites as if they were derived from a single sample, thereby ignoring the heterogeneity of design, population, intervention, and setting characteristics across studies. Without careful conceptualization regarding how constructs are grouped, this could lead to spurious findings. Interpretation of the findings of the cross-site evaluation requires attention to the designs of the different program contributing to the database. However, the use of common predictors and adherence outcomes helped to reduce variability in measurement.

Target population. All programs targeted people at increased risk for non-adherence, but there was considerable variability in participant characteristics both within and across programs. While cross-site evaluation allows assessment of the impact of different population risk factors, such as ethnicity, gender, age, active substance use, and mental illness, this evaluation approach greatly complicates the evaluation of particular clusters of adherence support program interventions.

Eligibility criteria. The different enrollment criteria across programs added complexity to the data analysis plan. A major difference among programs was whether or not clients were receiving ART at study enrollment. Some programs enrolled participants who were considering initiation of ART and started them in a readiness program, while others enrolled participants at the point of beginning or changing an ART regimen. All sites (except one), however, did conduct "prospective enrollment", that is, enroll clients new to their programs.[4] These variations in enrollment eligibility criteria resulted in different populations receiving adherence support services across the programs. The resulting variability in sample size for comparisons of similar participants affected statistical power, permitting us to address some questions and not others. Even within the sample of participants currently on medications, variations in eligibility criteria (e.g., participants who were less than 90% adherent, those who failed their first ART regimen, or those who changed their current ART regimen) created challenges in interpreting changes in adherence and biologic indicators from baseline to any of the successive follow-up assessments. This variability in eligibility criteria was further compounded by site investigator-initiated changes in the program over the course of the intervention, e.g., changes in recruitment strategy, or developing a plan to ensure that participants graduate from the program. In the cross-site evaluation, it is critical to document such changes and adjust for program variables that change over time.

Participant retention and attrition. In any longitudinal study, loss to follow-up is a major problem, but in a cross-site evaluation of multiple programs, the bias in favor of participants who stay in the program is even more of an issue if the attrition rate differs by site, program type, or client characteristics. For example, if it appears that participants in a peer-delivered intervention are more likely to complete their adherence support intervention than those in a provider-delivered HIV intervention, participants in peer-delivered interventions are overrepresented among the "completers" and the effectiveness of peer-driven interventions will be overstated. Therefore, we have to be cautious about drawing conclusions regarding the effectiveness of specific programs, proceeding only after examination of the specific relationships among participant attributes, retention, and adherence. As in any study, documenting participant characteristics and reasons for dropping out among those who do not complete a time-limited intervention is important for interpreting the findings. In our cross-site evaluation, the variations in program retention are being addressed by an attrition analysis to identify systematic biases attributed to non-completion of the adherence support intervention. Missing data techniques (e.g., imputation) will be employed in longitudinal analysis when certain assumptions about the missing patterns are met.

TOP

Implications for Practice

With the complexity of HIV treatment, evidence-based, multifaceted medication adherence support interventions are needed. This cross-site evaluation of a large sample of people with HIV/AIDS provided a method for addressing key questions about the effectiveness of interventions in supporting adherence for different populations, across different intervention models, and in different settings. We have identified the following methodological strengths of the cross-site evaluation process.
Ability to share adherence support program ideas. One of the major advantages of the cross-site evaluation process has been the ability of the 12 participating SPNS sites to compare intervention modalities during the collaborative development of the cross-site evaluation. In the process of developing standardized data collection instruments, several sites refined their interventions after learning about programs planned by other grantees. This undoubtedly contributed to the program refinements that were assessed during the qualitative interviews. The final analyses of the effectiveness of adherence intervention components will be greatly enriched by our collective understanding of each other's interventions.

Clinician-evaluator participation in the design and data analysis. The cross-site evaluation criteria and guidelines for analysis of pooled data were informed by the input of clinicians and evaluators from the collaborating sites. Therefore, the questions addressed in the cross-site evaluation were shaped by the clinical issues currently confronting providers who work with HIV-infected individuals, and our findings will be presented in a manner that are more likely to be useful to clinicians. Unlike meta-analyses which are post-hoc comparative evaluations, the cross-site evaluation used here had the comparison designed from the start, with evaluation instruments carefully tailored to the agreed upon objectives.

Feasibility of implementing adherence support interventions. Because the cross-site evaluation involved the collection of data about the context of the adherence support interventions, these contextual data can be helpful in understanding the quantitative findings regarding intervention effectiveness. Moreover, data about program costs, staffing patterns, and time estimates for program components may allow the HIV treatment community to adopt interventions that fit their settings and resources in a way that may decrease the typical lag in the adoption of best practices identified by research and evaluation projects.

Enhancement of adherence assessment. The cross-site evaluation examined the sensitivity and specificity of the various adherence assessment approaches as clinical screening tools. Finding significant associations among different measures of self-reported adherence as well as between self-reported adherence and biologic indicators in the pooled longitudinal data and identifying a parsimonious set of adherence measures may help clinicians assess medication adherence of a single participant in a single visit. With clinicians having limited time for adherence assessment, use of more efficient and effective adherence screening tools is pragmatic.

Generalizability of evaluation findings. Most importantly, the cross-site evaluation of adherence-support interventions contributed to our arsenal of methodologies by using real-world settings, grounding the evaluation within the uncertainties of clinical settings and participant behavior as well as changing medication regimens and treatment standards. Although this methodology does not provide the same stringent control over the research environment as a randomized controlled trial, the robustness of findings, despite methodological "noise", will be generalizable to the participants and providers in other clinical settings, and perhaps more feasible to implement. Understanding current adherence support as it is practiced also will suggest strategies for disseminating new approaches to assessment and optimizing support of antiretroviral medication-taking behavior. Understanding implementation parameters, such as the features of a setting and the service delivery context, is critical to transferring effective programs to new settings. [Ref. 68]

Development of standards for adherence support. The CASE implementation and evaluation blueprint can inform health policymakers about adherence support programs that maximize the benefits of ART and should be incorporated into routine HIV medical care. Critical program components that improve or maintain adherence form the basis for developing clinical practice guidelines and minimum performance standards for high-quality adherence support.

TOP

Conclusion

In the CASE evaluation, this method allowed for an integrated study design to ascertain multilevel relationships among characteristics of individuals, providers, interventions, and systems in affecting adherence to therapy. The cross-site evaluation methodology provides a viable alternative to meta-analysis and randomized controlled trials in determining adherence support intervention effectiveness. These findings will be useful in developing clinical standards for adherence assessment and support.

Participating sites are:
Action Point, San Francisco, CA
Health Services Center, Hobson City, AL
Chase-Brexton Health Center, Baltimore, MD
Dimock Community Health Center, Boston, MA
Harlem Hospital Center, NY, NY
Johns Hopkins University School of Medicine, Baltimore, MD
Mailman School of Public Health/Columbia University, NY, NY
Mission Neighborhood Health Center, San Francisco, CA
Multnomah County Health Department, Portland, OR
State University of New York/Downstate Medical Center, Brooklyn, NY
North Broward Hospital District, Fort Lauderdale, FL
Helena Hatch Special Care Center, Washington University School of Medicine, St. Louis, MO

TOP

References

1. Detels R, Munoz A, McFarlane G, et al: Effectiveness of potent antiretroviral therapy on time to AIDS and death in men with known HIV infection duration. JAMA 280:1497-1503, 1998.

2. Hogg RS, Heath KV, Yip B, Kully C, Craib KJ, O'Shaughnessy MV, Schechter MT, Montaner JS: Improved survival among HIV-infected individuals following initiation of antiretroviral therapy. JAMA 279:450-454, 1998.

3. Palella FJ, Delaney KM, Moorman AC, et al: Declining morbidity and mortality among patients with advanced human immunodeficiency virus infection. N Engl J Med 338:853-860, 1998.

4. Moore RD, Chaisson RE: Natural history of HIV infection in the era of combination antiretroviral therapy. AIDS 13:1933-42, 1999.

5. Ledergerber B, Egger M, Erard V, Weber R, Hirschel B, Furrer H, et al: AIDS-related opportunistic illnesses occurring after initiation of potent antiretroviral therapy: The Swiss HIV Cohort Study. JAMA 283:2653-2654, 2000.

6. Mocroft A, Katlama C, Johnson AM, et al: AIDS across Europe, 1994-98: The EuroSIDA Study. Lancet 356: 291-296, 2000.

7. Bangsberg DR, Hecht FM, Clague H, Charlebois ED, Ciccarone D, Chesney M, Moss A: Provider assessment of adherence to HIV antiretroviral therapy. J Acquir Immune Defic Syndr Hum Retrovirol 26: 435-442, 2001.

8. Bangsberg D: Is average adherence to HIV antiretroviral therapy enough? J Gen Intern Med 17: 812-813, 2002.

9. Hogg RS, Heath K, Bangsberg D, Yip B, Press N, O'Shaughnessy MV, Montaner JS: Intermittent use of triple-combination therapy is predictive of mortality at baseline and after 1 year of follow-up. AIDS 16(7): 1051-1058, 2002, 1998.

10. Paterson DL, Swindells S, Mohr J: Adherence to protease inhibitor therapy and outcomes in patients with HIV infection. Annals of Internal Medicine 133: 21-30, 2000.

11. Mostashari F, Riley E, Selwyn PA, Altice FL: Acceptance and adherence with antiretroviral therapy among HIV-infected women in a correctional facility. J Acquir Immune Defic Syndr Hum Retrovirol 18: 341-348, 1998.

12. Samet JH, Libman H, Steger, KA, et al: Compliance with zidovudine therapy in patients infected with human immunodeficiency virus, type 1: A cross-sectional study in a municipal hospital clinic. Am J Med 92: 495-502, 1992.

13. Singh N, Squier C, Sivek C, Wagener M, Nguyen MH, Yu VL: Determinants of compliance with antiretroviral therapy in patients with human immunodeficiency virus: Prospective assessment with implications for enhancing compliance. AIDS Care 8(3): 261-269, 1996.

14. Arnsten J, Demas P, Gourevitch M, Buono D, Farzadegan H, Schoenbaum E: Adherence and viral load in HIV-infected drug users: Comparison of self-report and medication event monitors (MEME). In 7th Conference on Retroviruses and Opportunistic Infections, San Francisco, CA, 2000.

15. Gifford AL, Bormann JE, Shively MJ, Wright B, Richman D, Bozzette S: Predictors of reported adherence and plasma HIV concentrations in patients on multidrug antiretroviral regimens. J Acquir Immune Defic Syndr Hum Retrovirol 23 (5): 386-395, 2000.

16. Mannheimer S, Friedland G, Matts J, Child C, Chesney M, for the Terry
Beirn Community Programs for Clinical Research on AIDS: The consistency of
adherence to antiretroviral therapy predicts biologic outcomes for human
immunodeficiency virus-infected persons in clinical trials. Clin Infect Dis
34: 1115-1121, 2002.

17. Ickovics JR , Meisler AW: Adherence in AIDS clinical trials: A framework for clinical research and clinical care. Clinical Epidemiology 50 (4) 385-391, 1997.

18. Haubrich RH, Little SJ, Currier JS, Forthal, DN, Kemper CA, Beall GN, Johnson D, Dube MP, Hwang JY, McCutchan A and The California Collaborative Treatment Group: The value of patient-reported adherence to antiretroviral therapy in predicting virologic and immunologic response. AIDS 13 (9): 1099-1107, 1999.

19. Broers B, Morabia A, Hirschel B: A cohort study of drug users' compliance with zidovudine treatment Arch Intern Med 154: 1121-1127, 1994.

20. Stein MD, Rich JD, Maksad J, Chen MH, Hu P, Sobota M, Clarke J: Adherence to antiretroviral therapy among HIV-infected methadone patients: Effect of ongoing illicit drug use. Am J Drug Alcohol Abuse 26 (2): 195-205, 2000.

21. Ferrando SJ, Wall TL, Batki SL, Sorensen JL: Psychiatric morbidity, illicit drug use and adherence to zidovudine (AZT) among injection drug users with HIV disease. Am J Drug Alcohol Abuse 22 (4): 475-487, 1996.

22. Falkin G, Strauss S: Drug-using women's communication with social supporters about HIV/AIDS issues. Journal of Drug Issues 30(4): 801-822, 2000.

23. Gielen A McDonnell KA, Wu AW, O'Campo P, Faden R: Quality of life among women living with HIV: The importance of violence, social support and self care behaviors. Soc Sci Med 52: 315-322, 2001.

24. Halkitis PN, Parsons JT, Wolitski RJ, Remien RH: Characteristics of HIV antiretroviral treatments, access and adherence in an ethnically diverse sample of men who have sex with men. AIDS Care 15(1): 89-102, 2003.

25. Burack JH, Barrett DC, Stall RD, Chesney MA, Ekstrand ML, Coates TJ: Depressive symptoms and CD4 lymphocyte decline among HIV-infected men. JAMA 270 (21): 2568-2573, 1993.

26. Siegel K, Raveis V, Karus D: Illness-related support and negative network interactions: Effects on HIV-infected men's depressive symptomatology. American Journal of Community Psychology 25 (3): 395-420, 1997.

27. Gordillo V, del Amo J, Soriano V, Gonzalez-Lahoz J: Sociodemographic and psychological variables influencing adherence to antiretroviral therapy. AIDS 13 (13): 1763-1769, 1999.

28. Leserman J, Petitto JM, Golden RN: Impact of stressful life events, depression, social support, coping and cortisol on progression to AIDS. American Journal of Psychiatry 157 (8): 1221-1228, 2000.

29. Fleishman J Sherbourne CD, Crystal S, Collins RL, Marshall GN, Kelly M, Bozzette SA, Shapiro MF, Hays RD: Coping, conflictual interactions, social support, and mood among HIV-infected persons. American Journal of Community Psychology 28 (4): 421-453, 2000.

30. Starace F, Ammassari A, Trotta, MP, Murri R, De Longis P, Izzo C, Scalzini A, d'Arminio Monforte A, Wu AW, Antinori A for the AdICoNA and the NeuroICoNA Study Groups: Depression is a risk factor for suboptimal adherence to highly active antiretroviral therapy. J Acquir Immune Defic Syndr Hum Retrovirol 31 (Supplement 3): S136-S139, 2002.

31. Catz SL, Kelly JA, Bogart LM, Benotsch EG, McAuliffe TL: Patterns, correlates, and barriers to medication adherence among persons prescribed new treatments for HIV disease. Health Psychology 19 (2): 124-133, 2000.

32. Power R, Koopman C, Volk J, Israelski DM, Stone L, Chesney M, Spiegel D: Social support, substance use and denial in relationship to antiretroviral treatment adherence among HIV-infected persons. AIDS Patient Care and STDs 17 (5): 245-252, 2003.

33. Singh N, Berman SM, Swindells S: Adherence of human immunodeficiency virus-infected patients to antiretroviral therapy. Clinical Infectious Diseases 29: 824-830, 1999.

34. Morse EV, Simon PM, Coburn M, Hyslop N, Greenspan D, Balson PM: Determinants of subject compliance within an experimental anti-HIV drug protocol. Soc Sci Med 32: 1161-1167, 1991.

35. Tuldrá A, Fumaz CR, Ferrer MJ, Bayés R, Arnó A, Balagué M, Bonjoch A, Jou A, Negredo E, Paredes R, Ruiz L, Romeu J, Sirera G, Tural C, Burger D, Clotet B: Prospective randomized two-arm controlled study to determine the efficacy of a specific intervention to improve long-term adherence to highly active antiretroviral therapy. J Acquir Immune Defic Syndr Hum Retrovirol 25: 221-228, 2000.

36. Eldred LJ, Wu AW, Chaisson RE, Moore RD: Adherence to antiretroviral and pneumocyctis prophylaxis in HIV disease. J Acquir Immune Defic Syndr Hum Retrovirol 18: 117-125, 1998.

37. Ostrow D, Fox K, Kleeberger C, Jacobson L, Chmiel J, Silvestre A, Visscher B, Strathdee S: Psychosocial correlates of incomplete adherence to HIV antiretroviral therapy (HAART): Mental health matters. In 8th Conference on Retroviruses and Opportunistic Infections Chicago, IL, 2001.

38. Stone VE, Clarke J, Lovell J, Steger KA, Hirschhorn LR, Boswell S, Monroe AD, Stein MD, Tyree TJ, Mayer KH: HIV/AIDS patients' perspectives on adhering to regimens containing protease inhibitors. J Gen Intern Med 13 (9): 586-593, 1998.

39. Altice F, Motashari F, Friedland G: Trust and acceptance of and adherence to antiretroviral therapy. J Acquir Immune Defic Syndr Hum Retrovirol 28: 47-58, 2001.

40. Thompson AS, Blankenship KM, Selwyn PA, Khoshnood K, Lopez M, Balacos K, Altice FL: Evaluation of an innovative program to address the health and social service needs of drug-using women with or at risk for HIV infection. J Community Health 23 (6): 419-440, 1998.

41. Besch CL: Adherence in clinical trials. AIDS 9: 1-10, 1995.

42. Besch CL, Morse E, Simon P, Hodges J, Franchino B: Preliminary results of a compliance study with CPCRA 007 combination nucleoside study (NuCombo). In 4th Conference on Retroviruses and Opportunistic Infections, Washington, DC, 1997.

43. Orr PR, Blackhurst DW, Hawkins BS: Patient and clinic factors predictive of missed visits and inactive status in a multicenter clinical trial. The macular photocoagulation study group. Control Clin Trials 13 (1): 40-49, 1992.

44. Gray L, Edmondson E, and Lemke AL: HIV Treatment Adherence: A Guide for Program Development. HIV/AIDS Project Development and Evaluation Unit, University of Washington School of Social Work, Seattle, WA, 1998.

45. Friedland GH, Williams A: Attaining higher goals in HIV treatment: The central importance of adherence AIDS 13 (Supplement 1): S61-S72, 1999.

46. Esch L, Hardy H, Wynn H, Shelton R, Hewitt R, Morse G: Intensive adherence interventions improve virologic response to antiretroviral therapy (ART) in naïve patients. Poster presented at the 8th Conference on Retroviruses and Opportunistic Infections, Chicago, IL, 2001.

47. Grohskoph, LA, et al: Feasibility of a two-way electronic pager to enhance adherence with highly active antiretroviral therapy [Abstract No. 489] 36th Annual Infectious Diseases Society of America Meeting. Denver, Colorado, 1998.

48. Saffren SA, Otto MW, Worth JL, Salomon E, Johnson W, Mayer K, Boswell S: Two strategies to increase adherence to HIV antiretroviral medication: Life-Steps and medication monitoring. Behaviour Research and Therapy 39: 1151-1162, 2001.

49. Graham K, Beeler LH, Renae S, Sension MG: Interventions and patient outcome from a pharmacist-based HIV medication adherence referral clinic. Twelfth World AIDS Conference, Geneva, Switzerland, 1998.

50. Ickovics JR, Meade CS: Adherence to antiretroviral therapy among patients with HIV: A critical link between behavioral and biomedical sciences. J Acquir Immune Defic Syndr Hum Retrovirol 31: 98-102, 2002.

51. Sorensen JL, Mascovich A, Wall TL, DePhilippis D, Batki SL, Chesney M: Medication adherence strategies for drug users with HIV/AIDS. AIDS Care 10: 297-312, 1998.

52. Fogarty L, Roter D, Larson S, Burke J, Gillespie J, Levy R: Patient adherence to HIV medication regimens: A review of published and abstract reports. Patient Educ Couns 568: 1-16, 2002.

53. Broadhead R, Heckathron DD, O'Connor PG, Selwyn PA: Increasing active drug users' adherence to HIV therapeutics. Abstract No. 32336. In: Proceedings of the 12th World AIDS Conference, Geneva, Switzerland, June 28-July 3, 1998.

54. Conway B, Prasad J, Reynolds R, et al: Nevirapine (NVP) and protease inhibitor (PI)-based regimens in a directly observed therapy (DOT) program for intravenous drug users (IDUs). Abstract 545. Program and abstracts of the 9th Conference on Retroviruses and Opportunistic Infections, Seattle, WA, February 24-28, 2002.

55. Arriola KRJ, Kennedy S, Coltharp JC, Braithwaite RL, Hammett TM, Tinsley, MJ: Development and implementation of the cross-site evaluation of the CDC/HRSA corrections demonstration project. AIDS Educ Prev 14 Supplement A: 107-118, 2002.

56. Huba GJ, Melchior LA, Panter AT, Brown VB, Larson TA: A national program of AIDS care projects and their cross-cutting evaluation: The HRSA SPNS Cooperative Agreements, in: Huba GJ, Melchior L, Brown VB, Larson TA, Panter AT (eds.): Evaluating HIV/AIDS Treatment Programs: Innovative Methods and Findings. New York: The Haworth Press, Inc., 2000, pp. 5-29.

57. Raveis VH, Karus D, Marconi K, Selwyn PA, Perrone M, Hanna B, Patterson E, Rosefield HA: Delivering palliative care to the medically-underserved with HIV/AIDS: Innovations in models of care. Session No. 5037.0, Abstract #46744, American Public Health Association Meeting, Philadelphia, PA, November 9-13, 2002.

58. Datta L: Multimethod evaluation: Using case studies together with other methods, in Chelimsky E. Shadish WR (eds.): Evaluation for the 21st Century: A Handbook. Thousand Oaks, CA: Sage. Publications, Inc., 1997, pp. 344-359.

59. Rallis SF, Rossman SB: Mixed methods in evaluation contexts: A pragmatic framework, in Tashakkori A, Teddle C (eds.): Handbook of Mixed Methods in Social & Behavioral Research. Thousand Oaks, CA: Sage. Publications, Inc., 2003, pp. 491-512.

60. Chesney MA, Ickovics JR, Chambers DB, Giddord AL, Neidig J, Zwickl B, Wu AW: Self-reported adherence to antiretroviral medications among participants in HIV clinical trials: The AACTG adherence instruments. Patient Committee and Adherence Working Group of the Outcomes Committee of the Adult AIDS Clinical Trials Group (ACTG): AIDS Care 12 (3): 255-266, 2000.

61. van Servellen G, Chang B, Garcia L, Lombardi E: Individual and system level factors associated with treatment nonadherence in human immunodeficiency virus-infected men and women. AIDS Patient Care & STDs 16: 269-281, 2002.

62. Blashfield RK, Aldenderfer MS: The literature on cluster analysis. Multivariate Behavioral Research 13: 271-295, 1978.

63. Gore PA, Jr: Cluster analysis, in Tinsley HE, Brown, SD (eds.): Handbook of Applied Multivariate Statistics and Mathematical Modeling, NY, Academic Press, 2000, pp. 297-321.

64. Raudenbush SW, Bryk AS: Hierarchical Linear Models: Applications and Data Analysis Methods, Second edition, Thousand Oaks, CA: Sage, 2002.

65. Singer JD, Willett JB: Applied Longitudinal Data Analysis. New York: Oxford University Press, Inc., 2003.

66. Waters M, Weiss L, French T, Finkelstein R, Agins B, Novello A: Implementing adherence programs: Lessons learned from the New York State adherence demonstration project. Oral Presentation ThPeF8185, XIV International AIDS Conference, Barcelona, Spain, July 7-12, 2002.

67. Bakken S, Holzemer WL, Brown N, Powell-Cope G, Turner J, Inouye J, Nokes
K, Corless I: Relationship between perception of engagement with health care provider and demographic characteristics, health status, and adherence to therapeutic regimen in persons with HIV/AIDS. AIDS Patient Care & STDs 14: 189-196, 2000.

68. Bauman LJ, Stein REK, Ireys HT: Reinventing fidelity: The transfer of social technology among settings. American Journal of Community Psychology 19 (4): 619-639, 1991.

TOP

Authors

Joanne E. Mantell, MS, MSPH, PhD
New York Academy of Medicine

John Dougherty, PhD
Multnomah County Department of Health

Sally Findley, PhD
Joseph Mailman School of Public Health /Columbia University

Julie Franks, PhD
Harlem Hospital Center, New York, NY

Lynn Wenger, MPH
University of California San Francisco

Ricardo Alvarez, MD
Mission Neighborhood Health Center

Barbara Aranda-Naranjo, RN, PhD
Health Resources and Services Administration

G. Stephen Bowen, MD, MPH
North Broward Hospital District \

Theresa Cassidy, MPH
Mt. Sinai Medical Center

Dan Ciccarone, MD, MPH
University of California San Francisco

Barbara Hanna, MD
Health Services Center, Inc.

Lisa R. Hirschhorn, MD, MPH
Dimock Community Health Center

Sharon B. Mannheimer, MD, MPH
Harlem Hospital Center

Richard Moore, MD
Johns Hopkins University School of Medicine

Rajat Mukherjee, MS,
New York Academy of Medicine

Linda M. Mundy, MD
Washington University School of Medicine

Emily Richie, MD
Chase-Brexton Health Services, Baltimore, MD

Nancy Van Devanter, DrPH
Joseph Mailman School of Public Health/Columbia University

Mark Waters, RN, MPH
New York State Department of Health/AIDS Institute

Tracy Wilson, PhD
State University of New York-Downstate Medical Center

Lois Eldred, DrPH, MPH
Health Resources and Services Administration,

Ruth Finkelstein, ScD
New York Academy of Medicine

TOP

Footnotes

This allowed us to assess the comparability of the CASE and NYSAI adherence and other measures.

Two sites were excluded from the cross-site analysis. The Mailman School of Public Health/Columbia University program targeted caregivers of children with HIV/AIDS who reported ART adherence on behalf of the children and the measures were deemed incompatible with one's own self-reported adherence. The entire NYSAI - a consortia of 14 sites - was not included because enrollment far exceeded the size of the CASE sample and is itself a cross-site evaluation with well documented findings (Waters, Weiss, French, Finkelstein, Agins, & Novello, 2002). However, two sites in the NYSAI project were included in the analysis: Harlem Hospital's evaluation was independently funded by SPNS and its intervention by the NYSAI; SUNY-Downstate's intervention was funded by the NYSAI but used CASE evaluation instruments for comparison. A second comparison site, St. Luke's -Roosevelt Hospital Center program, was not refunded by NYSAI; therefore, the number of follow-up interviews was limited.

Demographic information also was collected in participants' baseline interview.

The exception was a site in a rural community funded to evaluate adherence support, but was not anticipating enrollment of new participants during the study period.