Tools for Grantees: Quality Management Manual < Previous | Home | Next >  III. Step-By-Step Quality Management Guide: Nine Critical Steps   Determine Performance Measures & Collect Baseline Data   •  Based on QM priorities, determine performance measures.    •  Develop indicators to measure performance.    •  Define measurement population and delineate eligibility criteria.    •  Create a data collection plan, including: a method of data collection, i.e., chart abstraction, interviews; sampling strategy, and potential data sources.    •  Create data collection tools: create instructions for data collection, train personnel who will collect data, conduct pilot test of tools.    •  Establish process of communicating with staff about measurement process.

Based on QM priorities, determine performance measures.  TOP

Once the QM plan is set and the priorities have been identified, the performance measures must be determined in order to put the plan into motion. Performance measures are designed to serve as yardsticks on which to measure quality. In order to measure a particular element of care, process, or outcome, indicators are selected to assess performance within a particular area of focus. Indicators are quantitative measures that can be used to assess and improve performance. While not a direct measure of quality, indicators are tools that can be used to direct attention to potential performance issues that may require more intense review.

Develop indicators to measure performance.  TOP

Indicators chosen should reflect key aspects of care which impact on patient outcomes and can be either clinical or service-oriented. Indicators can be grouped by type and address issues such as accessibility, appropriateness of care, effectiveness, continuity of care, etc. An example of a performance measure, indicator, and indicator type is provided below:

Together, these measures assess the ease with which new patients access care in a timely fashion. Appendix G provides examples of other indicator types, performance measures, and indicators. Potential data sources are also identified.

Define measurement population and delineate eligibility criteria.  TOP

Define the target population: Specific eligibility criteria must be established in order to clearly define the population to which you want to apply the performance measures. If this is not done properly, the end result may be useless data. It is important to determine who is part of the target population and who is not. Timing and individual patient characteristics also need to be considered.

In respect to timing, key questions to consider include: a) at what specific points in time should the data be collected; b) should the data be collected as an individual enters the program; or c) does the individual need to be receiving services for three months, six months, or one year. This is determined primarily by the standard that is being measured. For instance, if you want to know if patients have laboratory monitoring performed on a quarterly basis, patients might need to be enrolled in care for at least one year in order to adequately assess this standard. On the other hand, if you want to know whether an individual was referred to a case manager during their first visit as per your standard, any individual entering the system could be included in the population. Individual patient characteristics should also be considered, such as gender, age, location of service provision, and presentation or treatment status.

Gender: You’ll need to determine if the measurement applies to both women and men. Standards measuring gynecological care or treatment for HIV-positive pregnant women would obviously not apply to men.

Age: Some programs provide services to both adult and pediatric individuals. A particular measure may not apply to both. Some clinical standards of care, such as mammograms and colon cancer screening, are based on the age of the individual.

Site/Location: If you have more than one service location, you’ll need to determine if all sites will be included in your review. This is of particular interest at sites that utilize multiple sub-contractors. If multiple sites are providing similar services, a uniform standard of care is usually determined and measured consistently across sites, i.e. clinical care, case management. There is an opportunity in these instances to compare site performances and develop benchmarks. Benchmarks, or target performance levels, are based on “best practice” standards. If you do this, you will need to assure that you include all providers and all locations equally, and present comparison data. Comparative data should be used to improve performance and should not be used in a punitive manner.

Presentation/Treatment Status: Some standards of care apply to a particular patient/ client need or clinical status. In clinical care this would apply to a particular diagnosis. If you are measuring how an opportunistic infection was managed, the diagnosis of an opportunistic infection would need to be present.

In a case management setting, if you are measuring the degree to which a need has been met, i.e. housing, only those clients with this specific unmet need would be included in the sample.

Create a data collection plan, including: a method of data collection, i.e., chart abstraction, interviews; sampling strategy, and potential data sources.  TOP

Data Collection Method

When determining a data collection method, the following key questions should be asked:

 Is the data already available in some form?

 Is the data collection method feasible and not overly expensive?

 Is there a less time-intensive or less expensive way to collect this information?

 Will the data be useful, and will it address the desired performance measure?

 Will the resulting data be credible?

 Is the data reliable?

After the population has been defined and eligibility criteria outlined, a data collection plan should be formulated to include a sampling strategy and a process for collecting data. The plan can include random sampling or targeted/focused review. This will be determined by the purpose or aim of study, the question you want to answer, and the size of the population that you are studying.

Sampling Strategy

Sampling is a simple, efficient way to help understand how a system is performing. In most cases it is not necessary or feasible to look at 100 percent of your population. Sampling allows you to make presumptions about your larger population based on observations of a smaller subset of that group. The total number of “eligible” cases determines the size of the sample. Many sites use a standard methodology of 30 percent or 30 cases, whichever is greater, of the eligible population. This should be adequate to provide you with the information that you need. For programs with a small number of HIV-positive clients (<100), a 100 percent sample may be desirable and feasible. It is important to remember that your QM project is not research. You are trying to establish a snapshot of performance in the most current point in time that you have available.

Random sampling, a method to assure that each record has an equal chance of being included in the sample, should be applied. Some organizations will have sophisticated software programs to generate a random sample while others will have to manually select the sample from a printed list of eligible records. Many spreadsheet programs have a random number table to assist in this process.

Targeted or Focused Reviews

A targeted or focused review may be undertaken to provide additional specific information around a particular topic. These reviews can help establish all of the root causes contributing to a problem or to establish the extent of a problem. They are usually short term, require limited analysis, and provide immediate feedback.

Data Sources

We all wish that we had one source of data to answer all of our questions. Unfortunately, most programs have multiple sources of data, and a combination of electronic and paper sources. The appropriate source will need to be identified to assure that you are measuring the intended performance indicator. At times, data collected from various sources will be merged. For instance, data generated from a clinical database can be augmented with data obtained from a review of medical records.

Potential sources of data that should be considered include the following:

• Clinical database
• Medical record (paper or electronic health record)
• Client/patient satisfaction surveys
• Client/patient intake forms and questionnaires
• Billing records
• Client/patient/staff interviews
• Client comments, suggestions and complaints
• Laboratory database
• Case management/social work records
• Scheduling programs

Create data collection tools: create instructions for data collection, train personnel who will collect data, conduct pilot test of tools.  TOP

Collect data

Depending on the source of the data, an abstraction method will need to be identified. For data collected through paper record abstraction, a data collection tool should be developed. Abstractors will need to understand the purpose of the tool and be trained on each and every data element to assure standardized data collection. The tool should be tested prior to full implementation. No matter how carefully the data collection tool was developed, some problems will be discovered only after the tool has been used and tested. Each abstractor should try the tool on a limited basis, with one or two records, to clarify the data elements.

Based on the results of the pilot test, further revision of the data collection tool may be required before full implementation occurs. To maintain reliability in data extraction processes, it is imperative that a straightforward instruction guide be established along with introducing the tool that will be used for data extraction. This will establish a consistent data collection methodology and ensure that data are extracted the same way each and every time.

Inter-rater reliability is a term used to describe the process of having all of the source/chart abstractors review the same source/chart(s) and compare the results. The number of charts to review is dependent on the size of your sample and the number of abstractors. If all abstractors have the same responses, you can be sure you will have consistency in the data collection. If the responses vary, review the discrepancies with the abstractors and review methods of data collection to understand why the discrepancies occurred.

For data information that will be obtained from an electronic source, work closely with an expert in that system. This could be an information technology (IT) staff person, a computer analyst or specialist, or the data entry person who runs the data reports.

When requesting data runs, be very specific about the information you are requesting. The IT staff person may have to run the report from one or many systems. Always review the report closely. If the report is what you want, you may put in a request to have monthly data reports generated. Trust your intuition. Usually we have a good sense of what the results will be ahead of time. If the results vary widely from what you expected take the time to validate the data.

It is likely that you will be obtaining data from various sources if one system does not house all the data you need. Be careful what you ask for. Some requests for data reports can generate too much information that does not answer the question that you want to answer, or will take too much of your time to analyze. Some reports are coded and can only be deciphered by those trained in decoding data. You can always ask for sample data reports to get you started.

Most importantly, KNOW YOUR DATA. Understand the source of where it is coming from; understand how the report was generated and the components of the report; and validate the data through some random checks. Incorrect data, or data that doesn’t answer the question can be both a waste of time and lead you down an unnecessary path. This will frustrate staff involved in the process and reduce credibility in the overall QM program.

Establish process of communicating with staff about measurement process.  TOP

Once the data collection plan has been finalized, it is important to establish a process whereby staff are routinely updated about the measurement process and how data will be handled.

Another important issue to address is the confidentiality of data. As data are being collected, procedures to protect client confidentiality need to be strictly adhered to. HIV providers already have a heightened awareness and concern for the confidentiality of HIV-infected individuals yet it helps to be reminded of standard approaches used to protect confidentiality during the data collection process. In most cases, patients’ names are not required and data collection is often conducted by HIV program staff who are already trained in issues related to HIV confidentiality. If external abstractors are to be used, it is important to discuss the ethical implications and the legal consequences of disclosing medical information, particularly the HIV status of patients. All external reviewers should be required to sign a confidentiality statement indicating they understand and will comply with confidentiality standards.

Once the data collection is complete, all related forms and reports should be kept in a locked file and access restricted to those directly involved in the data collection process. If computers or laptops are used for data collection, appropriate steps (password protections, firewalls) must be taken to safeguard the information and ensure patient confidentiality.

Health care organizations must begin following regulations outlined in the Health Insurance Portability and Accountability Act (HIPAA). This Act outlines specific requirements around privacy and security which health care providers must use to ensure a patient’s medical information remains private and secure. The Act also has a component that requires that health care organizations have a standardized process for the exchange of electronic information. It is advisable to discuss with your organization steps that they have put into place to assure your organization is in compliance with HIPAA regulations.