CHAPTER 6

Tools and Methods for Data Collection and Data Analysis

In the previous chapter, we discussed issues related to data sources and data systems. In order to proceed with our quality assurance model, we need to understand concepts and methods involved in data collection and analysis, including some statistical fundamentals. In our quality assurance model, Step 5 involves defining the range of values to expect for a performance measure. The expected range of values are those which are reported for the performance measure. For many performance measures, the range of values expected would be from 0 to 100 percent. Step 6 of our quality assurance model involves defining the target value, or expected unadjusted value, for a performance measure. The concept of target value in greater detail later in this chapter.

Quantitative and Qualitative Data

Typically, the term data refers to information that answers questions for the purpose of making decisions. When measuring the ongoing performance of programs, we need to ask ourselves, What kinds of data are we going to need? Perhaps we want to know the number of children who have used emergency room services or how many of these children received outpatient services in addition to emergency room services. These types of data consist of numbers and percentages, and are referred to as quantitative data. Quantitative data tend to be succinct and numeric and reflect incidents or occurrences. On the other hand, we may want to gather more detailed information about a familys experience in the emergency room or when receiving outpatient health services. We may want to understand why or how families are accessing or not accessing certain services. These types of data consist of information about personal experiences, beliefs, perspectives, or situations, and are referred to as qualitative data. Qualitative data tend to be rich in both depth and detail and reflect personal experience. The most important thing to remember in choosing a type of data to use to answer a question or set of questions is that the type of data must correlate with the question you are attempting to answer.

CHARACTERISTICS OF QUANTITATIVE DATA

Generally quantitative data and data reports are in the form of numbers. Facts and figures are fairly concise, and consequently quantitative data reports can be brief and to the point. In addition, an organizations information system often contains a good deal of quantitative information that can be used for performance measurement; for example, service utilization and cost information may already be collected. Conclusions drawn from quantitative data are limited by parameters of the data.
23

CHARACTERISTICS OF QUALITATIVE DATA

Qualitative data consist of narrative description. The data obtained in qualitative inquiry is detailed and rich, and include information about personal experiences, beliefs, perspectives, or situations. Qualitative data provides a high level of information to assist in decisionmaking. If need be, one can almost paint a picture with words. The information needed by decisionmakers is often very specific, very detailed and very unique, and qualitative data techniques can be customized to meet their needs. Qualitative data, however, are not routinely collected by agencies or health care providers and can require additional resources. Because of the level of detail characteristic of qualitative data, data collection and analysis can be time-consuming and costly to collect and analyze. Many agencies already have quantitative data collection systems in place, so those tend to be used more frequently, even when qualitative information would be more informative. In a comprehensive quality assurance system, both qualitative and quantitative data will be necessary, since the questions that people need answered usually involve both types of information. In addition, integrating qualitative and quantitative data in a quality assurance system is generally the best way to provide a holistic picture of the status of the service delivery system.

Consumer Data
Two methods for obtaining information on quality from consumers of health care services are structured or quantitative methods and naturalistic methods.

STRUCTURED METHODS
Structured methods conform to traditional research protocols for data collection and include classic patient satisfaction surveys. The most well-known set of surveys currently in use is the Consumer Assessment of Health Plans (CAHPS) survey. The CAHPS surveys were developed through funding provided by the Agency for Health Care Policy Research for a five-year study, begun in 1995, to construct a set of replicable surveys. These surveys are designed to be practical, standardized, and flexible enough to provide an integrated system of components that users can assemble into new questionnaires tailored for a variety of populations. The CAHPS surveys consist of: a common core set of standard items; a small group of supplementary targeted items to be added to the core set (for use with specific populations in Medicaid, Medicare, persons with chronic conditions, and children); and a survey designed for those who disenroll from plans. The CAHPS questionnaires are designed to collect data on: access, communication, and interaction with health care professionals continuity and coordination of care preventive care administrative burden health plans customer service
24

enrollment personal contribution toward the premium utilization of health services health status respondent characteristics

The core survey items are designed to collect information from all persons currently enrolled in a health plan, who have been continuously enrolled for at least six months, and who consider the plan to be their main source of insurance. This six-month timeframe is shorter than that recommended for similar surveys developed by other organizations, which generally use a minimum continuous enrollment of twelve months as the norm. The sampling unit for the CAHPS surveys is the individual, not the family, and parents must answer surveys for their children. Sampling issues become more complex when the CAHPS materials are used with special populations because sample sizes must be increased for some of these populations in order to get numbers large enough to produce statistically significant results. In many states, the CAHPS surveys are being modified to meet the Medicaid programs specific needs.

NATURALISTIC METHODS
Naturalistic methods use lived experiences, and include direct observation of clients, analysis of client comments and complaints, personal in-depth interviews, and focus groups. A wide variety of approaches exist for conducting qualitative inquiry. In general, all qualitative methods utilize one of three approaches to collecting data: personal interview, direct observation, or examination of written documents. Personal interviews can include one or two individuals; focus groups include three or more individuals. Typically, a set of questions is developed for the use of all inteviewers to assure greater consistency across the interview or focus group. Both interviews and focus groups should be taped recorded, transcribed, and analyzed to assure accuracy. While questions are structured and determined prior to the inteview, additional, spontaneous discussion will most likely occur. This information should be included and noted as supplemental. Examination of written documents: Written documents, such as data files on appeals, complaints, and grievances may be available. in these cases, the researcher systematizes a method for reviewing and extrapolating information from the data file. Structure is once again important to assure the credibility of data collected. The pattern or themes that are uncovered from an analysis of written documents may indicate the need for additional inquiry or the be can used in conjunction with other types of daa to create a more thorough synopsis of what is happening. Another approach used for collecting data is the case study. Case studies most often include personal interview combined with direct observation and/or examination of written documents and are developed to take an in-depth and highly detailed look at a relatively small number of cases. Naturalistic methods are especially useful for collecting information from consumers whose knowledge and expectations may be very different from those of the general population, such
25

as HIV-positive pregnant women, or some parents of children with special health care needs. These methods can help your organization to determine how clients define quality, how clients access necessary services, and what are client expectations for health care. The information gathered through naturalistic methods can highlight areas in which further data collection and analysis is needed.

Sampling Methods for Collecting Data on Quality Assurance

Sampling is a statistical process used to calculate the probability that a small group of cases, i.e., a subset of the population, accurately represents the total population we want to study. Sampling is used when it is difficult, costly, or not possible to collect information from the entire population that is being studied. Sampling methods are used to determine what group of cases to select, which statistical methods to apply to these cases, as well as when and how to apply sampling methods to the group of cases. It is important to be aware of considerations regarding the sampling methods used in the collection of quality assurance data to help us make decisions about the target population for performance measures. For each performance measure we have two groups of individuals that must be identified: the group comprising the numerator and the group comprising the denominator. The denominator may contain either the target population for the performance measure or the total number of target population with specific characteristics. The numerator contains the group with the characteristic we want to know about for our performance measure. The reporting of quality assurance measures requires that adequate sampling procedures be implemented. Depending upon the size of the program, we may want to include every member of a particular group and not apply sampling methods. However, at other times this is impractical and we will need to sample a smaller population. When sampling a population, we must ask two questions: Who should be in the sample? How many should be in the sample? Who should be in the sample depends on our performance measure. How many should be in the sample depends on how the characteristic you want to measure varies in the population. Regardless of the characteristic of interest, the smaller the sample size, the more variability you will have in your measure. There are many types of sampling methods from which to choose that address specific challenges that our data needs may present. The three most prevalent sampling methods typically used in quality assurance systems include simple random, stratified random, and purposeful sample.

SIMPLE RANDOM SAMPLE

A simple random sample selects participants totally at random so that every person in the pool has an equal chance of being selected. The HEDIS guide has suggested approaches to pulling a sample for items such as childhood and adolescent immunization status, prenatal care, and well-child visits.
26

STRATIFIED RANDOM SAMPLE

A stratified random sample involves organizing the population into subsets and then drawing a simple random sample within each subset. If we wish to ensure that each group is represented, for example, by age, sex, race, ethnic background, or geographic location, this is the sampling technique to use. For HEDIS, if a plan wants to use this method they must get permission from the National Committee for Quality Assurance, which has a committee of statisticians and health experts review the methodology.

PURPOSEFUL SAMPLE
This is not random at all. This method involves choosing a specified group of people, and the information collected from this group cannot be extrapolated to describe a broader population. For example, if we want to interview only the people who indicated on a survey that they were very dissatisfied with a provider service, the resulting data could not be interpreted to reflect the entire population using that provider service.

Statistical Significance
Statistical significance is an important concept for comparing a health care plans performance to standards, and therefore should be calculated before sampling. It is important to know that data are statistically significant; i.e., whether the data result by chance or in connection with an event. In this context of data analysis context, the term significant means discernible, rather than important. Statistical significance detects differences from the standard or among plans; we do not have the evidence to hold a plan accountable for change if the data are not statistically significant. The HEDIS guide addresses the issue of statistical significance by specifying the exact number of persons required for a sample. If we develop new performance measures, we will need to do address the issue of statistical significance ourselves, or hire experts to do so. Probably the single greatest barrier for those working with CSHCN/MCH populations is that many of the target populations of interest to us (such as adolescents, CSHCN, or children age 0-3, to name a few) are small. Therefore, in order to have statistical significance, we need to draw an adequate sample from each population of interest. In fact, for many of these smaller groups, no sampling should be done at all because the groups are too small to be statistically significant.

Target Value
As described at the beginning of this chapter, Step 5 of our quality assurance model involves defining the range of values to expect for a performance measure. The expected range of values are those which are reported for the performance measure. For many performance measures, the range of values expected would be from 0 to 100 percent. The range of values may also be expressed as a scale, e.g., from poor (0) to excellent (10) on a ten point scale. Knowing the range of values for all of our performance measures is important in determining the level of quality associated with each measure, which is Step 6 in our quality assurance model.
27

A target value is the measure of quality you want to achieve. The target value defines where in the range of values the health care providers should strive to be. Without an expected target level for a measure, it is difficult to interpret and analyze your indicator data. The target value helps answer the question: Did the plan do well or poorly on this measure? As an example, a Healthy People 2000 objective for infant health is to: reduce low birth weight to an incidence of no more than 5 percent of all live births. The target value in this case is 5 percent. In Healthy People 2000, many of the measures have target values that were developed based on the knowledge of current and historical data on the measure. In quality assurance, however, because the quality assurance process is new and plans have not been collecting data for the process longer than a year or two, we have few benchmarks for many of our measures. It is difficult to develop meaningful objectives and surveillance strategies when we have no idea what to expect from our measures. It is typical to use the data we obtain in the first two years of a new quality assurance process as baselines from which to develop meaningful objectives. Many plans report the average as their standard. There are problems with using the mean or average if the data values are not consistent over time. Some health plans that use the average as a target value also report the standard deviation of the average, which tells us the stability of the average across populations, or across plans or over time. Remember that any standard we set for a measure could not apply to everyone served by the agency or plan. We know that income, residence, and culture all contribute to the selection of a particular target value. In the absence of historical data to help us decide on a target value, state or national survey data become very useful resources. Expert panels may also help us to determine a level of acceptability or whether using the mean or average is adequate.

Confidence Intervals
After the sample is drawn and the performance measure is reported, a confidence interval for the measure should be calculated. Confidence intervals are statistical estimates of certainty that define the degree to which we can be confident that the data occur within a specified range. To illustrate the significance of a confidence interval, we will consider the following example, using a measure of: the proportion of children under age 6 that had a well-child visit in the past twelve months. Assume that our target value for this measure is 80 percent. If a health care plan reports that 75 percent of children under age 6 had a well-child visit in the past twelve months, we should ask: Should the plan develop a corrective strategy? Should we break the contract with the plan? The first step in answering these questions is to calculate a confidence interval for this measure. A confidence interval is used to assess the reliability of statistics by estimating a
28

range of values that includes the unknown population value with a given probability. A narrow confidence interval indicates that the population value is probably quite close to the sample estimate; a wide confidence interval indicates that the population value may be quite far from the sample estimate. Assuming we have calculated the confidence interval for our above example to be 65 to 85 percent, we would be fairly certain that the true value we are seeking for our measure lies somewhere between 65 and 85 percent. (Remember, we stated that the target value for our measure is 80 percent.) In this case, the confidence interval is so wide we cannot say that the health plan has not met the target value because 80 percent is within the calculated confidence interval (of 65 to 85 percent). This situation could have been corrected by taking a larger sample to ensure a narrower confidence interval around the value. With a larger sample, the plan may have reported the same value of 75 percent for the measure but with a narrower confidence interval of 72 to 78 percent. With this new interval we would be fairly certain that the plan did not meet the target value of 80 percent because the 80 percent value is not included in the interval. Therefore, we can now hold the health plan accountable for not achieving the target value. The target level we choose for a measure can come from experience with the data, a state or national average, a recommendation by an expert group, and so on. Regardless of the target value, however, to use data to track a health care plans performance over time or to compare different plans, we must be aware that sampling is important and that the variability of a measure, which can be affected by sampling, can affect our interpretation of plan performance or meeting a target.

Data Collection from the Practitioners Perspective

Many of the data collection methods discussed in this chapter are from the administrative plan perspective. Some quality assurance data, however, e.g., data on effectiveness of care, can only be gathered from the primary care physicians who actually see the patients. The following is a summary of the article, 10 Practical Pointers for Tracking HEDIS Data (Carolyn Buppert, C.R.N.P ., JD, Managed Care Daily Briefing), which discusses data collection from the practitioners perspective. What should providers do to prepare for collecting data related to quality assurance? Develop a tracking form A quality measure tracking form is a tickler, i.e., a reminder for health care providers and staff to address the issues of health care maintenance with patients. The tracking form is also a record of what has been done. Health plans often gather this type of data through referral and billing forms. A chart audit verifies data collected previously from other sources. Whether we use a tracking form as a reminder that generates a referral, as a document of record, or as a time-saving device, the each health care provider should complete the tracking form as part of the review of a patients chart at each visit.

29

 Appoint a staff member to be the practices quality expert Whether it is a physician, nurse, or practice manager, this expert must know how performance measures are constructed, the data sources from which to collect the data, and how to overcome any data system incompatibility problems. The quality assurance expert must often remind health care providers to complete the simple tracking form. The expert may need to monitor the software used to track the data, and perform quarterly chart audits to determine whether people are completing the tracking form. When auditors arrive on site, it is the experts responsibility to provide them with the both paper and electronic records and to assist them in any way. Do internal audits regularly Some health care providers may be diverted from health care maintenance issues when they are involved in complicated diagnostic issues. Some health care providers will need feedback to keep them on track with the quality assurance requirements. The quality assurance expert should select fifty charts per quarter and audit them to ensure that indicator tracking forms are up to date. Health care providers who have not kept their patients forms up to date should reminded of the importance of the tracking form. Make the pertinent questions part of the intake routine For example, if a triage nurse takes vital signs and chief complaint, have the nurse ask such questions as Are you smoking? When did this child last get his tetanus booster? When did you have your last Pap smear and what were the results? The nurse can update the tracking form or alert the physician about what is needed. Complete the form Practices with incomplete or blank tracking forms will fair poorly when the report cards are issued. If a health care provider or a quality assurance expert finds incomplete forms, he or she should make an effort to eliminate the omissions on the form. Appoint a person to follow up with labs Tests that are ordered and completed, but that have no follow up are a potential legal liability. Implement a policy that requires each health care provider to be responsible for following up any quality-related diagnostic test that he or she orders, and allow time for this task. Alternatively, appoint an individual who is responsible for making sure all laboratory results are noted on the patients chart, documented on the flow sheet, and followed up. Give specific follow-up for diagnoses targeted by HEDIS Develop forms and procedures that incorporate specific HEDIS targets. For example, HEDIS targets otitis media in children. Develop a form for this diagnosis that documents the appropriate treatment and follow-up.

30

 Keep current on the quality assurance data required Have the quality assurance expert check the required information a several times a year. Make sure that health maintenance organizations (HMOs) auditors know the practices plan for quality compliance Each health plan that contracts with a practice will send its own auditors to the practice. Have a staff member knowledgeable about their required measures greet the auditors, brief them about the practices compliance plan, show them where the tracking form is positioned in a chart, and where to find follow-up data. Incorporate compliance with quality assurance data tracking into job descriptions and include rewards for excellent performance in the bonus formula. Some HMOs and preferred-provider organizations (PPOs) give bonuses to practices that score high on performance measures.

Exercise 1
You represent a health service program that provides for services to a population composed primarily of women and children, including CSHCN. At a meeting organized to determine the requirements for developing a quality assurance health care system you have been asked to present the concerns of the MCH/CSHCN community. As part of your presentation, you decide to create a framework to develop and evaluate quality assurance indicators. To do so, you must first identify a health issue in the MCH/CSHCN population that you wish to use in your presentation. Create the framework for your presentation using the first six steps of the quality assurance process discussed in Chapters Two through Six of this guide. Record your responses on the Quality Assurance Measures WorksheetExercise One that can be found at the end of this chapter. A sample of a completed Worksheet using the example of children with asthma can be found in the Answer Key in the Appendix of this guide. Review the steps discussed in Chapters Two through Six: Step One Identify the health issue to be addressed. Step Two Describe the target populations(s) in detail (e.g., age, sex, income, and location) affected by your issue (e.g., children ages 0-3, women of childbearing age). Step Three Define the desired goal for the target population you selected in Step Two. Step Four Define one or more performance measures for the goal. Remember that quality assurance measures should address the structure, (i.e., care delivery), process, (i.e., providing appropriate health care in an appropriate manner), and outcomes (i.e., end of result care) of the goal. For each of these three dimensions of quality, identify one to three measures that affect the goal and ensure quality. Step Five Define the range of values for each of the measures. Step Six Define the target value. Be sure to identify the data sources (actual or hypothetical) that could be used to develop a target value.
31

32

Quality Assurance Measures WorksheetExercise 1

Step One
Health issue to address:

Step Two

Describe the target population(s) in detail (age, sex, income, location, etc.) for this health issue (e.g., 0-3 Hispanics, women of childbearing age, children with asthma, etc.)

Step Three

Define your desired goal

Step Four
Define Measures A. Structure

Step Five
Range of values

Step Six
Expected target value

33

Step Four
Define Measures B. Process

Step Five
Range of values

Step Six
Expected target value

C. Outcome

35