VOIC

E
AN SRC TOOL

©2004 The Lewis Group. All Rights Reserved.

How to Evaluate a Research Proposal
Whenever you use research findings to make decisions, you need to evaluate the process
used to generate the findings. If the research process is flawed then you need to be
cautious about using the research findings. The SRC works with the state agency to
conduct research and you have an obligation to determine if the research process will
produce valid and reliable findings. The Statewide Consumer Needs Assessment and
Customer Satisfaction Survey are just two examples of research projects that you and
agency management undertake together. The findings from The Statewide Consumer
Needs Assessment and Customer Satisfaction Survey are two of the most important
inputs to decisions made by the SRC and agency management. It is imperative that any
research studies you rely on to draw conclusions about consumers’ needs and the
agency’s performance follow widely accepted guidelines for consumer market research.

The purpose of this SRC Tool is to:

1. Provide you with an overview of the market research process.

2. Familiarize you with some of the technical terms researchers use so that you
can have more meaningful dialogue with them.
3. Help you make decisions about research studies you shape.
4. Help you evaluate the quality of research findings from studies conducted by
others.

Market research is a profession. There is a considerable body of scientific knowledge that

influences researchers’ decisions about every aspect of research design. Generally
accepted practices guide researchers in making decisions about what data to collect, who
to collect data from, and how and when to collect the data. These professional norms
also determine if a census or a sample is appropriate. If sampling is used, the researcher
needs to decide how to sample the population. Should a convenience or a random sample
be drawn? Does a proportional sample best fit the objectives of the research or is there a
reason to over sample some subgroups? A variety of analysis methods exist and the
researcher must determine which methods are most appropriate for each study.

The research process determines the quality of the findings from a study. If you want to
enhance your credibility and influence then you need to be disciplined about insisting on
the use of research professionals to design your consumer research studies, to supervise
the research and, to interpret the research findings. Remember that the evidence based
decision making process is grounded in the appropriate collection, analysis and
application of information.

When others don’t agree with your thinking or don’t like the findings from a research
study, they usually attack the process you used in your research study. The research
process needs to be absolutely defensible in that it is based on widely accepted research
principles and guidelines for consumer research studies. Anything less will leave you
open to criticism that will be difficult to neutralize.

11/2004 RSA SRC Training Initiative Page 2 of 9 Created by M. Christine Lewis, PhD
Why Do Research?

Consumer research is conducted to support decision making. The Statewide Consumer

Needs Assessment and Consumer Satisfaction Survey help the SRC and agency
management make decisions about policies and practices. The research provides
information useful in making decisions about what services to provide, how to make
services available, what the service delivery system should look like, how to
communicate the agency’s services to consumers and referral sources, what to focus on to
improve consumer satisfaction and other similar decisions.

Questions You Need To Answer

Six questions need to be answered before a decision to conduct a research study can be
made. The SRC should work with the state agency personnel responsible for conducting
prior studies on the subject or your research. You may also engage an independent
research professional to help you answer these questions. Often, this is an iterative
process, involving the client [You] and the Researcher. The client, which would be you
and the VR agency, would explain to the Researcher what decisions you would like to
make. The Researcher is responsible for translating your objectives into research terms.
The questions that must be answered are:

1. What decision [or decisions] is this research study supposed to support?

2. What information is needed to make a good decision?
3. What do we already know?
4. What information do we need but don’t have?
5. Can we get this information by conducting this research study?
6. Does the value of the information justify the cost of the research?

What’s The Research Plan?

If you decide to conduct a research study then a research plan is developed that lays out
in substantial detail how the data will be collected and analyzed. The research
professional you’re working with is responsible for developing the research plan. The
plan describes all of the following elements:
• Background [What Are the Circumstances That Led To the Study?]
• Management Decision[s] The Research Study Will Support
• Research Objectives
• Research Design
• Report And Presentation Plan And Format

11/2004 RSA SRC Training Initiative Page 3 of 9 Created by M. Christine Lewis, PhD
Do the stated objectives refer to the management decision[s] the
study is supposed to support? Do the objectives spell out what
information will be gleaned from the research findings and how
this information will help management make these decisions?

The researcher is responsible for translating management decisions into researchable

issues, information needs and data requirements. This usually occurs through a series of
discussions between the client [the SRC and the state agency] and the researcher. The
researcher should be asked to prepare a written summary of the discussion that describes
his understanding of your objectives and the specific management decisions that the
findings from the research study will support. This written summary is reviewed by
members of the SRC and agency management.

If you determine that the researcher does not fully understand your objectives then it is
incumbent upon you to arrange for another discussion with the researcher to clear up any
misunderstanding. The researcher should again be asked to prepare a written summary of
the discussion that describes his understanding of your objectives and the specific
management decisions that the findings from the research study will support. This
written summary is reviewed by members of the SRC and agency management. This
process continues until you are satisfied that the researcher fully understands your
objectives and knows what information you expect the research findings to provide.

One important benefit of this process is that it helps members of the SRC and agency
management clarify their own thinking and ensures a more precise set of research
objectives. Quite often you will find that engaging in this iterative process with the
researcher helps you deepen understanding of the research objectives among board
members and management. It strengthens support for the research study and ultimately
produces more useful information.

Target Audience

Who will we get this information from? It is important to understand how the target
audience will be identified and reached. Will the research study target consumers who
are or were served by the state agency? If so, the agency has a record for each consumer
served. The record should include basic contact information for each consumer such as
their mailing address, phone number and email address if they have one.

Most agencies verify the consumer’s contact information when the case is closed. Even
so, the contact information for many consumers is incorrect. In part, this results from the
lack of a permanent residence for some consumers. Another cause of “dirty data” or
incorrect consumer contact information includes an inconsistent effort to gather this
information at case closure. Data entry errors can also contribute to the problem of
incorrect contact information. All databases contain some incorrect contact information.
This only becomes a source of concern if it causes one of two problems.

11/2004 RSA SRC Training Initiative Page 4 of 9 Created by M. Christine Lewis, PhD
The first problem is that time and money are wasted trying to reach consumers who can’t
be reached because the contact information is incorrect or missing. Of course any effort
to clean up the database should be weighed against the cost of trying to reach consumers
who are unreachable due to incorrect contact information. If the “unreachables” are a
microcosm of the overall consumer population you’re better off accepting this as a
normal condition of any database.

The second problem is that “dirty data” can skew the results if contact information is
more frequently missing for a specific segment of consumers. For example, let’s assume
that African Americans represent 30 percent of total clients served but represent 65
percent of the consumers with incorrect contact information. In this case, African
Americans will be under represented in the study. This happens because we can’t reach
almost two thirds of these clients. If we’re using a random sample, we may find that only
about ten percent of the sample is African American. This is unacceptable and should be
addressed.

The SRC can help improve the quality of consumer research by determining if the state
agency has a problem with incorrect or missing contact information. Begin by asking
about response rates in previous studies. Ask the agency to determine what percentage of
non respondents could not be contacted and why. If the survey was mailed, the agency
receives information on all undeliverable surveys. If the survey was conducted by
telephone, the agency should have a record of how many consumers targeted for a study
could not be reached due to an incorrect or missing telephone number. You should also
ask agency management to review response rates for important segments of the consumer
population to see if any group is underrepresented.

Research Design

The next step in the research process is to choose a research method best suited to meet
the objectives of the research study. The researcher can select from a myriad of research
design alternatives such as focus groups, individual depth interviews, mail, telephone, in-
person or web based surveys.

In general, research studies are classified as either qualitative or quantitative. Qualitative

research usually takes the form of focus groups. Focus group research helps illuminate
how the target audience thinks about and makes decisions about the subject of interest.
In the context of vocational rehabilitation, a focus group research study may help you
understand how consumers think about their employment related needs and make
decisions about them, including decisions to seek services from the state VR agency.

Focus group studies are often characterized as exploratory. These studies are useful in
understanding why people do what they do. Focus groups give us insight into the
motivations that drive consumers’ actions. Focus group studies are usually small,
involving as few as 20 or 30 people. Participants are not chosen through a random

11/2004 RSA SRC Training Initiative Page 5 of 9 Created by M. Christine Lewis, PhD
sampling process where every member of the target population has an equal chance of
being selected for the study. The research findings take the form of a narrative summary
of how participants think and behave; they cannot be subjected to statistical analysis.
Technically, the findings cannot be generalized to the total population of interest but
often are in spite of the limitations of the research design.

Quantitative research studies are more useful in helping us understand what consumers
think or do with regard to the subject of interest. Quantitative studies often take the form
of surveys. Surveys are almost always designed to be representative of the population of
interest. Some form of random sampling is employed and the data is subjected to
statistical analysis. This sounds daunting but it isn’t if you’re a trained researcher. The
chief benefit of survey research is that the findings can be generalized to the population
of interest.

Often focus groups are conducted to gain insight into how the target audience thinks and
makes decisions about the subject of interest as a prelude to a larger survey. The focus
group research is exploratory and helps shape the survey objectives, sampling plan,
questionnaire and other important aspects of the study. Survey research is often referred
to as “confirmatory”. This does not mean that the survey research does confirm the
findings from the focus groups. It means that the survey is capable of validating the
focus group findings. The survey may confirm or disconfirm the findings from the focus
groups.

Sampling Plan

Census or Sample When a research study is designed to include all members of the
target population it is called a census. It is rarely advisable to conduct a census because
sampling methods enable you to lower the costs of the research while maintaining a high
degree of validity. This means that the research findings represent the population of
interest and can be generalized to the entire population. In addition to being less costly,
sampling is often faster.

When determining the sampling plan, a key consideration is how the data will be used.
You should carefully consider the costs of making an incorrect management decision. In
addition to monetary costs, there could be political or public relations fallout. When the
costs of making a mistake are high, a higher degree of confidence is called for and a
larger sample is required. Even so, it would be very difficult and often impossible to
justify the cost of taking a census.

Determining the correct sampling plan can require extensive work. Laypersons may be
left to engage in guesswork to make the correct choice. This guesswork, if incorrect, will
yield survey results that inaccurately reflect the sentiments of the population, and
therefore compromise any conclusions drawn based upon the results.

11/2004 RSA SRC Training Initiative Page 6 of 9 Created by M. Christine Lewis, PhD
Many factors should be considered when determining a suitable sampling plan, such as
the total population size, appropriate confidence level, anticipated participation level, and
other technical issues.

A key consideration is whether survey results need to be interpreted at the sub-group

level. Examples of sub-groups are cultural or racial minorities such as African
Americans or Hispanics. Other examples could include consumers with a specific
disability or combination of disabilities or other characteristics such as whether they have
internet access or work history. Unless proper sampling strategies are implemented, sub-
group results may not be representative of the views held by members of sub-groups. If
subgroup results are not representative, developing and implementing effective strategies
will be challenging at best.

Judgment Samples Focus group studies do not require use of a random sample because
the objective is not to generalize the findings to the entire population. Rather, the goal of
a focus group study is to help us deepen our understanding of why consumers behave as
they do or think as they do about their rehabilitation and employment needs. The
findings will help shape a larger survey. Participants are usually chosen from a judgment
or convenience sample.

The SRC is well advised to consult an experienced survey researcher or statistician for
advice on sampling issues. These are not decisions that members of the SRC are
qualified to make, unless, of course, they are professional survey researchers.

Questionnaire Design

Questionnaire design is quite a science and should also be undertaken by a research

professional. Experienced researchers understand the characteristics of marketing data,
levels of measurement and consequences of improper manipulation of research data.

An experienced researcher will help you avoid common errors in questionnaire

construction. The researcher will make decisions about what to ask and how, the flow of
the questions, question structure and wording. The researcher will also make decisions
about the use of open vs. close-ended questions. In general, you want to use open ended
questions when doing qualitative research but closed ended questions on a survey.

Open ended questions allow participants to share their thinking and practices. In
exploratory studies you want to encourage the participants to share in-depth. In contrast,
surveys are structured to facilitate statistical analysis. Closed-ended questions yield more
useful information and are subject to a wide array of statistical analyses. Open-ended
questions yield limited information on a survey and are time consuming and costly to
analyze. One open-ended question is fine but more than that is a poor use of resources.

It is best to let the researcher make decisions about alternative procedures for obtaining
data from respondents; choosing multiple choice, ranking procedures, rating scales,

11/2004 RSA SRC Training Initiative Page 7 of 9 Created by M. Christine Lewis, PhD
constant sum and other procedures that will affect the types of analyses the data can be
subjected to later.

You should insist on pre-testing the questionnaire to discover any problems with flow,
question structure, wording and the ability of participants to understand what you’re
asking. Usually this can be accomplished with a small sample and is well worth the time
and cost, both of which should be relatively minor.

Data Collection Method

Data collection methods differ for qualitative and quantitative research studies.
Qualitative research relies on focus groups or in-depth personal interviews. Both have
similar goals and use a similar method. A focus group is a group of 8 to 12 participants
who share similar characteristics of interest. They may share similar needs or experience
paths. One example of an experience path might include clients whose rehabilitation plan
included education or training. A minimum of three focus groups should be conducted
with each sub-group of interest. A trained but unbiased moderator should conduct and
analyze the focus groups.

Quantitative research most commonly takes the form of a survey and the data can be
collected using all of the following methods:

• Face-to-face
• Telephone
• Internet or email
• U. S. mail

Multiple data collection methods are needed when surveying people with
disabilities and should be carefully considered when planning the research study.
It is unlikely that you will use face-to-face as a primary data collection method in
survey research due to the expense. This makes it imperative that you provide an
800 HELP telephone number that consumers can call for assistance in completing
the questionnaire.

Bias

Bias can be introduced into a research study quite innocently, usually because lay people
make decisions about research design without consulting a trained researcher. All of
your great work can be negated if the research results are biased in some preventable
way. The most common source of preventable bias is the use of state agency personnel
to conduct the research and analyze the findings. One could argue that the use of state
agency personnel to complete certain steps in the research process introduces more
egregious bias. The steps most often mentioned as being particularly vulnerable to bias
are data collection and data analysis. If the agency feels that it cannot avoid using staff
to complete these steps in the research process, it is absolutely essential that direct
service staff, especially counselors not be selected for this purpose. Staff asked to

11/2004 RSA SRC Training Initiative Page 8 of 9 Created by M. Christine Lewis, PhD
complete any part of a research study should be research professionals with the possible
exception of staff used to answer the 800 HELP telephone line. These staff, however,
must be trained to take the phone calls and may need to pass these calls to more skilled
personnel.

The best approach avoids even the appearance of bias by limiting the use of state agency
staff to design and conduct research studies. You need to take this issue seriously if you
want to use the evidence based decision approach. You need to produce credible
evidence that you can use to build a case – a defensible case to explain positions you take
and decisions you make.

Research Report and Presentations

The research proposal should also describe how the report will be organized and
distributed. If the research findings will be formally presented to you and or other
audiences, the research proposal should describe the general presentation plan.

A Regional Approach

Lower Costs and Greater Insights

The SRC should evaluate the possibility of pursuing some research projects on a regional
basis. The potential benefits include lower costs and greater insight into the subject of
your research. Region I is already using a regional approach to conduct consumer
satisfaction research. Region IV used a regional approach to conduct research with
employers. Regional studies enable you to compare and contrast the findings within your
state to other states and to the region. Regional studies often provide more insight into
sub-groups of interest because the sample size is larger and supports more detailed and
sophisticated analysis.

A Unified Voice

Using a regional or even a national approach to consumer research produces a more

unified voice. A unified voice makes the entire vocational rehabilitation community
more effective as an advocate for people with disabilities.

11/2004 RSA SRC Training Initiative Page 9 of 9 Created by M. Christine Lewis, PhD