Professional Documents
Culture Documents
Go Beyond Basic Behavioral And Analytics Data By Using Surveys To Really Understand The Who And The Why Of Your Site Visitors
AN ORANGE PAPER BY
TABLE Of CONTENTS
PART I: WHAT IS A SCALE? PART II: WHY USE A RESPONSE SCALE? PART III: CHOOSING A RESPONSE SCALE - GRANULARITY PART IV: CHOOSING A RESPONSE SCALE - SCALE POINTS PART V: CHOOSING A RESPONSE SCALE - BAGGAGE & BALANCE CONCLUSION ABOUT CROWD SCIENCE 4 5 6 8 10 12 13
The most effective and efficient way to influence these elements is to gain a deep demographic level understanding of a particular website audience. This position paper has been developed to shed some light on the importance of creating a high quality survey questionnaire. Since the inception of the Web, the Crowd Science team has had extensive online survey experience and learned a lot about survey best practices. One of the most important aspects of a survey questionnaire is the response scale. The response scale has a key role in the type and quality of the analysis that can be drawn from your survey research. Sandra Marshall, Crowd Science VP of Research, explains why the response scale matters.
RESPONSE SCALE
Very unlikely 1
Somewhat unlikely 2
Somewhat likely 4
Very likely 5
ORANGE PAPER RESPONSE SCALES PART II: WHY USE A RESPONSE SCALE?
A key aspect of quantitative, closed-ended questionnaire design involves not only how the question is worded, but also how we design the response categories. Why ask one question to get one response, when you can ask one question to address several items? For example: Instead of asking whether a respondent plans to buy a new car in the next 12 months, ask the degree of likelihood to buy within the next 12 months. Rather than asking whether a respondent is merely satisfied or dissatisfied with a product or service, ask their level of satisfaction or dissatisfaction.
Scales that go beyond this or that response options open up a new set of possibilities to better understand respondents. The proper use of response scales can inform where risks lie and provide the opportunity to be in a better position to minimize them. Shades of Grey A response scale offers more detail than a simple black or white type of response. It digs into the range of space between the black and the white: Yes or No? Do You or Dont You? Will You or Wont You?
A scale is almost like having a dimmer switch the respondent can dial up or dial down the intensity of their response. You end up with more detail on the degree of activity, intent, or attitude. You also have more options when it comes time to conduct the statistical analyses. With this level of detail, smarter decisions can now be made on how or where to allocate resources and efforts to maintain and grow customer bases/audiences/etc. So next time you catch yourself asking whether, see if the question translates into a scale format that now allows you to ask how much. Itll go a long way in providing a full picture, and better answers.
ORANGE PAPER RESPONSE SCALES PART III: CHOOSING A RESPONSE SCALE - GRANULARITY
Once youve decided to use a scale, you are now faced with the decision of what kind of scale to use, and what it should look like. There are many ways to present response scales, and each one has its particular strengths and purpose. More Points = Greater Sensitivity As with everything in research, always go back to the objectives: What is the desired goal (or research hypothesis)? How do you intend to use the results?
This will drive every aspect of questionnaire design, including the appropriate response scale to use. For example, lets say you want to understand customer satisfaction to track changes between specific time periods which are fairly short (hourly, perhaps). You may therefore want the response scale to be more sensitive in order to pick up on the smallest of changes. This will require the use of a more finely tuned scale - one with more points on it. This would enable you to act nimbly on any change that was recorded from hour to hour. But perhaps you may not want to be bothered by minute fluctuations, in cases where they may be distracting, less meaningful, or in some way not worthwhile to act upon. Perhaps the measurement normally fluctuates a great deal, and you need to obtain a sense of the bigger picture, where fluctuations go beyond the norm. In these cases, a smaller number of scale points might be in order. Again, there may be other factors affecting your decision when choosing the right scale range. Even or Odd? When deciding on how many points to include on the scale, another decision to make is whether to have an odd or even number of points. An odd number provides a midpoint, which may or may not be desirable, depending on your objectives, and how you intend to use the information. You may not want to use an odd number in order to avoid a neutral response. For example, perhaps you are measuring satisfaction, and you really want to know whether your customers are satisfied you want to force the issue and get them to choose a positive or negative rating, without sitting on the fence. You may decide on a scale that includes the following even number of points:
Somewhat dissatisfied 2
Somewhat satisfied 3
Very satisfied 4
By offering no point between dissatisfied and satisfied, you force the respondent to commit to one side or the other, presumably in order to know whether/how much you need to improve your offering. However, lets say youre asking about some hypothetical future behavior, such as intent to purchase. Its possible that the respondent truly does not know whether they will carry out this behavior, because they require more information (costs, benefits, their future needs/situation, etc.) before making a decision. Similarly, asking opinions on topics the respondent has never before encountered, means it is possible that they have not yet formed any opinion on the subject at all. In such cases a midpoint will offer a position where they can respond without committing to a positive or negative answer.
Somewhat dissatisfied 2
Somewhat satisfied 4
Very satisfied 5
Alternatively, if an even-numbered scale is offered in addition to a separate Dont know category, this could achieve the same result. Either way, its important to be sensitive to situations where forcing an answer one way or the other may be inappropriate. In the end, when deciding on the number of points in your scale, think ahead to the analysis and how you want to present results, think about the nature of the measurement youre taking, and think of the respondent experience in answering your question.
ORANGE PAPER RESPONSE SCALES PART IV: CHOOSING A RESPONSE SCALE - SCALE POINTS
Anchors Aweigh! So far, weve addressed why to use a response scale, and some things to consider in deciding the number of points in the scale. Now lets discuss how to label the scale points. A Picture Is Worth A Thousand Words Some scales dont even use verbal descriptions, and rely on diagrams to conveying meaning, such as simple line drawings of faces with various expressions (e.g., big smile, medium smile, straight-line mouth, medium frown, and big frown). These types of scales may be successful with audiences (like children) that are preliterate, or for whatever reason have difficulty expressing themselves with words.
Such situations may require an interviewer to assist in administering the questions if this is the case. However, even with respondents able to read the questionnaire, such scales may make the experience more fun or engaging, depending on the situation. Too Many Words Spoil The Broth Most scales use words to define their points. You might consider putting verbal definitions on every point in the scale (a.k.a. fully anchored), or to use a partially anchored scale, say, only defining the two opposite ends. Some people wonder about labeling the two ends, plus a label for the midpoint if there is one. A general rule of thumb is to fully anchor or label a scale if there are five or fewer points on it, and for scales with six or more points, a bipolar anchor can do nicely.
Somewhat satisfied 5
Very satisfied 7
Very satisfied 7 6
Somewhat satisfied 5
Mostly satisfied 6
Very satisfied 7
Twos Company, Threes A Crowd You may wonder if it is helpful, when using bipolar anchors on each end of a scale, to also label the midpoint to provide another point of reference for the respondent. You are arguably introducing risk by calling unnecessary attention to the midpoint. This emphasis on the midpoint may tempt respondents to use it as an out from having to think about their response and provide an opinion. Midpoints are also tempting fallback for lazy or fatigued respondents, so the less attention to the halfway point in the scale, the better the results.
ORANGE PAPER RESPONSE SCALES PART V: CHOOSING A RESPONSE SCALE - BAGGAGE & BALANCE
Lose The Baggage, Keep The Balance When constructing a response scale, its important to achieve a nice variety of response options across the scale. A scale that tends to consistently elicit responses skewed toward one very small part of the scale doesnt tell you much. It would be like measuring agreement with the statement adequate sleep contributes to good health. Most people would agree this statement is true; there would probably be minimal variability in the results, either across time or across respondents. On A Scale Of One To Ten In many cases, the nature of the question causes responses to skew a certain way. But sometimes it is the nature of the scale that skews the data in a particular direction. For example, consider 10-point scales, e.g. On a scale of one (or zero) to ten, please rate your agreement/satisfaction/etc. with the following statement. The potential problem with a 10-point scale is that it carries a lot of baggage from prior experience. This baggage can include familiarity with other questionnaires that have used this scale. It also can stem from the popularization of various commonly used 10-point rating or ranking systems. For example: David Lettermans Top 10 The movie 10 [Hello, Bo!] School tests scored out of 10 Casual discussions On a scale of one to ten, what did you think of that movie?
You may be saying, So what? These examples demonstrate how respondents are already conditioned look at 10-point scales in a predetermined way. For instance, think of those tests or quizzes that everyone took as kids in school, graded out of ten. Here we have been conditioned to focus on the upper end of the scale. Specifically, most of these tests required at least a 5 to pass. Anything below 5 was a fail. It didnt really matter by how much you failed, either - after all, if you failed the test, you failed! There is little importance or meaning attributed to the differentiation of any scale points below 5. But for points 5 and upward, we attribute differences for each of the scores, even corresponding the numbers to different letter grades, A through F. This is one situation that can be conditioned into respondents from a fairly early age and contribute to our reluctance to use the lower part of the 10-point scale.
7 POINT SCALE
Very dissatisfied 2 1
Very satisfied 7 6
OUR PRODUCTS
Get accurate intelligence about audiences that cant be uncovered using standard analytics, behavioral inferences, or low quality surveys. Built on SmartSample technology Questionnaires pre-built by our researchers Rich & statistically valid results
AUDIENCE PROfILER
AUDIENCE TARGETER
Audience Targeter provides publishers and networks with direct paths to segmenting, targeting and monetizing their audiences. Measurement collect deep, customized audience data Segmentation organize audiences into sellable subgroups Optimization bridge segmentation metrics with ad serving Extension grow frequency & reach via re-targeting
CAmPAIGN mAxImIzER
Delivering real-time results that allow you to understand and report key brand campaign metrics: Brand Lift favorability & usage Awareness aided & unaided recall Intent purchase intent
SITE SATISfACTION
Site Satisfaction research provides publishers with actionable information about their audiences. Services include: Full service program management Cost-effective research & actionable deliverables Innovative technology & real-time dashboards
Full suite of traditional custom online research services, including: Research consultation & design Questionnaire design & programming Data collection & fielding Reporting, analysis, & insights
2011 Crowd Science. All Rights Reserved. 13