Questionnaire Validation in a Nutshell

1. Generally speaking the first step in validating a survey is to establish face validity. There
are two important steps in this process. First is to have experts or people who understand
your topic read through your questionnaire. They should evaluate whether the questions
effectively capture the topic under investigation. You might have them pretend to fill out
the survey while scribbling notes. Second is to have a psychometrician (i.e., one who is
expert on questionnaire construction) check your survey for common errors like double-
barreled, confusing, and leading questions.
2. The second step is to pilot test the survey on a subset of your intended population.
Recommendations on sample size for pilot testing vary. Some academicians are staunch
supporters of things like a 20 participant per question. Well if your survey has 30
questions, that means that you’ll need at least 600 respondents! I think this standard
should be relaxed – perhaps it has been. Trust me, it is possible to validate with far fewer
participants. I’ve taught classes where I gave students questionnaires containing a small
proportion of somewhat irrelevant questions. After they filled out the form I pointed out
which questions were weak – they had no idea. Then we ran the statistics on their
responses and guess what? The analysis revealed that the somewhat irrelevant questions
be dropped. It worked time and time again with about 35 students. (I don’t recommend
telling a journal reviewer that you only need 35 pilot testing participants because some
guy on Methodspace said so. The more participants the better, but if all you can get are
60 participants, it may be enough, especially if your survey is short [about 8-15
questions].)
3. After collecting pilot data, enter the responses into a spreadsheet and clean the data. Here
is an important tip: Have one person read the values while another enters the data. Having
one person read and enter data is highly prone to error. After entering the data you will
want to reverse code negatively phrased questions. When used sparingly, negatively
phrased questions can be very useful for checking whether participants filled out your
survey in a reckless fashion. If they read the question carefully, their responses to
negatively phrased questions should be consistent with responses to similar positively
phrased questions. If they are not consistent you might consider tossing out a person’s
responses. It is also wise to check maximum and minimum values for the entire dataset.
If you used a 5-point Likert-style scale but find a response of 6 you’ve probably
identified a data entry error.
4. Identify underlying components using principal components analysis (PCA). Component
or factor loadings, as they are sometimes called, tell you what factors are being measured
by your questions. Questions that measure the same thing should load onto the same
factors. Factor loadings range from -1.0 to 1.0. When grouping factor loadings I usually
look for values that are ±0.60 or higher, although this varies depending on what the rest
of the loadings look like. Sometimes there will be surprises. Occasionally a question will
not load onto any factors very well. The fun part is determining what the factors represent
by looking for common themes in the questions that load onto the same factors. If you
identify 3 factor-themes, you can be assured that your survey is at least measuring three
things. Validity is measuring what you purport to be measuring, therefore this step
validates what your survey is really measuring. Finally, questions loading onto the same
factors can be aggregated (i.e., combined) and compared during the final data analysis
 phase. (A word of caution: Don’t attempt PCA by yourself if you are inexperienced.
Have someone skilled in PCA analysis guide you through the process or have good
resources on hand.)
5. Check the internal consistency of questions loading onto the same factors. This step
basically checks the correlation between questions loading onto the same factor. It is a
measure of reliability in that it checks whether the responses are consistent. A standard
test of internal consistency is Cronbach’s Alpha (CA). Cronbach Alpha values range
from 0 – 1.0. In most cases the value should be at least 0.70 or higher although a value
from 0.60 to 0.70 is acceptable. What should you do if you have a low value? A nice
function in some programs is telling you the CA value after removing a question. IBM
SPSS calls it “scale if item deleted.” You might consider deleting a question if doing so
dramatically improves your CA. (As with PCA, you should seek assistance from a
statistician or a good resource if you are new to testing internal consistency).
6. The final step is revising the survey based on information gleaned from the PCA and CA.
Consider that even though a question does not adequately load onto a factor, you might
retain it because it is important. You can always analyze it separately. If the question is
not important you can remove it from the survey. Similarly, if removing a question
greatly improves a CA for a group of questions, you might just remove it from its factor
loading group and analyze it separately. If your survey undergoes minor changes it is
probably ready to go. If there are major changes you may want to repeat the pilot testing
process. Repeat pilot testing is warranted whenever you start with many more questions
than are included in the final version (e.g., pilot testing 50 questions and then narrowing
the field to 10 questions).

(I strongly recommend running PCA and CA again after completing the formal data collection
phase [i.e., after you use your questionnaire to collect “real” data]. You want to make sure that
you get the same factor loading patterns.)

When reporting the results of your study you can claim that you used a questionnaire whose face
validity was established experts. You should also mention that it was pilot tested on a subset of
participants. Report the results of the PCA and CA analyses. Should you report the results from
the pilot testing or formal data collection? I think reporting PCA and CA results on the formal
data is most useful. When reporting PCA results you may say something like “Questions 4, 6, 7,
8, and 10 loaded onto the same factor which we determined represents personal commitment to
employer.” When reporting CA results you may say something like “The Cronbach’s Alpha for
questions representing personal commitment to employer was 0.91, indicating excellent internal
consistency in the responses.”

Summary of Steps to Validate a Questionnaire.

1. Establish Face Validity

2. Pilot test
3. Clean Dataset
4. Principal Components Analysis
5. Cronbach’s Alpha
6. Revise (if needed)
7. Get a tall glass of your favorite drink, sit back, relax, and let out a guttural laugh
celebrating your accomplishment. (OK, not really.)