DEVELOPING A QUESTIONNAIRE FOR USE IN OUTCOMES ASSESSMENT

Kathleen A. Martin, D.P.E.

Office of Institutional Research & Assessment Bucknell University

POINT OF CLARIFICATION
Terms survey and questionnaire are often used interchangeably Survey refers to a descriptive research technique or methodology (also includes interviews, and focus groups) Questionnaire refers to a paper and pencil instrument through which information is obtained For the purposes of outcomes assessment, questionnaires are frequently developed to survey a population

QUESTIONNAIRES & ASSESSMENT

Useful for surveying groups of any size Typically designed to determine opinions, attitudes or present practices Limitation: rely on self report information v. observed behavior Challenges to validity become a concern

VALIDITY CHALLENGES
Validity: the extent to which the test measures what it purports to measure Internal Validity: Relates to instrument (questionnaire) quality. Are the appropriate questions asked clearly and logically? External validity: can the results be generalized? Measurement error: difference between the characteristics of your sample and the characteristics of the population Non-response bias: answers of respondents do not mirror the characteristics of those who refused participation.

MAXIMIZING VALIDITY
Create a questionnaire that is clear and logical Support content relevance and content representativeness Content Relevance: are the questions relevant to the purpose of the questionnaire? Unimportant content should be eliminated. Content Representativeness: Are the questions an adequate representation of the universe of possible questions? Solicit the input of experts in scale development and the content of the questionnaire for an objective evaluation Select a large, representative random sample from the population of interest or census the finite population Increase response rate using formal follow up procedures

INITIAL CONSIDERATIONS
Is the development of a new questionnaire . . . Necessary? Feasible? Select mode of administration Web Based? Paper and pencil? Combination?

QUESTIONNAIRE PLANNING
1. Determine the Objective What do you want to ask and of whom? Consider how the data will be analyzed. What information is needed? 2. Determine the Mode of Administration Paper and pencil Cost effective, but low response rate Web / E-mail based Cost effective, higher response rate, some suspicion of the medium Combination Can be administratively cumbersome

QUESTIONNAIRE PLANNING
Determine the Sampling Methodology
To whom will you send the questionnaire? Who can supply the information? Will everyone be surveyed? ALL current students? ALL alumni? If yes, this is a census of the finite population The selection of a representative sample of the population may be more effective based on time and cost of administration Decision depends on the size of the finite population

CONSTRUCTING the QUESTIONNAIRE

Consider what objective each question measures Three to five questions per objective is recommended

How many questions?

Find a balance: long enough to obtain the necessary information but short enough so respondents wont lose interest. Questionnaires that are too long wont be completed Goal: 25-30 minutes MAX

DESIGNING THE QUESTIONNAIRE

Question Formats: Open Ended Questions
Allow respondents to express feelings and expand on ideas. Question should be phrased to avoid a one word response. What aspects of the Sociology program best prepared you for your current position? Open ended questions at the beginning of the questionnaire may be discouraging to respondent include one or two at the end Analysis of responses can be challenging

QUESTIONNAIRE DEVELOPMENT
Question Formats: Closed Questions Ranking
Forces the respondent to rank order responses based on a value judgment. Rank the following items with regard to how you spend time at Bucknell from 1 (spend the most time) to 5 (spend the least time). _____ Studying _____ Socializing _____ Volunteering _____ Working at a part time job _____ Exercising

QUESTIONNAIRE DEVELOPMENT
Question Formats: Closed Questions Checklist
A number of possible answers are provided and the respondent is asked to choose one or choose all that apply. How did you finance your Bucknell University education? Choose all that apply. _____ Parental contribution _____ Student Loan _____ Work Study _____ Off campus employment _____ Other (please specify): _____________

QUESTIONNAIRE DEVELOPMENT
Question Formats: Closed Questions Scaled items
Very common Indicate strength of agreement or disagreement with a statement. Numbers are assigned to each response on the continuum The English curriculum at Bucknell University adequately prepared me for my current position.

QUESTIONNAIRE DEVELOPMENT
Question Formats: Closed Questions Suggestions for writing scaled items
Use positively worded or negatively worded items consistently Use an odd number of responses along the continuum to allow for a neutral response Anchors may be different based on focus of the item (Strongly Agree Strongly Disagree; Frequently Never; Very Good Very Poor) Keep items short

QUESTIONNAIRE DEVELOPMENT
Question Formats: Categorical
Typical demographic questions: Yes/No/?, Male/Female, F/S/J/S/G Answers fall into certain categories respondents pick the one that applies to them Analyzed using frequency, proportion, percentages 2 considerations: Questions on a continuum should stay on a continuum if they will be directly analyzed Example: Age allow respondent to write in their age rather than check off a box for their age range Categories must be collectively exhaustive & mutually exclusive (next slide)

QUESTIONNAIRE DEVELOPMENT
Problematic: Number of hours spent studying per week: 18 21 21 24 24 27 Categories are not mutually exclusive Corrected: Number of hours spent studying per week: 18 21 22 25 26 29

IMPORTANT CONSIDERATIONS
APPEARANCE AND DESIGN Have clear directions that are age appropriate & easy to follow Format, size, reproduction should facilitate completion No less than 12 pt Times, Arial, Courier PILOT STUDY Give the questionnaire to a few people to try it out Have them evaluate readability and clarity of questions Estimate time required to complete the questionnaire Suggestions for revisions

SENDING IT OUT
INITIAL CONTACT / COVER LETTER Short & to the point Identify person by name Convince respondent that participation is important Ensure confidentiality Deadline for return of questionnaire SASE for return (mailed questionnaires only)

Institutional approval
Institutional Review Board Research involving human subjects must be approved by the Bucknell University IRB For more information, visit the IRB website at http://www.departments.bucknell.edu/inst_research/IRB/ irb.htm Campus Survey Coordination The timing of all university-wide surveys must be coordinated through the Campus Survey Coordination Group. For more information, visit the CSCG website at http://www.departments.bucknell.edu/inst_research/cscg .htm

FINAL THOUGHTS
FOLLOW UP PROCEDURES
Send to all participants to preserve confidentiality (If you havent already returned the questionnaire..) May include a second copy if available Usually done in 2 week intervals

RESPONSE RATE FOR MAILED QUESTIONNAIRES

Typically low - 30% is considered good Higher response rate with web-based questionnaires Good follow-up increases response rate

SUMMARY OF STEPS
Determine the Objective (consider analyses) 2. Determine the Mode of Administration 3. Determine the Sampling Methodology 4. Construct the Questionnaire 5. Institutional Approval 6. Conduct the Pilot Study 7. Write the Initial Communication 8. Send the Questionnaire 9. Follow up 10. Analyze the Results
1.

FOR MORE INFORMATION

Contact Kathy Martin @ 71960 or kmartin@bucknell.edu