You are on page 1of 9

GWU PAd/PPol 202: Fall 2010

Research Methods & Applied Statistics

Paper copy to be distributed in class.


GWU PAd/PPol 202: Research Methods and Applied Statistics
Semester: Fall 2010 Textbooks and Software:
Instructors: Bill Adams (adams@gwu.edu) P William Adams, Election Night News and Voter Turnout: Solving
Office: MPA Bldg. #601-D Phone: 202-994-7494 the Projection Puzzle (Lynne Rienner Publishers, 2005).
Hours: Wednesdays 1:00-5:00; Thurs. 3:30-5:30
P R. Schutt, Investigating the Social World (6th ed, Pine Forge, 2009).
Lab: Veronica Brooks (vdbrooks@gwmail.gwu.edu)
P W. Wagner, Using SPSS (2nd ed., Pine Forge Press, 2010).
Course Learning Objectives: P SPSS Student Version for Windows, 18.0 (2010).
PAd/PPol 202 focuses on practical skills for conducting and evaluating
P Plus, additional readings to be posted online.
empirical and quantitative research. The course explores the strengths
and weaknesses of experimental, quasi-experimental, and nonexperi-
mental research; it also covers the most widely used statistical soft- Assignments and Grades:
ware (SPSS) and the most widely used statistics (univariate stats, con-
tingency table stats, chi-square, t-tests, bivariate regression, correla- The overall course grade is the sum of the final exam (30%), two
tion, multiple regression). In particular, the course targets as student quizzes (first 10%, second 20%), five SPSS projects (25%), and the
learning objectives these ten important research skills: weekly worksheets 15%. The lowest unrounded grade for an A is
exactly 94.0; A! 90.0; B+ 87.0; B 83.0; B! 80.0; C+ 77.0; C 73.0; and
Skill 1: How to design/conduct/critique meta-analyses C! 70.0. The quizzes and final exam are all cumulative.
Skill 2: How to design/conduct/critique survey research
Skill 3: How to create an SPSS data set Please post weekly worksheets on Blackboard at least one hour be-
Skill 4: How to analyze basic univariate statistics fore class; no late submissions. Late SPSS projects entail letter grade
Skill 5: How to analyze basic bivariate statistics deductions. Class attendance (no more than one unexcused absence)
Skill 6: How to design/conduct/critique controlled experiments is required to successfully complete the course without penalty.
Skill 7: How to design/conduct/critique quasi-experiments
Skill 8: How to analyze correlation and regression statistics If for any reason a class is missed, in whole or in part, please obtain
Skill 9: How to design/conduct/critique focus groups all announcements and assignments (especially those due at the next
Skill 10: How to design/conduct/critique content analyses session), along with class notes and handouts, from your class
colleagues before the next class. Please note: Genuine collaboration
Class Web Site: http://gwu202.blogspot.com on the weekly worksheets is encouraged, but analyses of the SPSS
This course site will include posts of updates and links to readings. computer output for the SPSS projects must be independent. (See
Please be sure to check it at least each Friday afternoon or weekend. GW’s Code of Academic Integrity: gwu.edu/~ntegrity/code.html)
Work- (1) Introduction to Pad/PPol 202;
Class 202 Class Topics Lab SPSS Projects Quiz sheets Trends in social research methods;
Hypotheses; Research questions;
Intro; Trends, Research questions;
1: Sept. 1-2 T intro Literature review; Meta-analyses
Literature review; Meta-analyses

Questionnaire design; Readings: ! Adams, Chapters 1-2


2: Sept. 8-9 T intro T ! Schutt, Chapters 1-2, 15 (pp. 543-546)
Survey sampling; Response rates
! Adams, chapter on “Internet Research”
Mail, telephone, e-mail surveys; T data set
3: Sept. 15-16 T
Measurement validation; SPSS creation Broad 20 th century research trends
“Traditionalism”
Univariate descriptive statistics;
4: Sept. 22-23 T univar T Behavioralism / basic research
Univariate inferential statistics
Classic model of scientific research steps
5: Sept. 29-30 SPSS and PowerPoint tips and tricks [no lab]
Theory
Crosstabs (contingency tables); T tables 1: Dataset and Hypothesis
6: Oct. 6-7 T
X 2 (and measures of association); & X2 univariate analysis Independent variable (X)
Dependent variable (Y)
7: Oct. 13-14 Causal inference & experiments [no lab] T
Operationalize concepts; operational definition
8: Oct .20-21 Complex experimental designs [no lab] 2: Crosstabs Î T
Applied research / policy analysis
9: Oct. 27-28 T-tests; NEC group designs T t-test T Model of applied research steps

10: Nov. 3-4 Time series & correlational designs [no lab] 3: T-tests T Main goals of literature review
Some tips for the literature review
T bivar.
11: Nov. 10-11 Bivariate regression & correlation T
corr/reg
Basic structure of the written research report:
12: Nov. 17-18 Multiple regression & correlation T #5 4: Bivariate correlation Ï T Intro; lit review; methodology; findings; discussion

13: Dec. 1-2 Ethical issues; Content analysis T #5 T Meta-analysis purpose/strengths


Focus groups; Qualitative analysis Steps in conducting a meta-analysis
14: Dec. 8-9 Capstone tools; Class summary [no lab] 5: Synthesis Project T
Triangulation
Final exam review session (optional): Monday, December 13, 5:15-7:15 pm
Inductive vs. deductive research
Final examination: Wednesday, December 15, 5:15-8:00 pm (approx. time)
(2) Survey Research (4) Univariate Statistics (5) SPSS & PowerPoint Tips and Tricks

Readings: ! Schutt, Chapters 5 and 8 Readings: ! SPSS, Chapters 4 and 5 Readings: ! SPSS, Ch. 11 & Ch.2 (pp. 18-19)
! Schutt, Chapter 14 (pp. 486-513)
Principles for designing good individual questions (6) Crosstabulations and Chi Square
Ways to filter or minimize "random responses" Good data analysis requires good data, plus the
Best practices for overall questionnaire flow: recognition that: all summary statistics are Readings: ! Schutt Chapter 14 (pp. 513-530)
short intro; easy start; broader to more detailed; reductionist, context dictates interpretation, ! SPSS, Chapters 6 and 8 (pp. 89-93)
sensitive questions later; demographics at end small differences should not be exaggerated,
Closed-ended vs. open-ended questions correlation does not prove causation, start with Interpreting crosstabulations (aka: crosstabs or con-
Likert item (strongly agree/agree/disagree/strongly disagree) univariate analysis before bi and multivariate. tingency tables) using counts, row percents, col-
Census vs. sample; Random vs. nonrandom samples Nominal univariate statistics – percents and mode umn percents, total percents, and marginals
Simple random sampling Interpretation pitfalls include:
Systematic random sampling Chi square (X 2 ) test of statistical significance:
misleading pictograms; confusing absolute and
Stratified random sampling Interpreting significance levels: .05, .01, .005, .001
relative percents; misinterpreting mode as mid-
(proportionate vs. nonproportionate) point; and misleading modal composites. Is it permissible to use .10 as the criteria?
Sampling frame Nonresponse bias Response bias Plurality vs. majority Statistical significance vs. substantive significance
Nonrandom (nonprobability) sampling such as Type I Error vs. Type II Error
convenience, snowball, and purposive sampling
Measures of central tendency: mean and median,
Evaluating response rates; how high is high enough? Chi square steps:
plus trimmed mean
Mode (not necessarily a central tendency) state the null hypothesis,
Measures of dispersion: standard deviation and state the research hypothesis,
(3) Survey Research & Measurement state decision rule (alpha level),
interquartile range
Positive skew (high values pull the mean above me- examine the result to reject or not reject the null
Readings: ! Schutt, Chapter 4 ! SPSS, Ch. 1-2
dian) versus negative skew
Interpretation caveats:
Factors for the optimum size of a completed sample
Survey participation as a quick cost-benefit decision Normal curve "Statistically significance" does not mean it is
Benchmark confidence intervals (95% level) for important, powerful, strong; it just means odds
± 1 standard deviation = 68.3% of normal curve
are good that there’s some kind of relationship.
n=100 (±10%); 600 (±4%); 1100 (±3%) ± 2 standard deviations = 95.4% of normal curve
Best practices for improving survey response rates ± 3 standard deviations = 99.7% of normal curve Large samples often show statistical significance
Measurement reliability & measurement validity Value of examining frequency distribution charts. for weak, trivial relationships. Conversely, failure
Subjective validity: face validity & content validity to detect a relationship, especially in a small sam-
Criterion validity: concurrent & predictive validity ple, does not prove there is no relationship.
Descriptive vs. inferential statistics
Operationalize; operational definition Confidence intervals, sampling error, margin of error The null is the assumption, but it is never proven.
Measurement levels: nominal, ordinal, interval, ratio Standard error of the mean (formula optional)
Combined interval and ratio = scale or interval Measures of association (strength) for crosstabs
Standard error of the proportion (formula optional)
Unobtrusive measures Gamma (most common measure of association)
(7) Causal Inference & Experiments (8) More on Experimental Designs (9) T-Tests & Nonequivalent
Comparison Group Designs
Readings: ! Schutt, Chapter 6 Readings: ! Adams, Ch. 5 ! Schutt, Chapter 7
Readings: ! Adams, Chapter 6
Three elements of causal inference... Factorial designs (simple or complex) ! SPSS, Chapter 8 (pp. 83-89)
( 1) X & Y covary; [statistical] association; concomitant variation Dosage/sensitivity designs
(2) X precedes Y; direction; time sequence; temporal order Natural experiments T-tests – significance test for the difference in the
(3) Rule out the Zs; no plausible alternatives; nonspuriousness Complex X means of two randomized groups – either two
Antecedent variables; intervening variables
Multiple Ys means from a sample (random selection) or from
Campbell & Stanley's system for diagraming designs
an experiment (random assignment).
External validity (generalizability)
Single group posttest only T-test steps: state the null hypothesis, the research
Random assignment from pool of subjects to groups
Single group pretest-posttest (aka before-and-after) hypothesis, decision rule (alpha), F-test, look at
strengthens internal validity.
Static group design result to reject or not reject the null hypothesis
Random selection from the relevant population
Threats to internal validity (partial list): Usual issues in statistical significance apply...
strengthens external validity.
History
stat. significance vs. substantive significance;
Maturation
Reactivity small vs. large samples; careful interpretations if
Practice
Hawthorne effects fail to reject null; customary .05 criterion; only
Instrumentation
Placebo difference is the extra F-test step.
Regression to the mean
Selection Practical reasons why true randomized, controlled
Big four sets of validity issues: experiments often may not be conducted
Intragroup history
Mortality (plus ways to cope with attrition problems) construct (measurement) validity
Quasi-experiments
internal validity
statistical conclusion validity Nonequivalent comparison group designs
Elements of a true experiment: random assignment
(1) of subjects from pool to groups and (2) of X external validity Pretest-posttest nonequivalent comparison group
design
Reason for the power of experimental designs: Watch for between-group reactivity as well as other
Posttest only nonequivalent comparison group design
Comparability of the groups (i.e., only real differ- types of reactivity
Key inescapable threat to the internal validity of
ence between the groups is X, so X is the best
Solomon Four-Group design these NEC designs: selection
explanation for differences in the groups)
Rotation experiment / counterbalanced design Retrospective matching design (ex post facto with
Classic experimental design nonrandom posttreatment matching)
(aka pretest-posttest control-group design) Causal-comparative is a term sometimes used for
Posttest only experiment studies that try to infer causality using compar-
(aka posttest-only control-group design) ison groups that were not randomly assigned
(10) Tim e Series & Correlational Designs (11) Bivariate regression & correlation (13-14) Ethics; Other Methods:
Content Analysis, Focus Groups,
Readings: ! Adams, Chapter 7 Readings: ! SPSS, Chapter 7 (pp. 73-77) and Qualitative Analysis
! Schutt, Chapter 14 (pp. 531-534)
Time series (aka longitudinal) research Readings: ! Adams, Chapters 3 and 8-10
Simple interrupted time series r (correlation coefficient) ! Schutt, Chapters 9-10, 13, 15 (pp. 555-561)
r 2 (coefficient of determination) ! Adams, Ch. on “Semi-Structured Interviews”
Reiterative time series
Scattergram / scatterplot
Multiple time series
Linear and curvilinear patterns Content analysis
Panel data (aka "panel-back") Positive/direct relationship Inter-coder reliability testing
Cross-sectional data Negative/inverse relationship
Steps in content analysis — define scope; operation-
Deceptive time series charts (truncated base) alize variables to code; refine and test coding
Bivariate regression: í = a + bx
system; code content; analyze data
í (estimate of y); a (intercept); b (slope)
Retrospective pretests; proxy pretests
Analysis of residuals; Outliers
Fallacy of time series inferences from one survey Homoscedasticity and heteroscedasticity Qualitative Research
Standard error of the estimate More exploratory than hypothesis testing
Process and logic of correlational designs (analogous to standard deviation) Small, purposive sample, not large random
Correlational design problems: often big selection Ecological fallacy Extended, intense observations or interviews
threats due to motivation and self-selection, plus Aggregate data (units of analysis are groups)
Unstructured or semi-structured data gathering
difficulty in statistically controlling all Zs; so the
Reports with little or no quantitative data
findings may vary widely depending on the avail-
Often note the researchers' subjective impact
ability and choice of control variables (12) Multiple regression & correlation

Checklist of key questions to consider conducting


Readings: ! SPSS, Ch. 7 (pp. 78-82) Focus groups
and evaluating research Goals (probing; reaction testing; brainstorming)
Multiple Regression
Overall assessment of causal designs: Issues in participant recruitment; focus group size;
ì = a + b 1 X 1 + b 2 X 2 ... b n X n session length and agenda; moderator style; and
Lab experiments: strong on internal validity but
ì = â 1 X 1 + â 2 X 2 ... â n X n ideal focus group facilities.
weak on external validity
Multiple R / multiple correlation coefficient
Nonexperimental field studies: strong external Multiple R 2 / multiple coefficient of determination
but weak internal validity Recap for capstone research, especially
Unstandardized partial regression coefficients (aka:
Field experiments: strong in internal/external unstandardized regression coefficients; unstandard- Suggestions for semi-structured interviews con-
validity but often not feasible to conduct ized partial coefficients) ducted in-person.
The logic of inferring causality by coupling lab experi- Standardized partial regression coefficients Structure of capstone research reports.
ments with nonexperimental field studies (aka: betas; beta weights)
Dummy variables; Multicollinearity
PAd/PPol 202
Stats - Bill Adams Categorical Scale

(Session 4) (Session 4)
Measures of central tendency:
Percent; Mode Mean, median, trimmed mean

Confidence intervals
Confidence intervals
(around a sample mean – calculated
Univariate (around a sample percent – calculated
from standard error of the proportion)
from the standard error of the mean)

Measures of dispersion:
Interquartile range, standard deviation

Other: Mode, skewness

(Session 6) (Session 9) (Session 11)

Crosstabulations ¹ Two groups Bivariate correlation (r & r2)


Bivariate (Col/row/tot %) Two means ¸ Bivariate regression (í = a + bx)
Chi-Square T-tests Standard error of the estimate
(Measures of strength, Statistical significance
e.g., gamma and lambda)

(Session 6) (Session 12)

Crosstabs with controls Multiple correlation


Chi Square (Multiple R & Multiple R2)
Multivariate
Multiple regression
(ì = a + b 1 X 1 + b 2 X 2 ... b n X n )
(ì = $ 1 X 1 + $ 2 X 2 ... $ n X n )
Statistical significance
SUMMARY OF RESEARCH DESIGNS PAd/PPol 202 - Bill Adams

True experiments Quasi-Experimental Designs aka Causal Comparative Designs


Single group designs Other Selected
(Randomized, controlled) Nonequivalent Comparison Time Series Designs Correlational Designs (mainly descriptive)
(Session 7)
(Sessions 7-8) Group Designs (Session 9) Methodologies
(Session 10) (Session 11)

Pretest-posttest w/control Pretest-posttest NEC group Simple interrupted t.s. Meta-analysis (1)
Single group posttest only R) O X O O X O O O O X O O O O
- - - - Survey research (2-3)
XO R) O O O O Correlational designs
Reiterative time series
Posttest only w/control Posttest only NEC group (aka data analytic designs)
OOXOOXOOXOO Content analysis (13)
Single group pretest-posttest often based on controls
Solomon four-group Retrospective matching design
(aka before-and-after study) (aka ex post facto design) Multiple/comparison t.s. using multiple regression
Qualitative research (14)
0XO Factorial designs O O O X O O O O
(¹Plus other designs using - - - - - - - - - - -
Dosage/sensitivity tests NEC, not randomized, groups) O O O O O O O Focus groups (13)
Simple before-and-after Randomization of subjects &
Selection (subjects with X
studies are not necessarily treatment(s) makes X only Selection is always a big threat
History is always a threat not distributed randomly);
bad – if it is reasonable to explanation for changes in Y; (due to no randomized controls). Is the study systematic,
- plus, if using panel data data for all needed control
suppose that there are no best design to test causality! So, these designs may be fairly clear, representative,
(attrition and reactivity), variables often not avail-
alternative explanations Controls for internal validity strong or very bad depending on and replicable?
if comparison (selection). able, so hard to rule out
intervening between the threats, but still must watch the similarity of the groups.
alternative explanations.
pretest and posttest. attrition, intragroup history.
September October November December
“To-Do” Checklist
Before Next Session 2 3 4 5 6 7 8 9 10 11 12 13 14

“To-do” almost every week..

Internalize key concepts listed on the


syllabus and covered the prior session.

Read textbook chapters listed on the


syllabus accompanying prior session.

Read the Friday or weekend post and


all its new links at the class web site.

Prepare and submit weekly worksheet


via Blackboard (at least one hour before class) .

As needed & scheduled on syllabus...

Attend (optional) SPSS computer lab.

Bring a paper copy of your


SPSS project to submit in class.
â ã ä å æ

Review for the next quiz/exam. ì í

You might also like