Professional Documents
Culture Documents
10: Nov. 3-4 Time series & correlational designs [no lab] 3: T-tests T Main goals of literature review
Some tips for the literature review
T bivar.
11: Nov. 10-11 Bivariate regression & correlation T
corr/reg
Basic structure of the written research report:
12: Nov. 17-18 Multiple regression & correlation T #5 4: Bivariate correlation Ï T Intro; lit review; methodology; findings; discussion
Readings: ! Schutt, Chapters 5 and 8 Readings: ! SPSS, Chapters 4 and 5 Readings: ! SPSS, Ch. 11 & Ch.2 (pp. 18-19)
! Schutt, Chapter 14 (pp. 486-513)
Principles for designing good individual questions (6) Crosstabulations and Chi Square
Ways to filter or minimize "random responses" Good data analysis requires good data, plus the
Best practices for overall questionnaire flow: recognition that: all summary statistics are Readings: ! Schutt Chapter 14 (pp. 513-530)
short intro; easy start; broader to more detailed; reductionist, context dictates interpretation, ! SPSS, Chapters 6 and 8 (pp. 89-93)
sensitive questions later; demographics at end small differences should not be exaggerated,
Closed-ended vs. open-ended questions correlation does not prove causation, start with Interpreting crosstabulations (aka: crosstabs or con-
Likert item (strongly agree/agree/disagree/strongly disagree) univariate analysis before bi and multivariate. tingency tables) using counts, row percents, col-
Census vs. sample; Random vs. nonrandom samples Nominal univariate statistics – percents and mode umn percents, total percents, and marginals
Simple random sampling Interpretation pitfalls include:
Systematic random sampling Chi square (X 2 ) test of statistical significance:
misleading pictograms; confusing absolute and
Stratified random sampling Interpreting significance levels: .05, .01, .005, .001
relative percents; misinterpreting mode as mid-
(proportionate vs. nonproportionate) point; and misleading modal composites. Is it permissible to use .10 as the criteria?
Sampling frame Nonresponse bias Response bias Plurality vs. majority Statistical significance vs. substantive significance
Nonrandom (nonprobability) sampling such as Type I Error vs. Type II Error
convenience, snowball, and purposive sampling
Measures of central tendency: mean and median,
Evaluating response rates; how high is high enough? Chi square steps:
plus trimmed mean
Mode (not necessarily a central tendency) state the null hypothesis,
Measures of dispersion: standard deviation and state the research hypothesis,
(3) Survey Research & Measurement state decision rule (alpha level),
interquartile range
Positive skew (high values pull the mean above me- examine the result to reject or not reject the null
Readings: ! Schutt, Chapter 4 ! SPSS, Ch. 1-2
dian) versus negative skew
Interpretation caveats:
Factors for the optimum size of a completed sample
Survey participation as a quick cost-benefit decision Normal curve "Statistically significance" does not mean it is
Benchmark confidence intervals (95% level) for important, powerful, strong; it just means odds
± 1 standard deviation = 68.3% of normal curve
are good that there’s some kind of relationship.
n=100 (±10%); 600 (±4%); 1100 (±3%) ± 2 standard deviations = 95.4% of normal curve
Best practices for improving survey response rates ± 3 standard deviations = 99.7% of normal curve Large samples often show statistical significance
Measurement reliability & measurement validity Value of examining frequency distribution charts. for weak, trivial relationships. Conversely, failure
Subjective validity: face validity & content validity to detect a relationship, especially in a small sam-
Criterion validity: concurrent & predictive validity ple, does not prove there is no relationship.
Descriptive vs. inferential statistics
Operationalize; operational definition Confidence intervals, sampling error, margin of error The null is the assumption, but it is never proven.
Measurement levels: nominal, ordinal, interval, ratio Standard error of the mean (formula optional)
Combined interval and ratio = scale or interval Measures of association (strength) for crosstabs
Standard error of the proportion (formula optional)
Unobtrusive measures Gamma (most common measure of association)
(7) Causal Inference & Experiments (8) More on Experimental Designs (9) T-Tests & Nonequivalent
Comparison Group Designs
Readings: ! Schutt, Chapter 6 Readings: ! Adams, Ch. 5 ! Schutt, Chapter 7
Readings: ! Adams, Chapter 6
Three elements of causal inference... Factorial designs (simple or complex) ! SPSS, Chapter 8 (pp. 83-89)
( 1) X & Y covary; [statistical] association; concomitant variation Dosage/sensitivity designs
(2) X precedes Y; direction; time sequence; temporal order Natural experiments T-tests – significance test for the difference in the
(3) Rule out the Zs; no plausible alternatives; nonspuriousness Complex X means of two randomized groups – either two
Antecedent variables; intervening variables
Multiple Ys means from a sample (random selection) or from
Campbell & Stanley's system for diagraming designs
an experiment (random assignment).
External validity (generalizability)
Single group posttest only T-test steps: state the null hypothesis, the research
Random assignment from pool of subjects to groups
Single group pretest-posttest (aka before-and-after) hypothesis, decision rule (alpha), F-test, look at
strengthens internal validity.
Static group design result to reject or not reject the null hypothesis
Random selection from the relevant population
Threats to internal validity (partial list): Usual issues in statistical significance apply...
strengthens external validity.
History
stat. significance vs. substantive significance;
Maturation
Reactivity small vs. large samples; careful interpretations if
Practice
Hawthorne effects fail to reject null; customary .05 criterion; only
Instrumentation
Placebo difference is the extra F-test step.
Regression to the mean
Selection Practical reasons why true randomized, controlled
Big four sets of validity issues: experiments often may not be conducted
Intragroup history
Mortality (plus ways to cope with attrition problems) construct (measurement) validity
Quasi-experiments
internal validity
statistical conclusion validity Nonequivalent comparison group designs
Elements of a true experiment: random assignment
(1) of subjects from pool to groups and (2) of X external validity Pretest-posttest nonequivalent comparison group
design
Reason for the power of experimental designs: Watch for between-group reactivity as well as other
Posttest only nonequivalent comparison group design
Comparability of the groups (i.e., only real differ- types of reactivity
Key inescapable threat to the internal validity of
ence between the groups is X, so X is the best
Solomon Four-Group design these NEC designs: selection
explanation for differences in the groups)
Rotation experiment / counterbalanced design Retrospective matching design (ex post facto with
Classic experimental design nonrandom posttreatment matching)
(aka pretest-posttest control-group design) Causal-comparative is a term sometimes used for
Posttest only experiment studies that try to infer causality using compar-
(aka posttest-only control-group design) ison groups that were not randomly assigned
(10) Tim e Series & Correlational Designs (11) Bivariate regression & correlation (13-14) Ethics; Other Methods:
Content Analysis, Focus Groups,
Readings: ! Adams, Chapter 7 Readings: ! SPSS, Chapter 7 (pp. 73-77) and Qualitative Analysis
! Schutt, Chapter 14 (pp. 531-534)
Time series (aka longitudinal) research Readings: ! Adams, Chapters 3 and 8-10
Simple interrupted time series r (correlation coefficient) ! Schutt, Chapters 9-10, 13, 15 (pp. 555-561)
r 2 (coefficient of determination) ! Adams, Ch. on “Semi-Structured Interviews”
Reiterative time series
Scattergram / scatterplot
Multiple time series
Linear and curvilinear patterns Content analysis
Panel data (aka "panel-back") Positive/direct relationship Inter-coder reliability testing
Cross-sectional data Negative/inverse relationship
Steps in content analysis — define scope; operation-
Deceptive time series charts (truncated base) alize variables to code; refine and test coding
Bivariate regression: í = a + bx
system; code content; analyze data
í (estimate of y); a (intercept); b (slope)
Retrospective pretests; proxy pretests
Analysis of residuals; Outliers
Fallacy of time series inferences from one survey Homoscedasticity and heteroscedasticity Qualitative Research
Standard error of the estimate More exploratory than hypothesis testing
Process and logic of correlational designs (analogous to standard deviation) Small, purposive sample, not large random
Correlational design problems: often big selection Ecological fallacy Extended, intense observations or interviews
threats due to motivation and self-selection, plus Aggregate data (units of analysis are groups)
Unstructured or semi-structured data gathering
difficulty in statistically controlling all Zs; so the
Reports with little or no quantitative data
findings may vary widely depending on the avail-
Often note the researchers' subjective impact
ability and choice of control variables (12) Multiple regression & correlation
(Session 4) (Session 4)
Measures of central tendency:
Percent; Mode Mean, median, trimmed mean
Confidence intervals
Confidence intervals
(around a sample mean – calculated
Univariate (around a sample percent – calculated
from standard error of the proportion)
from the standard error of the mean)
Measures of dispersion:
Interquartile range, standard deviation
Pretest-posttest w/control Pretest-posttest NEC group Simple interrupted t.s. Meta-analysis (1)
Single group posttest only R) O X O O X O O O O X O O O O
- - - - Survey research (2-3)
XO R) O O O O Correlational designs
Reiterative time series
Posttest only w/control Posttest only NEC group (aka data analytic designs)
OOXOOXOOXOO Content analysis (13)
Single group pretest-posttest often based on controls
Solomon four-group Retrospective matching design
(aka before-and-after study) (aka ex post facto design) Multiple/comparison t.s. using multiple regression
Qualitative research (14)
0XO Factorial designs O O O X O O O O
(¹Plus other designs using - - - - - - - - - - -
Dosage/sensitivity tests NEC, not randomized, groups) O O O O O O O Focus groups (13)
Simple before-and-after Randomization of subjects &
Selection (subjects with X
studies are not necessarily treatment(s) makes X only Selection is always a big threat
History is always a threat not distributed randomly);
bad – if it is reasonable to explanation for changes in Y; (due to no randomized controls). Is the study systematic,
- plus, if using panel data data for all needed control
suppose that there are no best design to test causality! So, these designs may be fairly clear, representative,
(attrition and reactivity), variables often not avail-
alternative explanations Controls for internal validity strong or very bad depending on and replicable?
if comparison (selection). able, so hard to rule out
intervening between the threats, but still must watch the similarity of the groups.
alternative explanations.
pretest and posttest. attrition, intragroup history.
September October November December
“To-Do” Checklist
Before Next Session 2 3 4 5 6 7 8 9 10 11 12 13 14