Quantitative

Research in
Language
Education

Experimental
Design
What is an experiment?

A test or investigation to examine the validity of hypothesis and also it is used

to determine whether it influences an outcome or dependent variable.
We use an experiment when we want to establish possible cause and effect
between our independent and dependent variables.
When did experiments develop?

Some researchers did an experiment began in the late 19th and early 20th
centuries. The second researcher completed and developed the first
researchers experiments and the other did the same continously. Until in
the 1980s, the experiments have grown in complexity and largely because
of computers and improved statistical procedures.
Random Assignment

Random assignment is the process of assigning individuals at random to

groups or to different groups in an experiment. By randomization, you
provide control for extraneous characteristics of the participants that might
inuence the outcome. The experimental term for this process is equating
the groups.
Control Over Extraneous Variables

Extraneous factors are any inuences in the selection of participants, the

procedures, the statistics, or the design likely to affect the outcome and
provide an alternative explanation for our results than what we expected.
Other control procedures you can use both before and during the
experiment are pretests, covariates, matching of participants,
homogeneous samples, and blocking variables.
Other control procedures you can use
both before and during the experiment:

1. Pretests
2. Covariates
3. Matching of participants
4. Homogeneous samples
5. Blocking variables.
1. Pretests and Posttests

A pretest provides a measure on some attribute or characteristic that you

assess for participants in an experiment before they receive a treatment.
After the treatment, you take another reading on the attribute or
characteristic. A posttest is a measure on some attribute or characteristic
that is assessed for participants in an experiment after a treatment.
2. Covariates

Covariates are variables that the researcher controls for using statistics and
that relate to the dependent variable but that do not relate to the
independent variable. This test allows the researcher to assess accurately
the relationship between the treatment and the outcome (i.e., rate of
smoking) because of a reduction in the amount of error.
3. Matching of participants

Another procedure used for control in an experiment is to match

participants on one or more personal characteristics. Matching is the
process of identifying one or more personal characteristics that inuence
the outcome and assigning individuals with that characteristic equally to
the experimental and control groups.
4. Homogeneous Samples
This approach is used to make the group comparable by selecting people
who vary little in their personal characteristics.

5. Blocking Variable
A blocking variable is a variable the researcher controls before the
experiment starts by dividing (blocking) the participants into subgroups
(or categories) and analyzing the impact of each subgroup on the
outcome.
Manipulating Treatment Conditions
In experimental treatment, the researcher physically intervenes to alter the
conditions experienced by the experimental unit.
The procedure would be:
Identify a treatment variable
Identify the conditions (or levels) of the variable
Manipulate the treatment conditions
WHAT ARE KEY CHARACTERISTICS
OF EXPERIMENTS?

Random assignment Outcome measures

Control over extraneous

Group comparisons
variables
Manipulation of the
Threats to validity
treatment conditions
Outcome Measure

In experiments, the outcome (or response, criterion, or posttest) is the dependent

variable that is the presumed effect of the treatment variable. It is also the effect
predicted in a hypothesis in the cause-and-effect equation.

Example:

o Achievement scores on a criterion-referenced test

o Test scores on an aptitude test

Group Comparisons

A group comparison is the process of a researcher obtaining scores for

individuals or groups on the dependent variable and comparing the means
and variance both within the group and between the groups
Threads to Validity

Threats to validity refer to specific reasons for why we can be wrong when we make an
inference in an experiment because of covariance, causation constructs, or whether the
causal relationship holds over variations in persons, setting, treatments, and outcomes.
Four types of validity

Statistical conclusion validity

Construct validity
Internal validity
External validity
Two primary threats:

Threats to Threats to
internal external
validity validity
Threats to internal validity

Threats to internal validity are problems in drawing correct inferences about whether the
covariation (i.e., the variation in one variable contributes to the variation in the other
variable) between the presumed treatment variable and the outcome reflects a causal
relationship ( Shadish, Cook, & Campbell, 2002 ).
Categories
History, maturation,
regression, selection,
Participant mortatlity, interaction with
selection

Diffusion of treatments.
Compensatory equalization,
Treatment Compensatory rivalry,
Resentful demoralization

Procedure Testing, instrument

Threats to external validity

Threats to external validity are problems that threaten our ability to draw correct
inferences from the sample data to other persons, settings, treatment variables, and
measures.
According to Cook and Campbell (1979) , three threats
may affect this generalizability:

Interaction of selection and treatment

Interaction of setting and treatment
Interaction of history and treatment
 WHAT ARE THE TYPES OF
EXPERIMENTAL DESIGNS?
Between Group Designs
True experiments (pre- and posttest, posttest only)
Quasi-experiments (pre- and posttest, posttest only)
Factorial designs
Within Group or Individual Designs
Time series experiments (interrupted, equivalent)
Repeated measures experiments
Single subject experiments
Here are the several characteristics of the differentiated designs:
The random assignment of participants to groups
The number of groups or individuals being compared
The number of interventions used by the researcher
The number of times the dependent variable is measured or observed
The control of extraneous variables.
 True experiments

True experiments comprise the most rigorous and strong experimental designs
because of equating the groups through random assignment. The procedure for
conducting major forms of true experiments and quasi-experiments, viewing them
in terms of activities from the beginning of the experiment to the end. In true
experiments, the researcher randomly assigns participants to different conditions
of the experimental variable.
 Quasi-experiments

Quasi-experiments include assignment, but not random assignment of participants

to groups. This is because the experimenter cannot artificially create groups for the
experiment.
The quasi-experimental approach introduces considerably more threats to internal
validity than the true experiment. Because the investigator does not randomly
assign participants to groups, the potential threats of maturation, selection,
mortality, and the interaction of selection with other threats are possibilities
Factorial designs

Factorial designs represent a modification of the between group design in which

the researcher studies two or more categorical, independent variables, each
examined at two or more levels (Vogt, 2005). The purpose of this design is to study
the independent and simultaneous effects of two or more independent treatment
variables on an outcome.