Experimental Design - A blueprint of the procedure that enables the researcher to test his hypothesis by reaching valid conclusions

about relationships between independent and dependent variables. It refers to the conceptual framework within which the experiment is conducted. Steps involved in conducting an experimental study Identify and define the problem. Formulate hypotheses and deduce their consequences. Construct an experimental design that represents all the elements, conditions, and relations of the consequences. 1. Select sample of subjects. 2. Group or pair subjects. 3. Identify and control non experimental factors. 4. Select or construct, and validate instruments to measure outcomes. 5. Conduct pilot study. 6. Determine place, time, and duration of the experiment. Conduct the experiment. Compile raw data and reduce to usable form. Apply an appropriate test of significance. Validity of Experimental Design Internal Validity asks did the experimental treatment make the difference in this specific instance rather than other extraneous variables? External Validity asks to what populations, settings, treatment variables, and measurement variables can this observed effect be generalized? Factors Jeopardizing Internal Validity History - The events occurring between the first and second measurements in addition to the experimental variable which might affect the measurement. Example: Researcher collects gross sales data before and after a 5 day 50% off sale. During the sale a hurricane occurs and results of the study may be affected because of the hurricane, not the sale. Maturation - The process of maturing which takes place in the individual during the duration of the experiment which is not a result of specific events but of simply growing older, growing more tired, or similar changes. Example: Subjects become tired after completing a training session, and their responses on the Posttest are affected.

Pre-testing - The effect created on the second measurement by having a measurement before the experiment. Example: Subjects take a Pretest and think about some of the items. On the Posttest they change to answers they feel are more acceptable. Experimental group learns from the pretest. Measuring Instruments - Changes in instruments, calibration of instruments, observers, or scorers may cause changes in the measurements. Example: Interviewers are very careful with their first two or three interviews but on the 4th, 5th, 6th become fatigued and are less careful and make errors. Statistical Regression - Groups are chosen because of extreme scores of measurements; those scores or measurements tend to move toward the mean with repeated measurements even without an experimental variable. Example: Managers who are performing poorly are selected for training. Their average Posttest scores will be higher than their Pretest scores because of statistical regression, even if no training were given. Differential Selection - Different individuals or groups would have different previous knowledge or ability which would affect the final measurement if not taken into account. Example: A group of subjects who have viewed a TV program is compared with a group which has not. There is no way of knowing that the groups would have been equivalent since they were not randomly assigned to view the TV program. Experimental Mortality - The loss of subjects from comparison groups could greatly affect the comparisons because of unique characteristics of those subjects. Groups to be compared need to be the same after as before the experiment. Example: Over a 6 month experiment aimed to change accounting practices, 12 accountants drop out of the experimental group and none drop out of the control group. Not only is there differential loss in the two groups, but the 12 dropouts may be very different from those who remained in the experimental group. Interaction of Factors, such as Selection Maturation, etc. - Combinations of these factors may interact especially in multiple group comparisons to produce erroneous measurements. Factors Jeopardizing External Validity or Generalizability Pre-Testing -Individuals who were pretested might be less or more sensitive to the experimental variable or might have "learned" from the pre-test making them unrepresentative of the population who had not been pre-tested. Example: Prior to viewing a film on Environmental Effects of Chemical, a group of subjects is given a 60 item antichemical test. Taking the Pretest may increase the effect of the film. The film may not be effective for a nonpretested group. Differential Selection - The selection of the subjects determines how the findings can be generalized. Subjects selected from a small group or one with particular characteristics would limit generalizability. Randomly chosen subjects from the entire population could be generalized to the entire population.

Example: Researcher, requesting permission to conduct experiment, is turned down by 11 corporations, but the 12th corporation grant permission. The 12th corporation is obviously different then the others because they accepted. Thus subjects in the 12th corporation may be more accepting or sensitive to the treatment. Experimental Procedures - The experimental procedures and arrangements have a certain amount of effect on the subjects in the experimental settings. Generalization to persons not in the experimental setting may be precluded. Example: Department heads realize they are being studied, try to guess what the experimenter wants and respond accordingly rather than respond to the treatment. Multiple Treatment Interference - If the subjects are exposed to more than one treatment then the findings could only be generalized to individuals exposed to the same treatments in the same order of presentation. Example: A group of CPAs is given training in working with managers followed by training in working with comptrollers. Since training effects cannot be deleted, the first training will affect the second. Tools of Experimental Design Used to Control Factors Jeopardizing Validity Pre-Test - The pre-test, or measurement before the experiment begins, can aid control for differential selection by determining the presence or knowledge of the experimental variable before the experiment begins. It can aid control of experimental mortality because the subjects can be removed from the entire comparison by removing their pre-tests. However, pre-tests cause problems by their effect on the second measurement and by causing generalizability problems to a population not pre-tested and those with no experimental arrangements. Control Group -The use of a matched or similar group which is not exposed to the experimental variable can help reduce the effect of History, Maturation, Instrumentation, and Interaction of Factors. The control group is exposed to all conditions of the experiment except the experimental variable. Randomization - Use of random selection procedures for subjects can aid in control of Statistical Regression, Differential Selection, and the Interaction of Factors. It greatly increases generalizability by helping make the groups representative of the populations. Additional Groups - The effects of Pre-tests and Experimental Procedures can be partially controlled through the use of groups which were not pre-tested or exposed to experimental arrangements. They would have to be used in conjunction with other pre-tested groups or other factors jeopardizing validity would be present.

True Experimental Design - greater control and refinement, greater control of validity Aim of the Research Name of the Design Pretest-posttest control group Notation Paradigm R--[OXO [O-O Comments

To study the effect of an influence on a carefully controlled

This design has been called "the old workhorse of traditional

sample

experimentation." If effectively carried out, this design controls for eight threats of internal validity. Data are analyzed by analysis of covariance on posttest scores with the pretest the covariate. Solomon fourgroup design R--[OXO [O-O [- X O [--O This is an extension of the pretest-posttest control group design and probably the most powerful experimental approach. Data are analyzed by analysis of variance on posttest scores. An adaptation of the last two groups in the Solomon four-group design. Randomness is critical. Probably, the simplest and best test for significance in this design is the t-test.

To minimize the effect of pretesting

To evaluate a situation that cannot be pretested

Posttest only control group

R--[ XO [-O

Experimental Design
We are concerned with the analysis of data generated from an experiment. It is wise to take time and effort to organize the experiment properly to ensure that the right type of data, and enough of it, is available to answer the questions of interest as clearly and efficiently as possible. This process is called experimental design. The specific questions that the experiment is intended to answer must be clearly identified before carrying out the experiment. We should also attempt to identify known or expected sources of variability in the experimental units since one of the main aims of a designed experiment is to reduce the effect of these sources of variability on the answers to questions of interest. That is, we design the experiment in order to improve the precision of our answers.

Experimental Design in Statistics

The term experimental design refers to a plan for assigning experimental units to treatmentconditions. A good experimental design serves three purposes.

Causation. It allows the experimenter to make causal inferences about the relationship between independent variables and a dependent variable.

Control. It allows the experimenter to rule out alternative explanations due to the confoundingeffects of extraneous variables (i.e., variables other than the independent variables).

Variability. It reduces variability within treatment conditions, which makes it easier to detect differences in treatment outcomes.

Independent Variable
In an experimental design, the independent variable is the variable that is manipulated by the experimenter to determine its relationship to an observed phenomenon, called the dependent variable. More generally, the independent variable is the cause, while dependent variable is the effect of the independent variable.

Dependent Variable
In an experimental design, the independent variable is the variable that is manipulated by the experimenter to determine its relationship to an observed phenomenon, called the dependent variable. More generally, the independent variable is the cause, while dependent variable is the effect of the independent variable.

There are six major classes of information with which an experimental designer must cope. They include:

post -treatment behavior or physical measurement [P1] pre-treatment behavior or physical measurement [P2] internal threats to validity [I] comparable groups [C] experiment errors [E] relationship to treatment [R]

Basic Experimental Designs

Eleven commonly used experimental designs will be described. They include: 1. 2. 3. 4. 5. 6. One-Shot One-Group, Pre-Post Static Group Random Group Pre-Post Randomized Group Solomon Four Group

7. Randomized Block 8. Factorial 9. One-Shot Repeated Measures 10. Randomized Groups Repeated Measures 11. Latin Square Obviously in a treatment of experimental design of fewer than 15 pages, not all possible designs are covered. Some of those not treated include, incomplete block designs, Youden square designs, lattice square designs, Taguchi designs, fractional factorial designs, Graeco-Latin square designs, split-plot designs, covariance designs, or time-series designs. The reader is referred to the references at the end of this paper, many of which treat these more specialized designs. One-Shot The One-Shot is a design in which a group of subjects are administered a treatment and then measured (or observed). In experimental research, an experimental treatment should be given to the subjects, and then the measurement or observation made. Usually, with this design, an intact group of subjects is given the treatment and then measured or observed. No attempt is made to randomly assign subjects to the groups, nor does the design provide for any additional groups as comparisons. Thus, one group will be given one treatment and one "observation." This design is diagrammed as follows: GP--T--O The One-Shot Design is highly useful as an inexpensive measure of a new treatment of the group in question. If there is some question as to whether any expected effects will result from the treatment, then a one-shot may be an economical route. In cases where other studies, or the cumulative knowledge in the field provide information about either pre-treatment baseline measurements or behavior, the effects of other kinds of treatments, etc., the experimenter might sensibly decide that it si not necessary to undertake a more extensive design. Simplicity, ease, and low cost represent strong potential advantages in the oft-despised one-shot. This design answers only one question and that is in reference to post-treatment behavior, P1-1. It will describe the information about the behavior of the subjects shortly after treatment. One-Group, Pre-Post

In this design, one group is given a pre-treatment measurement or observation, the experimental treatment, and a post-treatment measurement or observation. The posttreatment measures are compared with their pre-treatment measures. This design is diagrammed as follows: GP--O--T--O The usefulness of this design is similar to that of the one-shot, except that an additional class of information is provided, i.e., pre-treatment condition or behavior. This design is frequently used in clinical and education research to determine if changes occurred. It is typically analyzed with a matched pairs t-test. This design will answer the same question as the one-shot design P1-1, so that not only the post-treatment behavior of the subjects is answered, but it will also answer some questions in pre-treatment condition or behavior, namely P2-1 and P2-2. Static Group In this design, two intact groups are used, but only one of them is given the experimental treatment. At the end of the treatment, both groups are observed or measured to see if there is a difference between them as a result of the treatment. The design is diagrammed as follows: GP--T--O GP------O This design may provide information on some rival hypotheses. Whether it does or not depends on the initial comparability of the two groups and whether their experience during the experiment differs in relevant ways only by the treatment itself. Whether the groups were comparable or not is crucial in determining the extent of information yielded by this design. The design is could be used to compare the value of a drug. If the designer cannot, on the basis of information outside the experiment itself, assume the comparability of the groups, the design will yield only information regarding P1-1 and P1-2. However, IF additional information is available to equate the two groups initially, it may handle some of the class I questions. Without additional information, on the basis of the design alone, it cannot. Random Group

This design is similar to the Static Group design except than an attempt is made to insure similarity of the groups before treatment begins. Since it is difficult to have exactly similar subjects in each of two groups (unless you separate identical twins), the design works toward a guarantee of comparability between groups by assigning subjects to groups at random. If the researcher does this there is likely to be reasonable comparability between the two groups. This design can be diagrammed as: R--GP--T--O R--GP------O This design is a real workhorse. It was developed by the great statistician, Sir Ronald Fisher as part of his work in agricultural statistics and has been a primary experimental design for more than 100 years. It is economical. It provides fairly clearcut information as to the relationship between treatment and post-treatment measurement or behaviors. Since this is often the sole reason for the research, the randomized group design is frequently the appropriate selection. Good that it is, it does not provide information about pre-treatment behavior. This design will answer questions P1-1 and P1-2. Since the groups are randomized, the design will cope with the internal threats to validity. Subject changes due to other causes should affect the control group so that a comparison of the post-treatment behavior should reveal any differential effects of the treatment. Class C, comparable group questions C-1 and C-2 are also answered in the randomization provided that there are no probable differences between the groups entering the experiment except for the treatment. The design will not cope with class P1-3, P1-4, P1-5, P2's questionss of pre-treatment behavior, class E questions, or class R questions. It is frequently analyzed with a two-sample t-test assuming equal variances of the groups. Pre-Post Randomized Group This design adds a pre-test to the previous design as a check on the degree of comparability of the control and experimental groups before the treatment is given. This experimental design could be diagrammed as: R--GP--O--T--O R--GP--O------O

This yields information of P1-1, P1-2 as to post-treatment behavior and a comparison of post-treatment behavior between groups. It also answers must P2 questions on pretreatment behavior, and questions P2-1 through P2-4. It answers most of the class I questions, that is, threats to internal validity. It handles class C questions, the groups are comparable because thar are randomized. The design does not answer P1-3, P1-4, or P1-5; nor the class E questions relating to experiment errors. It does not answer the class R questions regarding the relationship of the treatment nor P2-5, the effect of the pre-treatment observations on the subsequent behavior or measurements of the subjects. Solomon Four Group The Solomon Four Group design attempts to control for the possible "sensitizing" effects of the pre-test or measurement by adding two groups who have not been a part of the pre-test or pre-measurement process. This design can be diagrammed as: R--GP--O--T--O R--GP--O------O R--GP-------T--O R--GP-----------O Although this design is not frequently used in clinical studies, it is frequently used in both behavior and educational research and in medical studies involving the physical activities of patients (physical therapy, for example where the pre-measurement involves some sort of physical activity or testing). The additional cost of this design must be justified by the need for information regarding the possible effects of the pretreatment measurement. The Solomon Design answers the P1 questions, P1-1, P1-2, and P1-3. It answers the P2 pre-treatment questions, including the effect of the pre-treatment measurement process. It handles threats to internal validity, the class I questions. It also handles the class C questions because the groups are randomized. It does not handle P1-4, P1-5, the class E questions nor the class R questions. Randomized Block

This design is of particular value when the experimenter wishes to determine the effect of a treatment on different types of subjects within a group. This design can be diagrammed as: ----------BLK--T--O R--GP ----------BLK--T--O ----------BLK------O R--GP ----------BLK------O Typically, this design refers to blocking or grouping of subjects with similar characteristics into treatment subgroups. The group to be used in an experiment is usually given some pre-treatment measure, or previous records are examined, and the entire group is blocked or sorted into categories. Then equal numbers from each category are assigned to the various treatment and/or control groups. While blocking according to subject characteristics is most typical of this design, blocking could be based on other relevant attributes. For example, if subjects are to be treated during different times of the day, such as morning and afternoon, we might block a morning and an afternoon group within each treatment condition. The importance of this design lies in the probability that the variable upon which the blocking is based may interact with the treatment. Frequently, no overall treatment effect is observed because subjects with different characteristics react differentially to treatment. If they were blocked on the appropriate attributes, differential treatment effects would be revealed. The randomized block design will handle the P1-1, P-2 and P1-3 questions. It does handle the class I questions and the class C questions of comparable groups. It does not handle the P1-4 and P1-5 questions or the P2 questions of pre-treatment behavior. It does not handle class E questions and only one of the class R questions, R-1, the relationship of the treatment to subjects characteristics, by virtue of blocking. Factorial

As you saw above in the blocking design, the subjects were assigned to different groups on the basis of some of their own characteristics such as age, weight, or some other physical characteristic. Sometimes we wish to assign different variations of the treatment as well, and the procedure is similar. For example, we may wish to try two kinds of treatments varied in two ways (called a 2x2 factorial design). Some factorial designs include both assignment of subjects (blocking) and several types of experimental treatment in the same experiment. When this is done it is considered to be a factorial design. A diagram of a 2x2 factorial design would look like: R--GP--T-------O A1 B1 R--GP--T-------O A1 B2 R--GP--T-------O A2 B1 R--GP--T-------O A2 B2 The factorial design as we are describing is really a complete factorial design, rather than an incomplete factorial, of which there are several variations. The factorial is used when we wish information concerning the effects of different kinds or intensities of treatments. The factorial provides relatively economical information not only about the effects of each treatment, level or kind, but also about interaction effects of the treatment. In a single 2x2 factorial design similar to the one diagrammed above, information can be gained about the effects of each of the two treatments and the effect of the two levels within each treatment, and the interaction of the treatments. If all these are questions of interest, the factorial design is much more economical than running separate experiments. The factorial handles some of the same classes of information as the previously described randomized group design: P1-1, P1-3 and class C questions. It provides little support for class I questions, but some weak inferences might be drawn. The design also answers class R questions, R-2, R-3 and R-4 concerning relationships among treatments, factors, or levels. It does not answer questions P1-2, P1-4, P1-5, or P2 questions, nor does it answer either class E or questions R-1 or R-5. Factorials can

run to enormous and uninterpretable proportions. 2x3x4 designs are not uncommon, particular in agricultural experimentation. One-Shot Repeated Measures This design, or variations of it, is used to assess the effects of a treatment with the same group or the same individual over a period of time. A measure, or observation is made more than once to assess the effects of the treatment. This design can be diagrammed as: GP--T--O--T--O--T--O This design is an extension of the simple one-shot and adds only information regarding the effects of repeated or continued treatment. Often an economical trial balloon, the design can acquire high yield when other extra-design sources of knowledge can be related to it. This design handles only the questions related to class P1 or post-treatment behavior, question P1-1 and P1-4. It answers questions about behavior or effect shortly after treatment and the longer-term effects related to subsequent treatments. It might handle class R questions, R-5 relationships of the treatment in that the effect of the repeated treatments may be observed. It will not answer questions P1-2, P1-3, P1-5 nor any class P2, I, C, E, or R-1 through R-4 questions. Randomized Groups Repeated Measures The Randomized Groups Repeated Measures design is a variant of the previous design in which two or more experimental methods are compared and repeatedly measured or observed. Although we are diagramming but two groups, more groups may be used. R--GP--T---O--T----O--T----O R--GP--T2--O--T2--O--T2--O This design handles answers P1-1, P1-3, and P1-4 questions. It handles class C questions. The groups are comparable because of random assignment. Some light is shed upon the relationship of repeated treatments, question R-5. It fails to handle P1-2, P1-5, or class P2 pre-treatment behavior or measurement questions. It does not handle class I questions on internal threats to validity unless a

non-treatment group is added to yield class I information. It further fails to handle class E and most class R questions except for R-5. Latin Square A researcher may wish to use several different treatments in the same experiment, for example the relative effects of an assortment of perhaps three or more drugs in combination in which the sequence of administration may produce different results. A diagram of a three treatment Latin Square design is: R--GP--TA--O--TB--O--TC--O R--GP--TB--O--TC--O--TA--O R--GP--TC--O--TA--O--TB--O If all possible sequencing permutations had to be addressed, there would be 3x2=6 possible arrangements--a doubling of cost. The original 3-treatment design would, however, answer the question whether sequencing made a difference without testing all possible sequences. The Latin Square design answers questions P1-1, P1-3, C-1, C-2, and R-5. It will not provide information on P1-2, P1-4, P1-5 nor any questions in class I or class E. Questions R-1 through R-4 are not addressed as well. Sometimes in psychological, physiological or medical research, individual subjects form the rows, time-the columns, and the cells form the treatments with an observation taken at each treatment cell. When this is done, and the measurements between each cell are to be compared, clear interpretation may be difficult because of possible carry-over effects of previous treatments (e.g., earlier treatments affecting later ones).