Lesson 1.13 · Unit 1 · CED topic 1.13
Experimental design
An experiment is the only kind of study that can show that one thing causes another. What gives an experiment that power is not the size of the sample or the care with which the data were measured, but how the treatments were assigned. This lesson lists the elements of a well-designed experiment, describes the three standard designs (completely randomized, randomized block and matched pairs), and explains exactly which conclusions a well-designed experiment supports and why.
Learning objectives.
- 1.13.A. Identify elements of a well-designed experiment.
- 1.13.B. Identify experimental designs.
- 1.13.C. Justify the appropriateness of a particular experimental design.
- 1.13.D. Justify the appropriateness of the conclusions based on a well-designed experiment.
Essential knowledge.
- 1.13.A.1. A well-designed experiment should include the following:
- 1.13.A.1.i. Comparisons of at least two treatment groups, one of which could be a control group
- 1.13.A.1.ii. Random assignment of treatments to experimental units
- 1.13.A.1.iii. Replication
- 1.13.A.1.iv. Direct control of potential extraneous sources of variation in the response
- 1.13.A.2. A control group is a collection of experimental units that are created for comparison. A control group may be given a treatment different from the treatment of interest to determine if the treatment of interest has an effect (e.g., a treatment with an inactive substance, a placebo, may be given).
- 1.13.A.3. The placebo effect is the difference between the average response to a placebo and the average response to no treatment.
- 1.13.A.4. In a single-blind, also called single-masked, experiment, participants do not know which treatment they are receiving, but members of the research team who interact with them know which treatment each participant is receiving, or vice versa.
- 1.13.A.5. In a double-blind, also called double-masked, experiment, neither the participants nor the members of the research team who interact with them know which treatment each participant is receiving.
- 1.13.A.6. An extraneous source of variation, also referred to as an extraneous variable, is a variable that is known (or believed) to affect the response but is not an explanatory variable being studied.
- 1.13.A.7. The purpose of random assignment is to create treatment groups that are as similar as possible with respect to extraneous sources of variation. If random assignment is successful, the respective distributions of each extraneous variable will be approximately the same for all the treatment groups.
- 1.13.A.8. A confounding variable in an experiment is a variable that is related to the explanatory variable in such a way that it is difficult to determine which variable, explanatory or confounding, is influencing the change in the response variable. However, in a well-designed experiment, the potential for confounding variables is reduced.
- 1.13.A.9. Replication within an experiment means more than one experimental unit is assigned to each treatment.
- 1.13.A.10. Direct control in an experiment means keeping the settings of certain potential extraneous sources of variation in the response variable the same from experimental unit to experimental unit.
- 1.13.B.1. In a completely randomized design, treatments are assigned to experimental units completely at random. Often the number of experimental units assigned to each treatment will be the same, but the sample sizes in each treatment do not have to be the same.
- 1.13.B.2. A blocking variable is a source of extraneous variation in the response variable. In a randomized block design, the experimental units are first grouped according to similar values of a blocking variable. These groups are called blocks. Units within the same block are homogeneous with respect to the blocking variable. After the blocks are formed, the treatments are randomly assigned to experimental units within each block so that all treatments occur within every block.
- 1.13.B.3. The purpose of blocking is to separate the variation in the response caused by the blocking variable from the rest of the extraneous variation in the response. Blocking allows for more precise comparisons of the response across the treatments. Within a block, the treatments can be compared without having to worry about variation in the response caused by changes in the blocking variable.
- 1.13.B.4. A matched pairs design is a randomized block design with only two treatments. Experimental units are arranged in pairs by matching on one or more extraneous sources of variation in the response variable. Each pair receives both treatments by randomly assigning one treatment to one member of the pair and the other treatment to the second member of the pair. Alternatively, each experimental unit may get both treatments while the order of the treatments is randomized.
- 1.13.C.1. One experimental design may be more appropriate than another experimental design based on the goals of the investigative study, the characteristics of the population, and the sample and variables involved.
- 1.13.D.1. Using random assignment of treatments to experimental units allows for cause-and-effect conclusions between the explanatory and the response variables because the potential for confounding variables is reduced.
- 1.13.D.2. Depending on the experimental unit, it may be unethical or difficult to randomly select experimental units to participate in an experiment. In that case, the study’s experimental units are obtained from volunteers and will represent the population of experimental units similar to those who participated in the study.
Elements of a well-designed experiment
A well-designed experiment should include:
- Comparison of at least two treatment groups, one of which could be a control group.
- Random assignment of treatments to experimental units.
- Replication.
- Direct control of potential extraneous sources of variation in the response.
Comparison and the control group. A control group is a collection of experimental units created for comparison. It may receive a treatment different from the treatment of interest, to find out whether the treatment of interest has an effect: for example an inactive substance, a placebo. The placebo effect is the difference between the average response to a placebo and the average response to no treatment; people often improve simply because they believe they are being treated.
Blinding. In a single-blind (single-masked) experiment, the participants do not know which treatment they are receiving, but members of the research team who interact with them do (or the reverse). In a double-blind (double-masked) experiment, neither the participants nor the members of the research team who interact with them know which treatment each participant receives. Blinding keeps expectations from influencing the responses or how they are measured.
Extraneous variables and confounding. An extraneous source of variation (extraneous variable) is a variable that is known or believed to affect the response but is not an explanatory variable being studied. A confounding variable in an experiment is related to the explanatory variable in such a way that it is difficult to tell which of the two is influencing the change in the response. In a well-designed experiment the potential for confounding is reduced.
Random assignment. The purpose of random assignment is to create treatment groups that are as similar as possible with respect to extraneous sources of variation. If random assignment is successful, the distribution of each extraneous variable is approximately the same in all the treatment groups, so differences in the response are not explained by those variables. Random assignment is done with a random mechanism: a random number generator, drawing numbered slips, or the flip of a coin.
Replication means that more than one experimental unit is assigned to each treatment. With replication we can see how much the response varies from unit to unit and can tell a real difference between treatments from chance variation.
Direct control means keeping the settings of certain potential extraneous sources of variation the same from experimental unit to experimental unit (the same soil, the same time of day, the same equipment).
Completely randomized design
In a completely randomized design the treatments are assigned to the experimental units completely at random. Often the same number of units is assigned to each treatment, but the groups do not have to be the same size.
Randomized block design
A blocking variable is a source of extraneous variation in the response that we can identify ahead of time. In a randomized block design the experimental units are first grouped, according to similar values of a blocking variable, into blocks; units within a block are homogeneous with respect to the blocking variable. After the blocks are formed, the treatments are randomly assigned to units within each block, so that every treatment occurs in every block.
The purpose of blocking is to separate the variation in the response caused by the blocking variable from the rest of the extraneous variation. It allows more precise comparisons of the treatments: within a block, treatments can be compared without worrying about changes in the response caused by the blocking variable. (Blocking is to experiments what stratifying is to sampling.)
Matched pairs
A matched pairs design is a randomized block design with only two treatments. The units are arranged in pairs by matching on one or more extraneous sources of variation, and within each pair one unit gets one treatment and the other gets the other, decided at random. Alternatively, each experimental unit may receive both treatments, with the order of the treatments randomized.
A gardener wants to know whether a new fertilizer increases tomato yield compared with the usual fertilizer. Thirty tomato plants are available. Describe a completely randomized design.
Number the plants 1 to 30. Use `randIntNoRep(1,30,15)` to choose 15 plants at random to receive the new fertilizer; the other 15 receive the usual fertilizer (the control group). Each treatment is given to 15 plants, so the experiment is replicated. Use direct control: the same soil mix, the same amount of water and sunlight, and the same planting date for all the plants. After a set time, measure the yield of each plant in kilograms (the response variable) and compare the two groups. The random assignment spreads other differences among the plants (such as natural vigor) roughly evenly across the two groups.
A school compares two reading apps with 60 students. Reading ability strongly affects reading scores. Describe a randomized block design and explain why it is better than a completely randomized design here.
Block the students by reading level: 20 low, 20 middle and 20 high. Within each block, randomly assign 10 students to App 1 and 10 to App 2, so every treatment appears in every block. Comparing the apps within a block removes the large differences between a strong and a weak reader from the comparison. A completely randomized design could, by bad luck, put more strong readers in one group; blocking guarantees balance on reading level and gives a more precise comparison. The blocking variable (reading level) is a source of variation, not a treatment.
A coach wants to compare two types of running shoes using 20 runners. Describe a matched pairs design.
Each runner will run a timed mile in both types of shoes, so each runner is a "pair" with himself or herself, and runner-to-runner differences in speed are removed from the comparison. For each runner, flip a coin to decide which shoes are worn first, to prevent the order (for example, fatigue) from favoring one shoe type. Record the difference in the two times for each runner (the response) and analyze the 20 differences.
A drug company tests a new medicine for headaches in 200 volunteers. Explain how a placebo and a double-blind design improve the experiment.
Half of the volunteers, chosen at random, get the new medicine and half get a placebo that looks the same. The placebo group controls for the placebo effect: some people feel better just because they are treated. In a double-blind design neither the volunteers nor the doctors who rate their headaches know who received which pill. That prevents participants' expectations and the evaluators' hopes from affecting the reported pain, so a difference in pain relief can be credited to the medicine itself.
What can be concluded
Using random assignment of treatments to experimental units allows cause-and-effect conclusions between the explanatory and the response variable, because the potential for confounding variables is reduced: the groups are alike on average in every other way. If the response differs between groups by more than we would expect from chance alone, the treatment is the plausible explanation.
Random assignment is not random selection. Random assignment says who gets which treatment; random selection says who is in the study. Depending on the experimental unit, it may be unethical or difficult to randomly select units for an experiment (you cannot randomly select people and force them into a trial). In that case the units are obtained from volunteers and represent the population of experimental units similar to those who participated in the study.
One design may be more appropriate than another depending on the goals of the study, the characteristics of the population, and the sample and the variables involved. When you justify a design on the exam, tie your reasoning to those features of the situation.
In a study, 80 volunteers are randomly assigned to meditate daily or not, and after 8 weeks the meditators have significantly lower blood pressure. What can be concluded, and about whom?
Because the volunteers were randomly assigned to the treatments, the groups are expected to be similar on other variables, so the study supports the conclusion that meditation caused the lower blood pressure. Because the participants were volunteers, not a random sample, the conclusion applies only to people similar to those who volunteered, not to everyone. The exam answer should cite random assignment for the causal claim and the volunteer sample for the limit on generalization.
- A well-designed experiment has at least two treatment groups (possibly a control/placebo), random assignment, replication and direct control.
- Designs: completely randomized; randomized block (group by a blocking variable, then randomize within blocks); matched pairs (blocks of two, or each unit gets both treatments in random order).
- Random assignment \(\Rightarrow\) cause-and-effect conclusions; random selection \(\Rightarrow\) generalization to the population. Do not mix up the two.
- Blinding (single or double) and placebos guard against the placebo effect and biased measurement.
- Describe how to randomly assign 40 mice to two diets using a random number generator.
Show answer
Number the mice 1 to 40. Use `randIntNoRep(1,40,20)` to choose 20 mice for Diet A; the remaining 20 get Diet B. This is a completely randomized design with replication (20 mice per treatment).
- What is the purpose of random assignment in an experiment?
Show answer
To create treatment groups that are as similar as possible on extraneous variables, so the distributions of those variables are about the same in each group and differences in the response can be attributed to the treatments rather than to confounding variables.
- A researcher compares two exercise programs for 90 adults. Age strongly affects fitness. Explain how blocking by age (under 30, 30 to 50, over 50) would work.
Show answer
Group the adults into three age blocks of 30. Within each block, randomly assign 15 people to each program. Each program then appears in every age block, and the comparison within each block is not affected by age.
- What is the difference between a single-blind and a double-blind experiment?
Show answer
In a single-blind experiment, either the participants do not know their treatment (or the evaluators do not), but the other group knows. In a double-blind experiment neither the participants nor the researchers who interact with them or evaluate them know which treatment each participant received.
- A study randomly assigns 50 volunteers to a new teaching method or the usual method and finds that scores are significantly higher with the new method. Can we conclude that the new method causes higher scores for all students everywhere? Explain.
Show answer
We can conclude that the new method caused the higher scores for participants like the volunteers, because treatments were randomly assigned. We cannot generalize to all students because the participants were volunteers and not a random sample of all students.
- A student says, "Random assignment means we randomly picked the participants from the population." Correct the statement.
Show answer
Random assignment means treatments were assigned to the experimental units at random. Picking the participants from the population at random is random selection, which allows generalization to the population; random assignment allows cause-and-effect conclusions.