Section 1.5: The Design of Experiments

1. 1.1 Introduction to the Practice of Statistics
2. 1.2 Observational Studies versus Designed Experiments
3. 1.3 Random Sampling
4. 1.4 Bias in Sampling
5. 1.5 The Design of Experiments

For a quick overview of this section, watch this short video summary:

Designed Experiments

Before we can talk about what to design an experiment, we first need to know what an experiment is in a statistics context.

A designed experiment is a controlled study in which one or more treatments are applied to experimental units (subjects). The experimenter then observes the effect of varying these treatments on a response variable.

You can see already that we've got quite a few terms. You may want to get the definition sheet that we started back in Section 1.1.

To help clarify all this terminology, let's consider a simple example:

Example 1

Many designed experiments are double-blind. This means that neither the subjects nor the experimenters know who is receiving which treatment. Typically, subjects are assigned to two groups, with one receiving the treatment (like a new medical drug), while the other receives a placebo. This can be key to avoid researcher bias. Suppose, for example, that the previous study was done by the Atkins Institute and researchers new who was on which diet. Don't you think they'd be tempted to try to influence the results somehow?

In some cases, though, a single-blind experiment is preferable. One good example of this might be a study involving a heart medication. In this case, the doctors involved should be aware of who is taking the drug, and who is taking the placebo.

The Steps in Designing an Experiment

Step 1: Identify the problem or claim to be studied.
The statement of the problem needs to be as specific as possible. In order to be complete, you must identify the response variable and the population to be studied.

Step 2: Determine the factors affecting the response variable.
This is best done by an expert in the field, but we'll be able to do this for most examples we'll be looking at.

Step 3: Determine the number of experimental units.
In general, more experimental units is better. Unfortunately, time and money will always be limiting factors, so we have to decide an appropriate number. We'll talk more about this later on in the course.

Step 4: Determine the level(s) of each factor.
We split factors up into three categories:

1. Control: If possible, we try to fix the level of factors that we're not interested in.
2. Manipulate: This is the treatment - we manipulate the levels of the variable that we think will affect the response variable.
3. Randomize: Often, there are factors we just can't control. To mitigate their effect on the data, we randomize the groups. By randomly assigning experimental units, these factors should be equally spread among all groups.

Step 5: Conduct the experiment.

Step 6: Test the claim.
We'll focus on this step much later in the course - Chapters 9-12. It uses inferential statistics, where we look at information from a sample and try to make a generalization about the population.

OK, now that we have the basic process down, let's look at an example using various designs.

 

We're going to focus on three particular experimental designs - completely randomized, matched-pairs, and randomized block. I'll illustrate all three in the context of determining whether a practice exam helps improve student learning..

Example 2

 

Completely Randomized Design

A completely randomized design is when each experimental unit is assigned to a treatment completely at random. (This is similar to simple random sampling.)

In this design, we would randomly select 60 students and randomly split them into two groups with 30 each. One group does not take the practice exam, while the other does. We have the two groups then take the actual exam and we compare results.

Here's a visual:

Matched-Pairs Design

A matched-pair design is when the experimental units are paired up and each of the pair is assigned to a different treatment.

There are a couple ways to do matched-pairs - we could find people who are very similar somehow, and have one do the practice exam and the other not. Unfortunately, there are so many factors affecting performance on the exam, this pretty impractical.

Another way to do a matched-pair design is to have the same individual before and after the treatment. In this case, we could do just that - give the exam, have students study the practice exam, and then give the exam again. The problem with this design is that we don't know if the improvement (if any) is from the practice exam or just from seeing the material again.

A better plan would be to have all individuals take the exam as a "pre-test", then have 30 students take the practice exam, while the rest do not. Then we have the students all take the exam again, and we compare the "before" and the "after".

Randomized Block Design

A randomized block design is used when the experimental units are divided into homogeneous groups called blocks. Within each block, the experimental units are randomly assigned treatments. (This is similar to stratified sampling.)

Student maturity is a huge factor in college success. Another idea might be to split our sample by academic year - those in their first year versus those in their second. Essentially, we're stratifying the sample, and then doing a completely randomized design on each of the "strata".

<< previous section | next section >>

Chapter: 1 2 3 4 5 6 7 8 9 10 11 12 13

This work is licensed under a Creative Commons License.