Lesson 1.12 · Unit 1 · CED topic 1.12
Potential problems with sampling: bias
A random sample is the best protection against a sample that misrepresents the population, but random selection is not enough by itself. A random method applied to the wrong list, a question worded to push an answer, or a survey that only some chosen people answer can still produce results that are consistently wrong. This lesson names the common sources of bias, shows how to describe the direction of the error, and explains why a bigger sample does not fix a biased method.
Learning objective.
- 1.12.A. Identify potential sources of bias in sampling methods.
Essential knowledge.
- 1.12.A.1. Bias in a sampling method is a systematic error in the sampling procedure that results in a statistic being consistently larger or consistently smaller than the parameter the statistic is used to estimate.
- 1.12.A.2. Voluntary response bias is a bias that may occur when a sample consists entirely of volunteers.
- 1.12.A.3. Undercoverage bias may occur when the sampling method fails to include part of the population or a part of the population is less likely to be selected based on the sampling method.
- 1.12.A.4. Nonresponse bias may occur because of a failure to obtain responses from some individuals chosen to be sampled. The respondents and nonrespondents could differ significantly in ways that are important for the study.
- 1.12.A.5. Response bias may occur when responses to a survey or measurements of observational units tend to differ from the “true” value in one direction. Examples include questions that are confusing or leading (question wording bias) or self-reported responses.
- 1.12.A.6. Nonrandom sampling methods (e.g., samples chosen by convenience or voluntary response) introduce potential bias because they do not use random chance to select the individuals.
What bias is
Bias in a sampling method is a systematic error in the sampling procedure that results in a statistic being consistently larger or consistently smaller than the parameter it is used to estimate. Bias is not the same as the ordinary chance variation between random samples. Every sample statistic differs a little from the parameter; a biased method misses in the same direction almost every time.
In the figure each dot is the percentage computed from one sample. With an unbiased method the dots scatter evenly around the true value; with a biased method they pile up on one side of it. Taking a larger sample makes the dots in each panel cluster more tightly, but it does not move the center, so a biased method gets a precise but wrong answer.
On the AP exam a complete answer about bias names the type of bias, explains how it affects the sample, and says whether the statistic will probably be too high or too low compared with the parameter. A claim about bias without a direction is not enough.
Types of bias
| Type | What goes wrong | Example |
|---|---|---|
| Voluntary response bias | The sample consists entirely of volunteers, who tend to have stronger opinions | A radio station asks listeners to call in |
| Undercoverage bias | The sampling method fails to include part of the population, or part of it is less likely to be selected | A phone survey that dials only landlines |
| Nonresponse bias | Individuals chosen for the sample do not respond, and the respondents differ from the nonrespondents in important ways | A mail survey with a low return rate |
| Response bias | Responses or measurements tend to differ from the "true" value in one direction: confusing or leading questions (question wording bias), or self-reported answers | "Don't you agree that taxes are too high?"; asking people how much they exercise |
Nonrandom methods
Samples chosen by convenience (whoever is easiest to reach) or by voluntary response are nonrandom methods. They introduce potential bias because they do not use random chance to select the individuals: the people who are easy to reach or who choose to answer may differ from everyone else. Notice the difference between undercoverage (some people had no chance to be chosen) and nonresponse (people were chosen but did not answer).
A radio station asks listeners to call in to say whether they support a new tax, and 78% of the 1,200 callers oppose it. Is this a good estimate of the proportion of all voters in the city who oppose the tax?
No. This is voluntary response bias: the people who call in are those who care most strongly, and people who oppose a tax are often more motivated to call. The sample may not represent the voters in the city. The sample proportion of 0.78 is probably too high compared with the true proportion of voters who oppose the tax. The large number of callers does not help, because the method is biased.
To estimate the proportion of adults in a city who own a smartphone, a researcher randomly dials landline phone numbers and calls adults who answer. Describe the bias and its likely direction.
The method has undercoverage bias: adults without a landline have no chance of being selected. People who have no landline tend to be younger, and younger adults are more likely to own a smartphone. The sample will therefore be missing many smartphone owners, so the estimated proportion is probably too low.
A store emails a satisfaction survey to 500 randomly chosen customers; 60 reply. Explain why the result may be biased and in which direction.
This is nonresponse bias: 440 of the chosen customers did not respond. People who had a bad experience may be more motivated to reply, so the respondents may differ from the nonrespondents in satisfaction. The percentage of satisfied customers among the 60 respondents is therefore probably too low as an estimate of the satisfaction of all 500 customers. The argument must say how respondents differ from nonrespondents and which way that pushes the estimate.
(a) A survey asks, "Don't you agree that the city should stop wasting money on parks?" (b) A study asks 100 students how many hours they study per week and records the answers. Identify the bias in each.
(a) Question wording bias (a type of response bias): the leading words push respondents to agree, so the percentage who favor cutting park funding is probably too high. (b) Response bias from self-reported data: students may exaggerate how much they study to look responsible, so the mean reported hours is probably too high. A neutral question and, when possible, an actual measurement would reduce the bias.
- Bias is a systematic error that makes a statistic consistently too high or too low; a larger sample does not remove it.
- Know the four types: voluntary response, undercoverage, nonresponse, response (wording and self-reporting). Convenience and voluntary response samples are nonrandom.
- A complete bias answer names the type, explains the mechanism in context, and states the direction (too high or too low).
- Do not confuse bias with natural sample-to-sample variability, and do not confuse undercoverage with nonresponse.
- A student newspaper posts an online poll asking, "Do you support the new dress code?" and reports the percentage of the 300 students who clicked a response. Identify the type of bias and its likely direction.
Show answer
Voluntary response bias: only students who care enough to click respond, and students who dislike a new rule are usually more motivated to respond. The reported support is probably too low.
- A researcher selects a random sample of students from the list of students enrolled in on-campus classes to find the mean number of hours per week students work at a job. Why might the estimate be biased, and which way?
Show answer
Undercoverage: students who take only online classes are not on the list and cannot be selected, and they probably work more hours. The estimated mean is probably too low.
- A random sample of 400 residents is mailed a survey about a new park, and 70 return it. Explain how nonresponse could bias the result.
Show answer
Only 17.5% responded. The respondents may feel more strongly about the park (for example, those who dislike the park may respond more often) than the nonrespondents, so their percentage may not match that of all 400 residents. To justify a direction, argue about which group is more likely to respond.
- Which is more serious for bias: a voluntary response sample of 5,000 or a random sample of 200? Explain.
Show answer
The voluntary sample, in general. Increasing the sample size does not reduce bias, because the problem is how the units were chosen. A random sample of 200 may vary more from the true value, but its errors are not systematically in one direction.
- Rewrite the question "Wouldn't you say that the unfair school lunch prices should be lowered?" as a neutral question.
Show answer
"Do you think school lunch prices should be lowered, kept the same, or raised?" The new version does not suggest the answer, so it avoids question wording bias.
- A survey uses a random number generator to choose students from a complete list, and every chosen student answers. Does this guarantee the estimate will not differ from the true value? Explain.
Show answer
No. A random method with full response avoids systematic bias, but the sample statistic still varies from sample to sample, so it will usually not equal the parameter exactly. Random selection removes bias; it does not remove sampling variability.