Lesson 1.10 · Unit 1 · CED topic 1.10
The investigative question revisited and data collection
So far we have only described data we were handed. This lesson steps back to the beginning of a study, where the most important decisions are made: what exactly are we asking, how will the data be gathered, and what will the answer be allowed to say? You will learn the three parts of a well-built investigative question, the difference between a census, an experiment and an observational study, and what random selection and random assignment each make possible.
Learning objectives.
- 1.10.A. Determine the components of an investigative question within a statistical study.
- 1.10.B. Identify a census.
- 1.10.C. Identify an experiment.
- 1.10.D. Identify an observational study.
- 1.10.E. Justify the appropriateness of generalizations for a statistical study.
Essential knowledge.
- 1.10.A.1. The first component of an investigative question should guide the data collection process and should be phrased in terms of the variable(s) of interest in the study.
- 1.10.A.2. The second component of an investigative question should guide the data analysis choice.
- 1.10.A.2.i. In the case of a hypothesis test, the investigative question should make clear the parameter and the direction of the alternative hypothesis (i.e., not equal, greater than, less than, association, not independent).
- 1.10.A.2.ii. In the case of a confidence interval, the investigative question should make clear the parameter and the goal of estimation of that parameter within a range of potential values.
- 1.10.A.3. The third component of an investigative question should indicate the type(s) of conclusion(s) applicable from the study. The investigative question should provide the population to which the conclusions will be applicable and, in the case of an experiment that uses random assignment, a cause-and-effect conclusion.
- 1.10.B.1. A census consists of recording information from all items or individuals in a population.
- 1.10.C.1. An experiment is a study in which a researcher assigns conditions, or treatments, to experimental units to explore an investigative question of interest about the population.
- 1.10.C.2. The experimental unit is the observational unit to which the treatment is assigned. When experimental units consist of people, they are sometimes referred to as subjects or participants.
- 1.10.C.3. An explanatory variable, or factor, is a variable whose different categories, or levels, are imposed on the experimental units. The different categories, or levels, of the explanatory variable are called treatments. When there is more than one explanatory variable, the combinations of the categories, or levels, of the explanatory variables are called treatments.
- 1.10.C.4. A response variable is an outcome measured on each experimental unit after the treatment has been administered.
- 1.10.D.1. An observational study is a study where treatments are not imposed. The researcher records the values of the variables of interest in order to explore an investigative question of interest.
- 1.10.D.2. A prospective study is one in which the observational units of study are selected at a point in time, and data are gathered both at that time and into the future.
- 1.10.D.3. A retrospective study is one in which the observational units of study are selected at a point in time and data from the past are gathered.
- 1.10.D.4. A survey is an observational study in which the data are collected from humans using a standard set of questions.
- 1.10.D.5. A confounding variable in an observational study provides an alternative explanation for the observed relationship between the explanatory and response variables determined in the study, thereby reducing the possibility of concluding a causal relationship between the explanatory and response variables of interest. To be a confounding variable, a variable must be associated with both the explanatory variable and the response variable.
- 1.10.E.1. A sample is considered random when all observational units in the sample are selected from the population using some type of random mechanism, such as a random number generator.
- 1.10.E.2. When observational units, or experimental units, in a sample are randomly selected from a population, it is appropriate to make generalizations about the entire population of individuals from which the sample was selected.
- 1.10.E.3. A sample is not randomly selected when observational units are deliberately chosen or volunteer themselves to be in the sample.
- 1.10.E.4. When observational units, or experimental units, in a sample are not randomly selected from a population, it is appropriate to make generalizations only about a population of individuals that are similar to those used in the study.
The three components of an investigative question
An investigative question for a specific study has three components.
- The variable(s) of interest. The first component should guide the data collection process and should be phrased in terms of the variable or variables of interest in the study.
- The analysis. The second component should guide the data analysis choice. For a hypothesis test, the question should make clear the parameter and the direction of the alternative hypothesis (not equal, greater than, less than, association, not independent). For a confidence interval, it should make clear the parameter and the goal of estimating that parameter within a range of plausible values.
- The conclusion. The third component should indicate the type or types of conclusion that apply from the study. It should name the population to which the conclusions can be applied, and, for an experiment that uses random assignment, a cause-and-effect conclusion.
A student government wants to know whether the proportion of students at their school who ride the bus is greater than 40%. A random sample of 150 of the school's 1,800 students will be surveyed. Write the investigative question and identify its three components.
Question: "Is the proportion of all students at our school who ride the bus greater than 0.40?" (1) The variable of interest is whether a student rides the bus (categorical), which tells us what to ask each sampled student. (2) The parameter is the proportion \(p\) of the school's students who ride the bus, and the question points in one direction (greater than 0.40), so the analysis is a one-sided hypothesis test about a proportion. (3) The sample is random, so the conclusion can be applied to the population of all students at this school, and because it is a survey, not an experiment, it cannot be a cause-and-effect conclusion.
Census, experiments and observational studies
A census consists of recording information from all items or individuals in a population. A census is possible only for a small or accessible population (every student in one classroom, for example); for most questions it is too costly or difficult, which is why we sample.
Most studies that use a sample are one of two types, and the key question is whether the researcher imposes conditions on the units.
Experiments. An experiment is a study in which a researcher assigns conditions, or treatments, to experimental units to explore an investigative question about the population. The vocabulary:
- The experimental unit is the observational unit to which the treatment is assigned. When the units are people they are often called subjects or participants.
- An explanatory variable, or factor, is a variable whose different categories, or levels, are imposed on the experimental units. The levels of the explanatory variable are the treatments. When there is more than one explanatory variable, the combinations of levels are the treatments.
- A response variable is an outcome measured on each experimental unit after the treatment has been given.
Observational studies. An observational study is one in which treatments are not imposed. The researcher records the values of the variables of interest in order to explore the question. There are several kinds:
- A prospective study selects the units at one point in time and gathers data both then and into the future.
- A retrospective study selects the units at one point in time and gathers data from the past.
- A survey is an observational study in which data are collected from people using a standard set of questions.
Confounding in observational studies
Because an observational study cannot assign who gets what, other differences between the groups may offer an alternative explanation for what we see. A confounding variable in an observational study provides an alternative explanation for the observed relationship between the explanatory and response variables, which reduces the possibility of concluding that one causes the other. To be a confounding variable, a variable must be associated with both the explanatory variable and the response variable.
For example, suppose a study finds that students who eat breakfast have higher grades. Hours of sleep could be a confounder: students who sleep more may both eat breakfast more often and do better in school. A confounder must be linked to both variables; a variable that affects only the grades does not, by itself, explain why breakfast and grades are related.
What a study can generalize to
The conclusions we can draw depend on how the units were chosen. A sample is random when all observational units in it are selected from the population using some random mechanism, such as a random number generator.
- When units in a sample are randomly selected from a population, it is appropriate to make generalizations about the entire population from which the sample was selected.
- A sample is not random when units are deliberately chosen or volunteer themselves. When the sample is not random, it is appropriate to make generalizations only about a population of individuals similar to those in the study.
Combining this with how treatments are given (which the experimental design lesson covers in detail) gives the standard picture of what a study can conclude.
| Units randomly selected | Units not randomly selected | |
|---|---|---|
| Treatments randomly assigned | Can generalize to the population and conclude cause and effect | Cause and effect, but only for units similar to those in the study |
| No random assignment (observational) | Can generalize to the population, but only an association (no cause and effect) | Neither: only an association, only for similar units |
Classify each study and name the type: (a) 60 volunteers are randomly assigned either a new diet or their usual diet, and weight change is recorded after 12 weeks; (b) researchers read the medical records of 500 adults to compare lung disease rates for smokers and nonsmokers; (c) researchers choose 2,000 five-year-olds and follow them until age 18, recording daily screen time and grades; (d) 800 adults answer a poll about a proposed law; (e) every one of the 28 students in a class is measured for height.
(a) Experiment: the researchers assign the diet. (b) Observational, retrospective: past data are gathered and nothing is imposed. (c) Observational, prospective: the children are selected now and data are gathered into the future. (d) Observational, a survey (standard questions asked of people). (e) A census of the class, since all of the individuals in the population (the class) are measured.
A teacher randomly assigns 40 students to either study with flashcards or reread their notes, and compares quiz scores. Identify the experimental unit, the explanatory variable, the treatments and the response variable.
The experimental units are the 40 students (the objects to which the treatments are assigned). The explanatory variable (factor) is the study method, with two levels, and those two levels are the treatments: flashcards and rereading. The response variable is the quiz score, which is measured on each student after the treatment. Because the treatments are assigned randomly, the study can support a cause-and-effect conclusion about the effect of study method on quiz score for students like these.
(a) A random sample of 500 registered voters in a state is polled. (b) A school posts a link and 40 students who choose to respond are used. For each, what population can the results be applied to?
(a) The voters were randomly selected, so the conclusions apply to all registered voters in the state. (b) The sample is a volunteer sample, so it is not random, and the results apply only to students similar to those who responded, not to the whole school.
- A good investigative question names the variable(s) (guiding data collection), the parameter and direction or goal (guiding the analysis), and the population and type of conclusion.
- Census: every unit. Experiment: the researcher assigns treatments (explanatory variable = factor, levels = treatments, response variable). Observational study: nothing imposed (survey, prospective, retrospective).
- A confounding variable is associated with both the explanatory and the response variable.
- Random selection lets you generalize to the population; a nonrandom sample generalizes only to similar individuals. Random assignment allows cause-and-effect conclusions.
- A researcher wants to know whether the mean time teenagers spend on homework is less than 2 hours, using a random sample from one school district. State the parameter and the direction of the alternative hypothesis, and identify the population to which the conclusion applies.
Show answer
The parameter is the mean time \(\mu\) spent on homework by all teenagers in the district. The alternative hypothesis is \(\mu<2\) hours (one-sided, less than). The conclusion applies to all teenagers in the district, since the sample is random.
- Classify as experiment, prospective observational study, retrospective observational study or census: researchers record the age at which every one of the 120 employees of a company started working there.
Show answer
A census: information is recorded from every individual in the population (all 120 employees).
- In a study of 300 adults, researchers ask them about past exercise habits and then record their current blood pressure. Is this an experiment or an observational study? Why?
Show answer
An observational study (and retrospective for the exercise data): the researchers did not assign anyone to exercise levels; they recorded values.
- A researcher randomly assigns 90 plants to receive no fertilizer, fertilizer A, or fertilizer B and measures growth. Identify the experimental units, the explanatory variable, the treatments and the response variable.
Show answer
Experimental units: the 90 plants. Explanatory variable: the type of fertilizer (three levels). Treatments: no fertilizer, fertilizer A, fertilizer B. Response variable: the growth of the plant.
- A study finds that people who own a pet live longer, on average, than people who do not. Suggest a possible confounding variable and explain why it is one.
Show answer
One possibility is general health or exercise level (for example, healthier, more active people may be more likely to own a dog and also live longer). To be a confounder it must be associated with both pet ownership and life span, giving an alternative explanation for the relationship.
- A radio station asks listeners to call in and says that 78% of callers favor a new stadium. For what population can the result be generalized?
Show answer
The callers volunteered, so the sample is not random. The result can be generalized only to people similar to those who called in, not to the whole city.