Lesson 1.4 · Unit 1 · Science practice skills 3.C
Correlation, effect size, and statistical significance
The last lesson summarized one variable at a time. Most psychological questions ask how two variables relate: do people who sleep more remember more, and does a treatment make a difference? This lesson covers the three tools the exam uses to describe those relationships and results: the correlation coefficient, effect size, and statistical significance. Together with the descriptive statistics, they are what you need to read the data tables and graphs in the multiple-choice questions and the statistics in the Article Analysis Question, where you must say what a number means in the context of the study.
Science practice skills.
- 3.C. Interpret quantitative or qualitative inferential data from a given table, graph, chart, figure, or diagram.
What the exam asks you to do.
- 3.C: Describe trends in and relationships between the variables as depicted in the data presented, such as interpreting correlational data from a scatterplot, including the correlation coefficient.
- 3.C: Interpret results from research that could be expressed as effect sizes or statistical significance.
- 3.C: Effect size indicates the strength of the relationship between or among variables in a non-experimental study or the size of the difference between or among conditions in an experimental study. The effect size statistic is often interpreted as follows: a small effect size statistic is usually 0.2 and below; a medium effect size statistic is usually between 0.3 and 0.7; a large effect size statistic is usually 0.8 or greater.
- 3.C: Statistical significance indicates that the results of the study are not likely due to chance and instead likely due to the variables being studied. This statistic can be calculated for all types of research methods.
Correlation and the scatterplot
A correlation is a measure of how strongly two variables are related. Each person is measured on both variables, and the data can be shown in a scatterplot, where each point is one person: one variable is on the horizontal axis and the other on the vertical axis.
In a positive correlation, the two variables increase together (more hours of studying, higher scores): the points trend up to the right. In a negative correlation, as one variable increases the other decreases (more hours of screen time before bed, less sleep): the points trend down to the right. When there is no pattern, the correlation is near zero.
The correlation coefficient, written \(r\), is a number from \(-1.00\) to \(+1.00\). The sign gives the direction. The distance from zero gives the strength: \(r=+1.00\) or \(r=-1.00\) is a perfect straight-line relationship, and \(r=0\) means no linear relationship. Rough guidelines are that values around \(\pm0.2\) are weak, around \(\pm0.5\) moderate, and \(\pm0.8\) or more strong, but use the pattern: how tightly the points cluster around a line. Remember that a correlation of \(-0.80\) is stronger than one of \(+0.30\); the minus sign says nothing about strength.
Three cautions apply to every correlation. First, correlation does not equal causation: it may reflect a directionality problem or a third variable (see Lesson 1.1). Second, a correlation tells you about a group, not about what will happen to any one person. Third, an outlier can change the value of \(r\) a great deal.
Inferential statistics and statistical significance
Descriptive statistics summarize the data in hand. Inferential statistics are used to decide whether a result from a sample is likely to be true of the population, or whether it might just be a coincidence of who happened to be in the sample. Suppose a researcher randomly assigns 30 students to a sleep condition and 30 to stay awake, and the sleep group recalls an average of 2 more words. Even if sleep did nothing, random variation in who was in each group would produce some difference. The question is how likely a difference that big would be by chance alone.
A result is statistically significant if it is unlikely to have happened by chance. The exam describes it this way: statistical significance indicates that the results of the study are not likely due to chance and instead likely due to the variables being studied. It is reported as a \(p\) value, the probability of getting results at least this extreme if only chance were operating. By convention, researchers call a result significant when \(p\lt .05\) (less than a 5% probability). A report of "\(p=.01\)" means that if there were really no effect, a difference this large would show up only about 1 time in 100. Statistical significance can be calculated for any kind of research method, not only experiments.
Two things significance does not say. It does not say the effect is large or important, and it does not prove the hypothesis, because there is always a small chance that the result is a fluke. A larger sample makes it easier to get a significant result, which is why effect size is also reported.
Effect size
Effect size says how big the effect or relationship is. In a non-experimental study, it indicates the strength of the relationship between variables (the correlation coefficient is itself an effect size). In an experimental study, it indicates the size of the difference between conditions. A common effect size for a difference between two groups is the difference in means divided by the typical SD, so it tells you how many SDs apart the groups are. The AP Exam interprets effect-size statistics this way: a small effect size is usually 0.2 and below; a medium one is usually between 0.3 and 0.7; and a large one is usually 0.8 or greater.
The two ideas answer different questions. Significance asks, "Is there probably a real effect?" Effect size asks, "How big is it?" A tiny effect can be significant if the sample is huge, and a large effect can fail to be significant if the sample is tiny.
Reading tables and graphs
When a question shows a table or graph, work through the same steps each time. First identify the variables and what is on each axis or in each column, including units. Second, describe the trend or comparison: which group is higher, what direction a relationship goes, and how strong. Third, look for the statistic (means, \(r\), \(p\), effect size) and say what it means in context. Fourth, ask what the design allows: an experiment can support a causal claim; a correlation cannot. Be careful with graphs whose vertical axis does not start at zero, which can make a small difference look dramatic.
A researcher measures hours of sleep and score on a memory test for 8 students and finds \(r=+0.97\). A second study of 400 adults finds a correlation of \(r=-0.62\) between hours of daily social media use and hours of sleep. Interpret each.
Step 1, study 1: the sign is positive, so more sleep goes with higher memory scores. The size, 0.97, is very close to 1, so the relationship is very strong. With only 8 students, one or two unusual people could affect \(r\) a lot, so the researcher should be cautious about generalizing.
Step 2, study 2: the sign is negative, so more social media use goes with less sleep. The size, 0.62, is moderate to strong.
Step 3, causation: neither study can say that one variable caused the other. In study 2, heavier use could cut into sleep, or people who cannot sleep might turn to their phones (directionality), or stress could cause both (third variable).
Step 4, compare strength: study 1's relationship (0.97) is stronger than study 2's (0.62), even though study 2 is negative.
In an experiment, students who slept after studying (mean recall 14.2 words, SD 2.1) are compared with students who stayed awake (mean 11.8 words, SD 2.4). The result is reported as \(p=.01\), and the effect size is 1.06. Interpret the results.
Step 1, the difference between the group means is \(14.2-11.8=2.4\) words in favor of the sleep group.
Step 2, significance: \(p=.01\) is below .05. If sleep had no effect, a difference this large would occur by chance only about 1% of the time, so the result is statistically significant and is probably due to the sleep condition.
Step 3, effect size: 1.06 is above 0.8, so it is a large effect. The group averages are about one SD apart. (The effect size here is the mean difference, 2.4, divided by the typical SD, about 2.25.)
Step 4, cause: because participants were randomly assigned to conditions in an experiment, the researchers can conclude that sleep, rather than a confound, probably caused the better recall, within the limits of the sample.
Two groups of 10,000 people each take a reading test. The group that used a new app averages 100.6 and the group that did not averages 100.0, with an SD of 15 in both groups. The difference is statistically significant (\(p\approx.005\)). Should the app be considered a major improvement?
Step 1, the difference is 0.6 points, and the SD is 15, so the groups are \(0.6\div15=0.04\) SD apart. That is an effect size of 0.04, far below the 0.2 that counts as small.
Step 2, the difference is significant only because the samples are enormous: with 10,000 people per group, even a tiny difference is unlikely to be chance.
Step 3, conclusion: the effect is real in a statistical sense but trivial in size, so the app is not a major improvement. Always read the effect size as well as the \(p\) value.
- \(r\) runs from \(-1\) to \(+1\); the sign is direction and the size is strength (\(-0.80\) is stronger than \(+0.30\)). Correlation does not equal causation.
- Statistical significance (\(p\lt .05\)) means a result is unlikely due to chance alone. It does not mean the result is large or important.
- Effect size shows the strength or size of an effect: small about 0.2 or less, medium 0.3 to 0.7, large 0.8 or more.
- For any table or graph: identify the variables, describe the trend, interpret the statistic in context, and state what the design allows.
- Which shows the stronger relationship: \(r=-0.75\) or \(r=+0.40\)? Explain.
Show answer
\(r=-0.75\) is stronger. Strength depends on how far the value is from zero (0.75 versus 0.40); the sign shows only direction.
- A study reports a correlation of \(r=+0.55\) between hours of exercise per week and a happiness score. Describe the relationship, and say why the study cannot conclude that exercise makes people happier.
Show answer
It is a moderate to fairly strong positive correlation: more exercise goes with higher happiness. A correlation cannot show cause. Happier people might be more motivated to exercise (directionality), or a third variable such as good health or a supportive social life could increase both.
- A study reports \(p=.03\). Explain what that means in plain words.
Show answer
If only chance were operating, a result this extreme would happen about 3% of the time. Because that is below the .05 convention, the result is considered statistically significant, so it is likely due to the variables studied rather than chance.
- A result is statistically significant but has an effect size of 0.05. What can you conclude?
Show answer
The effect is probably real (not due to chance) but it is very small, much less than even a small effect (0.2), so it likely has little practical importance. A very large sample can make a tiny effect significant.
- An experiment reports a mean of 80 for a treatment group and 70 for a control group, with an SD of 10 for each, and a significant \(p\) value. About how large is the effect, and how would you describe it?
Show answer
The means differ by 10, which is 1 SD (\(10\div10=1.0\)), so the effect size is about 1.0. That is greater than 0.8, so it is a large effect.
- A bar graph's vertical axis runs from 70 to 74, and one group's bar looks twice as tall as the other's. What should you check before concluding the difference is large?
Show answer
Check the axis scale and the actual numbers. Because the axis does not start at zero, a difference of a point or two looks huge; look at the effect size or the means and SDs to judge the real size.