Lesson 1.7 · Unit 1 · CED topic 1.7 (part 2)
Variability, units, outliers and choosing a summary
Two data sets can have exactly the same center and still be very different: a bus that always takes 20 minutes and a bus that takes 5 minutes some days and 40 minutes on others both average about 20. The variability (or spread) of a distribution says how much the values differ from each other. This lesson defines the three standard measures of spread, shows what changing units does to every summary statistic, gives two common rules for calling a value an outlier, and then asks the question the exam cares about: which summary should you report, and why?
Learning objectives.
- 1.7.B. Calculate measures of variability for quantitative data.
- 1.7.C. Calculate different units of measurement for summary statistics.
- 1.7.D. Calculate outliers for quantitative data.
- 1.7.E. Compare multiple quantitative one-variable summary statistics.
- 1.7.F. Justify the selection of a summary statistic for describing quantitative data.
Essential knowledge.
- 1.7.B.1. Three commonly used measures of variability (or spread) in the distribution of a quantitative variable are the range, interquartile range, and standard deviation.
- 1.7.B.2. The range is the difference between the maximum data value and the minimum data value.
- 1.7.B.3. The interquartile range (IQR) is the difference between the third and first quartiles: \(Q_3-Q_1\).
- 1.7.B.4. The standard deviation is a typical deviation of the data values from their mean and can be found with and without using technology. The sample standard deviation is denoted by \(s\) and calculated by \(s=\sqrt{\dfrac{1}{n-1}\sum(x_i-\bar x)^2}\), where \(x_i\) is the data value, \(\bar x\) is the mean, and \(n\) is the number of data values in the sample. The square of the sample standard deviation, \(s^2\), is called the sample variance.
- 1.7.C.1. Changing units of measurement affects the values of the calculated statistics.
- 1.7.D.1. There are many methods for determining potential outliers. Two methods frequently used are as follows:
- 1.7.D.1.i. An outlier is a value located more than \(1.5\times\text{IQR}\) above the third quartile or more than \(1.5\times\text{IQR}\) below the first quartile.
- 1.7.D.1.ii. An outlier is a value located more than 2 standard deviations above, or below, the mean.
- 1.7.E.1. Summary statistics can be used to compare features of two or more independent samples, including center, variability, shape, and outliers.
- 1.7.F.1. The median and IQR are considered a resistant (or robust) measure of center and measure of variability, respectively, because outliers do not greatly (if at all) affect their values. Because outliers can affect their values greatly, the mean is considered a nonresistant (or non-robust) measure of center, and the range and standard deviation are considered nonresistant (or non-robust) measures of variability.
- 1.7.F.2. Summary statistics of a quantitative variable may reveal information that can be used to justify claims about the variable in context.
The range and the interquartile range
Three commonly used measures of variability for a quantitative variable are the range, the interquartile range and the standard deviation.
- The range is the difference between the maximum and minimum values: \(\text{range}=\max-\min\). It uses only two values and is easily distorted by one extreme observation.
- The interquartile range (IQR) is the difference between the third and first quartiles: \(\text{IQR}=Q_3-Q_1\). It is the length of the interval that holds the middle 50% of the data.
For the 30 commute times, the range is \(90-5=85\) minutes and the IQR is \(32-15=17\) minutes.
The standard deviation
The standard deviation is a typical deviation of the data values from their mean. The sample standard deviation is written \(s\): \[s=\sqrt{\frac{1}{n-1}\sum_{i=1}^{n}\left(x_i-\bar x\right)^2},\] where \(x_i\) is a data value, \(\bar x\) is the mean and \(n\) is the number of values. The square of the standard deviation, \(s^2\), is called the sample variance. It can be found with or without technology. The steps by hand: find \(\bar x\); find each deviation \(x_i-\bar x\); square the deviations; add them; divide by \(n-1\); take the square root. The deviations are squared so that those below the mean do not cancel those above it, and the divisor is \(n-1\) (not \(n\)) for a sample.
An interpretation must be in context: "the number of text messages sent by a student typically differs from the mean of 10 by about 3.4 messages." The standard deviation is zero only when all the values are equal, and it is large when many values are far from the mean.
Changing units
Changing the units of measurement affects the values of the calculated statistics. Two kinds of change matter.
- Adding a constant \(c\) to every value shifts the whole distribution. The mean, median, quartiles, minimum and maximum all increase by \(c\), but the range, IQR, standard deviation and variance do not change.
- Multiplying every value by a constant \(k\) (for example converting units) stretches the distribution. The center and position measures multiply by \(k\), and the range, IQR and standard deviation multiply by \(|k|\) (the variance multiplies by \(k^2\)).
Converting the commute times from minutes to hours means multiplying by \(1/60\): the mean \(26.03\) minutes becomes \(0.434\) hours, the standard deviation \(18.04\) minutes becomes \(0.301\) hours and the IQR \(17\) minutes becomes \(0.283\) hours.
Outliers: two common rules
There are many methods for identifying potential outliers. Two are used frequently, and a good answer always says which one it uses.
- The 1.5 × IQR rule. A value is an outlier if it is more than \(1.5\times\text{IQR}\) above \(Q_3\) or more than \(1.5\times\text{IQR}\) below \(Q_1\). The cutoffs \(Q_1-1.5\,\text{IQR}\) and \(Q_3+1.5\,\text{IQR}\) are called the fences.
- The 2-standard-deviation rule. A value is an outlier if it is more than 2 standard deviations above or below the mean.
The rules do not always agree. They are guides for flagging unusual values; the decision about what to do with an outlier depends on the context (an error, a different kind of case, or a real and important extreme).
Five students sent 6, 8, 9, 13 and 14 text messages. Find the standard deviation.
The mean is \(\bar x=50/5=10\). The deviations are \(-4,\,-2,\,-1,\,3,\,4\). The squared deviations are \(16,\,4,\,1,\,9,\,16\), and they sum to 46. Then \[s^2=\frac{46}{5-1}=11.5,\qquad s=\sqrt{11.5}\approx3.39.\] In context: the number of messages a student sent typically differs from the mean of 10 messages by about 3.4 messages.
For the 30 commute times, \(\bar x\approx26.03\), \(s\approx18.04\), \(Q_1=15\) and \(Q_3=32\). Which values are outliers by (a) the 1.5 × IQR rule and (b) the 2-standard-deviation rule?
(a) \(\text{IQR}=17\), so \(1.5\,\text{IQR}=25.5\). The fences are \(15-25.5=-10.5\) and \(32+25.5=57.5\). The values 60 and 90 are above 57.5, so there are two outliers: 60 and 90 minutes. (b) \(\bar x\pm2s=26.03\pm36.09\), which gives \(-10.05\) to \(62.12\). Only 90 is outside that interval, so there is one outlier, 90 minutes. The two rules disagree about the 60-minute commute (60 is above the fence of 57.5 but less than 2 standard deviations above the mean), which is why you should name the rule you used.
Remove the 90-minute commute from the 30 values and the remaining 29 have mean 23.83, median 20, standard deviation 13.64, IQR 16.5 and range 55. Compare these with the full-data values (mean 26.03, median 21, standard deviation 18.04, IQR 17, range 85).
Removing the one outlier changed the mean by about 2.2 minutes and the standard deviation by about 4.4 minutes, and it dropped the range by 30 minutes; the median changed by 1 minute and the IQR by 0.5 minute. The median and IQR are resistant (robust): outliers do not greatly affect them. The mean, standard deviation and range are nonresistant (non-robust): an outlier can change them a great deal.
Choosing and using summary statistics
Summary statistics let us compare features of two or more independent samples: their center, variability, shape and outliers. To justify the choice of a statistic, use resistance and shape:
- If the distribution is skewed or has outliers, describe it with the median and IQR, because they are resistant and are not dominated by the tail.
- If it is roughly symmetric with no outliers, the mean and standard deviation are appropriate and use all of the data.
Summary statistics can also be used to justify claims about the variable in context, as long as the claim cites the numbers. "Half of these students commute 21 minutes or less, and the middle 50% commute between 15 and 32 minutes" is better than "commutes are around 20 minutes."
Two sections of a course took the same exam. Section A (15 students) had mean 77.5, standard deviation 7.9, median 78 and IQR 11. Section B (15 students) had mean 77.1, standard deviation 15.4, median 78 and IQR 24. Compare the sections.
The centers are almost identical: the means are 77.5 and 77.1 and both medians are 78. The variability is very different: the scores in Section B are about twice as spread out as in Section A (standard deviation 15.4 compared with 7.9, and IQR 24 compared with 11). So a student's score in Section B is much less predictable than one in Section A. Every sentence names both sections and uses a comparison word.
1-Var Stats on the list gives \(S_x\) (use this for a sample) and \(Q_1\), \(Q_3\). Compute the fences by hand (or store them) and compare each suspect value. To see outliers, draw a modified boxplot (STAT PLOT, the boxplot icon with the dots; the dots mark values beyond the fences).
- Range (max \(-\) min), IQR (\(Q_3-Q_1\)) and standard deviation \(s=\sqrt{\sum(x_i-\bar x)^2/(n-1)}\) measure variability; \(s^2\) is the variance.
- Adding a constant shifts center and position measures but not spread; multiplying by \(k\) multiplies center and spread by \(|k|\).
- Outlier rules: more than \(1.5\times\text{IQR}\) beyond \(Q_1\) or \(Q_3\), or more than 2 standard deviations from the mean. Say which rule you used.
- Median and IQR are resistant; mean, standard deviation and range are not. Use median and IQR for skewed data or data with outliers.
- Always interpret in context and compare two groups with sentences that name both.
- Find the standard deviation of 2, 4, 4, 6 and 9.
Show answer
\(\bar x=25/5=5\). Deviations: \(-3,-1,-1,1,4\); squared: \(9,1,1,1,16\); sum \(=28\). \(s^2=28/4=7\), so \(s=\sqrt7\approx2.65\).
- A set of 50 delivery times (in minutes) has mean 45, standard deviation 12 and IQR 18. Find the mean, standard deviation and IQR in seconds.
Show answer
Multiply by 60: the mean is 2700 seconds, the standard deviation is 720 seconds and the IQR is 1080 seconds.
- A delay of 5 minutes is added to every delivery time in the previous problem. What happens to the mean, median, standard deviation and IQR?
Show answer
The mean and median each increase by 5 minutes. The standard deviation and the IQR do not change, because adding a constant shifts the data but does not change how spread out they are.
- A data set has \(Q_1=20\) and \(Q_3=32\). Use the 1.5 × IQR rule to decide whether 55 and 45 are outliers.
Show answer
\(\text{IQR}=12\), \(1.5\times12=18\). Fences: \(20-18=2\) and \(32+18=50\). 55 is above 50, so it is an outlier. 45 is below 50, so it is not.
- Exam scores have mean 70 and standard deviation 8. Which of 52 and 88 are outliers by the 2-standard-deviation rule?
Show answer
The interval is \(70\pm16=(54,86)\). Both 52 (below 54) and 88 (above 86) are outliers.
- The incomes in a city are strongly skewed right, with a few very large incomes. Which summary statistics should be reported for the center and spread, and why?
Show answer
The median and IQR. They are resistant, so the few very large incomes do not distort them, while the mean and standard deviation would be pulled up by the tail and give a misleading picture of a typical household.