Lesson 1.2 · Unit 1 · Science practice skills 2.B to 2.D

Samples, measurement, and ethics

A study's design tells you what kind of conclusion it can support. Three other questions decide whether you should believe the conclusion: who took part, how the variables were measured, and whether the participants were treated properly. This lesson covers all three, and then shows how a single study becomes part of scientific knowledge. These are the questions behind the research-methods items on the multiple-choice section and behind several of the points on the Article Analysis Question, which asks about the sample, the measures, the ethical guidelines, and whether the results can be generalized.

Required skills · 2.B and 2.C, 2.D

Science practice skills.

2.B and 2.C. Evaluate the appropriate use of research design elements in experimental and non-experimental methodologies (sampling, measurement and scientific process).
2.D. Evaluate whether a psychological research scenario followed appropriate ethical procedures.

What the exam asks you to do.

2.B and 2.C: In a research scenario, identify the participants of a study who make up the sample that is drawn from a population.
2.B and 2.C: Determine whether the sample used is representative of the population and is selected by either random sampling or convenience sampling.
2.B and 2.C: Evaluate whether the sampling procedure involves sampling bias.
2.B and 2.C: Determine to what extent the sampling method allows for results of the study to be generalized.
2.B and 2.C: Determine whether the measurement instrument for research used qualitative (such as structured interviews) or quantitative measures (such as Likert scales).
2.B and 2.C: Describe the impact of appropriate representation of participants in conducting research and on the outcomes of research.
2.B and 2.C: Explain how conclusions from research evolve via scientific processes such as peer review and replication.
2.D: Explain the importance of institutional review for research involving human and non-human animals.
2.D: Determine whether proper informed consent and/or informed assent was used in a research scenario.
2.D: Determine whether researchers took steps to protect participants from harm.
2.D: Determine whether researchers kept participant information confidential and/or anonymous.
2.D: Determine whether deception was used in the research (possibly with the help of research confederates).
2.D: Determine whether researchers used proper debriefing techniques with participants at the conclusion of the research.

Population and sample

The population is the entire group a researcher wants to draw conclusions about, such as all U.S. high school juniors. Studying everyone is almost never possible, so researchers study a sample, a smaller group drawn from the population. The sample is the people who actually took part, and a good one lets researchers say something true about the population.

A representative sample has the same mix of characteristics as the population (age, gender, region, income, and so on). The best way to get one is random sampling: every member of the population has an equal chance of being chosen, so chance, not the researcher's convenience, decides who is in. A convenience sample is made of people who are easy to reach, such as the students in a psychology class, shoppers at one mall, or volunteers who answer an online ad. Convenience samples are cheap but risky, because the people who are easy to reach may differ from the population.

When the way a sample is chosen leaves out or over-includes some kinds of people, the result is sampling bias. A phone survey that only calls landlines will miss people with only cell phones. An online survey about internet habits will only reach people online. Volunteers differ from non-volunteers as well, since the people who choose to answer a questionnaire about stress may be those who feel stress most.

Generalizability

Generalizability is the extent to which findings apply beyond the sample to the wider population and to other settings. To judge it, read the description of the participants and ask: how many were there, who were they (age, gender, culture, country), how were they recruited, and where was the study done? A large, randomly chosen, diverse sample supports broad generalization; a small group of volunteer undergraduates at one university supports only cautious generalization.

The exam also expects you to understand why appropriate representation of participants matters. Much of the early research in psychology was done on college students in Western countries, and many studies left out women, people of color, older adults, and people from other cultures. Findings from a narrow group can be wrong or misleading for everyone else, and using them to make decisions about education, health care, or policy can lead to unfair or ineffective results. Including diverse participants improves the accuracy of the conclusions and the fairness of how they are used.

A reminder from the last lesson: random sampling is about who gets into the study and affects generalizability; random assignment is about which condition each participant gets and affects whether the study can show cause and effect. A study can have either, both, or neither.

Measuring variables

Researchers need a way to turn a construct into data. The exam separates two kinds of measurement:

Quantitative measures produce numbers: test scores, reaction times in milliseconds, counts of behaviors, heart rate, or ratings on a Likert scale (for example, from 1 = strongly disagree to 5 = strongly agree).
Qualitative measures produce descriptions in words, such as answers to open-ended questions, a structured interview with set questions, or narrative notes from observations. They capture detail and context that a number can miss, but are harder to compare across people.

Whatever the measure, two properties matter. A measure is reliable if it gives consistent results when repeated, and valid if it measures what it is supposed to measure. A bathroom scale that reads 5 pounds heavier every time is reliable but not valid. (Lesson 2.9 develops these ideas for tests.) Self-reports are a common quantitative measure, and they are open to the self-report and social desirability biases from the last lesson, which is why researchers often add direct observation or physiological measures.

How a finding becomes knowledge: peer review and replication

One study is never the last word. Before a study is published in a scientific journal it goes through peer review: independent experts in the field read it and judge whether the methods and conclusions are sound, and they can ask for changes or reject it. Peer review is a filter, not a guarantee, and it does not replace the next step. Replication means that other researchers repeat the study, ideally with different participants and settings, and see whether they get the same result. If a finding replicates across samples, confidence in it grows; if it does not, researchers revise or abandon the claim. Meta-analyses combine many such studies. This ongoing process of review and repeating is how conclusions from both experimental and non-experimental research evolve, and it is why a single exciting result should be described as "suggests," not "proves."

Research ethics

Psychologists work with people and animals who can be harmed, so research follows ethical guidelines established by the American Psychological Association (APA) and by federal rules. You should be able to recognize each one in a scenario.

Institutional review board (IRB): a committee that reviews any proposed study with human participants before it begins, to make sure the risks are justified and participants are protected. For studies using non-human animals, an institutional animal care and use committee reviews the plan and requires humane treatment of the animals (housing, care, minimizing pain, and using as few animals as needed).
Informed consent: participants are told what the study involves, including any risks and the right to quit, and agree to take part voluntarily. For children and others who cannot legally consent, a parent or guardian gives permission and the participant gives informed assent, their own agreement to take part.
Protection from harm: researchers must minimize physical and psychological risk (stress, embarrassment, distress) and weigh it against the expected benefit of the knowledge.
Confidentiality and anonymity: confidentiality means the researchers know who the participants are but keep their data private and report results without identifying anyone. Anonymity means the researchers cannot connect the data to any person at all.
Deception: misleading participants about the true purpose is allowed only when the question cannot be studied any other way, the risk is minimal, and the participants are told the truth afterward. A confederate is someone who appears to be a participant but is really working with the researcher, a common way to carry out deception.
Debriefing: after the study, participants are told the real purpose, any deception is explained, their questions are answered, and they are checked for lingering distress.
Voluntary participation and the right to withdraw: participants can quit at any time without penalty.

Study spotlight · Milgram (1963)

Stanley Milgram recruited 40 men from New Haven, Connecticut, by newspaper ad and told them the study was about learning and memory. An experimenter in a lab coat told each man (the "teacher") to give a series of increasing shocks, on a generator labeled from 15 to 450 volts, to another man (a confederate, the "learner") every time the learner made a mistake. No shocks were actually delivered. As the learner protested, the experimenter used standard prods such as "The experiment requires that you continue." In the original study 26 of the 40 participants (65%) went all the way to 450 volts.

Reading the research: a laboratory study of obedience with an actual measure (the highest voltage given). It also became a landmark case in research ethics. Participants were deceived about the purpose and about the shocks, many showed obvious distress, and the right to quit was effectively undermined by the prods. The study helped drive the requirements for review boards, informed consent, minimizing distress, and debriefing that are in place today. (Unit 4 returns to what the study shows about obedience.)

Worked example · Evaluating sampling and generalizability

A team surveyed 1,200 teenagers about vaping. They posted the survey link on a popular social media site for two weeks. Participants were mostly girls (72%) and 88% were from large cities; 60% were 17 or 18 years old. The researchers concluded, "Most U.S. teens have never vaped." Evaluate the sample and the conclusion.

Step 1, identify the population: U.S. teenagers. The sample is the 1,200 respondents.

Step 2, how was it chosen? A link on a social media site, so people chose themselves: a convenience (volunteer) sample, not random.

Step 3, sampling bias: the sample over-represents girls, urban teens, and older teens, and it only includes teens who use that site, so it does not match the population.

Step 4, generalizability: the 1,200 respondents are a decent number, but they are not representative, so the conclusion about "most U.S. teens" is not well supported. It is better to describe only the results for the people surveyed. A stronger design randomly selects teens from school lists across regions and uses a measure that protects privacy to reduce social desirability.

Worked example · Applying the ethical guidelines

A researcher tells students they are in a study of "group problem solving." In fact, the study measures how participants react when a confederate on the team insults them. The researcher records the students' stress with a questionnaire and publishes the data with each student's name. The students were never told they could leave. Identify at least four ethical problems and a fix for each.

Step 1, deception: students were told the wrong purpose. Deception is allowed only if no other way exists and the participants are debriefed. Fix: justify the deception to the IRB and debrief afterward.

Step 2, informed consent and right to withdraw: students were not told about the insults or that they could leave. Fix: give a consent form that describes the risks in general terms and states they can stop at any time without penalty.

Step 3, protection from harm: insults cause stress. Fix: keep the stress mild, monitor participants, and offer support or counseling contacts.

Step 4, confidentiality: publishing names breaks confidentiality. Fix: report only group data or use code numbers.

Step 5, debriefing: none was described. Fix: explain the real purpose and the confederate's role, and check how each student feels.

Step 6, review: the plan needs IRB approval before it begins.

Exam focus

Random sampling helps a sample represent the population (generalizability); convenience samples risk sampling bias. Random assignment is a separate idea about creating equal groups.
To judge generalizability, cite specific evidence about participants: number, age, gender, culture, recruitment, setting.
Quantitative measures are numbers (Likert scales, scores); qualitative measures are words (structured interviews, open-ended answers).
Peer review and replication are how findings are checked and conclusions revised.
Ethics checklist: IRB (and animal care committee), informed consent or assent, protection from harm, confidentiality or anonymity, deception only when justified, debriefing, right to withdraw.

Practice

A researcher wants to know how stressed students at a large public high school are. She puts the name of every student in a hat and draws 200 names. Is this a random sample or a convenience sample? Which group is the population?
Show answer
It is a random sample, because every student had an equal chance of being picked. The population is all the students at that high school (and the 200 chosen students are the sample).
A psychologist studies memory using only 20 first-year students from her own university who took part for course credit. Give two reasons the results might not generalize.
Show answer
The sample is small (20), and it is a convenience sample of young, college-educated students from one school, so it may not represent older adults, people who did not attend college, or other cultures.
Classify each as quantitative or qualitative: (a) a 1 to 5 satisfaction rating, (b) a participant's spoken description of a recent stressful day, (c) reaction time in milliseconds.
Show answer
(a) quantitative (a Likert scale); (b) qualitative (a description in words); (c) quantitative (a number).
A study of a new reading program for 8-year-olds requires a signed form from each child's parent, and each child also says "yes" when asked if they want to play the reading game. Name the two kinds of agreement.
Show answer
The parent's signed form is informed consent (permission), and the child's own agreement is informed assent.
Researchers collected survey answers using code numbers, and the key linking codes to names is stored in a locked cabinet that only two researchers can open. Is this anonymity or confidentiality? Explain.
Show answer
Confidentiality. The researchers can identify who gave which answers, but they keep that information private. In true anonymity no one could link answers to a person.
Explain why replication matters even when a study was published after peer review.
Show answer
Peer review checks the methods and logic, but it cannot show whether the result will hold with new participants or settings. Replication tests that: if other researchers get the same result, confidence grows, and if they do not, the claim must be revised. One study can be affected by chance, a biased sample, or a hidden confound.