1.4 Experimental Design and Ethics

Does aspirin reduce the risk of heart attacks? Is one brand of fertilizer more effective at growing roses than another? Is fatigue as dangerous to a driver as the influence of alcohol? Questions like these are answered using randomized experiments. Proper study design ensures the production of reliable, accurate data.

The purpose of an experiment is to investigate the relationship between two variables. When one variable causes a change in another, we call the first variable the explanatory variable. The affected variable is called the response variable. In a randomized experiment, the researcher manipulates values of the explanatory variable and measures the resulting changes in the response variable. The different values of the explanatory variable are called treatments. An experimental unit is a single object or individual to be measured.

Video: “Observational Studies and Experiments” by ProfessorMcComb [3:06] is licensed under the Standard YouTube License.Transcript and closed captions available on YouTube.

Suppose we want to investigate the effectiveness of vitamin E in preventing disease. We recruit a group of subjects and ask them if they regularly take vitamin E. We notice that the subjects who take vitamin E exhibit better health on average than those who do not. Does this prove that vitamin E is effective in disease prevention? No, it does not. There are many differences between the two groups compared, in addition to vitamin E consumption. People who take vitamin E often take other steps to improve their health, such as exercise, diet, other vitamin supplements, or choosing not to smoke. Any one of these factors could be influencing a person’s health. As described, this study does not prove that vitamin E is the key to disease prevention.

Additional variables that can cloud a study are called lurking variables. In order to prove that the explanatory variable is the cause of a change in the response variable, it is necessary to isolate the explanatory variable. The researcher must design their experiment in such a way that there is only one difference between the groups being compared: the planned treatments. This is accomplished by the random assignment of experimental units to treatment groups. When subjects are assigned treatments randomly, all of the potential lurking variables are spread equally among the groups. At this point, the only difference between groups is the one imposed by the researcher. Therefore, different outcomes measured in the response variable must be a direct result of the different treatments. In this way, an experiment can prove a cause-and-effect connection between the explanatory and response variables.

The power of suggestion can have an important influence on the outcome of an experiment. Studies have shown that the expectation of the study participant can be as important as the actual medication. When participation in a study prompts a physical response from a participant, it is difficult to isolate the effects of the explanatory variable. To counter the power of suggestion, researchers set aside one treatment group as a control group. This group is given a placebo treatment—a treatment that cannot influence the response variable. The control group helps researchers balance the effects of being in an experiment with the effects of the active treatments. Of course, if someone is participating in a study and they know that they are receiving a pill which contains no actual medication, then the power of suggestion is no longer a factor. Blinding in a randomized experiment preserves the power of suggestion. When a person involved in a research study is blinded, they do not know who is receiving the active treatment(s) and who is receiving the placebo treatment. A double-blind experiment is one in which both the subjects and the researchers involved with the subjects are blinded.

Ethics

The widespread misuse and misrepresentation of statistical information often gives the field a bad name. Some say that “numbers don’t lie,” but the people who use numbers to support their claims often do.

A recent investigation of famous social psychologist Diederik Stapel has led to the retraction of his articles from some of the world’s top journals, including the Journal of Experimental Social Psychology, Social Psychology, Basic and Applied Social Psychology, British Journal of Social Psychology, and the magazine Science. Diederik Stapel is a former professor at Tilburg University in the Netherlands. Recently, an extensive investigation involving three universities where Stapel worked concluded that the psychologist is guilty of fraud on a colossal scale. Falsified data taints over 55 papers he authored and 10 Ph.D. dissertations that he supervised.

The committee investigating Stapel concluded that he is guilty of several practices, including:

• creating datasets, which largely confirmed the prior expectations;
• altering data in existing datasets;
• changing measuring instruments without reporting the change and
• misrepresenting the number of experimental subjects.

Clearly, it is never acceptable to falsify data the way this researcher did. Sometimes, however, violations of ethics are not as easy to spot.

Researchers have a responsibility to verify that proper methods are being followed. Many of Stapel’s co-authors should have spotted irregularities in his data. Unfortunately, they did not know very much about statistical analysis, and they simply trusted that he was collecting and reporting data properly.

Many types of statistical fraud are difficult to spot. Some researchers simply stop collecting data once they have just enough to prove what they had hoped to prove. They do not want to take the chance that a more extensive study would complicate their lives by producing data contradicting their hypothesis.

Professional organizations, like the American Statistical Association, clearly define expectations for researchers. There are even laws in the federal code about the use of research data.

When a statistical study uses human participants, as in medical studies, both ethics and the law dictate that researchers should be mindful of the safety of their research subjects. Most countries have federal regulations that protect participants in research studies. When a university or other research institution engages in research, it must ensure the safety of all human subjects. For this reason, research institutions establish oversight committees, commonly known as Institutional Review Boards (IRB). All planned studies must be approved in advance by the IRB. Key protections that are mandated by law include the following:

• Risks to participants must be minimized and reasonable with respect to projected benefits.
• Participants must give informed consent. This means that the risks of participation must be clearly explained to the subjects of the study. Subjects must consent in writing, and researchers are required to keep documentation of their consent.
• Data collected from individuals must be guarded carefully to protect their privacy.

These ideas may seem fundamental, but they can be very difficult to verify in practice. Is removing a participant’s name from the data record sufficient to protect privacy? Perhaps the person’s identity could be discovered from the data that remains. What happens if the study does not proceed as planned and risks arise that were not anticipated? When is informed consent really necessary? Suppose your doctor wants a blood sample to check your cholesterol level. Once the sample has been tested, you expect the lab to dispose of the remaining blood. At that point, the blood becomes biological waste. Does a researcher have the right to take it for use in a study?

It is important that students of statistics take time to consider the ethical questions that arise in statistical studies. How prevalent is fraud in statistical studies? You might be surprised—and disappointed. There is a website dedicated to cataloguing retractions of study articles that have been proven fraudulent. A quick glance will show that the misuse of statistics is a bigger problem than most people realize.

Vigilance against fraud requires knowledge. Learning the basic theory of statistics empowers users and observers to analyze statistical studies critically.

Exercises

1. Discuss potential violations of the rule requiring informed consent.
   1. Inmates in a correctional facility are offered good behaviour credit in return for participation in a study.
   2. A research study is designed to investigate a new children’s allergy medication.
   3. Participants in a study are told that the new medication being tested is highly promising, but they are not told that only a small portion of participants will receive the new medication. Others will receive placebo treatments and traditional treatments.
   Click to see Answer

   1. Inmates may not feel comfortable refusing participation or may feel obligated to take advantage of the promised benefits. They may not feel truly free to refuse participation.
   2. Parents can provide consent on behalf of their children, but children are not competent to provide consent for themselves.
   3. All risks and benefits must be clearly outlined. Study participants must be informed of relevant aspects of the study in order to give appropriate consent.

    

2. How does sleep deprivation affect your ability to drive? A recent study measured the effects on [latex]19[/latex] professional drivers. Each driver participated in two experimental sessions: one after normal sleep and one after [latex]27[/latex] hours of total sleep deprivation. The treatments were assigned in random order. In each session, performance was measured on a variety of tasks, including a driving simulation. Use key terms from this chapter to describe the design of this experiment.
   
   Click to see Answer

   • Explanatory variable: the amount of sleep.
   • Response variable: performance measured in assigned tasks.
   • Treatments: normal sleep and 27 hours of total sleep deprivation.
   • Experimental units: 19 professional drivers.
   • Lurking variables: none – all drivers participated in both treatments.
   • Random assignment: treatments were assigned in random order; this eliminated the effect of any “learning” that may take place during the first experimental session.
   • Control/Placebo: completing the experimental session under normal sleep conditions.
   • Blinding: researchers evaluating subjects’ performance must not know which treatment is being applied at the time89. You cannot assume that the numbers of complaints reflect the quality of the airlines. The airlines shown with the greatest number of complaints are the ones with the most passengers. You must consider the appropriateness of methods for presenting data; in this case, displaying totals is misleading.

    

3. An advertisement for Acme Investments displays the two graphs in the figure below to show the value of Acme’s product in comparison with the Other Guy’s product. Describe the potentially misleading visual effect of these comparison graphs. How can this be corrected?
   

   
   

   

   Click to see Answer

   There is no scale on either graph, making it impossible to tell which one produced the best return on investments. The results are shown in separate graphs, making it difficult to compare the results directly. To fix this, plot both graphs on the same set of axes and include a scale on the axes.

    

4. The graph in the figure below shows the number of complaints for six different airlines as reported to the US Department of Transportation in February 2013. Alaska, Pinnacle, and Airtran Airlines have far fewer complaints reported than American, Delta, and United. Can we conclude that American, Delta, and United are the worst airline carriers since they have the most complaints?
   

   
   

   Click to see Answer

   Because American, United and Delta are the largest carries, with the largest number of total passengers, and so will have the largest number of complaints. A more accurate graph would be to show the proportion of complaints out of the total number of passengers.

    

“1.5 Experimental Design and Ethics” and “1.6 Exercises” from Introduction to Statistics by Valerie Watts is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted.