Section 5.3: True Experiments | Research


Fundamentals of Social Research

Adam J. McKee, Ph.D.


DRAFT - Do Not Distribute

This content is released as a draft version for comment by the scholarly community.  Please do not distribute.


The big idea behind true experimental designs is that they maximize randomization.  True experimental designs control selection of participants, the assignment of participants to groups, and the assignment of treatments to groups—all through randomization.  In addition to this control for extraneous variables through randomization, the true experiment also requires the use of a control group. There are several variations on this theme, but these elements are common to them all.

Perhaps the most common true experimental design is the pretest posttest control group design.

Remember that the assumption of experimental designs is that each of the groups (control and treatment) are essentially equal at the beginning of the experiment because of the randomization involved in the design.  Thus, any differences between the groups observed at the end of the experiment are likely due to the experimental treatment.

The design depicted here is the simplest form.  A researcher can easily add more groups to the design in such a way that several treatments can be compared in a single experiment.  For instance, if you wanted to evaluate two different treatments, you would simply randomly select 3 groups so that you obtain a control group and two treatment groups.  Everything else is the same. The number of treatment groups really doesn’t make a difference; you are okay so long as you have a control group to serve as a baseline for comparison.  (Of course, there are limitations placed on the researcher by the statistical methods used to analyze experimental data, but more on that later.)

In some instances, nothing is done to the control group.  In other cases, the control group will receive the status quo treatment.  For example, let’s say that I have pioneered the XYZ method of teaching research methods.  I hypothesize that the XYZ method will improve students’ final grades in a research course.  To test my hypotheses, I divide my research methods class into two sections (randomly of course!) with the intent of using one section as my control group and the other as my experimental group.  Obviously, the treatment group will receive instruction via my new XYZ method. But, what do I do with the control group? Can I do nothing with them?  

That doesn’t seem like a very good baseline.  The real question is not whether students instructed by the new method will do better than those who were free to play video games or go fishing, but rather do they do better than students instructed under the competing (old) method.  What I need to do, then, is to teach them the way I’ve always taught my research methods class. The trick to deciding exactly what to do with your control group is to clearly establish exactly what you want to compare.

In other circumstances, especially in medicine, a placebo is used.  A placebo is something that participants cannot tell from the actual treatment—in medicine “sugar pills” are often used.

Another common experimental design is the posttest-only control group design.  It looks like this:

Note that all we have done here is to remove the pretest for both groups.  This may seem like a big sacrifice, but not so if randomization has done its job.  Because of randomization, both groups are very likely to be equal at the beginning of the experiment.  Still, the big disadvantage of this design is that if randomization has not done its job, the groups will not be equivalent at the start.  Another potential problem (depending on your specific design) is that you cannot use the pretest to assign participants to groups. This is often desirable when you want to form a group that represents high scores on a measure, and another group that represents low scores on that same measure.

Perhaps the Mother of All Experimental Designs is the Solomon Four-group Design.  This design aims to eliminate many of the flaws associated with the designs we’ve discussed previously.  In this design, you have four groups. You might be concerned that the very act of taking a pretest might have in impact on the posttest scores—an extraneous variable has entered into your design!  The Solomon Four-group design allows us to test for this. The design looks like this:

The reason that many researchers do not use this particular design (given its obvious superiority) is that it is expensive in terms of time and resources.  It takes many participants—more that researchers can effectively muster in many circumstances.

Threats to Validity

Why all these different experimental designs?  Obviously, a researcher must pick a design that can answer the research question.  In addition, the researcher needs to be confident that the results of an experiment will be valid.  Traditionally, social scientists have viewed the idea of validity as being of two types: Internal and external.  

Internal validity refers to the idea that changes in the dependent variable were caused by the experimental manipulation.  Simply put, internal validity is talking about control. If what you see (as measured by the DV) was caused by what you did (the IV), then the experiment was valid.  If there are several other plausible explanations for the observed outcome of the experiment, then the design does not have internal validity.

External validity is talking about the extent to which the results of a study (which are drawn from a sample) can be generalized to the population from which the sample was obtained.  This important element is often overlooked by researchers. But, considering its importance to the scientific endeavor, it is logical that we give it due consideration.

Threats to Internal Validity

Good scientists will try to reduce threats to both internal and external validity.  With internal validity, the researcher must eliminate everything other than the independent variable as a possible cause of changes in the dependent variable.  What follows is a brief explanation of some of the most common threats to internal validity.

History.  Many experiments are designed to take place over the course of a long time.  During the course of long experiments, things outside of the experiment can cause changes in the dependent variable.  This period of time when extraneous variables can do their dirty work is referred to as history.

Maturation.  Maturation is the effect of biological or psychological forces that act on a person over time.  To combat this “maturation effect,” researchers must be aware of things that change over the course of human development.  For example, it is commonly held that people “age out of crime,” meaning that after people reach a certain age they are far less likely to commit crimes.  If a researcher is interested in whether a particular treatment program reduces recidivism, then the “aging out” effect must be taken into consideration.

Selection.  Selection bias occurs when the members of experimental and control groups are not selected randomly and systematic biases exist between groups.  For example, it would be foolhardy to conduct a study of volunteerism among college students using a sample composed of volunteers.

Testing.  As we have already discussed, a pretest is a common feature of experimental designs.  When the pretest affects performance on later measures, then the pretest (testing) can be a threat to validity.  Let’s say that I’ve developed a new method of teaching students how to solve the logic problems that are found on law school admissions tests.  

If I take a group of students and give them a pretest, give them a treatment (teach them my new method) and then give them a posttest, what may I safely conclude if the posttest scores are, on average, higher than on the pretest?  The idea of doing the experiment was to demonstrate that my new method improved scores, which is a possibility. It is also possible, however, that the experience of taking the pretest introduced the students to a type of problem that they had never encountered before.  They could have learned a lot about solving logic problems simply by being asked to solve logic problems on the pretest.

Instrumentation.  Error can creep in when different participants are measured differently.  This is a problem when there is a subjective component to the measurement process, such as when a research design includes assessing the quality of an essay.  An example of how instrumentation may influence final decisions is Olympic judging. Most of the judges will evaluate an athlete’s performance similarly, but not exactly the same.  Research designs that include rating things are prone to this kind of subjective error.

Regression.  Often a researcher wants to conduct studies that use a sort of pretest to place people among the worst or best; this classification can easily be turned into experimental groups.  Regression (as a type of threat to validity, NOT a statistical method) says that people with extreme scores (either very high or very low) on a particular test will tend to score closer to the average if they are tested again.  That is, they “regress” toward the mean.

Mortality.  When people drop out of an experiment over time, it is known as mortality or attrition.  Of course, in our everyday language, mortality means a death has occurred.  Much of the language describing experiments comes to us from behavioral psychologists and others doing animal research.  If an experiment took place over a lengthy span of time, it was likely that several rats or pigeons would die before the experiment was complete.  

We now use the term mortality to refer to the threat to validity that arises when participants fail to complete an experiment, regardless of why.  The threat to validity arises when, for whatever reason, the group that remains in the experiment is systematically different than the group that began the experiment.  The features of the quitters may have an important impact on the observed level of the dependent variable at the end of the experiment.

Threats to External Validity

The previously discussed threats to validity concerned threats that prevent the researcher from confidently making causal statements.  In other words, how can we assert that X caused Y when we cannot exclude Z as a possible cause?  Now we will consider threats to validity that prevent the researcher from confidently making inferring things about a population from a sample.

Multiple Treatment Interference.  This threat to validity arises when the researcher (or an unknown person) introduces another variable into the experimental environment.

The Hawthorne Effect.  People tend to act differently when they know they are being observed.  Thus, the observation itself can be a treatment that can alter the level of the independent variable.

Experimenter effects.

Controlling Extraneous Variables

Extraneous variables are things that can decrease the internal validity of a study.  That is, they are variables that will—if not controlled—confound your results.  Confounding means that more than one thing is causing a change in some outcome, and you cannot separate the effects of those factors.

If you know that an extraneous variable is likely to have a big impact on your dependent variable, then you may want to consider matching your subjects.  Matching of subjects means that individuals in the experimental group are paired with someone in the control group with a very similar level of the extraneous variable that you want to control for.  Let’s say that a researcher is interested in improving the physical strength of police officers. She has developed a program that she thinks will increase strength faster than traditional physical training.

What other variables besides the training method will affect how fast officers put on lean muscle?  Her review of the literature indicates that females tend to put on muscle mass slower than males. For this reason, she decides to match her subjects on the variable gender.  That is, she will compare changes in female muscle mass with changes in other females, and male officers will be compared with other male officers.

In short, matching is a good way to control for variables that you are pretty sure will have an impact on your dependent variable.   The downside is that it can be expensive, time-consuming, and you may not be able to find a match for all subjects. Also, by fixing your groups so that subjects match, you have destroyed the benefits of random assignment.  In other words, you have established good control on one variable at the risk of bias from another unknown variable.

Previously, we have said that randomization is an effective way to control for extraneous variables.  The idea is that the randomly assigned groups will be equivalent on every variable. This gives us the advantage of not having to worry whether a particular variable is confounding our results.  Randomization is a form of control that is achieved through experimental design—the control is part of the way that we collect our data. Another method is to use statistical controls.

Statistical controls are methods of analyzing data that mathematically subtract the influence of an extraneous variable from the dependent variable in such a way that only the influence of the variable of interest remains.  A common method is the Analysis of Covariance (ANCOVA) technique, which equalizes any initial differences that might exist. Regression analysis is the more common method of accomplishing this.

Modification History

File Created:  07/25/2018

Last Modified:  07/25/2018

[ Back | Content | Next]


This work is licensed under an Open Educational Resource-Quality Master Source (OER-QMS) License.

Open Education Resource--Quality Master Source License


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.