Randomized experiments

As a reminder from the last lesson, remember that the relationship between the naive causal effect, the ATT, and selection bias was:

To simplify, we can write:

Now, get ready for some more magic. 🧙🏾‍♂️ Your friend with the naive causal effect has actually helped us find a way of estimating causal effects that circumvents the multiverse!

If we know what the selection bias is, we can calculate the naive causal effect (by comparing observed outcomes like your friend did) and then subtract from it the selection bias. And boom! We have the treatment effect (ATT).

Practice

In a causal study, if the selection bias term is positive, which statement do you think is correct?

ATT is smaller than the naive causal effect

ATT is larger than the naive causal effect

ATT and the naive causal effect are the same

Randomized experiments

We’ve now discovered two forms of magic: the magic of the multiverse (via the PCM model) and the magic of unearthing the ATT by subtracting selection bias from a naive causal estimate. But what if we don’t know how big the selection bias is? Have we reached a dead end?

Luckily, there’s more magic up our causal inference sleeves - a way of eliminating selection bias altogether by running randomized experiments.

If you remember from the previous lesson, selection bias exists because individuals choose (either intentionally or unintentionally) whether to receive the treatment or not. This choice is often correlated with individual characteristics and the outcome variables. As a result, our causal estimate will be biased.

In a randomized experiment, subjects are assigned randomly to treatment and control groups. Subjects don’t get to choose which group to be in. Because of the randomization, subjects are adequately shuffled between treatment and control groups. If groups are large enough, then we can say the control group as a whole is similar to the treatment group as a whole. Therefore we can compare the two groups outcomes and take it as the average causal effect.

Let’s return to the example of the cancer treatment from the previous lesson. Rather then having patients choose whether to undergo the surgery, imagine if subjects were randomly assigned to the treatment and control groups. We wouldn’t expect there to be any pre-treatment differences between the average outcomes of the treatment group and the average outcomes of the control group. For instance, both groups will have roughly 30 percent high-risk patients. Both have 45 percent male patients. The two groups are likely to have similar age distributions. Overall, the treatment group would be equivalent (on average) to the control group.

Keep in mind that for this to work, you have to have a large number of people in each group. In an extreme case, imagine your study only consisted of two people and you randomly assigned one to the treatment group and one to the control group. The randomization wouldn’t solve your problem. You would still be left with a comparison similar to that of Aliyah versus Connor in the previous lessons. The magic of randomization works because the two groups are balanced on average not because you have eliminated all differences between individuals in your study.

If subjects have been randomly assigned to treatment and control groups, the term below that reflects selection bias should equal zero.

If the selection bias is zero, we can find an estimate of ATT by computing the difference in means between the treated (went with the surgery) and control (didn’t go with the surgery) samples. In other words:

So, if we could run a randomized experiment for any causal question we have in mind, causal inference would not be called a science. Causal inference would simply boil down to calculating the difference between observed outcomes using one line of code and with the most basic software.

The trouble with randomized experiments

The trouble with randomized experiments is that they aren’t feasible most of the time:

• In many cases, they’re simply unethical. For instance, how would you design a randomized experiment to understand the health effects of smoking?
• In many cases, it’s costly. For instance, imagine you wanted to run a randomized experiment to find the causal effect of a universal basic income (say giving people an unconditional cash transfer of $1,000 a month) on employment decisions.
• In some cases, we’re just not patient enough. For instance, we would need to wait yeeaarrs to find the effect of breastfeeding babies on the babies’ future earnings.

As wonderful as randomized experiments, because they are hard to conduct, we typically cannot rely on them as a way of measuring causal effects. Instead, we are typically left studying observational data that does not come from a carefully randomized experiment. As we saw before, in observational studies, assignment to treatment and control groups is not random and is determined by external factors. We’re stuck dealing with selection bias!

Randomized experiments vs. observational studies

One useful approach in understanding how causal studies using observational data are different from causal studies using randomized experiments is to think of the eligibility criteria.

Given that your data is observational and as such subjects weren’t assigned to treatment at random, think thoroughly about the factors that determine who received the treatment and who didn’t. This set of external factors is called the eligibility criteria. Eligibility criteria determines who qualified for the treatment and who didn’t.

In a randomized experiment, the same eligibility criteria apply to those both in the treatment group and the control group. The eligibility criteria could be a computer randomization algorithm or simply flipping a coin. However, in our surgical example, the eligibility criteria could be being a high-risk patient or being old.

The good news is that working with observational data doesn’t mean we’re doomed and we can never estimate causal effects. Under certain assumptions, we can estimate causal effects using observational data.