A causal question

We begin this lesson with a question: Does a medical procedure A lead to better success rates (in terms of eliminating kidney stones among patients) compared to a medical procedure B?

We have an observational dataset of individuals who have taken one of only two available procedures for treating kidney stones. Procedure A is an open surgical procedure and procedure B involves only a small puncture.

So, let us ask this: Is this a causal question? And if it is, is it a forward causal question or a reverse causal question?

We can specify the following from the information above:

• The units are patients or individuals and are well defined. ✔️
• Even though we have two types of procedures in this study, we can define the “treatment group” as those who received procedure A and the “control group” as those who didn’t receive procedure A (therefore, received procedure B). If that’s our assumption, then the treatment variable is going with procedure A. Who gets treatment and who doesn’t can be manipulated. We can imagine giving the treatment to some and withholding it from some other. ✔️
• The outcome can be defined as success in eliminating kidney stones and can be quantified (whether kidney stones were removed or not). ✔️

Therefore, all the assumptions for having a well-defined causal question are met. This question is also a forward causal question in the sense that we are interested in the effect of a specific factor X (kidney treatment) on Y (success) rather than being interested in understanding why Y happens.

This is an important question for doctors, insurance companies, and individuals suffering from kidney stones, so it seems like a very worthwhile question to investigate. Now let’s take a closer look at the data.

The table below shows both the total number of treatments and the success rate for each treatment. Our population consists of 700 individuals with half (350 subjects) receiving procedure A and half receiving procedure B. The success rate for procedure A is 78% versus 83% for procedure B.

Looking at these numbers, can you easily determine whether procedure A is more or less successful than procedure B?

You may say, yes! It’s clear that A is inferior to B (B is more successful) because it has a higher success rate, but hang on. There’s a small detail you’ve overlooked.

Remember the data come from an observational study, therefore, they are not randomized. In an observational study, who receives the treatment and who doesn’t is determined by factors outside of the control of the researcher. For instance, patients may choose to receive the treatment or not. Or doctors might have prescribed procedure A to certain patients and procedure B to others, and the difference between patients in each group could significantly alter the inference we make.

Let’s assume the size of a patient’s kidney stone is a factor when a doctor is deciding whether to prescribe procedure A (open-surgery) versus procedure B (the less invasive procedure) and that the size of the stone is also a factor in determining the success of treatment.

If the size of a patient’s kidney stone affects which treatment the patient will receive and also affects the likelihood of success, then we need to update our information to take these facts into account.

If our data includes information about kidney stone size, we can include the information in our summary table like so:

Now, how has your conclusion about the relative success of procedure A to procedure B changed? Is your new answer straightforward? Or are you now more confused than ever? The third row in the table shows the aggregated information that we started with. Across all patients, procedure B seems to be the better treatment, but when you break the data down by kidney stone size procedure A is better, and not just for one group, but for BOTH patients with small kidney stones (93% versus 87%) and large kidney stones (73% versus 69%). How is this possible?

If you’re confused, you’re not alone. In fact, you have a causal way of thinking 😀 And by the way, this example comes from an actual scientific paper by Charig et al., so the numbers are real.

The kidney stone data is an example of a paradox called Simpson's paradox and there are many other examples of Simpson’s paradox across many different fields (medicine, sports, and economics). But let’s leave the discussion about the paradox for another time. Right now, we’re mainly interested in why this simple causal question is more complicated than what most people think.

Without getting into the details at the moment, one message is clear: statistics is insufficient for understanding the relationship between different kidney treatments and the success of the procedure and the paradox arises from not paying attention to the causal structure of these research problems and many others we’ll see in future.

With causal understanding, we’ll see that the paradox can be simply explained by what we call confounders.

To get you pumped up about the things we’re gonna learn in this course, think about the following paper published in the Proceedings of the National Academy of Sciences (PNAS), a major scientific organization in the US. The paper’s headline reads “Female hurricanes are deadlier than male hurricanes.” Start by thinking about what could have gone wrong with the analysis in this paper and go ahead and read the abstract of the paper if you want.