Randomized Controlled Trials (RCTs) are renowned as the gold standard in clinical trials, predominantly used to gauge the efficacy of new treatments. However, the applicability of RCTs extends far beyond merely clinical trials. In my line of work, we employ randomized experiments to evaluate the performance of varying credit underwriting and account management policies. While the stakes may not be as life-critical as in a medical context, I’ve always found myself intrigued by the theoretical underpinnings of RCTs. This article, though, will primarily focus on providing an introduction to this concept.
Often, treatments can have unintended adverse effects. Until the 20th century, bloodletting was still practiced by doctors as a means to cure disease. Similarly, in the bestseller “Baby and Child Care” by Benjamin Spock, the author suggested that babies should sleep on their stomachs to prevent the risk of choking. Although the reasoning seemed logically sound, the advice was tragically linked to the death of thousands of infants. In my own work, we frequently implement account freezes for users with low credit scores. Nevertheless, this action can inadvertently increase the default risk, as it may upset the users.
Humans are complex creatures, often ruled by emotions and biases. The placebo effect is one such bias. Both patients and healthcare professionals naturally hope for treatments to work. Likewise, we expect our policies to positively impact the business by reducing risk and improving unit economics. These optimistic expectations can cloud our judgment and distort our understanding of the actual causes.
Determining a cause-and-effect relationship between treatment and outcome necessitates a comparison of results, with and without the treatment. However, the problem lies in our inability to observe a hypothetical world where the treatment was not received, or reverse time. Adding to the complexity is the uncertainty of outcomes. If a treatment were to inevitably lead to a specific result, a simple observation would suffice. However, a treatment need not be 100% effective to be considered better than no treatment at all.
By incorporating randomization, we ensure comparability across all other factors between the treatment and control groups. Consequently, any difference observed can be attributed solely to the treatments. Let X represent the outcome random variable, with E(X) = u. The mean of the treatment group E(X_1) equates to the mean of the control group E(X_2) = u, resulting in a fair estimate of the treatment’s effect. If we could identify all the factors that may affect the outcome, we could control for them without utilizing RCTs. But, the question remains, how can we be sure that we have considered all potential factors?
In my upcoming article, I plan to delve deeper into the potential biases that may arise in RCTs. Stay tuned!
Matthews, J. N. S. Introduction to Randomized Controlled Clinical Trials. 2nd ed. Boca Raton, Fla: CRC Press, 2006.
Mike Clarke, Patricia Atkinson, Douglas Badenoch, Iain Chalmers, Paul Glasziou, Scott Podolsky, Ulrich Tröhler. ‘THe James Lind Library’s Introduction to Fair Tests of Treatments’, n.d.