By Randomized Controlled Trials (RCTs) are renowned as the gold standard in clinical trials, predominantly used to gauge the efficacy of new treatments. However, the applicability of RCTs extends far beyond merely clinical trials. In my line of work, we employ randomized experiments to evaluate the performance of varying credit underwriting and account management policies. While …
Revolutionizing Data Analytics with Text-to-SQL
A significant portion of a Data Analyst or Business Intelligence Analyst’s time is spent translating business questions into SQL queries. These analysts often serve as an interface between humans asking questions and the computers processing the data. In business settings, Software Engineers essentially act as translators, converting business requirements into executable code. With advancements in …
Markov Decision Process in Account Management of Consumer Lending
Recently, one of the main questions in my mind is how to adjust the user’s credit limit given a predicted risk. I consider myself a practitioner in the industry with limited experience. Currently, the account management process relies more on expert judgement. We may decrease the limit or freeze user accounts if they are not …
Creating A News Summarizer Using ChatGPT (Part 1)
Warren Buffet once said to Bill Gates after he said the computer is going to change everything: “I’ll stick to chewing gums, you stick to computers.”. We have to admit that we don’t understand a lot of things. It is wise to focus on our competencies. However, it should not stop us from exploring new …
Answering the What-if Questions
Is it possible that we can estimate the potential outcome — The outcome if we do things differently? We can observe what had happened., but not the counterfactual world. The formula can be read as the value that Y would have taken for individual u had X been assigned the value x. The answer to …
The Front and Back Door Adjustment Formula for Confounders
In fact, we have already seen the back-door adjustment when we discuss Simpson’s Paradox. We have calculated the weighted average of the rate of heart attacks with the drug in men and that in women to get the overall effect of the drug in the general population adjusted for the confounder, age. We can write …
Simpson’s Paradox
Last time, we have talked about Monty Hall Problem and Berkson’s Paradox. This time, let’s discuss another paradox called Simpson’s Paradox. Let’s consider the following table: This is not a randomized experiment. The data is observed. By looking at the data, you can observe the following: Female in Control Group has 5% chance of getting …
Why do the attractive ones tend to be jerks?
In the previous post, I have talked about the Monty Hall Problem. Let’s go through a similar problem called Berkson’s Paradox/ Admission Rate Bias. In a medical study, a guy studied two groups of diseases: respiratory and bone. In the general population, about 7.5% of the people have a bone disease but it does not …
Should You Switch Your Choice of Doors? The Month Hall Problem
I want to illustrate this problem using a causal diagram. Here is the rule of the game: Suppose you’re on a game show, and you’re given the choice of three doors. Behind one door is a car, behind the others, goats. You pick a door, say #1, and the host, who knows what’s behind the …
Cornfield’s inequality Causation v.s. Correlation
In the last post, I have shared about a debate: Today, I would like to show the proof on Corenfield’s Inequality, an inequality that settled the debate. I have read the original proof on the paper. Although the author said it is obvious, I don’t think so (always find reading maths discouraging LOL). I spent …
