When a government introduces a national tax reform or a universal health programme, it changes life for everyone at once. This creates a significant challenge for researchers: how can we tell if the policy actually worked? Usually, we compare the group of people who received a ‘treatment’ with a ‘control’ group of people who did not. But when a policy is universal, there is no one left to compare against.
We have developed a new tool called the forecasted average treatment effect (FAT) estimator to solve this problem (Botosaru et al, 2026). Instead of looking for a separate group of people, we look backwards at an individual’s (or a state’s) own history to predict what would have happened if the policy had never existed. This allows us to measure success even when every single person is affected by the change.
Using counterfactuals to evaluate policy effectiveness
We find that we can reliably measure the success of a policy by creating a ‘digital twin’ of the past. By using a short window of data from before the policy, we can obtain a predicted path – what we call the counterfactual or the path not taken. When we compare this prediction to the actual outcomes after the policy has been implemented, the difference tells us the true effect of the policy.
A key discovery in our research is that these predictions do not have to be perfect for every single person to be useful. In most forecasting, the goal is to be as accurate as possible. But for evaluating a national policy, we show that it is more important to be ‘unbiased’.
This means that while our predictions for one specific person might be slightly off, our predictions for the whole group are correct on average. We find that we do not need complex models to achieve this: using simple polynomial trends regressions is enough to produce reliable results.
We put our method to the test by revisiting several major studies. For example, we look at how no-fault divorce laws affected suicide rates among women across 36 US states. Even without using a separate control group, our method reaches the same conclusion as previous research: the laws did not lead to a significant change in suicide rates.
We also apply the tool to the legalisation of medical cannabis in 34 US states. While some hoped that these laws would reduce opioid overdose deaths, we find no significant evidence that this has happened.
A transparent tool for policy-makers and researchers
This tool is useful because many of our most important decisions are national in scope. Environmental regulations, major tax shifts and national health interventions affect the whole country, leaving no ‘untreated’ group for researchers to study.
In the past, experts often had to rely on models that made many hidden assumptions. Our approach is more transparent and easier for policy-makers to use and understand.
Our research also helps to solve problems when we have limited history of data. Because we only need a small amount of data from the period leading up to a policy change, we can evaluate the impact of new laws much sooner.
In addition, our method recognises that a policy affects people differently. A new labour law might help a worker in one city while having no effect on a worker somewhere else. Because we build a unique prediction for every person or region based on their specific history, we account for these individual differences naturally.
The method also serves as a vital safety check for other research. If a study that uses a control group finds the same result as our study – which uses no control group – we can be much more confident that the findings are accurate. It provides a way to double-check that an observed shift is actually caused by a policy and is not just something that would have happened anyway.
Tracking the long-term success of policies
We believe that this estimator should become a standard part of the toolkit for anyone evaluating the impact of changes or policies. Because it only requires a group’s own history, it allows for long-term monitoring. Governments can continue to track the success of a policy years after it was rolled out to the entire population, ensuring that it is still delivering the promised benefits to the public.
There is still work to be done in handling ‘common shocks’. These are sudden, unexpected events like a global pandemic or a sudden market crash, which hit everyone at once and can throw off our predictions.
While our current method can account for steady trends, it is harder to predict these sudden jumps. Future research could focus on using ‘auxiliary data’ – information from other variables that are not affected by the policy – to help to filter out these unexpected shocks.
Ultimately, we want to ensure that no policy is considered too big to be measured. By using our own history as a guide, we can finally get a clear and honest picture of whether the national paths that we choose are actually leading us to a better place.
