Control & Experimental Group Selection Methodology using STDEV and T-Test?

Question

I would like to know if my methodology was 'correct':

I am trying to conduct an experiment on my stores. I would like to find out the effect of a marketing campaign on the number of transactions.

Only about 20% of the stores are participating in the marketing campaign.

The original methodology was to use the entire 20% as the experimental group and the remaining 80% as the control group. Unfortunately, these two groups are incomparable in terms of number of transactions. when plotted as box and whisker plots next to each other, their distributions are incomparable (mean, median, quartlies, min, max, etc).

So what I did was filter out the 'outlier stores' at each end until the box and whisker plots for each group were practically identical. I then ran a t-test on the filtered groups, we failed to rej the null (meaning that these groups are statistically the same prior to the promotion).

Now that we have 2 comparable groups for time -1, we run the promotion for a month. after promotion month is over, we take the number of transactions from each group and run another t-test. We Rejected the null in favor of the alternate Hypo, which is that these 2 groups are now statistically different with an alpha of 0.05.

My first question is: is this methodology okay ?

My second question is: alternatively from using box and whiskers and removing outliers until both groups' descriptive stats are similar, can i use a normal distribution and STDEV to remove outliers and create comparable groups within my population ? The box and whisker method worked to get a comparable groups as confirmed by the t-test, but is very manual. So i would like to create an automated method and was wondering if using a normal distribution and removing outliers by STDEV would be plausible ?

Answer 1

I don't think weeding "outliers" will get you to a fair comparison, given that some stores have larger sales than others. A somewhat crude alternative is to compute two average sales per period figures for every store, one for a time period before the marketing campaign and the other for the time period containing the marketing campaign. Then compare the ratios of after-average divided by before-average. Hopefully the ratio is ~1 for the control group and > 1 for the experimental group.

A potentially more accurate but much more complicated approach uses what is known as "time series intervention analysis". The intervention here is the marketing campaign. You fit a time series model to the experimental stores (individually or in aggregate) using pre-campaign data, then use the post-campaign data to estimate the change due to the intervention. You don't really need the control group for this, although you might want to fit a time series model to the control group just to verify that it remains stable after the marketing campaign starts (ruling out the possibility that some other factor that coincides with the marketing campaign is causing the change -- a pandemic, arrival of a cicada brood, the Dodgers winning the World Series, ...). This approach requires facility with ARIMA models.