Although, two quick points.

Correct me if I am wrong, but Website Optimizer is using Z-scores not Chi Squares? Personally, I prefer Confidence Intervals because it is easier to explain to people and for people who don’t know statistics to see what is going on. Of course, it is three ways to do the same thing–find statistically significant differences. Although, like you showed in your own Chi Square, if it is anything but 2X2, you have to do another step to find which comparison is statistically significant. So, it isn’t always the most efficient way to get results.

The second point has to do with your comment regarding sample size and significance. I see this all the time with people talking about A/B testing and multivariate testing. While it is true that adding more samples will provide significance, it kind of defeats the purpose. Sure, if you only had a couple hundred samples, then sure, adding more people would be beneficial. But at a certain point, adding more participates simply waters down the results. Adding 21,000 respondents is just crazy. The p-value still tells you something if the result isn’t significant. Mainly, that the signal-noise ratio is weak. No result is still a result, i.e., you can’t reject the null hypothesis. Most likely, once you hit about 1,500 samples, you’ve pretty much reached a point of diminishing returns. Adding 7,000 more samples for each variation simply gives the illusion of significance when none really exists.

Regards,
Domen