Reading time: 2 – 3 minutes

We are coming into a digital / data age by all accounts, and have been for a while. Everything is based on data, data data and making decisions based on that data, especially for ecommerce sites. Really though, do we know what we’re looking at? Do we understand the numbers? The differences in conversion rates, visitor sources and the like?

The truth is that data has two major caveats that everyone needs to be aware of and needs to put their statistical hat on to understand:

Data can tell any story

Depending the on the way you cut your data, the way you include or exclude certain groups and the metrics you use, data can tell wildly varying stories to people. The data, of course, is always the same (hopefully). One pool of data, and we attach meaning to that data. It’s the meaning we attach that can differ so much!

If we go into data wanting to draw a particular conclusion, then we can find a way to do it! Be careful.

Data needs to be statistically relevant

The good old AB test. A dangerous tool at times. We’re comparing two different landing pages. One converts the visitor 10% better than the other one. We’re all happy with the work done. Is this number real? ‘What do you mean is it real?’ I hear you cry – how can numbers lie? They can’t. But they can hide the whole truth.

First of all, sample size. Did you test enough visits and conversions. It ran over 100 visits? Not enough! To get statistical relevance we need a large sample size, over a decent time period.

Secondly, if you try an AA test, comparing exactly the same creative to itself, in theory they should perform exactly the same. In practice, this does not always happen. In fact, over at Get Elastic they performed an AA test and one version outperformed the other by 4.97%! Shocking. So that 10% difference could of been 5%, or 15% – who knows!

If you perform tests make sure you have statistically relevant numbers, and that the results you get show a very clear difference between the two. If you can, perform an AA test also.

Filed under: Analytics & measurement, linkedin

Related Posts

• Do you just collect data? Or do you analyse?
• Shopping online ≠ Shopping offline
• Getting under Next customers’ skins