Statistical

Confidence Calculations

Statistical Methodology

Omniture Test&Target utilizes standard statistics to calculate confidence, confidence intervals, and lift for
each campaign. The students T-test is used for these calculations. For conversions and steps, calculations
for binomial distributions are used; for all other calculations a normal distribution is assumed.

Conversion Data Calculations

Conversion Rate
Conversion rate is the number of conversions divided by the number of visitors (or visits or impressions).

"#$%&'()#$!*+,&!-"*.!/!
!

"#$%&'()#$(
0)(),#'(

Confidence
Confidence is the result of the Students T-Test. Confidence indicates how likely it is that if the test were
repeated, the same results would be found.

To calculate confidence, the standard deviation must first be determined.

Standard Deviation
Standard deviation is a measure of the spread or dispersion of a set of data. The standard deviation is
simply a way to measure the dispersion of data. If many data points are close to the mean, then the
standard deviation is small; if many data points are far from the mean, then the standard deviation is large.
Standard deviation is also the square root of the variance.

The conversion rate results are a binomial distribution: visitors either convert or do not convert. Thus, we
use the binomial distribution formula for variance:

*
!"#$%#&'()!!! = +,-. " +,/

Since standard deviation is the square root of variance, we get:

!"#$%&$'&!()*+$#+,%-!!! = ./01 " ./2

Standard Error
The standard error is the estimated standard deviation of the error. This error is the noise in the results.
For the control experience:
1








! *)%#')+

"#$%&$'&!('')'*)%#')+ ,!!"( =
./0/#)'0*)%#')+
!

For alternative experiences:

! ,)%#')*
! $*#
"#$%&$'&!('')'$*# +!!"( =
+
./0/#)'0,)%#')* ./0/#)'0$*#
!

The standard error is then used to calculate the signal to noise ratio.

Signal to Noise Ratio

"#$%&' ,-&'. ! ,-/)%.0)'

=
()#*+
"1&'.
!

Confidence
This signal to noise ratio is then used to calculate confidence. Confidence is calculated with the Students T-
Test. It is a 2-tailed distribution, and the degrees of freedom is equal to visitorscontrol +visitorsalt-2.
In Excel this is calculated with the TDIST() function:

",5'6767#
"#$%&'#()!*!+,)#-,.$#,/'!0!1!2!*+3"*4
9:,),#/-)67# + :,),#/-);/'#-/7 ! <9<=
8/,)&67#
!

Confidence refers to the likelihood that the alternative experiences performance relative to control wasn't
due to noise. This determines how confident you can be that the results would be repeated if the test were
re-run.

Confidence Intervals
The confidence interval displayed in Test&Target is different than the confidence level. While the
confidence level shows the likelihood that the test results were not based on noise, the confidence interval
assumes a 95% confidence level and shows how much your results could vary and still be within that 95%
confidence level. Essentially, this calculation describes how large the standard deviation is in an easily
understood way.

The confidence interval is derived from the standard deviation and the sample size (# of visitors). The
smaller the standard deviation and the larger the sample size, the narrower your confidence interval.

"
%
! ./+
$
'
"#$%&'($)(!*$+(,-./!0!1234
$# 5&6&+#,6 '&
./+

!

where 1.96 is the interval on the normal distribution curve containing 95% of the area under the curve.

Then, determine the high and low bounds for your conversion rate by adding and subtracting this confidence interval
from your conversion rate:

!"#$%!&'()*!+,!-!+, + +.
!"#$!%#&'(!)*!+!)* ! ),

Lift
Lift is the percentage difference of your tested experience vs. control.

" '( ! '(+,-%.,* %

"#$%!&! $ )*%
'
'(+,-%.,* &
#
!

Lift Intervals
Lift intervals describe the best and worst lift a tested experience could have over control. To get this
information, the high and low bound of conversion rate for control is compared to the high and low bound
of conversion rate for the alternative (or tested) experience. The statistical confidence has already been
applied, so no additional statistical equations are involved here.

The worst lift the alternative could experience would be if the control performed at its high bound and the
alternative performed at its lower bound:

"#$!%#&'(!")*+!,!"#$-.+ /!0)123#'
!

Conversely, the best lift the alternative could experience would be if the control performed at its low bound
and the alternative performed at its high bound:

"#$%!&'()*!+#,-!.!"#$%/0- 1!+'23')
!

Revenue Data Calculations

Revenue Per Visitor








Revenue per visitor is the total sales number divided by the number of visitors (or visits or impressions).

Standard Deviation

Standard Error
For revenue calculations, the standard error calculation is the same for the control and alternate
experiences. This calculation is used for the signal/noise ratio.

Standard Error of the Difference
This calculates the difference in performance between the alternate experience and the control.

Signal to Noise Ratio

Confidence Revenue

Confidence Interval

Then, determine the high and low bounds for your revenue per visitor by adding and subtracting this
confidence interval from your revenue per visitor number, just like for conversion rate.








Lift
Lift is the percentage difference of your tested experience vs. control.

Lift Intervals
The lift intervals are calculated in the same manner as for conversion rate above.


These calculations describe the derivation of all the statistical values displayed in the Test&Target reports. With
these calculations, you can download the raw data directly from Test&Target or programmatically access it via
the API and run your own analyses.