Statistical Significance Testing

The concept of statistical significance is that some variation in the results of research findings is large enough to not be explained simply by chance. If a survey were given to 100 people and then given to a completely different group of 100 people, the results of the two surveys are going to be somewhat different. The question is, does this difference arise because of sampling error in choosing the participants for one of the surveys or is there a significant difference between the two groups. Statistical significance testing is a method of determining if marketing research findings are significant or incidental. There are several different types of significance tests. They include: 1. 2. 3. 4. Chi-square tests. z-tests. t-tests. F-tests.

Each of these methods of significance testing is described in this tutorial as well as when to apply the test and how to interpret the results of the test. Chi-Square Test The chi-square statistical test studies the relationship between two categorical variables. As explained in the Association Cross Tabulation tutorial, the association between two categorical variables is looked at by creating a table of all the possible combinations of responses of the two different variables. This table can be created in SPSS by a process called crosstabs. A chisquare test enables you to determine whether an observed pattern of frequencies in a crosstabs table corresponds to or fits an expected pattern. The simplest way to conduct a chi-square test is to use SPSS. A chi-square test using SPSS and a chi-square distribution table can be done by following these steps: 1. Enter the data into SPSS and perform a crosstabs analysis as explained in the SPSS Tool Kit tutorial. 2. Look at the table in the SPSS viewer labeled, Chi-Square Tests. There is a row in this column labeled Pearson Chi-Square. The two numbers in this row that you need to pay attention to are in the column labeled Value and df. The df stands for degrees of freedom. 3. Look up the chi-square test value in a chi-square test table. This table can be found in the appendix in the back of the textbook. You will notice in the test table there is a column labeled Degrees of freedom and one or more columns labeled with different levels of significance. For this class we will always use a 0.10 or 0.05 level of significance. These are the significance levels most commonly used in practice. The chi-square test table corresponding to these levels of significance is duplicated at the end of this tutorial.

Look up the number in the chi-square test table that corresponds to the df number given in the SPSS crosstabs analysis and the desired level of significance. 4. Compare the Value number given in the SPSS crosstabs analysis to the number looked up in the table. If the Value number given in the SPSS crosstabs analysis is greater than the number looked up in the table, then the results of the analysis are statistically significant at the level chosen. For a level of significance of .10, this means you are 90% confident the results are statistically significant. For example, suppose a survey asked a question about the respondents gender and another question about the respondents frequency of visits to a particular store. After collecting the data and running a crosstabs analysis in SPSS, you have the following two tables:
fre quency of visits * gende r Cros s tabulation Count gender male f emale 14 26 16 34 15 11 45 71 Total 40 50 26 116

f requency of visits Total

1-5 6-14 15 and abov e

Chi-Square Te s ts Asy mp. Sig. (2-s ided) .077 .081 .101

Pearson Chi-Square Likelihood Ratio Linear-by -Linear Ass ociation N of Valid Cases

Value 5.125a 5.024 2.685 116

df 2 2 1

a. 0 cells (.0%) have ex pected count less than 5. The minimum ex pec ted count is 10.09.

The crosstabs analysis yields two degrees of freedom. Looking up the chi-square statistic for two degrees of freedom and for a 0.10 level of significance gives you the value 4.605. Since the Pearson chi-square value of 5.125 is bigger than the value from the table, you would conclude with 90% confidence that there is a statistically significant difference in frequency of visits to the store between males and females. Alternatively, you can determine the significance level by looking at the value in the Pearson Chi-Square row under the column labeled Asymp. Sig. (2sided). From the example, this value is .077. This means we are 92.3% confident (1.0 - .077 = 0.923 or 92.3%) that there is a statistically significant difference in frequency of visits to the store between males and females. One other item to notice is the footnote immediately below the Chi-Square Tests output table. This footnote tells you how many cells in the crosstabs table have an expected count less than 5.

If more than 20% of the cells have an expected count less than 5, or if any cell has an expected count less than 1, then the results of the Chi-square test should not be used to test for statistical significance. The reason for this is because cells with low expected counts throw off the calculation of the Chi-square statistic. If too many cells have a low expected count the calculated Chi-square value is no longer accurate and should not be used to test for statistical significance. z-Test There are a couple of different types of z-tests you can conduct. Note that for any type of z-test you need to have at least 30 data points for the test results to be valid. One type of z-test is to test the statistical significance of a surveys results when used to estimate the characteristics of an entire population represented by the sample. Your marketing research textbook contains instructions for conducting and evaluating a z-test for this purpose. However, this is generally a less useful type of z-test and is therefore not explained any further in this tutorial. A more useful type of z-test is one that can be conducted to test the statistical significance of the difference in means between two sets of data. For example, suppose you asked a group of respondents a question about how often they make purchases from a particular store. You want to know if there is any difference between average frequency of purchases at the store for men and women. The statistical significance of the difference in means can be checked using a z-test. This can be done in Microsoft Excel by following these steps: 1. Enter a data set into one column. There must be at least 30 data points in this data set for the z-test to be valid. 2. Enter a second data set into another column. There also must be at least 30 data points in this data set for the z-test to be valid. However, the two data sets dont have to have the same number of data points. 3. Calculate the variance of the first data set. This is done by selecting a cell and using Excels var function. Select all data points in the set when calculating the variance. You can also calculate the standard deviation of the data set and then square the standard deviation to get the variance. 4. Calculate the variance of the second data set using the same procedure used to calculate variance of the first data set. 5. Click on the Tools menu at the top of the Excel screen. 6. Select Data Analysis from the drop down menu. If you dont have the Data Analysis option you need to perform the following steps (you shouldnt have this problem using one of the computers in the Tanner Building computer labs): a. Click on the Tools menu at the top of the Excel screen. b. Select Add-Ins from the drop down menu. c. In the window that pops up, check the box next to Analysis Toolpak. d. Click on OK. Depending on how Excel was originally installed on your computer, you may need your original Excel installation CD to complete this process. 7. Select z-test: Two Sample for Means from the list on the pop-up screen (it is the very last choice).

8. Click OK. 9. A new box will pop up asking for the information to use for conducting the z-test. For Variable 1 Range select the data set in the first column. 10. For Variable 2 Range select the data set in the second column. 11. For Hypothesized Mean Difference enter 0. This means you want to test the statistical significance of there being any difference between the means of the two data sets. 12. For Variable 1 Variance (known) enter the variance calculated in step 3 for the first data set. 13. For Variable 2 Variance (known) enter the variance calculated in step 4 for the second data set. Note that the two variance values must be manually entered. You cant just select a cell containing the values. 14. If you included column labels in the variable ranges selected in steps 9 and 10, check the Labels box. Otherwise leave this box blank. 15. For Alpha enter the level of significance you wish to use. In this class, and in most instances in business, you will either use 0.10 or 0.05. 16. Under Output options either choose Output Range and select a cell or choose New Worksheet Ply and type the name of a new worksheet. You can also select New Workbook but you may find it more helpful to keep track of your results by placing the output as close to the data as possible. 17. Click on OK After following these steps, an output table will appear with the results of the z-test. The table consists of three columns and several rows. The first column contains the labels for each of the rows after the first three header rows. The following steps will help you interpret the output results: 1. The first row of the table (after the table header) is labeled Mean. The 2nd and 3rd columns contain the calculated mean value for the two sets of data. 2. The second row is labeled Known Variance. The 2nd and 3rd columns contain the values you input for the variance of each set of data on steps 12 and 13 above. 3. The third row is labeled Observations. The 2nd and 3rd columns contain a count of the number of data points included in each data set. 4. The fourth row is labeled Hypothesized Mean Difference. The 2nd column contains the value you input on step 11 above. 5. The fifth row is labeled z and contains the z-score for the test you conducted. 6. The sixth and seventh rows are labeled P(Z<=z) one-tail and z Critical one-tail. Ignore the information in these two rows. You are interested in the information for a two-tail test. You want to look at the two-tail test results because it looks at the probability that the one mean is neither higher nor lower than the other mean. The onetail test is only checking one of these two possibilities. 7. The eight row is labeled P(Z<=z) two-tail. The 2nd column should contain a number between 0 and 1. This number is the probability there is no statistically significant difference between the two means. If you take one minus this number, it will give you the statistical significance of the test. For example, if the value shown were 0.025 then you would be 97.5% (1 - 0.025 = 0.975) certain that there is a statistically significant difference in the means of the two data sets.

8. The last row is labeled z Critical two-tail. The 2nd column contains the minimum zscore necessary for the test to be statistically significant at your chosen level. If this number is less than the absolute value of the z-score indicated in the fifth row, then the results of your test are statistically significant at the level you chose. For example, if you chose a 0.05 level of significance, the last row should have a value of 1.96 (rounded). If the z-score in the fifth row is 2.05, then you would be 95% confident that there is a statistically significant difference in the means of the two data sets because z (2.05) is greater than z critical (1.96). t-Test A t-test is similar to a z-test. The difference is that a t-test is used if the sample size is 30 or less. A t-test can be used for the same purposes as a z-test (determine statistical significance of a sample projected onto an entire population or determine statistical significance of a difference in sample means). Methods for conducting a t-test are not described in this tutorial because you will almost always have sample sizes bigger than 30 to analyze and therefore will use z-tests far more often than t-tests. Sometimes an SPSS analysis will provide results that include a t-statistic as well as a Sig value. SPSS never provides a z-score as part of the results of an analysis. This is because if the number of degrees of freedom is large enough, a t-test and a z-test provide the exact same analysis of statistical significance. Therefore, SPSS will not conduct z-tests. It will only conduct t-tests. It is typically easier to conduct a z-test in Excel than to conduct a t-test in SPSS. However, when SPSS provides a t-statistic, it is not necessary to know the number of degrees of freedom because the Sig value associated with the t-score provides the level of confidence. For example, if SPSS provided a t-score with an associated Sig value of 0.046 then we would be 95.4% confident (1 .046 = .954 or 95.4%) in the results of the analysis and would say the results are statistically significant at this level. F-Test Another common test of statistical significance that you will come across when doing analysis with SPSS is the F-test. For example, if you run a linear regression test in SPSS you get a table in the SPSS viewer labeled, ANOVA. The last two columns of this table are labeled F and Sig. If you know the degrees of freedom associated with this F-score you can compare the Fscore to a table of F-values like the one in the back of the textbook. However, this is beyond the scope of this class. The important thing to remember is that with an F-score, like a Z-score or a t-score, the bigger the number, the higher the level of statistical significance. The Sig value listed in the last column of the ANOVA table tells you the level of significance associated with the F-score. For example, if the Sig value is .21 then you are 79% confident (1 - .21 = .79 or 79%) that the results of your analysis are statistically significant. This level of confidence is generally considered to be statistically insignificant. A general rule of thumb is that an SPSS Sig value must be 0.10 or less (90% or more confident) to be considered statistically significant.

Chi-Square Test Table for 0.10 and 0.05 Level of Significance Degrees of Freedom 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 40 50 60 70 80 90 100 Test Value at 0.10 Level of Significance 2.70554 4.60517 6.25139 7.77944 9.23635 10.6446 12.0170 13.3616 14.6837 15.9871 17.2750 18.5494 19.8119 21.0642 22.3072 23.5418 24.7690 25.9894 27.2036 28.4120 29.6151 30.8133 32.0069 33.1963 34.3816 35.5631 36.7412 37.9159 39.0875 40.2560 51.8050 63.1671 74.3970 85.5271 96.5782 107.565 118.498 Test Value at 0.05 Level of Significance 3.84146 5.99147 7.81473 9.48773 11.0705 12.5916 14.0671 15.5073 16.9190 18.3070 19.6751 21.0261 22.3621 23.6848 24.9958 26.2962 27.5871 28.8693 30.1435 31.4104 32.6705 33.9244 35.1725 36.4151 37.6525 38.8852 40.1133 41.3372 42.5569 43.7729 55.7585 67.5048 79.0819 90.5312 101.879 113.145 124.342

t-Test Table for 0.10 and 0.05 Level of Significance Degrees of Freedom 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 40 60 120 Test Value at 0.10 Level of Significance 3.078 1.886 1.638 1.533 1.476 1.440 1.415 1.397 1.383 1.372 1.363 1.356 1.350 1.345 1.341 1.337 1.333 1.330 1.328 1.325 1.323 1.321 1.319 1.318 1.316 1.315 1.314 1.313 1.311 1.310 1.303 1.296 1.289 1.282 Test Value at 0.05 Level of Significance 6.314 2.920 2.353 2.132 2.015 1.943 1.895 1.860 1.833 1.812 1.796 1.782 1.771 1.761 1.753 1.746 1.740 1.734 1.729 1.725 1.721 1.717 1.714 1.711 1.708 1.706 1.703 1.701 1.699 1.697 1.684 1.671 1.658 1.645