Cypriot Journal of Educational Sciences

4 (2009) 5869 www.world-education-center.org/index.php/cjes

Application of Ordinal Logistic Regression and Artifical Neural Networks in a Study of Student Satistaction
Meral Yaya*, Eylem Deniz Akncb
a Assistant Professor at Mimar Sinan Fine Arts University, Department of Statististics, Turkey b Assistant Professor at Mimar Sinan Fine Arts University, Department of Statististics, Turkey

Received December 20, 2008; revised May 11, 2009; accepted May 30, 2009

Abstract Measuring student satisfaction is an important issue especially for university administration, in order to improve student services and opportunities. The major objective of this study is to provide a solution for this issue. Consequently, student satisfaction has been measured with an ordered five-point Likert scale. A student satisfaction questionnaire was applied to a total of 314 university students, consisting of 208 female and 106 male students, and satisfaction was measured by asking students to respond to 19 questionnaire items. Ordinal regression and artifical neural network methods were applied to the collected data which emphasized the differences between the two methods in terms of the correct classification percentages. Keywords: artifical neural networks; ordinal logistic regression; student satisfaction 2009 Academic World Education & Research Center. All rights reserved.

1.

INTRODUCTION

The measurement of student satisfaction can be useful in universities, based on the notion that students have needs and rights when it comes to participating in quality programmes and should receive a satisfactory service. A true and thorough understanding of the complex learning experience requires a knowledge of both student satisfaction and the factors which may detract from this satisfaction. The results from the questionnaire survey used in this research have been

Corresponding Author. Meral Yay. Tel: +90 0 212 236 69 36 177, fax: +90 0 212 261 11 21 E-mail address: myay@msgsu.edu.tr (M.Yay), edeniz@msu.edu.tr (Eylem Deniz Aknc)

Meral Yay & Eylem Deniz AkncI / Cypriot Journal of Educational Sciences 4 (2009) 58-69

used as feedback to help university administrators to enhance the quality of the universitys programmes and services. Different statistical methods can be used to analyze satisfaction data. These methods include regression models, multilevel modelling, decision trees and artifical neural networks. Regresssion models such as linear, logistic and ordinal regression are useful for analyzing the relationship between multiple explanatory variables and student satisfaction results (Thomas & Galambos, 2002). These methods also permit researchers to estimate the magnitude of the effect of the explanatory variables on the outcome variable. In this study, ordinal regression and artificial neural networks are used to analyze the student satisfaction data. Ordinal regression is a very useful and powerful method for satisfaction classification. Compared to the other regression methods used in the literature, the ordinal regression method is the most suitable and practical technique for analyzing the effects of multiple explanatory variables on the ordinal outcome that cannot be assumed to be a continuous measure with normal distribution. Artificial neural networks are algorithms that can be used to perform nonlinear statistical modeling and to provide a new alternative to ordinal logistic regression methods. Neural networks have a number of advantages, including requiring less formal statistical training than other methods, the ability to implicitly detect complex relationships between dependent variables and explanatory variables, the ability to detect all possible interactions between predictor variables, and the availability of multiple training algorithms. 2. ORDINAL LOGISTIC REGRESSION The ordinal logistic method is a generalization of the linear regression method. The ordinal regression method is used to model the relationship between response (outcome) variables and a set of explanatory variables, which can be either categorical or numerical (Sentas, Angelis, Stamelos & Bleris, 2004). In ordinal regression (OR) analysis, the major link functions, e.g. logit, complementaryloglog (continuation ratio or proportional hazard), negative log-log, probit and Cauchit are used to build specific models. There is currently no universal method to help the researcher choose which link function best fits a given dataset - only basic heuristics. Generally, the logit link is considered suitable for analyzing ordered categorical data evenly distributed among all categories; the cloglog link is often used to analyze ordered categorical data when higher categories are more probable (Cheng, 2007). In a negative log-log link function, lower categories are more probable and our data set is approprate for this function. Five different link functions can be explained briefly as in Table 1. The unique feature of the cumulative logit model is that the odds ratio for each predictor is taken to be constant across all possible collapsings of the outcome variable. When a testable assumption is met, the odds ratios in a cumulative logit model are interpreted as the odds of being lower or higher on the outcome variable, across the entire range of the outcome. The wide applicability and intuitive interpretation of the cumulative logit model are two reasons for its being considered the most popular model for ordinal logistic regression.

59

Meral Yay & Eylem Deniz Aknc / Cypriot Journal of Educational Sciences 4 (2009) 58-69

The hallmark of the cumulative logit or POM is that the odds ratio for a predictor can be interpreted as a summary of the odds ratios obtained from separate binary logistic regressions using all possible cut points of the ordinal outcome. Whereas a binary logistic regression models a single logit, the cumulative logit models several cumulative logits (Hosmer & Lemeshow, 1989). Therefore, if the ordinal outcome has five levels (1, 2, 3, 4, and 5), four logits will be modeled, one for each of the following cut points: 1 vs. 2, 3, 4, 5 ; 1, 2 vs. 3, 4, 5; 1, 2, 3 vs. 4, 5 and 1, 2, 3, 4 vs 5. (Gameroff, 2005) Let p 1 ( x i ) ,...., p k ( x i ) denote response probabilities at values for a set of explanatory variables. First form cumulative probabilities are as follows:
Fk ( x i ) = p ( Y k / x i ) = p 1 ( x i ) + ..... + p k ( x i ) , k = 1,2,...., K - 1

(1)

Cumulative logits are then formed as follows:

F (x ) Lk = log it Fk ( x i ) = log k i , 1- F ( x ) k i k = 1,2,....,K - 1

(2)

Letting Lk ( x i ) = logit Fk ( x i ) , where Fk ( x i ) is the cumulative probability up to, and including, category k, the Proportional Odds Model (McCullagh, 1980) can be expressed as follows:
Lk ( x i ) = a k + b k x i , k = 1,2,.....,K - 1

(3)

The a1 ,....., a K -1 parameters are nondecreasing in k and are known as the intercepts or cutpoints. The parameter vector b contains the regression coefficients for the covariate vector x i . Inherent in this model is the proportional odds assumption, which states that the cumulative odds ratio for any two values of the covariates is constant across response categories. Its interpretation is that the odds of being in a category less than or equal to k is exp b ( x1 - x 2 ) times higher at x = x1 than at x = x 2 . The model constrains the K-1 response curves to have the same shape, and therefore we cannot fit it by fitting separate logit models for each cutpoint. We must maximize the multinomial likelihood, subject to constraints. The model assumes that effects of variables are the same for each cutpoint, k = 1,..., K-1. In ordinal regression (OR) analysis, the major link functions, e.g. logit, complementaryloglog (continuation ratio or proportional hazard), negative log-log, probit and Cauchit are used to build specific models. There is currently no universal method to help the researcher choose which link function best fits a given dataset - only basic heuristics. Generally, the logit link is considered suitable for analyzing ordered categorical data evenly distributed among all categories; the cloglog link is often used to analyze ordered categorical data when higher categories are more probable (Cheng, 2007). In a negative log-log link function, lower categories are more probable and our data set is approprate for this function. Five different link functions can be explained briefly as in Table 1.

60

Meral Yay & Eylem Deniz AkncI / Cypriot Journal of Educational Sciences 4 (2009) 58-69

Table 1. Different Link Functions and Forms in Ordinal Regression

Function Logit Complementary log-log Negative log-log Probit Cauchit

F (x ) log k i 1- F (x ) k i

Form

log - log (1 - Fk ( x i ) ) - log - log ( Fk ( x i ) )

( Fk ( x i )) tan (p ( Fk ( x i ) - 0.5 ) )
-1

In this study, the lower categories are more probable. That is why, the negative log-log function is used in the application of ordinal regression. In the negative log-log function, the form of the link is defined as follows:
log it Fk ( x j ) = log - log 1 - Fk ( x j ) = - log - log P ( Y = y j / X ) / P ( Y > y j ) / X

(4)

3. NEURAL NETWORKS Classification is one of the most active research and application areas with regard to neural networks. The literature is vast and growing. Neural networks have emerged as an important tool for classification purposes. The recent research activities in neural classification have established that neural networks are a promising alternative to various conventional classification methods. The advantage of neural networks lies in the following theoretical aspects. Firstly, neural networks are data-driven self-adaptive methods in that they can adjust themselves to the data without any explicit specification of functional or distributional form for the underlying model. Secondly, they are universal functional approximators in that neural networks can approximate any function with arbitrary accuracy. Since any classification procedure seeks a functional relationship between the group membership and the attributes of the object, accurate identification of this underlying function is doubtlessly important. Thirdly, neural networks are nonlinear models, which makes them flexible in modeling real world complex relationships. Finally, neural networks are able to estimate the posterior probabilities, which provide the basis for establishing classification rules and for performing statistical analysis (Zhang, 2000). Artificial neural networks (ANN) are relatively crude electronic networks of "neurons" based on the neural structure of the brain. They process records one at a time, and "learn" by comparing their classification of the record (which, at the outset, is largely arbitrary) with the known actual classification of the record. The errors from the initial classification of the first record is fed back into the network, and used to modify the networks algorithm the second time around, and so on for many iterations (Silva, 2006). Roughly speaking, a neuron in an artificial neural network is; 1. A set of input values (xi) and associated weights (wi),

61

Meral Yay & Eylem Deniz Aknc / Cypriot Journal of Educational Sciences 4 (2009) 58-69

2.

A function (g) that sums the weights and maps the results to an output (y).

Figure1. Output Map

Figure 2. Hidden Layers

The input layer is composed, not of full neurons, but rather consists simply of the values in a data record, that constitute inputs to the next layer of neurons. The next layer is called a hidden layer; there may be several hidden layers. The final layer is the output layer, where there is one node for each class. A single sweep forward through the network results in the assignment of a value to each output node, and the record is assigned to whichever class's node had the highest value. To use a neural network for classification, we need to construct an equivalent function approximation problem by assigning a target value for each class. For a two-class problem we can use a network with a single output, and binary target values: 1 for one class, and 0 for the other. The training of the network is commonly performed using the popular mean square error. For multiclass classification problems (1 - of - K, where K > 2 ) we use a network with K outputs, one corresponding to each class, and target values of 1 for the correct class, and 0 otherwise. Since these targets are not independent of each other, however, it is no longer appropriate to use the same error measure. The correct generalization is through a special activation function (the softmax) designed so as to satisfy the normalization constraint on the total probability. However, this approach does not retain the ordinality or rank order of the classes, and is not, therefore appropriate for ordinal multistate classification problems ( Costa & Cardoso, 2005).

62

Meral Yay & Eylem Deniz AkncI / Cypriot Journal of Educational Sciences 4 (2009) 58-69

4. CASE STUDY In this study, student satisfaction questionnaires were distributed to a total 314 senior university students consisting of 208 females and 106 males. Satisfaction was measured by asking the students to respond to 19 items. Student satisfaction is a response variable and it is measured on a five-point Likert scale (very dissatisfied, dissatisfied, dont know, satisfied, very satisfied). Gender is coded by 1 female and 2 male. 19 questionnaire items were measured using a five-point Likert scale (very dissatisfied = 1, dissatisfied = 2, dont know = 3, satisfied = 4, very satisfied = 5). OR and ANN which are frequently used for analyzing satisfaction were applied to our data. The OR model is anayzed using SPSS 15 and ANN is analyzed using Statistica. Cronbach Alpha is calculated for 19 questionnaire items. According to Nunnely (1998), the Cronbach Alpha of a scale should be greater than 0.70 for items to be used together as a scale.The high internal consistency for the survey might be demostrated based on the Cronbach Alpha reliability 0.894 (for all items combined). The student respondents indicated that they were very dissatisfied (43.6%) and very satisfied (2.2%) with the overall university experience. The majority of respondents seemed to be very dissatisfied with the universitys programmes and services, regardless of gender. This is indicated in Table 1.
Table 2. Different Link Functions and Forms in Ordinal Regression frequency 137 102 42 26 7 314 percent 43.6 32.5 13.4 8.3 2.2 100.0 valid percent 43.6 32.5 13.4 8.3 2.2 100.0 cumulative percent 43.6 76.1 89.5 97.8 100

very dissatisfied dissatisfied dont know satisfied very satisfied total

For OR and ANN analysis, 80% of the data were used as the training set, and the remaining 20% for the test set in order to confirm model validation. The OR model is applied for a negative log-log model because the lower categories are more probable. The negative log-log model containing satisfaction items revealed a number of interesting findings. The Pearsons chi-square ( c 2 = 971.681 and p=1.00) for the model with the negative log-log link, indicates that the observed data are consistent with the estimated values in the fitted model. The results of parameter estimates are indicated as in Table 2. According to this table, student satisfaction is significantly associated with the four explanatory variables (department, facilities, activities and library). These four significant explanatory variables exhibit positive regression coefficients.

63

Meral Yay & Eylem Deniz Aknc / Cypriot Journal of Educational Sciences 4 (2009) 58-69

Table 3. Explanatory Variables Associated with Student Satisfaction Based on Complete Model for Negative Log-log Link estimate 3.401 4.839 5.852 7.633 -0.060 0.403 -0.113 0.208 0.290 0.002 0.056 0.054 -0.033 0.064 0.072 0.006 0.015 -0.004 -0.132 0.284 -0.145 0.114 0.057 0.163 significance 0.000 0.000 0.000 0.000 0.743 0.000 0.181 0.021 0.004 0.981 0.561 0.564 0.785 0.496 0.448 0.946 0.861 0.965 0.148 0.002 0.091 0.184 0.575 0.058

satisfied=1 satisfied=2 satisfied=3 satisfied=4 gender department course facilities activities union scholarship exchange administration election website computerization language consultancy clean library food cafe sports toilet

The test of parallel lines is designed to make a judgement concerning the adequacy of the model. The null hypothesis states that the corresponding regression coefficient is equal across all levels of the response variable. The alternative hypothesis states that the corresponding regression coefficients are different across all levels of response variables. According to the test of parallel lines results, there was no significant difference for the corresponding regression coefficients across the response categories, suggesting that the models assumption of parallel lines is not violated in the complete model with the negative log-log link ( c 2 =658.422 and p=0.000). The model with the negative log-log link provide evidence that it satisfies the parallel lines assumption. The data is divided into two parts - train (250) and test (64) in order to select the model which is the best classification among the logit link, the clog log link and the ANN model. The crosstabulating method is used to categorize the classified and actual responses into a 5 by 5 classification table at the negative log-log link and the ANN model. Table 4 displays the accuracy of the classification results for the satisfaction response categories of the train data at the logit link model. The percentage of classification is 62.4% for all categories.

64

Meral Yay & Eylem Deniz AkncI / Cypriot Journal of Educational Sciences 4 (2009) 58-69

Table 4. Actual Satisfaction Category * Predicted Satisfaction Category Crosstabulation for the Negative Log-log Link Model (train)
very dissatisfied 93 84.5% 28 32.2% 6 18.2% 2 10.5% 0 .0% 129 51.6% predicted response category dont dissatisfied satisfied know 17 0 0 15.5 .0% .0% 56 2 1 64.4% 2.3% 1.1% 21 4 2 63.6% 12.1% 6.1% 11 4 2 57.9% 21.1% 10.5% 0 0 0 .0% .0% .0% 105 10 5 42.0% 4.0% 2.0% very satisfied 0 .0% 0 .0% 0 .0% 0 .0% 1 100% 1 4% total 110 100% 87 100% 33 100% 19 100% 1 100% 250 100%

very dissatisfied dissatisfied actual response category dont know satisfied very satisfied total

Table 5 displays the accuracy of the classification results for the satisfaction response categories of the test data using the negative log-log link model. The percentage of classification is 65.6% for all the categories. Table 5. Actual Satisfaction Category * Predicted Satisfaction Category Crosstabulation for the Negative Log-log Link Model (test)
very dissatisfied 23 85.2% 4 26.7% 0 .0% 0 .0% 0 .0% 27 42.2% predicted response category dont dissatisfied satisfied know 4 0 0 14.8 .0% .0% 11 0 0 73.3% .0% .0% 7 1 1 77.8% 11.1% 11.1% 2 2 3 28.6% 28.6% 42.9% 0 0 2 .0% .0% 33.3% 24 3 6 37.5% 4.7% 9.4% very satisfied 0 .0% 0 .0% 0 .0% 0 .0% 4 66.7% 4 6.3% total 27 100% 15 100% 9 100% 7 100% 6 100% 64 100%

very dissatisfied dissatisfied actual response category dont know satisfied very satisfied total

Table 6 displays the accuracy of the classification results for the satisfaction response categories of the train data using the ANN model. The percentage of classification is 70.4% for all the categories.

65

Meral Yay & Eylem Deniz Aknc / Cypriot Journal of Educational Sciences 4 (2009) 58-69

Table 6. Actual Response Category* Predicted Response Category Crosstabulation for the ANN Model (train)
very dissatisfied 106 96.4% 35 42.7% 10 29.4% 5 22.7% 0 .0% 156 62.4% predicted response category dont dissatisfied satisfied know 3 1 0 2.7% .9% .0% 39 6 2 47.6% 7.3% 2.4% 7 15 2 20.6% 44.1% 5.9% 1 2 14 4.5% 9.1% 63.6% 0 0 0 .0% .0% .0% 50 24 18 20.0% 9.6% 7.2% very satisfied 0 .0% 0 .0% 0 .0% 0 .0% 2 100% 2 .8% total 110 100% 82 100% 34 100% 22 100% 2 100% 250 100%

very dissatisfied dissatisfied actual response category dont know satisfied very satisfied total

Table 7 displays the accuracy of the classification results for the satisfaction response categories of the test data using the ANN model. The percentage of classification is 65.6% for all the categories. Table 7. Actual Response Category* Predicted Response Category Crosstabulation for ANN Model (test)
very dissatisfied 24 88.9% 8 40.0% 3 37.5% 0 .0% 0 .0% 35 54.7% predicted response category dont dissatisfied satisfied know 2 1 0 7.4% 3.7% .0% 8 1 3 40.0% 5.0% 15.0% 0 3 2 .0% 37.5% 25.0% 0 0 4 .0% .0% 100% 0 1 1 .0% 20.0% 20.0% 10 6 10 15.6% 9.4% 15.6% very satisfied 0 .0% 0 .0% 0 .0% 0 .0% 3 60.0% 3 4.7% total 27 100% 20 100% 8 100% 4 100% 5 100% 64 100%

very dissatisfied dissatisfied actual response category dont know satisfied very satisfied total

The research findings indicate that the ANN model has better satisfaction classification than the ordinal regression model with negative log-log link model for train and test data.

66

Meral Yay & Eylem Deniz AkncI / Cypriot Journal of Educational Sciences 4 (2009) 58-69

5. CONCLUSION In this study, student satisfaction is measured with an ordered scale and analyzed using OR and ANN in order to emphasize the difference of the two methods in terms of satisfaction classification. The principle of classification with a negative log-log link function in ordinal regression and ANN is adopted to build the candidate models and to search for the best model. The ANN and the OR models are used for classifications in various fields. In this study, these methods are applied to student satisfaction data. As a consequence, the ANN model provides a better classification for ordinal data. In other words, it has more correct percentage of classification than the ordinal regression method. REFERENCES Cheng, J. (2007). A Neural Network Approach to Ordinal Regression. Electrical Engineering and Computer Science, 1, 1028-1036. Costa, J. P., & Cardoso, J. S. (2005). Classification of Ordinal Data Using Neural Networks (pp.690697) Springer Berlin. Fujimoto, K. (2003). Application of Multinomial and Ordinal Regressions to the Data of Japanese Female Labor Market, Unpublished Masters Thesis, 18, University of Pittsburgh. Gameroff, M. J. (2005). Using the Proportional Odds Model for Health-Related Outcomes: Why, When, and How with Various SAS Procedures, New York State Psychiatric Institute, 205-30. Hosmer, D.W. & Lemeshow, S. (1989). Applied Logistic Regression, John Wiley & Sons, NewYork. MacCullagh, P. (1980). Regression Models for Ordinal Data. Journal of the Royal Statistical Society, 42, 109-142. Sentas, P., Angelis, L., Stamelos, I., & Bleris, G. (2005). Software Productivity and Effort Prediction with Ordinal Regression, Information and Software Technology, 17-29. Silva, D.J. (2006). Comparison of Artificial Neural Network and Regression Models in Software Effort Estimation, Laboratory for Computing and Applied Mathematics, 47, 960-977. Thomas, E.H., & Galambos, N. (2002). What Satisfies Students? Mining Student-Opinion Data with Regression and Decision-Tree Analysis, Stony Brook University, NewYork. Zhang, G. P. (2000). Neural Networks for Classification: A Survey , IEEE Transactions on Systems, 30(4), 451-465.

67

Meral Yay & Eylem Deniz Aknc / Cypriot Journal of Educational Sciences 4 (2009) 58-69

APPENDIX MIMAR SINAN FINE ARTS UNIVERSITY STUDENT SATISFACTION EVALUATION FORM Gender : Female Male

The measurement of student satisfaction (response variable): Very dissatisfied Dissatisfied Dont know Satisfied Very Satisfied Questionnaire Items 1- My department met my expectations. 2- The number of elective courses is adequate. 3- Classroom, workplace, equipment and techniqual facilities are adequate. 4- There is enough scientific, social and cultural activities at the university. 5- The student union works efficiently. 6- The scholarship opportunities are adequate. 7- It is easy to benefit from the student exchange programmes. 8- The university administration believes that it is important to delegate decision making to the students. 9- Im satisfied with the way in which student delegates are elected. 10- Im satisfied with my universitys website. 11- Im satisfied with my universitys computer facilities. 12- The foreign language education is adequate. 13- The course consultancy system works well. 14- The environment of university is clean. 15- The library oppurtinities are adequate. 16- The quality of food is good. 17- The variety of food in the cafe is adequate. 18- The sports opportunities are adequate. 19- The toilets are clean. 1 2 3 4 5

68

Meral Yay & Eylem Deniz AkncI / Cypriot Journal of Educational Sciences 4 (2009) 58-69

Turkish Abstract Ordinal Lojistik Regresyon ve Yapay Sinir Alarnn renci Memnuniyeti zerinde Uygulanmas z: renci memnuniyetini lme, niversite ynetiminin renci olanaklarn ve frsatlarn artrmalar iin nemli bir sorundur. almann temel amac, bu soruna bir zm salamaktr. Bu ama dorultusunda renci memnuniyeti beli Likert leinde lmtr. renci memnuniyeti anket sorular 208i kz 106s erkek olmak zere toplam 314 renciye sorulmu ve memnuniyet rencilere yneltilen 19 soru ile llmtr. Toplanan verilere ordinal regresyon ve yapay sinir a yntemleri uygulanm ve ardndan ile renci memnuniyetini doru snflandrma oran asndan yntemler arasndaki fark vurgulanmtr. Anahtar Kelimeler: Yapay sinir a, ordinal lojistik regresyon, renci memnuniyeti.

69