UNIVERSITY of PIRAEUS

Improving

the Quality of Customer Satisfaction Measurements of MUSA Method using Clustering Data Mining Techniques
Nikos Tsotsolas, Yiannis Siskos and Gerasimos Marketos University of Piraeus ntsotsol@unipi.gr

* Research partially supported by the HERACLETOS EPEAEK II Programme of the Greek Ministry of National Education and Religious Affairs, co-funded by the European Union.

Structure:

MUSA Method Research objectives Data mining approach The experiment Application of results Conclusions
2

MUSA Method (1)

MUSA Method
The main objective of the method is the aggregation of individual judgements into a collective value function assuming that clients global satisfaction depends on a set of n criteria representing service characteristic dimensions.

Customers Global Satisfaction

Satisfaction according to the 1-st criterion

Satisfaction according to the 2-nd criterion

Satisfaction according to the n-th criterion

Grigoroudis and Siskos (2002)

The MUSA method assesses global and partial satisfaction functions Y* and X*I respectively, given customers judgements Y and Xi.
n * * Y b X i i i 1 n b 1 i i 1

where the value functions Y* and X*I are normalised in the interval [0,100], and bi is the weight of the i-th criterion

MUSA Method (2)

MUSA Method
MUSA uses a preference disaggregation model. In the traditional aggregation approach, the criteria aggregation model is known a priori, while the global preference is unknown. On the contrary, the philosophy of disaggregation involves the inference of preference models from given global preferences.
Aggregation Model

aggregation

CRITERIA

GLOBAL PREFERENCE
disaggregation

Aggregation Model?

MUSA Method (3)

MUSA Method
Customer's global satisfaction
Y* y* y*m

Global Added Value Function

...
y*2 y*1 Y y1 y2 ... ym ... y

Satisfaction according to the 1st criterion

X1* x1*i

Satisfaction according to the 2nd criterion

Xi* xi*i xi*m

...

Satisfaction according to the n-th criterion

Xn* xn*n

Satisfaction Function for the 1st Criterion

Satisfaction Function for the 2nd Criterion

Satisfaction Function for the n-th Criterion

...

...

x1*m

...

...

x1*2 x1*1 X1 x11 x12 ... x1k ... x11 xi*1 Xi xi1 xi2 ... xik ... xii

xn*2 xn*1 xn1 xn2 ... xnk ...

...

xi*2

xn*m

...

...

xnn

Xn

Research objectives (1)

Research Objectives

Samples of customers participating in satisfaction surveys often appear to have a low degree of homogeneity The collective nature of MUSA method may result in poor quality measurements when it is used to analyse data of really farraginous samples

A methodological framework has been developed for the segmentation of an initial sample into more homogeneous subsets using clustering data mining techniques.
6

Research objectives (2)

Research Objectives
Lets say that we have two equal divided, farraginous groups of customers in our sample. The first is consisted of demanding customers and the second one has non-demanding customers. MUSA will produce a result describing neutral customers.

Demanding

Non-demanding

Neutral

Research objectives (3)

Research Objectives
In the case of different importance of the criteria (criteria wights) given by farraginous groups of customers will lead us to similar problems.

1st Group of Customers

50% 50%

2nd Group of Customers

50%

MUSA results

40%

35.0%

35.0%

40%

35.0%

35.0%

40%

30%

30%

30%

25.0%

25.0%

25.0%

25.0%

20%

15.0%

15.0%

20%

15.0%

15.0%

20%

10%

10%

10%

0%

0%

0%

Criterion 1 Criterion 2 Criterion 3 Criterion 4

Criterion 1 Criterion 2 Criterion 3 Criterion 4

Criterion 1 Criterion 2 Criterion 3 Criterion 4

Research objectives (4)

Research Objectives
MUSA gives as internal measures evaluating the quality of its results. The reliability evaluation of the results is mainly related to the following quantitative measures:

the fitting level to the customer satisfaction data (Average Fitting Index AFI and Overall Prediction Level -OPL)

the stability of the near-optimality analysis results (Average Stability Index ASI).

Research objectives (5)

Research Objectives
Predicted global satisfaction level ~ y1 y1 N11 C11 N21 C21 R21 R11 N12 C12 N22 C22 R22 ... ~ y2 R12 ... N1j C1j N2j C2j R2j ... ~ yj R1j ... ~ y N1 R1 C1 N2 R2 C2

The Overall Prediction Level (OPL) is based on the sum of the main diagonal cells of the prediction table, and it represents the percentage of correctly classified customers: : the number of customers that have declared to belong to global satisfaction level m1, while the model classifies them to level m2

N m1m2

Actual global satisfaction level

y2

...

...

yi

Ni1 Ci1

Ri1

Ni2 Ci2

Ri2

...

Nij Cij

Rij

...

Ni Ci

Ri

actual global satisfaction level m1, that the model classifies to level m2 : the percentage of customers of estimated global satisfaction level m1, that have declared to belong to level m2

Rm1m2 : the percentage of customers of

Cm1m2

...

...

Na1 Ca1

Ra1

Na2 Ca2

Ra2

...

Naj Caj

Raj

...

Na Ra Ca

OPL N m1m1
m1 1

m1 1 m2 1

m1m2

10

Data mining approach (1)

Data Mining Approach

Surveys data will be processed following a clustering (unsupervised learning or segmentation) approach.
Labeling: Based on demographic data

Data Mining
Data Preprocessing:

Data Selection: Valid Answers

Transformations

MUSA for each cluster

.. .. ..

Clusters (2,, n)

Data from questionnaires

Preprocessed Data

11

Data mining approach (2)

Transformations
Two kinds of transformation were developed in order to cover both cases: segments with different demanding level or with different given satisfaction criteria importance.
4 3 2 1 0 Total Cr 1 Cr 2 Cr 3 Cr 4
12

Customer 1 Customer 2 Customer 3 Customer 4

Data mining approach (3)

Transformations
For different Demand, DCr(i) nominal: If Cr(i)<(T-thr) Then DCr(i)=INT(Cr(i)-(T-thr)) If Cr(i)>(T+thr) Then DCr(i)=INT(Cr(i)-(T+thr)) If(T-thr)Cr(i)(T+thr) Then DCr(i)=0 where T is the declared total satisfaction of the customer, Cr(i) is the satisfaction regarding his/her satisfaction on i criterion and thr is a threshold. For different Criteria Weights, W1Cr(i), W2Cr(i), numeric: W1Cr(i)=ABS(Cr(i)-T) and W2Cr(i)= W1Cr(i)*(crc(a)-(ABS([(a-1)/2-T]/(a-1)/2)) where T is the declared total satisfaction of the customer, Cr(i) is the satisfaction regarding satisfaction on i criterion, crc(a) is a correction parameter and a is the number of global satisfaction levels

13

Data mining approach (4)

EM algorithm

The EM algorithm can be seen as a generalized version of K-means clustering

A hard membership is adopted in the K-means algorithm, (i.e., a data pattern is assigned to one cluster only). This is not the case with the EM algorithm, where a soft membership is adopted, (i.e., the membership of each data pattern can be distributed over multiple clusters)
14

Data mining approach (5)

EM algorithm

Similarly to K-means, first select the cluster parameters (A, A, P(A)) or guess the classes of the instances, then iterate

Each cluster A is defined by a mean (A) and a standard deviation (A) Samples are taken from each cluster A with a specified probability of sampling P(A)

Adjustment needed: we know cluster probabilities, not actual clusters for each instance. So, we use these probabilities as weights For cluster A:

Stop when the difference between two successive iteration becomes negligible (i.e. there is no improvement of clustering quality).

We measure that by:

15

Data mining approach (6)

EM algorithm

16

The experiment (1)

The experiment
For the development of the transformation procedure and for the evaluation of our research results we designed and we implemented an experiment.

Steps:
1. 2. 3. 4. Generation of synthetic data DataSet Generator Evaluate clusters generation WEKA DM tool Evaluate MUSA results on new segments Select the most appropriate transformations
17

The experiment (2)

Generation of synthetic data

A dataset generator, developed by our team for MUSA software evaluation, was used for the production of different data sets. The generator is able to produce data (answers to surveys) that have specific characteristics.

18

The experiment (3)

Generation of synthetic data

1st Data Set Produce two segments regarding different customers demand:
Weights Sets (500) (500) Satisfaction Levels A (Global) a(i) (per criterion) Criterion 1 25% Criterion 2 25% Criterion 3 25% Criterion 4 25%

Non Demanding Demanding

5 5

Non Demanding Demanding

Non Demanding Demanding

Non Demanding Demanding

2nd Data Set Produce two segments regarding different criteria weights:
Demand Sets (500) (500) Satisfaction Levels A (Global) a(i) (per criterion)

Criterion 1 Neutral
15% 35% 5 5

Criterion 2 Neutral
15% 35%

Criterion 3 Neutral
35% 15%

Criterion 4 Neutral
35% 15% 19

The experiment (4)

Evaluate clusters generation

WEKA, a Java Data Mining Tool developed in University of Waikato, was used for classes to clusters evaluation.

20

The experiment (5)

Evaluate clusters generation

Evaluation of 1st data set using DCr(i) transformation:
Assigned to Cluster Initial Classes

0
164 446

1
336 54

Cluster 0 <-- II Cluster 1 <-- I

Incorrectly clustered instances: 21.8%

I II

Evaluation of 2nd data set using W1Cr(i), W2Cr(i) transformations:

Assigned to Cluster Initial Classes

0
323 175

1
177 325

Cluster 0 <-- I Cluster 1 <-- II

Incorrectly clustered instances: 35.2%
21

I II

The experiment (6)

Evaluate MUSA results

Evaluation of 1st data set using DCr(i) transformation:
MUSA's Internal Quality Measures

Samples - Data Sets

Generator Data Set I
Generator Data Set II

Customers
500
500

AFI
97.17%
96.29%

ASI
96.59%
95.68%

OPL
95.00%
95.40%

Generator Data Set I + II

Cluster 1 --> I

1000
390

91.29%
95.21%

88.77%
96.13%

56.50%
86.15%

Cluster 0 --> II

610

94.75%

94.63%

92.95%

Criteria Weights
Samples - Data Sets
Generator Data Set I
Generator Data Set II Generator Data Set I + II Cluster 1 --> I Cluster 0 --> II

Demanding Indices Cr 4 Global Cr 1 Cr 2 Cr 3 Cr 4

Cr 1

Cr 2

Cr 3

25.04%
26.28% 25.34% 25.24% 25.54%

25.92%
25.43% 27.69% 26.23% 25.01%

24.61%
24.16% 23.61% 24.89% 24.86%

24.44%
24.13% 23.36% 23.64% 24.59%

-55.25%
41.82% -27.96% -52.34% 40.26%

-60.43%
59.962 -19.57% -57.61% 54.48%

-57.28%
65.41% -27.40% -57.92% 62.63%

-16.13%
30.55% -5.16% -20.09% 35.28%

-61.86%
12.99% -54.35% -65.90% 19.14%

22

The experiment (7)

Evaluate MUSA results

Evaluation of 2nd data set using W1Cr(i), W2Cr(i) transformations:
MUSA's Internal Quality Measures

Samples - Data Sets

Generator Data Set I
Generator Data Set II

Customers
500
500

AFI
96.45%
96.51%

ASI
97.64%
96.63%

OPL
89.60%
91.00%

Generator Data Set I + II

Cluster 0 --> I

1000
498

92.52%
94.35%

91.97%
94.85%

60.30%
80.92%

Cluster 1 --> II

502

95.32%

94.65%

93.43%

Criteria Weights

Demanding Indices Cr 4 Global Cr 1 Cr 2 Cr 3 Cr 4

Samples - Data Sets

Generator Data Set I
Generator Data Set II Generator Data Set I + II Cluster 0 --> I Cluster 1 --> II

Cr 1

Cr 2

Cr 3

16.45%
34.31% 25.31% 19.78% 33.91%

16.34%
34.23% 25.08% 17.65% 34.14%

33.67%
16.50% 24.35% 35.35% 16.11%

33.53%
14.96% 25.27% 27.22% 15.83%

28.92%
29.80% 27.06% 14.56% -6.48%

26.97%
-36.15% -7.87% 24.30% -17.50%

51.05%
32.27% 37.47% -20.66% -20.52%

33.58%
-49.20% 23.23% 22.10% 15.42%

36.29%
46.51% 31.45% 3.70%

25.44%

23

Application of results (1)

Application of results
The clustering procedure was applied on two real world surveys in order to be further evaluated. The measure of success would be the improvement of MUSAs internal quality measures through the proper segmentation of the initial sample. Survey 1: Policemen Satisfaction in Greece (sample: 1508, criteria: 8) Survey 2: Tourists Satisfaction in Skopelos Island (sample: 599, criteria: 5)

24

Application of results (2)

Survey 1
Evaluation of segments production using DCr(i) transformation:
MUSA's Internal Quality Measures Samples - Data Sets Initial Sample Customers 1508 AFI 93.02% ASI 78.02% OPL 56.63%

Cluster 0
Cluster 1 Cluster 0 Cluster 1

537
971 526 835

92.76%
95.16% 94.98% 96.07%

82.65%
74.46% 66.42% 83.14%

77.09%
72.81% 82.32% 88.38%

Cluster 2
Cluster 0 Cluster 1 Cluster 2

147
720 105 494

85.03%
96.87% 82.10% 96.03%

81.96%
80.19% 79.15% 85.06%

52.38%
90.83% 40.95% 90.28%

Cluster 3

189

93.51%

67.67%

82.01%

25

Application of results (3)

Survey 1
Evaluation of segments production using W1Cr(i), W2Cr(i) transformations:
MUSA's Internal Quality Measures Samples - Data Sets Customers AFI ASI OPL

Initial Sample
Cluster 0 Cluster 1

1508
579 929

93.02%
90.39% 95.60%

78.02%
79.65% 75.63%

56.63%
34.72% 77.40%

Cluster 0
Cluster 1 Cluster 2 Cluster 0

533
440 535 374

94.59%
90.29% 96.30% 93.99%

84.22%
80.52% 76.01% 82.40%

54.60%
47.50% 90.84% 42.51%

Cluster 1
Cluster 2 Cluster 3

188
547 399

85.98%
94.96% 97.04%

83.47%
74.41% 78.24%

27.13%
73.13% 91.73%

26

Application of results (4)

Survey 1
Evaluation of segments production using DCr(i) transformation:
Demanding Indices Clusters Initial Sample Cluster 0 Cluster 1 Global
33.46% 12.32% 35.00%

Cr 1 17.62% 12.67% 16.96%

Cr 2 24.72% 16.32% 32.08%

Cr 3
36.00% 29.34% 65.00%

Cr 4 11.82% 8.29% 6.07%

Cr 5
-2.80% 6.42% -22.76%

Cr 6 21.10% 13.47% 24.42%

Cr 7 36.00% 35.41% 36.00%

Cr 8
70.05% 74.14% 36.00%

Criteria Weights

Clusters
Initial Sample Cluster 0 Cluster 1

Cr 1
9.71% 9.16% 10.24%

Cr 2
10.63% 9.56% 11.78%

Cr 3
12.50% 11.32% 22.99%

Cr 4
9.07% 8.72% 9.05%

Cr 5
8.75% 8.55% 10.36%

Cr 6
10.14% 9.25% 10.59%

Cr 7
12.50% 12.50% 12.50%

Cr 8
26.71% 30.94% 12.50%

Labelling: More women belonging to the Greek Police forces seem to belong into Cluster 0. More lower-level officers seem to belong into Cluster 1.
27

Application of results (5)

Survey 2
Evaluation of segments production using DCr(i) transformation:
` Samples - Data Sets Initial Sample Customers 599 MUSA's Internal Quality Measures AFI 93.18% ASI 62.64% OPL 53.59%

Cluster 0
Cluster 1 Cluster 0 Cluster 1

214
385 186 342

89.48%
96.25% 89.04% 96.69%

60.51%
90.69% 59.65% 74.61%

57.48%
77.66% 56.45% 84.21%

Cluster 2

71

92.31%

93.86%

39.44%

28

Application of results (6)

Survey 2
Evaluation of segments production using W1Cr(i), W2Cr(i) transformations:
MUSA's Internal Quality Measures Samples - Data Sets Initial Sample Customers 599 AFI 93.18% ASI 62.64% OPL 53.59%

Cluster 0
Cluster 1 Cluster 0 Cluster 1

330
269 149 179

96.65%
89.84% 88.54% 92.60%

46.00%
68.95% 78.03% 54.06%

75.76%
29.74% 34.90% 34.64%

Cluster 2

271

97.50%

46.00%

76.01%

29

Application of results (7)

Survey 2
Evaluation of segments production using DCr(i) transformation:
Demanding Indices Clusters Initial Sample Cluster 0 Global -60.37% -64.50% Cr 1 -49.56% -54.00% Cr 2 -55.99% -54.00% Cr 3 -38.34% -54.00% Cr 4 -12.14% -54.00% Cr 5 -75.29% -54.00%

Cluster 1

-55.10%

-40.09%

-56.97%

-29.22%

-10.32%

-76.54%

Criteria Weights

Clusters Initial Sample

Cluster 0
Cluster 1

Cr 1 17.84%

Cr 2 20.45%

Cr 3 14.60%

Cr 4 11.24%

Cr 5 36.87%

20.00%
15.36%

20.00%
21.38%

20.00%
13.00%

20.00%
10.26%

20.00%
40.01%

Labelling: Tourists staying in hotels turn to belong into Cluster 1 while on the contrary the ones chose to stay in rooms to let seem to belong in cluster 0.
30

Conclusions (1)

Conclusions

Data Mining Clustering procedure led to more homogeneous segments of customers both in synthetic datasets and in real world surveys results. DCr(i) transformation seems to work better than W1Cr(i), W2Cr(i) transformations.

The labelling of the produced clusters segments is not always obvious Maybe more attention should be paid during the designing of the survey to include more demographical information.
31

Conclusions (2)

Future research
Some improvements regarding the data mining procedure may include:

Further experiments using the dataset generator evaluating the results should be undertaken. Real world surveys should be used as well. Other MUSA internal quality measures, recently proposed, should be also considered. The transformations regarding the different criteria weights should be improved, if it is possible.

Other or new similarity metrics should be studied. The labelling procedure should be thoroughly examined.
32

Thank you

33