You are on page 1of 5

Performance Evaluation of VoIP QoE

Monitoring Using RTCP XR

Masataka Masuda1 , Kodai Yamamoto2 , Souhei Majima1 , Takanori Hayashi1 ,


and Konosuke Kawashima3
1
NTT Service Integration Laboratories
2
NTT Network Service System Laboratories
NTT Corporation,
3–9–11 Midori-cho Musashino-shi, Tokyo 180-8585, Japan
3
Tokyo University of Agriculture and Technology
2–24–16 Naka-cho Koganei-shi, Tokyo 184-8588, Japan

Abstract. In next generation networks (NGNs), not only bearer qual-


ity management but also quality of experience (QoE) management is
needed. One means of QoE monitoring is IETF RFC3611 RTCP XR
which is a protocol for reporting the quality information measured at
VoIP terminals. In our previous study, we propose a QoE monitoring
method of VoIP using the XR. However, XR is not a fully mature tech-
nology, and some report parameters are vaguely defined to implement
on a VoIP terminal. We focus on the XR parameters of network factors,
loss rate, round trip delay, and jitter. We clarified the applicability to
QoE monitoring of these XR parameters based on experimental results.

Keywords: RTCP XR, Quality, QoE, VoIP, Monitoring.

1 Introduction

The primary features of next generation networks (NGNs) are QoS control for
networks, and public switched telephone network replacement support for VoIP
networks. In conventional networks, the bearer quality is managed. In NGNs,
it is also essential for the end user to manage Quality of experience (QoE) [1].
The differences in VoIP terminals’ implementations strongly affect the QoE.
Therefore, it is essential to monitor the QoE in networks on a call-by-call basis [2],
and a quality measurement function should be implemented at VoIP terminals.
QoE monitoring technologies that are embedded in VoIP terminals are be-
ing studied with keen interest [3] One of these technologies is IETF RFC3611
RTCP XR, which is a protocol for reporting the quality of information measured
at VoIP terminals. We proposed a QoE monitoring method of VoIP by using XR,
and clarified the parameters which are needed for monitoring QoE [4]. Some XR
parameters are vaguely defined to implement on a VoIP terminal. The definitions
of report parameters have not been extensively evaluated in terms of their ap-
plicability to QoE monitoring. We evaluate the applicability to QoE monitoring
using XR parameters of network factors; loss rate, round trip delay, and jitter.

C.S. Hong et al. (Eds.): APNOMS 2009, LNCS 5787, pp. 435–439, 2009.

c Springer-Verlag Berlin Heidelberg 2009
436 M. Masuda et al.

PESQ measurement Packet capture equipment


equipment Signal in VoIP VoIP Signal out
terminal Hub Network emulator Hub terminal

Fig. 1. Configuration of VoIP system

2 QoE Monitoring Technology of VoIP

QoE monitoring technologies that are embedded in VoIP terminals are stan-
dardized in RFC3611 XR. XR is an extended protocol of RTCP for reporting
the quality information measured at VoIP terminals. RFC3611 defines the packet
format and many report parameters, but the applicability of the report is not de-
fined. In our previous study, We proposed QoE monitoring method using XR [4].
The scenario represents the case where the operator of a carrier network responds
to a user complaint. The operator can monitor the current QoE from end user
terminals, identify factors causing degradation of QoE.
To enable the use of this QoE monitoring method, we focus on network and
terminal factors in various report parameters of XR. However, the report pa-
rameters are currently implemented on VoIP terminals using various definitions,
for instance, RFC3611 does not clearly define the report parameters.

3 Definition of Network Parameters

RFC3611 defines the report format of network factors; loss rate, round trip delay
(RTD), and jitter. The loss rate is the fraction of RTP data packets from the
source lost. This value is calculated by dividing the total number of packets lost
by the total number of packets expected, multiplying the quotient by 256, and
taking the integer part. The RTD is the round trip time between RTP interfaces,
expressed in milliseconds. This value is measured using RTCP, and the time of
receipt of the most recent RTCP packet, minus the last sender report (LSR) time
reported in its SR, minus the delay since last SR (DLSR) reported in its SR.
The jitter is the transit time between two packets “above” the sequence number
interval. All jitter values are measured as the difference between a packet’s RTP
timestamp and the reporter’s clock at the time of arrival, measured in the same
units. RFC3611 does not clearly show this above sequence number and the jitter
calculation equation, and the jitter packet format must be a positive number,
but a negative number may be calculated in the above definition.

4 Performance Evaluation

4.1 Experimental Conditions

We conducted a performance assessment experiment to demonstrate the effec-


tiveness of QoE monitoring using XR. We evaluate two commercial VoIP ter-
minals equipped with XR. The configuration for the VoIP system under test is
Performance Evaluation of VoIP QoE Monitoring Using RTCP XR 437

outlined in Fig. 1. The SR and XR are sent together as a compound packet, and
the transmission interval is 5 seconds, which is recommended for a fixed mini-
mum interval. The reporting range of an XR packet is implemented as follows:
– Terminal A: The RTP packets received at the interval between receptions
of the XR packets (interval),
– Terminal B: The RTP packets received at the interval between the begin-
ning of the RTP session and the transmitting of the XR packet (accumulate).
The network conditions are emulated by an network emulator, and the number
of conditions (combinations of packet-loss rate (0–10%), delay (10–150 ms) and
jitter (0–20 ms)) was 54. We repeatedly fed speech samples with a total time
of 120 seconds into the system under all testing conditions. We evaluated the
performance of QoE monitoring using XR (the XR values) and compared them
with the actual values.

4.2 Performance Requirement


To evaluate the performance of QoE monitoring, we used the root-mean-square
error (RMSE), which was calculated using the difference between the XR values
and the actual values. We set the performance requirement as follows: “The
RMSE in estimating quality should be less than or equal to the mean of the 95%
confidence intervals in subjective speech quality assessment.” This means that
a QoE monitoring algorithm accurately estimates subjective quality comparable
with the statistical ambiguity of subjective assessment results. In our previous
study, we found that the mean for the 95% confidence interval in subjective
quality assessment under similar testing conditions was about 0.29 [4]. When this
was mapped onto the R-scale, it became about 11.5 at minimum. The R value is
a quality evaluation metric, and calculated using ITU-T Recommendation G.107
“the E-model” [5]. The loss rate and the RTD can be mapped onto R-scale.
The jitter cannot be mapped onto R-scale. Therefore we set the performance
requirement, “The degradation of listening quality is discoverd from the jitter.”
In this experiment, we discoverd the degradation of listening quality by using
perceptual evaluation of speech quality (PESQ) [6]. In the test condition at
maximum jitter, which does not affect degradation of the PESQ value, the jitter
value is 1.144 ms (average), and 1.720 ms (maximum). In the test condition at
minimum jitter, which affects degradation of the PESQ value, the jitter value is
1.837 ms (average), and 2.466 ms (maximum). The value of difference between
the previous two conditions is the boundary. Therefore, we set the performance
requirement as follows: “The RMSE value should be less than or equal to 0.689
ms (the mean jitter) or 0.746 ms (the maximum jitter).”

4.3 Experimental Results


To evaluate the performance of loss-rate measurement using integer mapping,
we compared the XR value with the actual value. The actual value is the loss
rate, which is measured using RTP packet capture. Figures 2 and 3 plot the
438 M. Masuda et al.

Actual round trip delay (ms)


10 10 400
Avg. error=10.810
Actual loss rate (%)

Actual loss rate (%)


350 Max. error=119.610
8 8
300 RMSE=21.086
6 6 250
200
4 4 150
Avg. error=-0.076 Avg. error=-0.128
Max. error= 0.384 Max. error= 0.390 100
2 2
RMSE= 0.117 RMSE= 0.180 50
RMSE(R-scale)= 0.441 RMSE(R-scale)= 0.676 RMSE(R-scale)=0.706
0 0 0
0 2 4 6 8 10 0 2 4 6 8 10 0 50 100 150 200 250 300 350 400
Loss rate (XR) (%) Loss rate (XR) (%) Round trip delay (XR) (ms)

Fig. 2. Loss rate (A) Fig. 3. Loss rate (B) Fig. 4. RTD (A)

Actual maximum jitter (ms)


10
Actual round trip delay (ms)

Actual maximum jitter (ms)


400 Loss condition 10
Avg. error=4.551 Loss condition
350 Max. error=109.472 8 No loss condition
8 No loss condition
300 RMSE=16.497 Avg. error=1.229 Avg. error=2.661
250 6 Max. error=22.078 6 Max. error=22.501
200
RMSE=2.456 RMSE=4.709
150 4 4
100
2 2
50
RMSE(R-scale)=0.570
0 0 0
0 50 100 150 200 250 300 350 400 0 2 4 6 8 10
0 2 4 6 8 10
Round trip delay (XR) (ms) Maximum jitter (XR) (ms)
Maximum jitter (XR) (ms)

Fig. 5. RTD (B) Fig. 6. Max. jitter (A) Fig. 7. Max. jitter (B)

performance of loss-rate measurement using XR, and show the RMSE. A plot
signifies the average loss rate under testing conditions. The measurement error
is calculated, the XR value minus the actual value. Figures 2 and 3 show that
both terminals satisfy the requirement, i.e., the RMSE was 0.441 and 0.676,
respectively. The XR values are less than the actual values, because the decimals
are dolloped when the measured loss rate maps onto the integer.
The XR value of RTD is measured using RTCP, but the transmission delay
of RTP packets must be measured to monitor the QoE. Therefore, we evalu-
ate the performance of the estimation of the RTP transmission delay using the
XR value. We define the actual RTD which is calculated by the average of one-
way delay of RTP packets plus the average of one-way delay of RTP packets
which are transmitted in reverse direction of the same time. The RTP packets
are the same target packets for measurement as those of XR. Figures 4 and 5
plots the performance of RTD measurement by XR. Figures 4 and 5 show that
both terminals satisfy the requirement, i.e., the RMSE was 0.706 and 0.570,
respectively. In the test conditions which is large delay variation, the maxi-
mum estimation error was large depending on the measurement method using
RTCP.
We define jitter as the relative delay between two packets always above se-
quence number one to use under various conditions: packet loss and packet re-
ordering. If there is no packet above sequence number one, the packet is excluded
from the jitter calculation. The jitter packet format in XR must take a positive
number, therefore the actual jitter is an absolute value and calculated by

Jn = |(tb(n) − ta(n) ) − (tb(n−1)) − ta(n−1) )|, (1)


Performance Evaluation of VoIP QoE Monitoring Using RTCP XR 439

Jn where is the jitter of nth packet, ta(n) is the transmitting-time of nth packet, tb(n)
is the receiving-time of nth packet.

The mean jitter measurement by the both terminals satisfies the requirement,
i.e., the RMSEs were 0.395 and 0.086. The measurement error was less than 1
ms. Figs. 6 and 7 plot the performance of the maximum jitter measurement with
XR. The plot symbols are classified based on the testing conditions, which does
or does not consist of the lost packets. The maximum jitter measurement by the
both terminals does not meet the requirement, i.e., the RMSEs were 2.456 and
4.709. Therefore, the degradation of listening quality is not detected using XR
value of the maximum jitter. Figures 6 and 7 show that the measurement error
is large in the test conditions, including the packet loss.

5 Conclusion
We evaluated the performance of QoE monitoring by using XR. We will study
QoE monitoring performance by using the terminal factors in the future.

References
1. ITU-T Rec. P.10/G.100 Appendix 1, Definition of QoE (January 2007)
2. ETSI TIPHON TS101 329-5 Annex E, Method for determining an equipment im-
pairment factor using passive monitoring (November 2000)
3. IETF RFC3611, RTP control protocol extended reports (RTCP XR) (November
2003)
4. Masuda, M., et al.: End-to-end quality management method for VoIP speech using
RTCP XR. IEICE Trans. on Communications E90-B(11)(November 2007)
5. ITU-T Rec. G.107, The E-model (March 2005)
6. ITU-T Rec. P.862, Perceptual evaluation of speech quality (PESQ) (February 2001)

You might also like