AWTGMobileBenchmarkingService

Executive Summary
January 2013

www.awtg.co.uk

Content
OverviewandBackground
Equipment
Methodology
SampleResults

OverviewandBackground
AWTGhascarriedapilotmobileservicebenchmarking.Thebenchmarkingaimsatmultidimensionalandmultilevel
analysis

ScopeofWork

BenchmarkingfourmajoroperatorsintheUK
InabusyChristmasperiodandbusyareaofapproximately0.6km2 inLondonthroughmorethan12kmwalktest
Fullysynchronizedtestingusingstateofthearttestequipment anddedicatedservers
Forthefollowingscenarios
3Gvoiceandvariousdataservices
RecentlylaunchedLTE(4G)service
OutdoorandIndoortesting
ServicesonnewBorisbusandtraditionalLondonDoubleDeckerbus
PublicWiFi performanceanalysesagainstLTEnetwork

Page3

HardwareandSoftwareTools
AWTG used SwissQual benchmarking tool called Ranger for this project and the related post processing tools also
AWTG QoE and Broadband tester has been used to capture detailed log files especially for WiFi benchmarking
furthermore AWTGs spectrum interference analyzer tool has been used for the more detail analyses.

Qualityof
Experience

SwissQual
DiversityRanger

(SwissQual)

Qualityof
Experience

PostProcessing

4xGalaxyS2
2xO2SIM
2xTMobileSIM
2xThreeSIM
2xVodafoneSIM
4xHuaweiDonglesE398

SwissQual NQDI

MapInfo

WireShark

CascadePilot

(AWTG Tool)

Page4

Methodology
Threemaintestcasescoveredonthisprojectwhichincludeindoor,outdoorandbustests

2.4 km of Boris bus and traditional London

Double Decker bus route testing

1012 km (approximately) of pedestrian

outdoor walk testing

Multistoryinbuildingtesting

Page5

3GRadioSummaryReport(OutdoorWalkTest)
HeatMaps&Statistics
OP1

OP2

OP3

OP4

AverageRSCP
OP1

OP2

OP3

OP4

55
dBm

60
61.98

65
70
75

68.16
Min=127.26
Max= 21.45
SD=37.33

OP2hasthebestRSCPinthetestarea

68.78
Min=107.56
Max= 15.82
SD=26.04

Min=105.55
Max= 38.21
SD=26.57

70.12
Min=102.75
Max= 26.38
SD=38.92

Page6

3GRadioSummaryReport(OutdoorWalkTest)
HeatMaps&Statistics
OP1

OP2

OP3

OP4

AverageEc/Io

dB

OP1
0
2
4
6
8
10
12

8.04
Min=26.73
Max= 0.88
SD=9.32

OP3hasthebestEc/IoandthelowestSD

OP2

8.99
Min=26.76
Max= 1.39
SD=8.22

OP3

OP4

7.91
Min=31.192
Max= 0.112
SD=7.37

9.81
Min=26.8
Max= 0.51
SD=11.04

Page7

FTP(OutdoorWalkTest)
HeatMaps&Statistics
OP4FailedFTPUploadTest

AverageUplinkThroughput

Kbps

Min=11.1
Max= 1496.8

800
700
600
500
400
300
200
100
0

735

Min=11.1
Max= 2337 Min=11.5
Max= 2526
545
482

NA
OP1

OP2

OP3

OP4

FTPUploadTestSuccessRate

3000
2500

OP4FTPUplinkTest
Max=2526.3 Min=11.5Avg=482
SuccessfulTest=58FailedTest=47

73%

80%
57%

60%

55%

40%

Kbps

2000
1500

20%

1000
500

NA

0%
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105

LowsuccessrateandthroughputvalueforFTPUL

OP1

OP2

OP3

OP4

Page8

FTP(OutdoorWalkTest)
HeatMap&Statistics
OP4FTPDownlinkTest

Outdoor WalkTestFTPDownlinkSuccess
Rate
99%

100%

99%

OP2

OP3

OP4

100%
80%
60%
40%

29%

20%
0%
OP1

FTPDownlinkAverageThroughput
Min=235.3
Max= 6106.4 MiMin= 70.7
2,608 Max= 11502.3

4000

Kbps

3000
2000
1000

Min=179.2
Max= 1294.8

Min=31.9
Max= 10834.4
3,663

1,925

737

0
OP1

AcceptablesuccessrateandthroughputvalueforFTPDL

OP2

OP3

OP4

Page9

3GRadioSummaryReport(OutdoorWalkTest)
CallStatistics
100%
100%
98%
96%
94%
92%
90%
88%
86%
84%

Dropped

CallStatistic CompletedCallRate%
99%
96%

Failed

CallStatistic Dropped&FailedCall%
7.3%

8%
6%

90%

3.8%

4%

2.4%
1.4%

2%
0%

0%

0%

0%

0%

OP1

OP2

OP3

OP1

OP4

AccessibilityVoice
100%
100%
99%
98%
97%
96%
95%
94%
93%

OP2

OP3

OP4

Retainability Voice

100%

100%

100%

100%

OP3

OP4

100%
99%

97%

97.8%

98%

96%

97%
96%
95%

OP1

OP2

OP3

OP4

OP1

OP2

OverallradioperformanceforOP2islowerincomparisonwithother
operators

Page10

VoiceMOS(WalkTest)
HeatMap&Statistics
OP5MOSHeatMap

AverageMOS

5
4

3.37

3.23

3.41

OP2

OP3

3.45
2.82

3
2
1
0
OP1

Sec

OP5

AverageSetupTimeforcallscompleted

8
6

OP4

5.28

6.41

5.80

5.35
3.53

4
2
0
OP1

OP2

OP3

TheoverallMOSvalueforOP5ishigherthanotheroperators
howevercallsetuptimeishigher

OP4

OP5

Page11

HandoverStatistics
HeatMap&Statistics
100%

100%

98%

100%

100%

100%

100%

99.96%

OP1

OP2

OP3

OP4

100%

95%

95%

90%

90%

85%

3GOP4Handover

IntegritySuccessful3G
Handover

IntegritySuccessful2G
Handover

85%

NA

80%

80%
OP1

UMTS2100

OP2
GSM1800

OP3

OP4

UMTS900

GSM900

ServingTechnology
100%

100%
80%
60%

51.6% 48.4%

40%
20%

48.5%

47%

33.1%

27.5%
21.3%
0% 0%

4.2%

0% 0% 0%

9.6% 8.9%

0%
OP1

OP2

OP3

OP4

3GOP4istheonlyoperatorthathassomefailedhandovers

Page12

IndoorTest
Statistics
OP1

AverageThroughput(Kbps)

3GOP4MOS

OP2

OP3

OP4

WebBrowsingAverageThroughput

3000
2500

2,102

2,048

2000

1,635
1,377

1500

1,267

999

882

1000
411.6

267

500

342

732
123.2

1027.2

813.6
264

218

OP1

OP2

OP3

100%

100%
80%

OP4

100% 100%
79%

AverageSuccessRate
100%

94%

93%

100%
87%
76%

73%

100%

93%

93%

71%

60%
40%

27%

20%
0%

www.bbc.co.uk

www.google.com

www.msn.com

TheoverallwebexperienceforOP4usersislowercomparedwith
otheroperators

www.yahoo.com

Page13

BusTest
HeatMaps
3GOP4MOSNormalBus

3GOP4MOSBorisBus

3GOP4AverageRSCP
Walk

NormalBus

BorisBus

68
72
74
76

Sec

db

70
70.5

3GOP4MOSWalk

BorisBusAverageSetupTimeforcallscompleted
4.61

6.06

5.32

73.8

3.34

3.29

3.43

3.34

OP1

OP2

OP3

OP4

2
73.7

4
3

3.82

BorisBusAverageMOS

0
OP1

OP2

OP3

OP4

NosignificantdegradationbetweenNormalbusandBorisbus

Page14

LTEOPRadioReport
HeatMap&Statistics
LTEOPRSRP

RSRP Distribution
Ave =-91.7dbm Max= -65.9dbm Min=-121.4dbm
PDF

CDF

10%

120%

9%
100%

8%
7%

80%

5%

60%

4%
40%

3%
2%

20%

1%
0%
-46--44

-50--48

-54--52

-58--56

-62--60

-66--64

-70--68

-74--72

-78--76

-82--80

-86--84

-90--88

-94--92

-98--96

-102--100

-106--104

-110--108

-114--112

-118--116

3G

-122--120

-126--124

LTE

-130--128

0%

[dbm]

TherewasgoodLTEcoverageinmostofthecoveredarea

Page15

CDF

PDF

6%

LTEOPRadioReport
HeatMap&Statistics
LTEOPRSRQ

RSRQ distribution
Ave =-5.7dbm Max= -2dbm Min=-15.3dbm

5%

20%

0%

0%

3G

[db]

TherewasgoodLTERSRQinmostofthecoveredarea

Page16

CDF

40%

-4--3

10%

-6--5

60%

-8--7

15%

-10--9

80%

-12--11

20%

-14--13

100%

-16--15

25%

-18--17

120%

-20--19

LTE

CDF

30%

-22--21

PDF

PDF

LTE&3GOPsFTPDL/UL(WalkTest)
Statistics
FTPDownlinkAverageThroughput

FTPDownlinkSuccessRate
Min=1199.7
Max= 24648.1

9,423

10000

Kbps

Min=235.3
Max= 6106.4

6000
4000

2,608

Min=70.7
Max= 11502.3

Min=31.9
Max= 10834.4

100%

3GOP2

3GOP3

3GOP4

LTE

40%

1,925

20%

NA
3GOP1

0%

3GOP2

3GOP3

3GOP4

FTPUplinkAverageThroughput

8,825

NA
3GOP1

LTE
Min=2438.1
Max= 19030.7

10000

FTPUplinkSuccessRate
73%
57%

60%

6000
Min=11.1
Max= 1496.8

4000
2000

735

NA
3GOP1

Min=11.1
Max= 2337

545

Min=11.5
Max= 2526

3GOP3

20%

482

3GOP4

55%

40%

0%

3GOP2

100%

100%
80%

8000

Kbps

99%

60%

3,663

2000

100%

80%

8000

99%
100%

LTE

NA
3GOP1

3GOP2

3GOP3

3GOP4

FTPDL/ULThroughputforLTEOPismuchhigherthanall3GOPs

LTE

Page17

LTEOPRadioReport
HeatMaps
LTEOPRSRP

LTEOPPingTest

LTEOPFTPDL

LTE

3G

FailedpingtestsandlowerFTPthroughputmayhavebeencaused
bypoorcoverage

Page18

LTEOPand3GOP4RadioReport
HeatMaps
LTEOPRSRP

LTE

3GOP4RSCP

3G

3GOP4hasbettercoveragecomparedtoLTEOP

Page19

LTEOP&3GOP4FTPDownlink(WalkTest)
HeatMaps
LTEOPFTPDownlink

3GOP4FTPDownlink

FTPDownlinkThroughputforLTEOPismuchhigherthan3GOPs

Page20

LTEOP&3GOP4FTPUplink(WalkTest)
HeatMaps
LTEOPFTPUplink

3GOP4FTPUplink

FTPUplinkforLTEOPismuchhigherthan3GOPs

Page21

LTEOP&3GOP4WebThroughput(WalkTest)
HeatMaps
LTEOPWebThroughput

3GOP4WebThroughput

HigherwebthroughputforLTEOPcomparedto3GOPs

Page22

LTE&3GWebThroughputUsingHandset(WalkTest)
Statistics
LTEOP1

3GOP2

3GOP3

100

3GOP4

7.2

8.0

68.8

6.0

40
NA

7.7

15.1
NA

17.5

9.4 11.2

4.9 4.9 5.4

13.0

4.9 5.2 5.7

5.0

4.5

4.0
2.0

3.1
0.4

4.1 4.2

3.3 3.6 3.3

2.4

1.8
0.9

0.3

0.0

BBC

Google

MSN

Yahoo

98.8%

100%

95.3%

93.7%

BBC

Google

100%

99.6%

99.7%

92.3%

92%
90%

80%

LTEOP1

3GOP2

3GOP3

3GOP4

88%

LTEOP1

3GOP2

3GOP3

3GOP4

ms

94%

85%

Min=90
Max= 4943
SD=462

98.9%

96%
87.2%

Yahoo

AverageDelay

98%

90%

MSN

DelayTestSuccessRate

WebBrowsingSuccessRate
95%

3GOP3

TimetoLoad

60

20

3GOP2

WebThroughput

83.1

Sec

Kbps

80

LTEOP1

3GOP4

600
500
400
300
200
100
0

Min=53
Max= 4973
SD=561

480

Min=55
Max= 4918
SD=552

462

430

3GOP3

3GOP4

Min=40
Max= 877
SD=59

75

LTEOP1

3GOP2

WebthroughputforLTEOPismuchhigherthanall3GOps,alsoLTE
OPhasthelowestdelay

Page23

LTEandWiFiComparisonTest(AWTGBroadbandTester)
StaticTest
LTE

TestLocations

LTE

O2WiFi
OutdoorWiFi

FTPUplinkThroughput
12

10.9

FTPDownlinkThroughput

8
5.3
4.0

Location1

50
40

22.5

19.1
13.7

ms

Mbps

20

47.36

O2WiFi
OutdoorWiFi

26.87%

30%

43.75
32.29

Location2

PacketLoss%
23.87%

36.84
20%

16.12%
11.40%

20
10

Location2

Location1
LTE

30

10

Location1

Delay

36.2

30

Location2

OutdoorWiFi
O2WiFi

InternetSpeedDownload
40

5.4

LTE

6.7

6.1

OutdoorWiFi
O2WiFi

Mbps

10

Mbps

10

LTE

11.0

12

10.0

OutdoorWiFi
O2WiFi

10%

Location1

Location2

0%
Location1

OverallFTPDownlinkperformancewithLTEislowerthanWiFi whilethe
UploadPerformanceismuchhigher

Location2

Page24

LTEandWiFiComparisonTest(AWTGBroadbandTester)
StaticTestYouTubeSessionErrorAnalysis
LTE

TestLocations

O2WiFi
OutdoorWiFi

AverageNumberofErrors
189

200
150
100

172

Location1O2WiFiError
OutdoorWiFi

OutdoorWiFi
Location2O2WiFiError
5%

5%

7%

7%

125
71

13%

16%

50
0

70

1.2

64

63.4

60

1.02

Location1LTEError

Location2LTEError

1
45.18

45.8

Mbps

Sec

50

Location2

YouTubeThroughput

TimetoLoad

73%

75%
Location1

40
30
20

0.8

9%

0.4

10
0

Location2

16%

0.72

0.6

0.2

Location1

0.73

31%

44%

54%
20%
22%

Location1

4%

Location2

OverallYouTubeexperiencewithoudoor WiFi isbetterthanLTE

Page25

LTEandWiFiComparisonTest(AWTGBroadbandTester)
StaticTest
LTE

O2WiFi
OutdoorWiFi

LTE

Location1TimetoLoad

10

9.0

5.2

Mbps

Sec

5.3

5.2

4.8

Location1WebThrougput

7.3

OutdoorWiFi
O2WiFi

3.17
2.35

1.73

1.14

MSN

Yahoo

BBC

Location2TimetoLoad
6.4
4.5

7.6

6.9
4.3

Mbps

MSN

Yahoo

Location2WebThroughput

9.2

10

Sec

1.89

0
BBC

1.93

1.93

2.13

2.40

2.33
1.60
1.24

2
0

0
BBC

MSN

Yahoo

BBC

MSN

OverallperformanceofLTEandWiFi technologiesarecomparablein
bothlocationswhiletimetoloadforLTEisbetterthanWiFi

Yahoo

Page26

Thank You
Registered in UK No. 5793772HQ in London UK
Office: +44 (0) 208 799 0368
Fax: +44 (0) 208 728 9610
enquiries@awtg.co.uk