Definition KPIs EDGE

Definition of GPRS/EDGE End-to-End KPIs
Abstract
The document provides the definition of GPRS/EDGE end-to-end Key Performance Indicators (KPIs), which make a description of the end users perception of the service performance possible.
Introduction
Table of Contents 1 1.1 1.2 1.3 2 3 3.1 3.2 3.3 3.4 4 5 5.1 5.2 I TRODUCTIO SCOPE STRUCTURE REVISION HISTORY REMARKS O THE ATURE OF THE KPIS TERMI OLOGY USED FOR CATEGORIZI G THE KPIS ACCESSIBILITY RETAINABILITY INTEGRITY DISTINCTION BETWEEN KPIS AND PERFORMANCE INDICATORS (PIS) SERVICES DEFI ITIO S OF KPIS CATEGORIZATION OF THE KPIS ACCESSIBILITY KPIS 5.2.1 GPRS Unavailability 5.2.2 GPRS Attach Failure Ratio 5.2.3 GPRS PDP Context Activation Failure Ratio 5.2.4 Secondary PDP Context Activation Failure Ratio 5.2.5 GPRS Service Access Failure Ratio (for each service) 5.2.6 GPRS Accessibility Failure Ratio (for each service) 5.3 RETAINABILITY KPIS 5.3.1 GPRS PDP Context Cut-off Ratio 5.3.2 GPRS Data Transfer Cut-off Ratio 5.3.3 Cell Update Failure Ratio 5.3.4 Intra SGS RAU Failure Ratio 5.3.5 Inter SGS RAU Failure Ratio 5.3.6 PS Paging Failure Ratio 5.4 INTEGRITY KPIS 5.4.1 GPRS Attach Setup Time 5.4.2 GPRS PDP Context Activation Delay (per AP ) 5.4.3 GPRS Service Access Time (for each service) 5.4.4 GPRS Access Time (for each service) 5.4.5 GPRS Mean User Data Rate 5.4.6 Round Trip Time 5.4.7 Client-Server Interaction Time with Background Traffic 5.4.8 Packet Loss Ratio 5.4.9 Cell Update Time 5.4.10 Intra SGS RAU Time 5.4.11 Inter SGS RAU Time 6 7 8 8.1 8.2 9 10 OVERVIEW OF THE KPIS SUMMARY TERMI OLOGY ABBREVIATIONS GLOSSARY REFERE CES APPE DIX 4 4 4 5 6 6 6 7 7 7 9 11 11 12 12 13 15 17 19 24 26 26 27 29 31 32 34 35 35 36 37 41 42 44 46 47 49 50 52 53 54 56 56 57 58 59
Introduction
10.1
STATISTICAL GUIDELINES
59
Introduction
1
1.1
Introduction
Scope
One aspect of quality when it comes to GPRS/EDGE is the performance perceived by the end-user. Key Performance Indicators are those indicators that have big impact on the end-users perception of the service performance. Therefore the KPI definitions make a description of the end users perception of the performance possible. Additionally they enable operators to compare GPRS/EDGE performance between their own networks and with the performance of competitors networks. Purpose of this document is to define GPRS/EDGE end-to-end Key Performance Indicators (KPIs). In order to ensure comparable results when measuring KPIs the definition of each KPI has to outline what should trigger the start of the measurement and the stop of the measurement. This start and stop trigger can be certain signaling messages or a specific data packet. Therefore the definitions include these trigger points that are to be used when the KPIs are measured in a live network or in a lab environment.
The document does not: specify test cases that should be run in order to measure the end-to-end performance, give benchmark figures for the different KPIs, describe parameter settings for an optimized GPRS/EDGE network.
The above-mentioned items are covered in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing (see [1]), and in the Ericsson GPRS End-to-End Performance Guidebook (see [3]).
1.2
Structure
The document has the following structure: Nature of the KPIs Terminology used for categorizing the KPIs Definitions of the KPIs Appendix: Statistical Guidelines
Introduction
1.3
Revision History
Remarks on the Nature of the KPIs
Remarks on the Nature of the KPIs

Main focus of the KPI definitions is that the KPIs shall reflect the end-users perception of the service performance. The KPIs shall be measurable either with active or passive tests.
Active Measurements
Measuring with active tests means that the person who performs the tests actively generates traffic and only the performance for this generated traffic is measured. These measurements are typically limited to a certain period of time, the conditions of the system under test are known and detailed measurements and investigations are possible. Test cases for performing active measurements are defined in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1].
Passive Measurements
Measuring with passive tests means that the performance of traffic generated by all subscribers is continuously measured via counters in any node of the GPRS/EDGE network (refer to Measurements of GRPS End-to-End KPIs with Counters [6])1 or with specific tools for passive measurements. Detailed conditions are often unknown, e.g. cell load, number of used TS etc.
Terminology Used for Categorizing the KPIs

Terminology for grouping the KPI definitions follows the international ITU-T standards (E.800), in order to facilitate alignment of KPI definitions between Ericsson and vendors. Since standard ITU-T E.800 is valid for telephone network and ISDN only, the categorization of the GPRS/EDGE e2e KPIs definitions borrows the terminology used in the standard. This means that the KPI definitions are not standard-compliant, but that concepts deployed in the standard are transferred onto the grouping of the KPIs. According to ITU Recommendation E.800 [4] Quality of Service is: The collective effect of service performances, which determine the degree of satisfaction of a user of the service.
3.1
Accessibility
The ability of the user to obtain a service within specified tolerances and other given conditions. The accessibility of the service does include a chain of operations the user must be able to execute: the user must be able to access GPRS/EDGE as such, meaning he must be able to do a GPRS attach and a PDP context activation, and he must be able to access an IP service after he has an activated PDP context.
General remarks on measurement of KPIs with counters: The counters give information about the success ratios, but not about the failure cause code.
The accessibility KPIs are the most important set of KPIs regarding IP services, since the user is first of all interested in the fact if he is able to access the GPRS/EDGE network and the IP services or not. Accessibility KPIs are defined in chapter 5.2.
3.2
Retainability
Retainability indicates the ability of the user to keep a service once it was accessed under given conditions for a requested period of time. The retainability of a service or session does include that the user does not have to perform any additional manual operations that would not be necessary under stable network conditions, like for example manual re-activation of the PDP context. Retainability KPIs are defined in chapter 5.3.
3.3
Integrity
Integrity indicates the degree to which a service is provided without major interferences, once it was accessed. Integrity KPIs show the performance of successful service attempts. Even if a service was accessed successfully, the users perception of the performance may vary between very good and unacceptably bad. Integrity KPIs are defined in chapter 5.4.
3.4
Distinction between KPIs and Performance Indicators (PIs)

Some of the performance indicators are rated to be the primary performance indicators, because they reflect the end-users perception of the service performance in an outstanding manner. Therefore they are categorized as Key Performance Indicators, while the performance indicators that contribute to the measurement of a KPI are categorized as Performance Indicator (PI). This categorization into KPIs and PIs is deployed in cases in which such a distinction can be clearly drawn, i.e.:
Accessibility
KPI ACC 050 GPRS Accessibility Failure Ratio [%] PI ACC 010 GPRS Unavailability [%] PI ACC 020 GPRS Attach Failure Ratio [%] PI ACC 030 GPRS PDP Context Activation Failure Ratio [%] PI ACC 040 GPRS Service Access Failure Ratio [%]
Retainability
KPI RET 020 Data Transfer Cut-off Ratio PI RET 010 PDP Context Cut-off Ratio
Integrity
KPI I T 040 GPRS Access Time [s] PI I T 010 GPRS Attach Setup Time [ms] PI I T 020 GPRS PDP Context Activation Delay [ms] PI I T 030 GPRS Service Access Time [s]
Services
Services
The following services should be taken into consideration when defining KPIs and measuring them with specific test cases: GPRS access service WAP E-Mail Web browsing/HTTP FTP MMS Streaming
When measuring KPIs related to the above mentioned services in a live network file and e-mail sizes cannot be influenced, so that passive tests of KPIs will measure any file and e-mail size that appears in a live network. But when measuring the KPIs related to services with active tests the following file and e-mail sizes should be chosen: WAP: Fixed WAP pages to be downloaded have to be defined. WAP browser to be used has to be defined. E-Mail: Send e-mail: 120 Kbytes zip-file (e-Mail with attachment) Receive e-mail: 240 Kbytes zip-file (e-Mail with attachment) Send e-mail: 5 Kbytes plain text Receive e-mail: 5 Kbytes plain text No use of Java or Flash Web/HTTP : Fixed web pages to be downloaded have to be defined, e.g. www.ericsson.com Internet browser to be used has to be defined. Version of the internet browser has to be defined. FTP: Uplink: 100 Kbytes Downlink: 200 Kbytes
Services
Timeouts for Data Transfer per Service:
The timeout values summarized in Table 1 are only applicable for active measurements of KPIs related to services. The timeout values for e-mail download comprise only the download of the data content (e-mail text, or e-mail text plus attachment) but not the access procedure preceding the actual download of the data content, see chapter 5.2.4.
Data Transfer (content) FTP UL File Size Timeout 40kB 64s 120kB 192s FTP DL 40kB 32s 240kB 192s 120kB 192s E-Mail UL 5kB text + 5kB 120 kB attachment 28s 200s E-Mail DL 5kB text + 5kB 240 kB attachment 24s 196s HTTP frozen page
240kB 192s
Table 1: Timeout values for data transfer
Remark
Purely calculated timeout values for uploading 5kB and downloading 5kB would result in 8s and 4s respectively. But the actual timeout values being applied while measuring service-related KPIs with small files will be higher, because of the slow start behavior of TCP or TCP timeouts which increase the delay to several seconds. Therefore the timeouts are calculated as follows: 20s + <download size> / UL performance [5kbps] 20s + <download size> /DL performance [10kbps]
Timeouts are calculated assuming DL/UL via one timeslot (worst-case scenario). Example: FTP DL 240kB, 240 kB * 8 = 1920 Kbits, 1920 Kbits/10 kbps = 192 seconds
Definitions of KPIs
5
5.1
Definitions of KPIs
Categorization of the KPIs
The KPI definitions are divided into three categories as shown below:
QoS KPIs
Accessibility
GPRS/UMTS unavailability Attach failure ratio PDP context activation failure ratio Service access failure ratio e2e accessibility failure ratio
Retainability
PDP context cut-off ratio Data transfer cut-off ratio Cell Update Failure Ratio Intra SGSN RAU Failure Ratio Inter SGSN RAU Failure Ratio SGSN Paging Failure Ratio
Integrity
Attach setup time PDP context activation delay Service access time e2e access time Mean user data rate Round Trip Time Packet Loss Ratio Cell Update Time Intra SGSN RAU Time Inter SGSN RAU Time
Definitions of KPIs
5.2
5.2.1
Accessibility KPIs
GPRS Unavailability
TAG: ACC 010
Definition:
The PI GPRS/EDGE Unavailability [%] denotes the probability that GPRS/EDGE is not active in the cell used by a subscriber.
Trigger Point(s):
Check if the GPRS/EDGE-specific signaling (SI13) exists on cell selection. Check if the signaling message can be read out and if the required signaling exists on BCCH or on PBCCH.
GPRS Unavailability [%] =

Remark:
# unsuccessful attempts to read SI 13 100% total # attempts to read SI 13
If this PI is measured for example during drive tests information is provided about the GPRS/EDGE coverage on network level and coverage problems in special. Since this PI can only be measured in live networks the PI is measured for a random set of cells in the live network, since the GPRS/EDGE availability is checked before any test case can be executed.
Active Measurement:
PI not covered in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing.
Passive Measurement:
There is no means of passive measurement (counter or Moniq tool) for this PI.
Definitions of KPIs
5.2.2
GPRS Attach Failure Ratio

TAG: ACC 020
Definition:
The PI GPRS Attach Failure Ratio [%] denotes the probability that a subscriber cannot attach to the GPRS/EDGE network. The timeout interval for the attach procedure is 30 seconds, meaning: if the attach procedure was not completed after 30 seconds it is considered as failure. Note that the timeout value of 30 seconds is no system internal value.
GPRS Attach Failure Ratio [%] =

Trigger Point(s):
# unsuccessful GPRS attach attempts 100% total # of GPRS attach attempts
Check if the whole GPRS attach procedure from sending the Attach Request and receiving the Attach Accept message was executed successfully (Signaling messages 1 and 8 in Figure 1).
MS
BSS
new SGSN
old SGSN
GGSN
EIR
new MSC/VLR
HLR
old MSC/VLR
1. Attach Request 2. Identification Request 2. Identification Response 3. Identity Request 3. Identity Response 4. Authentication 5. IMEI Check 6a. Update Location 6b. Cancel Location 6c. Cancel Location Ack 6d. Insert Subscriber Data 6e. Insert Subscriber Data Ack 6f. Update Location Ack 7a. Location Update Request 7b. Update Location 7c. Cancel Location 7d. Cancel Location Ack 7e. Insert Subscriber Data 7f. Insert Subscriber Data Ack 7g. Update Location Ack 7h. Location Update Accept
8. Attach Accept 9. Attach Complete 10. TMSI Reallocation Complete
Figure 1: Signalling Messages during Attach Procedure
Definitions of KPIs
Remarks:
It might occur that the MS sends more than one Attach Request towards the SGSN, since retries are necessary. Maximum four retries are possible and the guard time between two requests is 15 seconds. These retries should not have impact on the Attach Failure Ratio, since only one GPRS/EDGE Attach Request message should be counted in the calculation.
Prerequisites:
GPRS/EDGE has to be active in the cell used by a subscriber (PI ACC 010)
Active Measurement:
This PI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: GPRS Attach,
TAG: ST 010.
This PI can be measured with a counter in the /// SGSN.
Definitions of KPIs
5.2.3
GPRS PDP Context Activation Failure Ratio

TAG: ACC 030
Definition:
The PI GPRS PDP Context Activation Failure Ratio [%] denotes the probability that a subscriber cannot activate a PDP context. The timeout interval for the PDP context activation procedure is 30 seconds, meaning: if the activation procedure was not completed after 30 seconds it is considered as failure. Note that the timeout value of 30 seconds is no system internal value.
PDP Context Activation Failure Ratio [%] =
# unsuccessful PDP context activation attempts 100% total # of PDP context activation attempts
Trigger Point(s):
Check if the PDP context activation procedure from sending the Activate PDP Context Request and receiving the Activate PDP Context Accept message was executed successfully. (Signaling messages 1 and 7 in Figure 2.)
MS
BSS
2G-SGSN
2G-GGSN
1. Activate PDP Context Request
2. Security Functions 4. Invoke Trace 5. Create PDP Context Request 5. Create PDP Context Response 6. BSS Packet Flow Context Procedures
7. Activate PDP Context Accept
Figure 2: Signaling Messages during the PDP Context Activation Procedure

Remarks:
It might occur that the MS sends more than one PDP Context Activation Request towards the SGSN, since retries are necessary. Maximum four retries are possible and the guard time between two requests is 30 seconds. These retries should not have impact on the Activation Failure Ratio, since only one GPRS PDP Context Activation Request message should be counted in the calculation.
Definitions of KPIs
PDP Context Activations that failed, because they were done towards an undefined APN should not be included in the calculation.
Prerequisites:
The MS is already GPRS attached. (PI ACC 020)

Active Measurement:
This PI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: PDP Context Activation,
TAG: ST020.
This PI can be measured with a counter in the /// SGSN-G 3.0.
Definitions of KPIs
5.2.4
Secondary PDP Context Activation Failure Ratio

TAG: ACC 035
Definition:
The PI Secondary PDP Context Activation Failure Ratio [%] denotes the probability that a subscriber cannot activate a Secondary PDP context. The timeout interval for the Secondary PDP context activation procedure is 30 seconds, meaning: if the activation procedure was not completed after 30 seconds it is considered as failure. Note that the timeout value of 30 seconds is no system internal value.
PDP Context Activation Failure Ratio [%] = # unsuccessful PDP context activation attempts 100% total # of PDP context activation attempts
Trigger Point(s):
Check if the PDP context activation procedure from sending the Activate Secondary PDP Context Request and receiving the Activate Secondary PDP Context Accept message was executed successfully. (Signaling messages 1 and 7 in Figure 2 and Figure 4.)
MS BSS 2G-SGSN 2G-GGSN
1. Activate Secondary PDP Context Request
2. Security Functions 3. Create PDP Context Request 3. Create PDP Context Response 5. BSS Packet Flow Context Procedures
7. Activate Secondary PDP Context Accept
Figure 3: Secondary PDP Context Activation Procedure for GPRS
Definitions of KPIs
MS
UTRAN
3G-SGSN
3G-GGSN
1. Activate Secondary PDP Context Request
3. Create PDP Context Request 3. Create PDP Context Response 4. Radio Access Bearer Setup 6. Update PDP Context Request 6. Update PDP Context Response
7. Activate PDP Context Accept
Figure 4: Secondary PDP Context Activation Procedure for UMTS
Remarks:
It might occur that the MS sends more than one PDP Context Activation Request towards the SGSN, since retries are necessary. Maximum four retries are possible and the guard time between two requests is 30 seconds. These retries should not have impact on the Activation Failure Ratio, since only one GPRS PDP Context Activation Request message should be counted in the calculation.
Prerequisites:
The MS is already GPRS attached and PDP context activated. (PI ACC 020 & PI ACC 030)
Active Measurement:
TAG: ST020.
Definitions of KPIs
5.2.5
GPRS Service Access Failure Ratio (for each service)

TAG: ACC 040
Definition:
The PI GPRS Service Access Failure Ratio [%] denotes the probability that a subscriber cannot access the service successfully, meaning that the data transfer of the content could not be started.3
Service Access Failure Ratio [%] =
# unsuccessful service access attempts 100% total # of service access attempts
The trigger points as well as the timeout values have to be defined per service for PI ACC 040.
Trigger Point(s):
The trigger points are defined per service: FTP (UL) Start trigger: MS sends its port (signaling message 2 in Figure 5). Stop trigger: Sending of the first data packet (message 11 in Figure 5).
Figure 5: FTP UL trace Note: The actual start trigger is the TCP SYN for port 21. If this SYN is not send the port is not opened and the service could not be accessed. This signalling message is not visible in the above log since the port was opened much earlier.
Content denotes different things depending on the service that is accessed. In case of an FTP session content is a file, in the case of an http session a web page and content of an E-Mail session is the text of the mail and so on.
Definitions of KPIs
FTP (DL) Start trigger: MS sends its port (signaling message 1 in Figure 6). Stop trigger: Reception of the first data packet (message 10 in Figure 6).
Figure 6: FTP DL trace Note: The actual start trigger is the TCP SYN for port 21. If this SYN is not send the port is not opened and the service could not be accessed. This signalling message is not visible in the above log since the port was opened much earlier.
E-Mail POP3 (Receiving) Start trigger: TCP SYN sent by the MS (signaling message 2 in Figure 7). Stop trigger: Reception of the first data packet containing mail content by the MS (message 18 in Figure 7)
Figure 7: E-Mail POP3 trace
Definitions of KPIs
E-Mail SMTP (Sending) Start trigger: TCP SYN sent by the MS (signaling message 2 in Figure 8). Stop trigger: Sending of the first data packet containing mail content (message 15 in Figure 8).
Figure 8: E-Mail SMTP trace
HTTP Start trigger: TCP SYN sent by the MS (signaling message 2 in Figure 9). Stop trigger: Reception of the first data packet and OK for http 1.1 or 1.0 (message 7 in Figure 9).
Figure 9: HTTP trace
Service Access Timeouts:
The timeout values summarized in Table 2 are only applicable for active measurements of the PI Service Access Failure Ratio.
Service Access FTP Timeout 30s E-Mail 60s HTTP 10s per attempt
Table 2: Service Access Timeouts
Definitions of KPIs
These timeout values are no system internal values but are expected to mirror the user behavior. If for example - an ftp file-transfer did not start after 30 seconds the end-user will stop this transaction. Similarly, if the end-user does not perceive that the loading of a web page started after 10 seconds he will try to reload the page. But if these reload attempts are not successful he will stop accessing the web page.
Remarks:
The above-mentioned trigger points for the different services are defined on a rather technical level and not from a total end-users point of view. An end-user would consider a service access as successful if the complete data transfer was successful. Measuring the PI Service Access Failure Ratio related to the transfer of the complete data content would mean that the measured ratio and as well the service access delay would depend on the file size.
Prerequisites:
The MS is already GPRS attached. (PI ACC 020) and a PDP Context is activated (PI ACC 030)
Active Measurement:
This PI can be measured by executing the following test cases described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: D S Lookup, FTP UL/DL, WAP Traffic, WWW Traffic, E-mail Traffic, TAG: SET 010, TAG: SET 020, TAG: SET 030, TAG: SET 040, TAG: SET 050.
This PI can be measured with the Moniq tool: WAP Connection Establishment Success Ratio (5.2.5.1) Web connection failure reasons: aborts, protocol error, congestion POP3, Mail Server Connection Success Ratio (5.2.5.2) Domain Name Service Query Success Ratio (5.2.5.3)
5.2.5.1
WAP Connection Establishment Success Ratio TAG: ACC WAPconnect
The number of successful connection establishments divided by the total number of connection attempts.
Definitions of KPIs
5.2.5.2
POP3, Mail Server Connection Success Ratio TAG: ACC POP3connect
The number of successful connection establishments divided by the total number of connection attempts.
5.2.5.3 Domain Name Service Query Success Ratio TAG: ACC DNS
The number of successful domain name (DNS) resolutions divided by the total number of DNS requests. A successful resolution means that the server has found the name and has responded to the request with a valid name.
Definitions of KPIs
5.2.6
GPRS Accessibility Failure Ratio (for each service)

TAG: ACC 050
Definition:
The KPI GPRS e2e Accessibility Failure Ratio [%] denotes the probability that a subscriber cannot access the service successfully, meaning that the data transfer of the content could not be started due to a failure that occurred within the complete chain of service access: GPRS Availability Attach PDP Context Activation Service Access
AccessFailureRatio[%]= [1 (1 PI ACC 010) (1 PI ACC 020) (1 PI ACC 030) (1 PI ACC 040)]
Trigger Point(s):
Trigger points and timeout values that are defined for the involved PIs: GPRS Unavailability (PI ACC 010) GPRS Attach Failure Ratio (PI ACC 020) GPRS PDP Context Activation Failure Ratio (PI ACC 030), and GPRS Service Access Failure Ratio (for each service) (PI ACC 040)
are valid for measuring GPRS e2e Accessibility failure ratio. See also Figure 15: Overview of the KPIs.
Example:
Assuming 1000 measurements and the following measured PIs ACC 010, ACC 020, ACC 030 and ACC 040: GPRS Unavailability (PI ACC 010): 0% GPRS was available all the time 1000 samples remain for the attach procedure GPRS Attach Failure Ratio (PI ACC 020): results in 700 samples for PDP context activation GPRS PDP Context Activation Failure Ratio (PI ACC 030): results in 595 samples for the service access, and 30%
15%
(PI ACC 040): 10% 535.5 service access attempts out of 1000 were successful and in 464.5 cases the service could not be access.
Definitions of KPIs
Access FailureRatio [%] = [1 (1 PI ACC 010) (1 PI ACC 020) (1 PI ACC 040) (1 PI ACC 060)] = [1 (1 0) * (1 0.3)* (1 0.15) * (1`0.1)]*100% = [1 (1* 0.7 * 0.85* 0.9)]*100%
] = [1 0.5355 *100% = 46.45%

Active Measurement:
KPI is not covered in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing.
This KPI can be measured with the Moniq tool: Web/FTP/E-mail service interrupts
Note:
The Web and SMTP success ratios are measured by dividing the successful service completions with the total number of service attempts (including those cases where the service access was already unsuccessful). The reason for this small difference is that the service access and data transfer are not separated in Moniq for these services.
Definitions of KPIs
5.3
5.3.1
Retainability KPIs
GPRS PDP Context Cut-off Ratio
TAG: RET 010
Definition:
The PI GPRS PDP Context Cut-off Ratio [%] denotes the probability that a PDP context is deactivated without being initiated by the MS.
PDP Context Cut Off Ratio [%] =
# PDP context losses 100% total # activated PDP contexts
Trigger Point(s):
Different trigger points for a PDP Context Deactivation not initiated by the MS are possible: SGSN failure or GGSN failure on which the PDP context will be deactivated by the SGSN or GGSN respectively. Expiry of the Mobile ot Reachable Timer leads to PDP Context Deactivation and is therefore another reason for PDP context cut-off.
Prerequisites:
Active Measurement:
PI not covered in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing.
This PI can be measured with the help of counters in the /// GGSN.
Definitions of KPIs
5.3.2
GPRS Data Transfer Cut-off Ratio

TAG: RET 020
Definition:
The KPI GPRS Data Transfer Cut-off Ratio [%] denotes the probability that a data transfer is incomplete after a timeout period or the transfer is aborted completely.
Data Transfer Cut Off Ratio [%] =
# incomplete data transfers 100% total # data transfers
Trigger Point(s):
1. Transfer abort file transfer is not complete. 2. PDP context cut-off (PI RET 010) file transfer is not complete. 3. FTP: Timeout counter of 90 seconds is triggered if no data reaches the receiver during 90 seconds file transfer is not complete.
Remarks:
In case of a data transfer cut-off it should be determined how much data was transmitted before this cut-off.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: Round Trip Time, FTP UL/DL, WAP Traffic, WWW Traffic, E-mail Traffic, TAG: TT 020, TAG: SET 020, TAG: SET 030, TAG: SET 040, TAG: SET 050.
This KPI can be measured with the Moniq tool: WAP Object Download Success Ratio (5.3.2.1) Web Object Download Success Ratio (5.3.2.2) FTP Download Success Ratio (5.3.2.3) POP3, E-mail Download Success Ratio (5.3.2.4) SMTP, E-mail Send Success Ratio (5.3.2.5)
Definitions of KPIs
5.3.2.1
WAP Object Download Success Ratio TAG: RET WAPobject
The number of successfully finished and acknowledged object downloads divided by all download attempts. Connection-mode WAP is measured. Precondition: established WAP session.
5.3.2.2 Web Object Download Success Ratio TAG: RET WEB
The number of successfully finished HTTP object downloads divided by all HTTP download attempts.
5.3.2.3
FTP Download Success Ratio TAG: RET FTP
The number of successfully finished file downloads divided by the number of all FTP download attempts. Condition: connection to FTP server was successful.
5.3.2.4
POP3, E-mail Download Success Ratio TAG: RET POP3download
The number of successfully finished mail downloads divided by all e-mail download attempts. Condition: connection to the server was successful, and a request to download (at least) one mail was initiated. This last condition is important since many sessions just check the mailbox and do not attempt any mail download.
5.3.2.5
SMTP, E-mail Send Success Ratio TAG: RET SMTPdownload
The number of successfully sent e-mails divided by the total number of attempts using the SMTP protocol. Web-based e-mail services are not considered. (Note: all common email applications Netscape, Outlook, and mobile phone integrated clients use the SMTP protocol.)
Definitions of KPIs
5.3.3
Cell Update Failure Ratio

TAG: RET 030
Definition:
The GPRS/EDGE KPI Cell Update Failure Ratio [%] denotes the probability that a Cell Update procedure fails.
Cell Update Failure Ratio [%] =

Note:
# unsuccessful Cell Update attempts 100% total # of Cell Update attempts
Since a cell update can only be observed in READY state the cell update failure ratio can be measured best during an ongoing FTP session.
Trigger Point(s):
The cell update procedure failed when the data transfer is not continued (sending or receiving) after the MS entered the new cell.
MS
BSS
SGSN
DL-UNITDATA (old cell)
UL-UNITDATA (old cell) DL-UNITDATA (old cell) DL-UNITDATA (old cell), TCP retransmission! UL-UNITDATA (new cell) FLUSH-LL (old BVCI and new BVCI) FLUSH-LL-ACK DL-UNITDATA (new cell)
Figure 10: Signalling diagram for cell update during DL FTP

Prerequisites:
The MS is already GPRS attached. (PI ACC 020) and a PDP Context is activated (PI ACC 030). Two different cells belonging to the same RA configured with GPRS/EDGE capability.
Definitions of KPIs
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: Cell Update during Downlink FTP, TAG: MT 010.
This KPI can be measured by executing the UDP based test solution: Transport Tests, IP layer outage as described in detail in Proposed Extensions to GPRS e2e performance Testing [2].
There is no means of passive measurement (counter or Moniq tool) for this KPI.
Definitions of KPIs
5.3.4
Intra SGSN RAU Failure Ratio

TAG: RET 040
Definition:
The KPI Intra SGSN RAU Failure Ratio [%] denotes the probability that an Intra SGSN RAU procedure fails.
Intra SGS RAU Failure Ratio [%] =

Trigger Point(s):
# unsuccessful Intra SGS RAU attempts 100% total # of Intra SGS RAU attempts
Check if the SGSN RAU procedure from sending the Routing Area Update Request and receiving the Routing Area Update Accept message was executed successfully. (Signaling messages 1 and 3 in Figure 11.)
MS
BSS
SGSN
1. Routeing Area Update Request 2. Security Functions 3. Routeing Area Update Accept C1 4. Routeing Area Update Complete
Figure 11: Intra SGSN RAU Procedure
Prerequisites:
The MS is already GPRS attached (PI ACC 020) and a PDP Context is activated (PI ACC 030). Two different cells belonging to different RAs served by the same SGSN configured with GPRS/EDGE capability.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: Intra SGS RA Update during Downlink FTP, TAG: MT 020
This KPI can be measured with counters in the /// SGSN.
Definitions of KPIs
5.3.5
Inter SGSN RAU Failure Ratio

TAG: RET 050
Definition:
The KPI Inter SGSN RAU Failure Ratio [%] denotes the probability that an Inter SGSN RAU procedure fails.
Inter SGS RAU Failure Ratio [%] =

Trigger Point(s):
# unsuccessful Inter SGS RAU attempts 100% total # of Inter SGS RAU attempts
Check if the SGSN RAU procedure from sending the Routing Area Update Request and receiving the Routing Area Update Accept message was executed successfully. (Signaling messages 1 and 11 in Figure 12.)
MS
BSS
new SGSN
old SGSN
GGSN
HLR
1. Routeing Area Update Request 2. SGSN Context Request 2. SGSN Context Response
3. Security Functions 4. SGSN Context Acknowledge
5. Forward Packets 6. Update PDP Context Request 6. Update PDP Context Response 7. Update Location 8. Cancel Location 8. Cancel Location Ack 9. Insert Subscriber Data 9. Insert Subscriber Data Ack 10. Update Location Ack
11. Routeing Area Update Accept
12. Routeing Area Update Complete
Figure 12: Inter SGSN RAU Procedure
Prerequisites:
Definitions of KPIs
The MS is already GPRS attached (PI ACC 020) and a PDP Context is activated (PI ACC 030). Two different cells belonging to different RAs served by the different SGSNs configured with GPRS/EDGE capability.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: Inter SGS RA Update during Downlink FTP, TAG: MT 030.
Definitions of KPIs
5.3.6
PS Paging Failure Ratio

TAG: RET 060
Definition:
The KPI PS Paging Failure Ratio [%] denotes the probability that an SGSN Paging procedure fails.
SGS PS Paging Failure Ratio [%] =

Trigger Point(s):
# unsuccessful SGS Paging attempts 100% total # of SGS Paging attempts
The network shall initiate the paging procedure for GPRS services using P-TMSI when GMM signaling messages or user data is pending to be sent to the MS while the Mobile Reachable timer is running. The network may page only GPRS MSs, which are GMMREGISTERED and identified by a local P-TMSI. The SGSN requests the RR sublayer to start paging. Upon reception of a paging indication, the MS shall respond to the paging with any LLC frame. [7] Check if the Paging for GPRS Services procedure from sending the Paging Request by the SGSN and receiving any LLC frame send by the mobile was executed successfully.
Prerequisites:
The MS is already GPRS attached (PI ACC 020) and a PDP Context is activated (PI ACC 030).
Active Measurement:
KPI not covered in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing.
Definitions of KPIs
5.4
5.4.1
Integrity KPIs
GPRS Attach Setup Time
TAG: INT 010
Definition:
The PI GPRS Attach Setup Time [ms] denotes the length of the time period it takes to attach to the GPRS/EDGE network. Note: Difference between an attach of a known subscriber and an unknown subscriber will be reflected in the length of the time period indicating the Attach Setup Time. In case of an attach of an unknown subscriber (meaning that the SGSN has changed since the GPRS detach, or if it is the very first attach of the mobile to the network), the SGSN contacts the HLR in order to receive the subscriber data. The attach setup time of an unknown subscriber will be slightly longer than the one of a known subscriber.
Trigger Point(s):
Measurement of the time [ms] from sending the Attach Request and receiving the Attach Accept message, see Figure 1.
Remarks:
While determining the average Attach Setup Time only successful attach attempts are included in the calculations. Failed attach attempts should not be part of the calculation denoting a timeout period of 30 seconds. Note that the timeout value of 30 seconds is no system internal value.
Prerequisites:
GPRS/EDGE has to be active in the cell used by a subscriber (PI ACC 010)
Active Measurement:
This PI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: GPRS Attach,
TAG: ST 010.
Definitions of KPIs
5.4.2
GPRS PDP Context Activation Delay (per APN)

TAG: INT 020
Definition:
The PI GPRS PDP Context Activation Delay Time [ms] denotes the length of the time period it takes to activate a PDP context.
Trigger Point(s):
Measurement of the time [ms] from sending the Activate PDP Context Request and receiving the Activate PDP Context Accept message, see Figure 2.
Remarks:
While determining the average PDP Context Activation Delay only successful activation attempts are included in the calculations. Failed activation attempts should not be part of the calculation denoting a timeout period of 30 seconds. Note that the timeout value of 30 seconds is no system internal value. The PDP Context Activation Delay should be determined per APN.
Prerequisites:
The MS is already GPRS attached. (PI ACC 020).

Active Measurement:
TAG: ST 020
Definitions of KPIs
5.4.3
GPRS Service Access Time (for each service)

TAG: INT 030
Definition:
The PI GPRS Service Access Time [s] denotes the length of the time period it takes to access a service successfully.
Trigger Point(s):
Measurement of the time [s] according to the trigger points defined for PI ACC 040 (see Figure 5 - Figure 9).
Remarks:
While determining the Access Time for a specific service only successful access attempts are included in the calculations. Failed access attempts should not be part of the calculation. The Access Time should be determined per service, since the actual access time depends strongly on the nature of the service. For e-mail and ftp the transmission rate is of higher interest than the actual service access time.
Remarks:
The above-mentioned trigger points for the different services are defined on a rather technical level and not from a total end-users point of view. An end-user would be rather interested in the time needed for transferring the complete data content. Measuring the PI Service Access Time related to the transfer of the complete data content would mean that the service access delay would depend on the file size.
Prerequisites:
Active Measurement:
This PI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: D S Lookup, FTP UL/DL, WAP Traffic, WWW Traffic, E-mail Traffic, TAG: SET 010, TAG: SET 020, TAG: SET 030, TAG: SET 040, TAG: SET 050.
Definitions of KPIs
This PI Service Access Time can be measured with the Moniq tool: WAP Connection Setup Delay (5.4.3.1) Web Server Response Time (5.4.3.2) POP3, Mail Server Connection Time (5.4.3.3)
Measuring the time for transmitting the data content starting after the service access itself (see Figure 13) reflects the end-users perspective and can be measured with the Moniq tool: WAP Object Download Delay (5.4.3.4) Web small object download time (5.4.3.5) POP3, Mail Download Time (5.4.3.6) SMTP, Mail Sending Time (5.4.3.7) Domain Name Service Response Delay (5.4.3.8)
Mobile requests service
Connection established/ Start of data transfer
Data transfer complete
Service Access Time
Data Transfer Time
Figure 13: Service Access & Data Transfer
5.4.3.1
WAP Connection Setup Delay TAG: INT WAPconnect
The average delay between the setup-packet from the mobile is received at the monitor until the mobile successfully acknowledges the server response packet and the ACK packet passes the monitor. The measured delay is the full setup round-trip time and it includes the following delay components: Monitor WAP Server Mobile Monitor.
Precondition:
Successful connection setup
Definitions of KPIs
5.4.3.2
Web Server Response Time TAG: INT WEBresponse
The average time until the server starts to transmit data measured from the first packet from the mobile. This measure includes the time it takes to set up a connection over the GPRS/EDGE and wired Internet, the client to deliver the request to the server, the server to process the request, and to respond to it. The condition is that only successful response is measured, but the content of the response may be either success or failure.
5.4.3.3 POP3, Mail Server Connection Time TAG: INT POP3connect
The average time it takes to connect to the POP3 e-mail server. The connection time includes the time to establish the TCP connection over the GPRS/EDGE and Internet path, the server to send a greeting message, and the time to log into the server. Condition: successfully established connections are measured.
PIs denoting the time for transmitting the data content starting after the service access itself that can be measured with the Moniq tool:
5.4.3.4 WAP Object Download Delay TAG: INT WAPdownload
The average delay between the request packet from the mobile until the server response is successfully acked by the mobile.
Precondition:
Successfully finished download

5.4.3.5 Web Small Object Download Time (9-11kbyte) TAG: INT WEBsmall
The average time it takes to download a small HTTP object. Only small (9-11 Kbytes) files are measured. Due to the TCP protocol, small objects are not able to utilize the full GPRS/EDGE throughput (timeslot capacity). The download time of such small objects depends more on the round-trip time
Remarks:
To have comparable results between different measurements and operators, only objects that fall into a narrow range are measured. The chosen range of 9-11 kbytes was chosen because the median of Web downloads falls in this range. Condition: object download was successful.
Definitions of KPIs
5.4.3.6
POP3, Mail Download Time TAG: INT POP3download
This KPI defines the average time to successfully download one or more e-mails from the POP3 e-mail server. The whole time is measured including server greeting, authentication, and time to download of all mails and quit. Condition: mail download was successful.
5.4.3.7 SMTP, Mail Sending Time TAG: INT SMTPsend
This KPI defines the average time for the mobile to send an e-mail using the SMTP protocol. Condition: e-mail sending was successful.
5.4.3.8 Domain Name Service Response Delay TAG: INT DNSdelay
This KPI defines the average delay for the DNS server to respond to a DNS request, counting retransmissions. The time is calculated between the request packet seen in the uplink direction and the response packet seen in the downlink direction at the monitor. Condition: resolution was successful.
Definitions of KPIs
5.4.4
GPRS Access Time (for each service)

TAG: INT 040
Definition:
The KPI GPRS e2e Access Time [s] denotes the length of the time period it takes to access a service successfully, measuring the time period for the complete chain: GPRS Availability Attach PDP Context Activation Service Access
AccessTime [s] = PI I T 010 + PI I T 020 + PI I T 030
Attach Setup Time
PDP Context Activation Time
Service Access Time
Trigger Point(s):
Trigger points and timeout values that are defined for the involved KPIs: GPRS Attach Setup Time (PI I T 010) GPRS PDP Context Activation Delay (per APN) (PI I T 020), and GPRS Service Access Time (for each service) (PI I T 030)
are valid for measuring GPRS e2e Access Time.

Remarks:
The lower the timeout values are for the different Access KPIs the higher the failure ratio is. At the same time a lower timeout value goes along with a lower access time, since the timeout value is the upper boundary for the measured time period it takes to perform a certain procedure, i.e. attach, PDP Context activation or service access.
Active Measurement:
KPI not covered in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing.
Definitions of KPIs
5.4.5
GPRS Mean User Data Rate

TAG: INT 050
Definition:
The KPI GPRS Mean User Data Rate [kbits/s] denotes the average data rate in kbits/s.
Mean Data Rate [kbits / s] =
File Size (kByte) 8 Time of Stop Trigger (s) Time of Start Trigger (s)
The trigger points have to be defined per service for KPI I T 050.
Trigger Point(s):
The average throughput is measured from opening the data connection to the end of the successful transfer of content (file, e-mail or web page). The trigger points are defined per service (FTP DL/UL, E-Mail DL/UL, HTTP) in the same way as they are defined for GPRS Data Transfer Cut-off Ratio (PI RET 020). Refer to chapter 5.3.2.
Prerequisites:
The MS is already GPRS attached (PI ACC 020), a PDP Context is activated (PI ACC 030) and a service was accessed successfully (PI ACC 040).
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: Round Trip, Time, Controlled PDCH Sharing, Cell Update during DL FTP, Intra SGS RA Update during DL FTP, Inter SGS RA Update during DL FTP, FTP UL/DL, WWW Traffic, E-mail Traffic, TAG: TT 020, TAG: TT 030, TAG: MT 010, TAG: MT 020, TAG:MT 030, TAG: SET 020, TAG: SET 040, TAG: SET 050.
This KPI can be measured by executing the UDP based test solution: Transport Tests, RAW throughput as described in detail in Proposed Extensions to GPRS e2e performance Testing [2].
Definitions of KPIs
This KPI can be measured for Web and FTP throughput in DL direction with the Moniq tool: FTP Download Rate (5.4.5.1) Web Large Object Download Rate (larger than 50kbyte) (5.4.5.2) End-to-End Achievable Throughput (5.4.5.3) TCP Goodput/Throughput Ratio (5.4.5.4)
5.4.5.1
FTP Download Rate TAG: INT FTPrate
The average rate is the size of the downloaded file divided by the time it takes to download it. Only large files (> 50 kbyte) are measured, when the available end-to-end path capacity dominates the measure and not the file size.
Precondition:
File download was successful

5.4.5.2 Web Large Object Download Rate (larger than 50kbyte) TAG: INT WEBlarge
The average rate is the size of the object divided by the time it takes to successfully download the object. This measure is also called goodput. Only large objects are measured, when the available end-to-end path capacity dominates the measure and not the object size. Condition: object download was successful.
5.4.5.3 End-to-End Achievable Throughput TAG: INT THRHPUT
This KPI measures the average achievable IP layer throughput end-to-end. This KPI is measured by tracing those mobiles that generate one or more parallel TCP downloads saturating the downlink channel. In an optimal GPRS/EDGE network this KPI shows the capacity of the available GPRS/EDGE data channel.
5.4.5.4 TCP Goodput/Throughput Ratio TAG: INT THRHPUT/GPUT
The ratio of user data payload in TCP connections (Web, e-mail, FTP, etc.) divided by the total IP layer data transmitted to the mobile terminal counted at the monitoring point. The ratio depends on the protocol overheads (TCP/IP) and the retransmissions. 100% is not possible; there is an upper limit even in an optimal network. The distance from this optimal value has to be evaluated.
Definitions of KPIs
5.4.6
Round Trip Time

TAG: INT 060
Definition:
The KPI Round Trip Time (RTT) [s] denotes the length of the time period it takes for a data packet to be sent from the MS and being returned to the sender. Note: The ping program is a suitable method for measuring RTT. Either a DNS server or an external Internet server can be pinged by the MS.
Trigger Point(s):
Since the ping program uses the ICMP echo request and echo reply messages (not using the transport layer TCP or UDP) the start trigger is the ICMP echo request message and the stop trigger the ICMP echo reply message.
Remarks:
The ping size has influence on the RTT, therefore measuring the KPI Round Trip Time should be done with different ping sizes. Furthermore the delay introduced by the Internet and the response time of the server influence the RTT.
Prerequisites:
The MS is already GPRS attached (PI ACC 020) and a PDP Context is activated (PI ACC 030).
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: Round Trip Time,
TAG: TT 010.
This KPI can be measured with the Moniq tool: End-to-End Round Trip Delay (5.4.6.1) Client-side Round Trip Delay (5.4.6.2) Server-side Delay (5.4.6.3)
Remarks:
Packet delay is measured for TCP protocol exchanges, when the mobile client initiates a TCP setup by sending a SYN packet. The server responds with a SYN/ACK packet, which is finally acknowledged by the client with an ACK packet. Thus the TCP setup includes a full end-to-end round-trip between the client and the server. The monitor calculates the times between these packets, thus it can separate the delay for client and server-side components.
Definitions of KPIs
5.4.6.1
End-to-End Round Trip Delay TAG: INT DELAYe2e
The average time measured on the full client-server-client round-trip path. It is usually dominated by the GPRS access delay (client-side delay).
Remarks:
Packet delay is measured for TCP protocol exchanges, when the mobile client initiates a TCP setup by sending a SYN packet. The server responds with a SYN/ACK packet, which is finally acknowledged by the client with an ACK packet.
5.4.6.2 Client-side Round Trip Delay TAG: INT DELAYgprs
The average time measured between the SYN/ACK and the ACK packets. This measure includes the time over the GPRS channel and the client processing time.
Remarks:
5.4.6.3 Server-side Delay TAG: INT DELAYinternet
The average time measured between the SYN and SYN/ACK packets. This measure includes the delay over the core and Internet to the server, and the processing time in the server. The propagation delay and the buffer congestion in the wired path of the connections dominate this measure. The propagation delay is usually proportional to the geographical distance (propagation time at speed of light). The processing time in servers is usually negligible.
Remarks:
Definitions of KPIs
5.4.7
Client-Server Interaction Time with Background Traffic

TAG: INT 065
Definition:
The KPI Client Server Interaction Time denotes the time [s] it takes to send a small data packet to the server and receive it back (RTT) during an ongoing DL TCP data transfer (FTP) in the background.
Remarks:
The typical use case for this KPI is a user that downloads his emails or a big file from the Internet in the background and starts at the same time a web session. For the web session an HTTP GET will be sent to the web server and the server will respond with an HTTP OK and start to send the content of the web page to the user. The user expects an answer to his request of the web page in a reasonable time. Note: If a big advertised TCP Window Size is used and the system buffers a lot of data, it might happen that the data from the background traffic delays the response from the web server to the user.
Trigger Point(s):
Since the ping program uses the ICMP echo request and echo reply messages (not using the transport layer TCP or UDP) the start trigger is the ICMP echo request message and the stop trigger the ICMP echo reply message.
Prerequisites:
The MS is already GPRS attached (PI ACC 020), a PDP Context is activated (PI ACC 030) and background traffic (FTP DL data transfer) is ongoing.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: Round Trip Time during Background Traffic, TAG: TT 025.
Definitions of KPIs
5.4.8
Packet Loss Ratio

TAG: INT 070
Definition:
The KPI Packet Loss Ratio [%] denotes the rate of data packets, which are lost regardless where they are lost in the network. Note: The ping program is a suitable method for measuring the packet loss rate.
Trigger Point(s):
The start trigger is the ICMP echo request message and the stop trigger the ICMP echo reply message. The ping program should be used with different packet sizes, since sometimes only packets of a special size are lost: The following packet sizes should be used: 32 byte, 548 byte and 1472 byte. A ping command with a length of 548 byte would result in a total IP packet size of 576 byte (548 byte + 28 byte ICMP header). For the case of 1472 byte ICMP data the total IP packet size would be 1500 byte. 576 and 1500 byte are typical IP packet sizes when using TCP.
Remarks:
The Packet Loss ratio has big impact on the application performance.
Prerequisites:
The MS is already GPRS attached. (PI ACC 020) and a PDP Context is activated (PI ACC 030).
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: Round Trip Time, TCP/IP Transfer, WAP Traffic, TAG: TT 010, TAG: TT 020, TAG: SET 030.
This KPI can be measured with the Moniq tool: WAP Retransmission Ratio (5.4.8.1) Downlink End-to-End Packet Loss (5.4.8.2) Server to Monitor Loss (5.4.8.3) Monitor to Client Loss (5.4.8.4)
Definitions of KPIs
5.4.8.1
WAP Retransmission Ratio TAG: INT WAPretransmission
The average number of packet retransmissions seen during WAP object downloads. Retransmissions greatly affect the delay of page downloads. All object downloads are measured regardless of their eventual result (success or failure).
5.4.8.2 Downlink End-to-End Packet Loss TAG: INT LOSSe2e
The ratio of TCP packets lost downlink, between the server and the client end-to-end. Typically the GPRS/EDGE channel access loss dominates this measure.
5.4.8.3 Server to Monitor Loss TAG: INT LOSSinternet
The ratio of TCP (Web, e-mail, FTP, etc.) packets lost between the Internet server and the monitoring point in the downlink direction. This KPI indicates the level of congestion over the wired path from the server up to the monitor.
5.4.8.4 Monitor to Client Loss TAG: INT LOSSgprs
The ratio of TCP (Web, e-mail, FTP, etc.) packets lost between the monitoring point and the mobile terminal in the downlink direction. This measure characterizes the GPRS access part of the path (GSN, PCU buffers and RLC/LLC loss).
Definitions of KPIs
5.4.9
Cell Update Time

TAG: INT 080
Definition:
The GPRS KPI Cell Update Time [s] denotes the length of the time period a cell update takes. Note: Since a cell update can only be observed in READY state the duration of a cell update can be measured best during an ongoing FTP session.
Trigger Point(s):
Start measuring the time interval when the last IP frame is sent/received in the old cell and stop measuring when the first IP frame is sent/received in the new cell by the MS.
Remarks:
In live networks parameters like the moving speed, the cell configuration and the MS have big influence on the time needed for the cell update.
Prerequisites:
The MS is already GPRS attached. (PI ACC 020) and a PDP Context is activated (PI ACC 030). Two different cells belonging to the same RA configured with GPRS/EDGE capability.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: Cell Update during DL FTP, TAG: MT 010
Definitions of KPIs
5.4.10
Intra SGSN RAU Time

TAG: INT 090
Definition:
The KPI Intra SGSN RAU Time [s] denotes the length of the time period an Intra SGSN RAU takes. Note: The duration of an Intra SGSN RAU can be measured best during an ongoing FTP session. In most of the case the RA belongs to a new LA, so that the MS performs a LAU as well.
Trigger Point(s):
Start measuring the time interval when the last IP frame is sent/received in the old cell and stop measuring when the first IP frame is sent/received in the new cell by the MS. In case the new cell belongs to a new LA, the MS enters dedicated mode and sends a Suspend message to the BSS. The BSS sends the Suspend message to the SGSN. The SGSN sends a Suspend Ack to the BSS, which will discard all LLC frames stored for the MS. After the MS has performed the LA update it will send a RA update in the new cell to resume the transfer, see Figure 14. This will be done in case that the BSS does not support the resume procedure. (Origin: [5]).
MS
BSS
SGSN
MSC/VLR
1. Dedicated Mode 2. Suspend 3. Suspend 3. Suspend Ack 4. Resume 4. Resume Ack 5. Channel Release 6. Routeing Area Update Request
Figure 14: Suspend/Resume Procedure
Remarks:
The continuation of the DL transmission after the RAU depends on the number of buffered frames in the SGSN: If there is still data buffered in the SGSN the transfer will continue immediately after the RAU complete.
Definitions of KPIs
If no more data is stored in the SGSN the continuation of the transfer depends on the application. For an FTP DL transfer a timeout will occur on TCP layer and the transmission continues afterwards.
In live networks parameters like the moving speed, the cell configuration and the MS have big influence on the time needed for the Intra SGSN RAU.
Prerequisites:
The MS is already GPRS attached (PI ACC 020) and a PDP Context is activated (PI ACC 030). Two different cells belonging to different RAs served by the same SGSN configured with GPRS/EDGE capability.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: Intra SGS RA Update during DL FTP, TAG: MT 020
Definitions of KPIs
5.4.11
Inter SGSN RAU Time

TAG: INT 100
Definition:
The KPI Inter SGSN RAU Time [s] denotes the length of the time period an Inter SGSN RAU takes. Note: The duration of an Inter SGSN RAU can be measured best during an ongoing FTP session. In most of the case the RA belongs to a new LA, so that the MS performs a LAU as well.
Trigger Point(s):
Start measuring the time interval when the last IP frame is sent/received in the old cell and stop measuring when the first IP frame is sent/received in the new cell by the MS.
Remarks:
In live networks parameters like the moving speed, the cell configuration and the MS have big influence on the time needed for the Intra SGSN RAU.
Prerequisites:
The MS is already GPRS attached (PI ACC 020) and a PDP Context is activated (PI ACC 030). Two different cells belonging to different RAs served by the different SGSNs configured with GPRS/EDGE capability.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]: Inter SGS RA Update during DL FTP,
TAG: MT 030
Overview of the KPIs
Overview of the KPIs

When measuring KPIs related in a live network file and e-mail sizes as well as timeout values cannot be influenced, so that passive tests of KPIs will measure any file and email size that appears in a live network as well as system internal timeout values. But when measuring the KPIs with active tests the following timeout values should be chosen:
KPIs: Accessibility Retainability Integrity

INT 050 Mean User Data Rate INT 060 Round Trip Time INT 070 Packet Loss Ratio
RET 010 PDP Context Cut-off Ratio
RET 020 Data Transfer Cut-off Ratio
max. 30s time out
max. 30s time out
service dependant
service dependant
max. 30s time out
max. 30s time out
time out
time out
Data (content) transfer

ACC 020 Attach Failure Ratio ACC 040 Service Access Failure Ratio
ACC 010 GRPS/UMTS Unavailability
ACC 030 PDP Context Activation Failure Ratio
INT 010 Attach Setup Time
INT 020 PDP Context Activation Time
INT 030 Service Access Time PDP Context Deactivation Wait for (max.) 30s, if PDP Context in not deactivated reset phone
Detach
Start trigger: Attach request Stop trigger: Attach Accept
Wait for (max.) 30s, if MS is not detached reset phone
Start trigger: PDP Context Activation request Stop trigger: PDP Context Activation Accept
Figure 15: Overview of the KPIs
Summary
Summary
In this chapter all KPIs are listed together with their tag and an indication whether the can be measured actively or passively. It is very important to use this chapter only for quick reference but not as ultimate input when looking for a specific KPI and its definition. Detailed information on each KPI is provided in chapters 5.2, 5.3, and 5.4.
en t
M e a s u r em
K P I C a t eg o r y
P a s s i ve
Accessibility KPIs:
GPRS/UMTS Unavailability Attach Failure Ratio PDP Context Activation Failure Ratio Secondary PDP Context Activation Failure Ratio Service Acces Failure Ratio GPRS Accessibility Failure Ratio
ACC 010 ACC 020 ACC 030 ACC 035 ACC 040 ACC 050 RET 010 RET 020 RET 030 RET 040 RET 050 RET 060 INT 010 INT 020 INT 030 INT 040 INT 050
counter counter Moniq tool Moniq tool Moniq tool counter counter counter Moniq tool Moniq tool
GTD: ST 010 GTD: ST 020 GTD: ST 020 GTD: SET 010, SET 020, SET 030, SET 040, SET 050 GTD: TT 020, SET 020, SET 030, SET 040, SET 050 GTD: MT 010 GTD: MT 020 GTD: MT 030 GTD: ST 010 GTD: ST 020 GTD: SET 010, SET 020, SET 030, SET 040, SET 050 GTD: TT 020, TT 030, MT 010, MT 020, MT 030, SET 020, SET 040, SET 050, Extension GTD: RAW throughput GTD: TT 010 GTD: TT 010, TT 020 & SET 030 GTD: MT 010, Extension GTD: IP layer outage GTD: MT 020, Extension GTD: IP layer outage GTD: MT 030
Retainability KPIs:
PDP Context Cut-off Ratio Data Transfer Cut-off Ratio Cell Update Failure Ratio Intra SGSN RAU Failure Ratio Inter SGSN RAU Failure Ratio PS Paging Failure Ratio
Integrity KPIs:
Attach Setup Time PDP Context Activation Delay Service Access Time End-to-End Access Time Mean User Data Rate
Round Trip Time Packet Loss Ratio Cell Update Time Intra SGSN RAU Time Inter SGSN RAU Time
INT 060 INT 070 INT 080 INT 090 INT 100
Moniq tool Moniq tool -
Figure 16: Overview: KPIs and Test Measurements
A c tiv e
T i tle
T a g
M e a s u r em
en t
Summary
K P I C a t e g o r y
Accessibility KPIs:
Service Acces Failure Ratio
ACC 040
Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool Moniq tool
WAP Connection Establishment Success Ratio ACC WAPconnect POP3, Mail Server Connection Success Ratio ACC POP3connect Domain Name Service Query Success Ratio ACC DNS Retainability KPIs: Data Transfer Cut-off Ratio RET 020
WAP Object Download Success Ratio RET WAPobject Web Object Download Success Ratio RET WEB FTP Download Success Ratio RET FTP POP3, E-mail Download Success Ratio RET POP3download SMTP, E-mail Send Success Ratio RET SMTPdownload Integrity KPIs: Service Access Time INT 030
WAP Connection Set-up Delay INT WAPconnect Web Server Response Time INT WEBresponse POP3, Mail Server Connection Time INT POP3connect WAP Object Downlaod Delay INT WAPdownload Web Small Object Download Time INT Websmall POP3, E-mail Download Time INT POP3download SMTP, E-mail Sending Time INT SMTPsend Domain Name Service Response Delay INT DNSdelay Mean User Data Rate INT 050 FTP Download Rate INT FTPrate Web Large Object Download Rate INT WEBlarge End-to-end Achievable Throughput INT THRHPUT TCP Goodput/Throughput Ratio INT THRHPUT/GPUT Round Trip Time INT 060 End-to-end Round Trip Delay INT DELAYe2e Client-side Round Trip Delay INT DELAYgprs Server-side Delay INT DELAYinternet Packet Loss Ratio INT 070 WAP Retransmission Ratio INT WAPretransmission Downlink End-to-End Packet Loss INT LOSSe2e Server to Monitor Loss INT LOSSinternet Monitor to Client Loss INT LOSSgprs
Figure 17: GPRS KPIs Based on Passive Measurements with Moniq
P a s s iv e
T i t le
T a g
Me a s u r e m
e n t
Terminology
8
8.1
Terminology
Abbreviations
BCCH DL e2e FTP GMM ICMP KPI LA LAU LLC PBCCH PDCH PDP PI P-TMSI RA RAU RR RTT UL WAP Broadcast Control Channel Downlink End-to-End File Transfer Protocol GPRS Mobility Management Internet Control Message Protocol Key Performance Indicator Location Area Location Area Update Logical Link Control Packet Broadcast Control Channel Packet Data Channel Packet Data Protocol Performance Indicator Packet Temporary Mobile Subscription Identity Routing Area Routing Area Update Radio Resource Round Trip Time Uplink Wireless Application Protocol
Terminology
8.2
Glossary
This chapter provides definitions of technical and other terms used in the document.
Accessibility
The ability of the user to obtain a service within specified tolerances and other given conditions. The accessibility of the service does include a chain of operations the user must be able to execute: the user must be able to access GPRS/EDGE as such, meaning he must be able to do a GPRS attach and a PDP context activation, and he must be able to access an IP service after he has an activated PDP context.
Integrity
Integrity indicates the degree to which a service is provided without major interferences, once it was accessed. Integrity KPIs show the performance of successful service attempts. Even if a service was accessed successfully, the users perception of the performance may vary between very good and unacceptably bad.
Retainability
Retainability indicates the ability of the user to keep a service once it was accessed under given conditions for a requested period of time. The retainability of a service or session does include that the user does not have to perform any additional manual operations that would not be necessary under stable network conditions, like for example manual re-activation of the PDP context.
Performance Indicator
Performance Indicators are those indicators that have impact on the end-users perception of the service performance.
Key Performance Indicator
Key Performance Indicators are those performance indicators that are rate to be primary performance Indicators reflecting the end-users perception of the service performance in an outstanding manner.
Goodput & Throughput
Goodput is the user data payload in TCP connections (Web, e-mail, FTP, etc.), in contrast to the total IP layer data transmitted to the mobile terminal. Throughput includes the protocol overhead (TCP/IP) and the retransmissions while the goodput is the raw data the end-user wants to receive or transmit.
References
References
[1] [2] [3] [4] [5] [6] [7] Generic Test Description for GPRS/EDGE End-to-End Performance Testing in a Live Network, EED/S/G-01:135 Uen, Revision C. Proposed Extensions to GPRS e2e performance Testing, (ETH). Ericsson GPRS End-to-End Performance Guidebook, EED/S/G-01:134 Uen ITU-T Recommendation E.800: Terms and Definitions Related to Quality of Service and Network Performance Including Dependability (08/94) GPRS Service Description Stage 2 (Release 1999), 3GPP TS 23.060 V3.7.0 Measurements of GPRS End-to-End KPIs with Counters, EED/S/G-02:044 Mobile radio interface layer 3 specification, 3GPP TS 24.008
Appendix
10
10.1
Appendix
Statistical Guidelines
When analyzing loss and success ratio KPIs the proportional frequency of a specific event is estimated from measurement data. Calculating the uncertainty of a measurement the correct sample size can be derived. Formula (1) represents the Gaussian approximation of the binomial distribution considering a 95% confidence interval:
1.96 *
with:
f * (1 f ) n
(1)
f = measured proportional frequency of the KPI one is interested in. n = length of the sample p = unknown proportional frequency of the KPI one is interested in
Thus, the corresponding proportional frequency p falls in the confidence interval (CI) with 95% probability:
f 1.96 *
f * (1 f ) , f + 1.96 * n
f * (1 f ) n
(2)
Example:
Assumption: Measured f = 20% for the GPRS Attach Ratio (KPI xx) with a sample size of 200 measurements. Using formula (1) shows that the uncertainty of the measurement is 5.54%. Therefore: with a 95%-probability the proportional frequency p is included in the interval [14.46: 25.54]

Definition KPIs EDGE

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Definition KPIs EDGE

Uploaded by

Copyright:

Available Formats

Definition of GPRS/EDGE End-to-End KPIs

Definition of GPRS/EDGE End-to-End KPIs

Definition of GPRS/EDGE End-to-End KPIs

Definition of GPRS/EDGE End-to-End KPIs

Definition of GPRS/EDGE End-to-End KPIs

Definition of GPRS/EDGE End-to-End KPIs

Remarks on the Nature of the KPIs

Remarks on the Nature of the KPIs

Terminology Used for Categorizing the KPIs

Terminology Used for Categorizing the KPIs

Definition of GPRS/EDGE End-to-End KPIs

Distinction between KPIs and Performance Indicators (PIs)

Definition of GPRS/EDGE End-to-End KPIs

Terminology Used for Categorizing the KPIs

Definition of GPRS/EDGE End-to-End KPIs

Definition of GPRS/EDGE End-to-End KPIs

Timeouts for Data Transfer per Service:

Table 1: Timeout values for data transfer

Definition of GPRS/EDGE End-to-End KPIs

Definition of GPRS/EDGE End-to-End KPIs

GPRS Unavailability [%] =

# unsuccessful attempts to read SI 13 100% total # attempts to read SI 13

Definition of GPRS/EDGE End-to-End KPIs

GPRS Attach Failure Ratio

GPRS Attach Failure Ratio [%] =

# unsuccessful GPRS attach attempts 100% total # of GPRS attach attempts

8. Attach Accept 9. Attach Complete 10. TMSI Reallocation Complete

Figure 1: Signalling Messages during Attach Procedure

Definition of GPRS/EDGE End-to-End KPIs

This PI can be measured with a counter in the /// SGSN.

Definition of GPRS/EDGE End-to-End KPIs

GPRS PDP Context Activation Failure Ratio

PDP Context Activation Failure Ratio [%] =

1. Activate PDP Context Request

7. Activate PDP Context Accept

Figure 2: Signaling Messages during the PDP Context Activation Procedure

Definition of GPRS/EDGE End-to-End KPIs

The MS is already GPRS attached. (PI ACC 020)

This PI can be measured with a counter in the /// SGSN-G 3.0.

Definition of GPRS/EDGE End-to-End KPIs

Secondary PDP Context Activation Failure Ratio

1. Activate Secondary PDP Context Request

7. Activate Secondary PDP Context Accept

Figure 3: Secondary PDP Context Activation Procedure for GPRS

Definition of GPRS/EDGE End-to-End KPIs

1. Activate Secondary PDP Context Request

7. Activate PDP Context Accept

Figure 4: Secondary PDP Context Activation Procedure for UMTS

Definition of GPRS/EDGE End-to-End KPIs

GPRS Service Access Failure Ratio (for each service)

Service Access Failure Ratio [%] =

# unsuccessful service access attempts 100% total # of service access attempts

Definition of GPRS/EDGE End-to-End KPIs

Figure 7: E-Mail POP3 trace

Definition of GPRS/EDGE End-to-End KPIs

Figure 8: E-Mail SMTP trace

Figure 9: HTTP trace

Service Access Timeouts:

Table 2: Service Access Timeouts

Definition of GPRS/EDGE End-to-End KPIs

WAP Connection Establishment Success Ratio TAG: ACC WAPconnect

Definition of GPRS/EDGE End-to-End KPIs