You are on page 1of 8

Wi-Fi, Bluetooth, and

Cellular Testing
Avkash Chauhan
Sr. Escalation Manager
Windows Embedded CE & Mobile
Microsoft Corporation

Microsoft Confidential

Agenda What Do Network Tests Test?


Benefits of network testing
Microsoft OS
Cellular Testing Overview
RadioMetrics Driver
Interface
RIL
WiFi
BT Stack
NDIS/WLAN
WiFi Testing Overview
Miniport
WiFiMetrics Driver Stack Radio Stack
Driver
HCI Drivers

Bluetooth Testing Overview


Q&A / Feedback Hardware Radio Chip NIC BT Module

Microsoft Confidential Microsoft Confidential

© 2007 Microsoft Corporation. All rights reserved.


This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
Why Network Testing? Agenda
Help you to catch Your Effort
Your Effort Benefits of network testing
bugs earlier. + Our Effort
Cellular Testing Overview
Help you to improve
quality of your drivers. RadioMetrics
Help you to reduce WiFi Testing Overview
chance of device WiFiMetrics
recall.
Bluetooth Testing Overview
Run recommended
tests as well as the Q&A / Feedback
mandatory tests.

Microsoft Confidential Microsoft Confidential

Cellcore/RIL Architecture Radio Test Process


Applications / UI BVT test pass on all new radio stacks
New Radio
Logic Module Basic Voice/Data Scenarios Drop
Windows API
Cellular API Connection Manager Scenario testing (RadioMetrics.exe) FAIL, traces
WTP
Core Provider Driven by real user scenarios provided to
partner
HW Interface Voice Data Etc WTLS In/out voice calls, data transfer, SMS…
Radio Software BVT
Stress oriented (~24 hours/scenario) test
WDP
RIL Test Pass
Winsock
NDISUIO API, Stress FAIL, bugs
ATCI UDP
SIM
Protocol Auto Bind LSP SMS API
Cellcore Test passes PASS Drive
ToolKit UI Driver CellTSP, SIMMgr, SMS…
TCP/IP
Real-world testing
SIMTK ExTAPI
Policy TAPI NDIS
Router „Drive‟ testing RQM
SIM test
WWAN
PPP Ideal route = handoffs, „dry‟ areas,
API SIMTK roaming… Hand-
Cellular TSP Miniport Text WDP Etc FAIL, bugs
Transport off
for GPRS Radio traces and GPS information
Hand-off testing PASS, no pri1 bugs
Radio Interface Layer (RIL) Intra-BTS & Inter-BTS handover
Control Channel Notification Channel Data Port(s) „Tunnel‟ testing
RIL &
MUX Driver Automated Signal Attenuators Tunnel
Cellcore
test
Radio Stack

Microsoft Confidential Microsoft Confidential

© 2007 Microsoft Corporation. All rights reserved.


This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
Radio Quality Metrics (RQM) Main Radio Functionalities
Measure radio quality as perceived by end
users Voice
Test using live network
Exercise functionality such as
incoming/outgoing voice calls, packet data,
SMS, and SIM manager Misc. Data
All test cases are scenario based.
Not intended to reproduce type approval tests
SMS

Microsoft Confidential Microsoft Confidential

RadioMetrics Scenarios Additional Metrics


Metric Name What’s tested? Approximate Metric Name What’s tested? Approximate
Duration Duration
LTK Combination of mandatory voice, data, SMS, SIM, and 7 hours Data Stability 10 iterations of downloading 25 blocks of 12 hours, but varies
(ltk.ini) RIL functionalities. This is the only one required for (gprs_stability_metric.ini random size data simulating sync scenarios. on data throughput
ODM to pass 100% to get our logo. csd_stability_metric.ini) rate
Outgoing Voice 1000 random iterations of outgoing calls that are 10 hours Multiple Contexts Multiple processes of radiometrics each 1 hour, but varies on
(outgoing_call_metric.ini) terminated by the device or the dialup server. (multiple_contexts.ini) downloading 20 blocks of 512K data. number of contexts
supported and data
Incoming Voice 1000 iterations of outgoing calls to the dialup server 22 hours
throughput rate.
(incoming_call_metric.ini) and incoming calls from the dialup server.
Voice and Data Interaction Repeated interactions of outgoing voice calls 2 hours, but varies
Data Performance 20 random iterations of downloading 4 blocks of 256K 1 hour, but
(voice_call_metric.ini) and pocket data download. Also tests on data throughput
(gprs_perf_metric.ini data, with or without suspend/resume. 20MB total varies on
suspend/resume and simultaneous voice and rate.
csd_perf_metric.ini) data download. throughput
data if supported.
SMS 100 iterations of sending random size message to self. 2 hours
Simultaneous Voice and Two processes of radiometrics testing voice Varies on data
(sms.ini)
Data and data simultaneously. throughput rate.
Supplementary Services 25 iterations for each supplementary service. 4 hours (svd.ini)
(supp_service_metric.ini)
Call Hold Conference 25 iterations for each voice call related scenario. 4 hours
(call_hold_conf_metric.ini)
All scenario suites are available through Jetstream:
https://eajs.partners.extranet.microsoft.com/ea20/mdd/WindowsMobile_5.0

Microsoft Confidential Microsoft Confidential

© 2007 Microsoft Corporation. All rights reserved.


This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
Sample RadioMetrics Results Radio Test Infrastructure
Operator 1 Operator 2 Average

Outgoing Call Metric 100% 100% 100%


Available Cellular Networks in Redmond
Incoming Call Metric 92% 100% 96% US EDGE: T-Mobile (1900MHz), Cingular
Data Throughput (Kbps) 56 120 88 (850/1900MHz)
Data Transfer 100% 100% 100%

SMS Stability 98% 100% 99%


European EDGE: Orange (1800MHz)
Supplementary Services 98% 98% 98% 1xRTT/EVDO: Verizon (850/1900MHz), Sprint
Call Hold/Conference 95% 77% 86% (1900MHz)
LTK 92% 95% 94%
Upcoming New Networks
Averages 96% 96% 96%
UMTS: Vodafone (2100MHz)
Green
Yellow
>=95%
UMTS: Cingular (1900MHz)
Red <50%

Microsoft Confidential Microsoft Confidential

Dedicated Radio Testing Lab Remote Test Station


Overview Traces from device
Over Serial Connection

Allow remote user to Bidirectional Serial


Connection to device

RDP into host machine Remote Station


Ethernet connection to device

Device under test


connected to host USB Relay

Sample Board
machine via KITL RDP connection
too remote set up
Internet
through the internet

USB relay controlled by


host machine for Remote Software
1. Platform builder
resetting device 2.
3.
Terminal
Flashing tools
4. Reset program
5. Tracing tools

Test/Dev Workstation

Microsoft Confidential Microsoft Confidential

© 2007 Microsoft Corporation. All rights reserved.


This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
Dial-up Server Drive Testing
Needed for automated
voice call tests Create a “coverage
Accepts/Rejects incoming
One phone number
map” with radio + GPS
calls
Ideal routes should
Generate outgoing calls Strong
have handoffs, „dry‟ DryMedium
area Handover

Accepts conference calls Hunt group


areas, roaming…
Remote hang up
4 lines 4 lines 4 lines 4 lines Collecting radio traces
In Redmond a “Hunt and GPS information
group” serves a group of Tunnel

servers Test Scenarios


Set up your own server Long-Call test,
locally for best results repeated
Package provided by MS- Large-File download,
cellcore on jetstream repeated

Microsoft Confidential Microsoft Confidential

Agenda WiFi Architecture


// Zero
// Config API NETUI // UI for entering user credentials
Application Winsock WZCSAPI ETHMAN // Display Network/Config Information
Benefits of network testing
Transport TCP/IP
Cellular Testing Overview
EAP/TLS
RadioMetrics EAP EAP/PEAP
// For 802.1X (EAP)
// Radius Login
EAPOL
WiFi Testing Overview Other EAP
WZCSVC // EXE for Wireless Zero Config (authentication)
Data Link
WiFiMetrics NDISUIO // Notify applications when adapters appear

NDIS
Bluetooth Testing Overview // Standard NDIS “wrapper”

// WLAN Hardware-specific driver


802.11 Miniport Driver
Q&A / Feedback // (Must cooperate with OS components!)
Public Code
MS Code
Non MS Code
802.11 NIC // 802.11 WLAN Hardware

Microsoft Confidential Microsoft Confidential

© 2007 Microsoft Corporation. All rights reserved.


This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
WiFi Quality Metrics (WQM) WiFiMetrics Scenarios
Scenario based tests to assess WiFi performance Metric Name What’s Tested? Approximate
Duration
and reliability: NDIS Data stability Confirms standard NDIS and WLAN functionality and 1 hour
data stability.
Identifies bugs and provides WiFi module vendors with NDIS Communication Confirms standard NDIS/WLAN functionality with 1 hour
an environment for bug reproduction. external NIC
NDIS Performance Confirm throughput numbers for peek and sustained. 1/2 hour
Tracks vendor and internal progress on improving WiFi
Authentication matrix Confirm authentication and security functions by 2 hours
quality. connecting using all supported security modes and
keys.
Provides a “quality bar” for WiFi module to attain before AP roaming/stress Measures and stresses AP-to-AP roaming. 8 hours
releasing.
Power consumption Measure power utilized by WiFi in various states from 3 hours
idle to high-speed data-communication.
VoIP Specific (optional) Confirms functionality such as QoS, WMM and 1 hour
Latency/Jitter required for proper VoIP operation
Miscellaneous Numerous other functional tests of features such as 2 hours
Ad Hoc, PMK caching, proper SSID handling and so
on

Microsoft Confidential Microsoft Confidential

WiFiMetrics Quality Score WQM Hardware / Lab Setup


Test Suite Score antennas

96%
7 1. Test Control Server
NDIS Data stability
2. Radius and DHCP Server *
NDIS Communication 87% RF Attenuator
3. WiFiMetrics DB and Web Site *
NDIS Performance 92%
6 8 4
RF Isolation Enclosure(s) USB
4. Device Under Test
Authentication matrix 100% RS232
5
Switch 5. WiFi Access Points
AP roaming/stress 43% USB
Mobile w/
and/or
KITL WiFi & Tux 6. RF Isolation Chambers
RS232
Power consumption 66%
1 KITL 7. RF Attenuators
95%
VoIP Specific 8. USB Switch(s)
Test Net Corp Net
Miscellaneous 92% * Optional
2 may coexist Optional
Averages 84% on one server Test Control 3
Server
Hardware Costs $10-15K
Green >=95%
Yellow Radius & DHCP WiFi Metrics Database
Server and Web Site
Red <50%

Microsoft Confidential Microsoft Confidential

© 2007 Microsoft Corporation. All rights reserved.


This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
Agenda Current Bluetooth LTK Tests
Most of them are manual so that 3 rd party
Benefits of network testing Bluetooth stack can also be covered.
Confirms:
Cellular Testing Overview Hands Free audio switching/routing from voice
RadioMetrics audio to Bluetooth audio (SCO) work.
A2DP audio quality is good.
WiFi Testing Overview
A2DP + Hands Free combination work.
WiFiMetrics Stability of Transport Driver
Bluetooth Testing Overview Good user experience with Suspend/Resume of
PPC/PE device
Q&A / Feedback
Headsets Required:
Motorola HT820
Plantronics Pulsar 590
Microsoft Confidential Microsoft Confidential

Headsets Microsoft Tested Automated WMTK Tests


Works only with Microsoft Bluetooth stack.
Headset Profile Tested
Metric Name What’s tested? Approximate
i-Tech Clip A2DP/HFP/AVRCP Profiles Duration

JABRA BT800 A2DP/HFP/AVRCP Profiles Bluetooth Winsock Test Assess Bluetooth Module & TDI drivers. 20 min

i-Phono mini BT450 A2DP/HFP/AVRCP Profiles Bluetooth Performance Measure Bluetooth performance at RFCOMM 10-20 min, varied on
Test level. Also finds issues with Transport driver, BT hardware
Motorola H500 Hands-Free Profile especially useful to find gaps/skips while throughput rate
streaming music with A2DP.
Motorola HS850 Hands-Free Profile BTHAPI Test Soon to be added into WMTK. Useful for 20-30 min
Motorola HT820 Hands-Free Profile testing Bluetooth Module and controller.

Plantronics Pulsar 590 Hands-Free Profile

Microsoft Confidential Microsoft Confidential

© 2007 Microsoft Corporation. All rights reserved.


This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
Future Automated LTK Tests BT Best Practices
Automated Blackbox tests
BT Module Transport Driver Audio Driver
Fake Hands-Free headset
• Scatternet • Stable • Routing/switching
Fake A2DP Sink (headset) • BT 1.2 features • Can handle faster works for outgoing &
• Wake on incoming data connection at incoming calls for
Very helpful to isolate problems with music call constant rate hands-free
connection
streaming, transport layer, or BT module.
Can work with 3rd party Bluetooth stack. OAL BT Stack/Profile Car Kit
Check out Preview at MEDC this year. • Run OAL timer test • If replacing Microsoft • Test interoperability
BT stack or profile, • Get on Auto-makers‟
test important compatible lists
scenarios • Attend UPF/IOTS
events

Microsoft Confidential Microsoft Confidential

Agenda
Benefits of network testing
Cellular Testing Overview
RadioMetrics
WiFi Testing Overview
WiFiMetrics
Bluetooth Testing Overview
Q&A / Feedback

© 2007 Microsoft Corporation. All rights reserved.


This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Microsoft Confidential Microsoft Confidential

© 2007 Microsoft Corporation. All rights reserved.


This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

You might also like