2014

SmallNetBuilder's
Wireless Testbed V2
WI-FI BENCHMARK TESTBED
BY TIM HIGGINS

Reprinted by octoScope, Inc.

www.octoscope.com
+1.978.222.3114

How We Test: SmallNetBuilder's Wireless Testbed V2

How We Test: SmallNetBuilder's Wireless Testbed V2

MON, 28 JUL 2014 06:00
TIM HIGGINS

Introduction
After spending the past year or so with our octoScopebased wireless testbed, I was ready to listen when
octoScope suggested upgrading.
The V1 testbed uses octoBox 26" wide chambers, which
have proved to be a bit too tight, especially as the size of
routers have grown. The desired 8" distance between
router under test and the chamber antennas has not
been achieved on more than one occasion.
Enter the 38" wide octoBox Stackable anechoic
chamber. It measures 24" x 38.35" x 31.2" vs. 18" x
26.35" x 25.2" for the 26" octoBox. The photo below
shows the new stack, with a new 38" wide octoBox MPE
(Multi Path Emulator) sandwiched between the upper
and lower chambers. The key features of the new
configuration are described in the octoScope video here.

The Stack
The new stack includes
octoScope's new quadAtten
programmable attenuator
module, which replaces the
multiple Vaunix Lab Brick attenuators in the old setup.
Dual-shielded cables are also used externally, providing
better shielding for test environments with a lot of RF
activity. The stack sits on a sturdy base with honkin' big
casters that provide very easy rolling. That's an APC
Back-UPS 550 and TRENDnet TEG-S80g Gigabit switch sitting on

SmallNetBuilder Wireless Testbed V2

the base in the photo.

Everything runs off the UPS so that power glitches don't interrupt testing. The TRENDnet switch connects the two
testbed computers to the SNB LAN and each other. RealVNC is used to remotely access the two testbed computers.

Reprinted by octoScope, Inc.

www.octoscope.com
Tel.: +1.978.222.3114

How We Test: SmallNetBuilder's Wireless Testbed V2

Note that the new testbed can handle four RF paths vs. three in the old testbed. This will let us test the new crop of
4x4 AC2350 routers, like ASUS' RT-AC87 that has been available for just about a week as I write this. The 38"
octoBoxes are much roomier inside as the photo below shows.

SmallNetBuilder Wireless Testbed V2 inside

Reprinted by octoScope, Inc.

www.octoscope.com
Tel.: +1.978.222.3114

How We Test: SmallNetBuilder's Wireless Testbed V2

Everything is connected as shown in the diagram below. As is our standard, the new process uses Ixia's IxChariot to
generate traffic and measure throughput.

octoBox based WLAN Testbed block diagram

Computers
The V2 testbed has gotten a computer upgrade, too. The Dell Optiplex 790 Small Form Factor (Core i5-2400 @ 3.1
GHz) formerly used in the lower chamber now connects to the upper chamber and runs theIxChariot console and the
IxChariot test endpoint for the Device Under Test (DUT). The Test STA computer is also a Dell Optiplex SFF
machine, but a 9010 model with Core i5 3570 CPU @ 3.4 GHz. After a briefly wrestling with Windows 8 quirks, I
decided to stay with Windows 7 Pro on both systems.
Both computers have a TP-LINK TG-3468 PCIe Gigabit Ethernet adapter (Realtek RTL8168B based) installed to
provide a second port. This adapter replaces the aging Intel CT's in the V1 testbed that were starting to become
unreliable. One port connects to the SNB LAN for control, the other to the DUT or reference STA for test data.

Reprinted by octoScope, Inc.

www.octoscope.com
Tel.: +1.978.222.3114

How We Test: SmallNetBuilder's Wireless Testbed V2

Upper Chamber
The signals from the device under test (DUT) are picked up by four omnidirectional unity-gain 2.4 / 5 GHz dualband antennas, shown on the right in the photo below. Because the test client is still 3x3, only three antennas are
currently used. The fourth signal path is sealed with connector caps on both ends. You can see there is now lots of
room in the cabinet, even with the largest router seen to date, NETGEAR's R8000 Nighthawk X6.
Some readers pointed out that the all-vertical antenna orientation in the V1 testbed might provide a performance
edge to routers with vertical antenna polarization. So in the V2 testbed, only one antenna remains vertical; the other
two are set at 45. The two Ethernet and USB connectors and quad power outlets are all filtered so that RF gets
neither in nor out.
That's a SNB-installed bamboo rod running across the top of the chamber. It is used to hang the power and Ethernet
DUT cables so that they don't get hung up as the turntable rotates the DUT. It's also handy to hang an LED worklight
used when setting up routers for test. octoScope provided a plastic crossbar for this, but it was too tall and barely
missed the top of the ASUS RT-AC68U's antennas.

NETGEAR R8000 router under test inside the upper test chamber

I ran many experiments before settling on the antenna configuration. I found the larger octoBox tended to yield
lower throughput than the smaller box, especially for the 5 GHz band. Experiments showed the lower throughput was
primarily due to higher path loss from the greater distance between the DUT and chamber antennas.
I found the slightly closer antenna spacing helped compensate for some of the distance-caused path loss. The
antenna array is centered on the chamber turntable. Distance from center of turntable to chamber antennas is 18
inches (45.72 cm).

Reprinted by octoScope, Inc.

www.octoscope.com
Tel.: +1.978.222.3114

How We Test: SmallNetBuilder's Wireless Testbed V2

Upper chamber antenna detail

Speaking of which, the octoBox-TT is a new feature in the upper chamber, used to rotate the DUT during testing. Its
exact use is described in How We Test Wireless Products - Revison 8. The center of the turntable is approximately
XX" from the chamber antennas.

Between
The three antennas are connected to octoScope's new quadAtten programmable attenuator used to adjust signal
level to simulate distance. The quadAtten can be seen in the first photo above, mounted on the side of the upper test
chamber. The quadAtten outputs are connected to a SMA connectors on theoctoBox MPE, which sits between the
upper and lower chambers. The MPE emulates multipath using IEEE 802.11n/ac indoor channel models B or C. The
lower set of SMA connectors on the MPE connect directly to the lower test chamber, which holds the
standard wireless test client (STA).

Lower Chamber
The V2 testbed retires the ASUS PCE-AC66 AC1750 class PCI-e board used as the reference client in V1 testbed.
After careful consideration, NETGEAR's R7000 Nighthawk running firmware V1.0.3.60_1.1.27 configured in client
bridge mode was selected as the new testbed reference client.
The R7000 is directly cabled to the MPE using dual-shielded cables. Direct cabling is necessary to ensure high
enough signal levels in both 2.4 and 5 GHz so that maximum link rates can be reached. Since the R7000 is an

Reprinted by octoScope, Inc.

www.octoscope.com
Tel.: +1.978.222.3114

How We Test: SmallNetBuilder's Wireless Testbed V2

AC1900 class router, we now can test the 2.4 GHz side of AC1900 routers to their full potential. The R7000 is
connected via Gigabit Ethernet to the Dell Optiplex. The R7000 is managed via web browser from the lower
chamber computer. The computer is accessed remotely via VNC.

Lower Test Chamber view

That concludes the tour of the V2 testbed. How We Test Wireless Products - Revison 8 describes how we use it.

Reprinted by octoScope, Inc.

www.octoscope.com
Tel.: +1.978.222.3114

How We Test Wireless Products - Revision 8

TUE, 29 JUL 2014 06:00
TIM HIGGINS

Introduction
NOTE:This article describes the wireless test procedure for products tested after July 25, 2014.
For wireless products tested between April 15, 2013 and July 24, 2014, see this article.
We described the technology used in our new wireless test process in this article. Now it's time to get into the exact
details of how we use the new testbed.

Environment
Our V8 process is similar to the V7 process in that it uses octoScope anechoic chambers to provide a repeatable RF
controlled test environment. The V2 testbed used by the V8 process has a turntable, which plays a key role in the
new process.

New SmallNetBuilder Wireless Testbed

Reprinted by octoScope, Inc.

www.octoscope.com
+1.978.222.3114

How We Test: SmallNetBuilder's Wireless Testbed V2

The V8 process has been designed to address some weaknesses of the V7 process. Specifically:

Small separation of DUT and chamber antennas

Bias toward products with vertically polarized antennas

The need to eyeball a "best" out of four test runs

Inability to fully test AC1900 class products in 2.4 GHz

The V8 process uses the same attenuation settings for "Location" values that are used in some benchmarks for
backward-compatibility with products tested with old open-air methods.
Key values for 2.4 GHz are shown in the plot below. Throughput at 0 dB attenuation is used for Maximum Wireless
Throughput Ranking and throughput at 60 dB attenuation is used for Wireless Range Ranking in
the Router and Wireless product Rankers.

octoBox MPE Test Location Attenuations - 2.4 GHz

Reprinted by octoScope, Inc.

www.octoscope.com
Tel.: +1.978.222.3114

How We Test: SmallNetBuilder's Wireless Testbed V2

Key 5 GHz values are shown in the plot below. Throughput at 0 dB attenuation is again used for Maximum Wireless
Throughput Ranking and throughput at 39 dB attenuation is used for Wireless Range Ranking in
the Router and Wireless product Rankers. Less attenuation is used for range ranking in 5 GHz vs. 2.4 GHz due to
higher path loss at those frequencies.

octoBox MPE Test Location Attenuations - 5 GHz

Reprinted by octoScope, Inc.

www.octoscope.com
Tel.: +1.978.222.3114

How We Test: SmallNetBuilder's Wireless Testbed V2

Test Setup
All wireless router performance testing is done on the LAN side of a router under test so that the product's routing
performance does not affect wireless performance results, i.e. the Ethernet client is connected to a router LAN port.
Access points are tested using their Ethernet port.
The wireless router / AP under test (DUT) is centered on the upper test chamber turntable in both X and Y axes. If the
DUT has external antennas, they are centered on the turntable. If the antennas are internal, the router body is
centered. Distance from center of turntable to chamber antennas is 18 inches (45.72 cm).
Initial orientation (0) is with the DUT front facing the chamber antennas. The photo below shows the 0 starting
position for a NETGEAR R8000 Nighthawk X6 router, which has six external antennas arrayed along the sides of
the product.

NETGEAR R8000 in Upper Test Chamber - "0 degree" Position

Reprinted by octoScope, Inc.

www.octoscope.com
Tel.: +1.978.222.3114

How We Test: SmallNetBuilder's Wireless Testbed V2

And here is an ASUS RT-AC68U in its starting test position. (The white object is an LED work light removed during
testing.) Power and Ethernet cables are routed to the DUT from above, to not interfere with DUT rotation.

ASUS RT-AC68U in Upper Test Chamber - "0 degree" Position

The router or AP under test is set as follows:

Reset to factory defaults

WPA2/AES security enabled

2.4 GHz band: 20 MHz bandwidth mode, Channel 6
5 GHz band: 40 MHz bandwidth mode for N devices, 80 MHz bandwidth mode for 802.11ac devices,Channel 153
We continue to use 20 MHz bandwidth mode for 2.4 GHz testing. Most environments have too many interfering
networks for reliable 40 MHz operation.

Reprinted by octoScope, Inc.

www.octoscope.com
Tel.: +1.978.222.3114

How We Test: SmallNetBuilder's Wireless Testbed V2

Test Process
The general test process is as follows:
1.

Check / update latest firmware

2.

Reset router to defaults

3.

Change router settings:

LAN IP: 192.168.1.1

2.4 GHz: Ch 6, Mode: 20MHz, WPA key: 11111111

5 GHz: Ch 153, Mode: Auto or 80 MHz, WPA key: 11111111

4.

Associate client. Check connection link rate.

5.

Run IxChariot manual test to check throughput

6.

Run test script

7.

Power cycle DUT and repeat steps 4-6

Two test runs are made in each band for dual-band devices.
Testing is controlled by a Tcl script modified from a standard script supplied by octoScope. The script executes the
following test plan:
1.

Move the turntable to starting position. This places the DUT at the "0 degree" starting position previously
described.

2.

Program the attenuators to 0 dB

3.

Move the turntable to -180 (counter-clockwise)

4.

Start 90 second IxChariot test (throughput.scr with 5,000,000 Byte test file size) simultaneous up and
downlink (0 dB only)

5.

Wait 30 seconds, then start turntable rotation to +180 at 1 RPM.

6.

Wait for test to finish

7.

Discard first 30 seconds of IxChariot data, calculate average of remaining data and save to CSV file. Save
entire IxChariot .tst file.

8.

Repeat Steps 3 to 7 for downlink only (DUT to STA)

9.

Repeat Steps 3 to 7 for uplink only (STA to DUT)

10. Increase attenuators by 3dB.

11. Repeat Steps 3 to 9 until attenuation reaches 45 dB for 5 GHz test; 63 dB for 2.4 GHz test.
We continue to use the throughput.scr IxChariot script instead of the High_Performance_Throughput.scrscript used
by many product vendors.
The main difference is the high performance script uses a 10,000,000 Byte default test file size, Microsoft's
Overlapped IO socket and sets the transmit and receive buffer sizes to 64 K bytes. The regular throughput script
doesn't use Overlapped IO, leaves the transmit and receive buffers at default and a 100,000 Byte test file size.
We found in testing that using the throughput script and adjusting the file size upward actually produced slightly
higher throughput than the high performance script.

Reprinted by octoScope, Inc.

www.octoscope.com
Tel.: +1.978.222.3114

How We Test: SmallNetBuilder's Wireless Testbed V2

The CSV files from the two test runs are merged into a single Excel file and the average of the two runs calculated.
The average of the two runs is the value entered into the Charts database.

Test Notes
Moving the DUT through an entire 360 while the IxChariot test is running removes any orientation bias from the test
and eliminates the need to determine the "best" run out of multiple fixed-position tests. Because throughput typically
varies during the test run, the averaged result tends to be lower. Rotation at lower signal levels, particularly in 5 GHz,
frequently causes the connection to be dropped sooner than if the DUT were stationary. For this reason, V8 5 GHz
test results will generally not extend beyond 39 dB.
The IxChariot plot below shows a typical test run. The first 30 seconds of the run include a throughput step-up that is
caused by an IxChariot artifact and does not reflect true device performance. The first 30 seconds also provides time
for DUT rate adaptation algorithms to settle. For both reasons, the test script excludes the first 30 seconds of data in
the average value saved in the Router Charts database.

IxChariot Test Data detail

Our testing focus has always been to provide results that provide the fairest relative comparison among products.
This new process achieves that goal by removing multiple sources of test setup bias to achieve even fairer productto-product comparison.

Reprinted by octoScope, Inc.

www.octoscope.com
Tel.: +1.978.222.3114