LANforge 5.4.1 Released

We’re happy to announce LANforge 5.4.1, that supports a number of new features:

  • Add TR-398 automated WiFi test support.
  • Support /AC on 2.4Ghz on /b/g/n/AC radios (2.4Ghz /AC)
  • Add support for wired traffic test to the Dataplane scenario test.
  • Support 5.2 kernel for latest drivers and kernel features.
  • Support IPSEC interfaces.
  • Support CT850a 2D attenuator turn-table.
  • Support ComXim MT series turn-table (for instance: MT200RUWL20)
  • Improve performance of creating virtual stations, tested 500 stations per /n radio.

Adding hostnames to dhclient configs

If you have hundreds of stations on your LANforge and want to give them all hostnames, this is certainly possible. Please get the script from the lanforge-scripts repository.

  1. First bring up your stations with DHCP enabled. This will create your dhclient config files in /home/lanforge/vr_conf.
  2. Admin-down the stations.
  3. With a root shell, go to that vr_conf directory and run this script on the files you want to give hostnames to.
    1. For just one station:
      root# ~lanforge/scripts/ dhclient_sta0000.conf
    2. For all the stations:
      root# ~lanforge/scripts/ dhclient_sta*conf
  4. Admin-up the stations.
    1. If you are sniffing, you will see a DHCP Request packet that has a hostname attribute.
    2. If you are inspecting a LANforge dhcpd server in a virtual router, you would check /home/lanforge/vr_conf/vrcx_eth1_dhcp_lease.db for entries with the setting client-hostname.

That should work.

Adjacent Wifi Channel Throughput Testing

What is the effect of wifi clients on adjacent channels when performing over-the-air wifi throughput testing?

We decided to investigate this question to help show why it happens and also to determine if the physical proximity of the wifi NICs could also be to blame.

After running the tests described below, we found that co-channel interference happens with Access Points using adjacent channels and with LANforge using physically adjacent wifi NICs. LANforge test results show better on-air throughput performance when the PCI-E NICs are physically separated by different systems rather than having the wifi NICs side by side in a single platform.

The test setup consists of two LANforge systems, a CT523c and a CT523b and two identical test APs. The CT523c, labeled LF1, has two 4×4 wifi NICs with clients wlan0 and wlan2. The CT523b, labeled LF2, has two 4×4 wifi NICs, but only one client, wlan2, is in use. The test APs are each setup with a single unique SSID using open authentication and are physically placed about 1m from each LANforge system. Each AP is also using its specified channel at 80MHz channel width.

Each test iteration is a 30 second download test, then 30 second upload test for each individual wifi client and then again for combinations of clients for download and upload.

First test: AP-A on ch100 and AP-B on ch116 – here is what it looked like on inSSIDer:

The baseline results of each individual client show that the test setup has ideal conditions with all clients reporting similar results:

The combination tests show that the clients’ throughput results suffer from the adjacent channel interference:

If we look at a spectrum analyzer while testing, we can see that the overlap at 5.57GHz between channels 112 and 116 is the reason for the interference.

The next test is a repeat of the first but on channels 36 and 52 which are also adjacent to each other.

Test 2: AP-A on ch36 and AP-B on ch52

Test 2 results:

As we move AP-B over to channels further away from AP-A, the combination tests show that the clients experience less interference and therefore show better throughput results:

Test 3: AP-A on ch36 and AP-B on ch100

Test 3 results:

Test 4: AP-A on ch36 and AP-B on ch149

Test 4 results:

Another important conclusion is that the throughput results for a single active NIC in each LANforge system is better than having both NICs active in a single LANforge system. We can show the difference by comparing the total on-air throughput.

In Test 1, channels 100 and 116:
The all-in-one on-air download throughput is 371Mbps + 674Mbps = 1045Mbps.
The separated on-air download throughput is 467Mbps + 963Mbps = 1430Mbps.
1430/1045 = 1.37 (separating the NICs shows a 37% increase).

In Test 2, channels 36 and 52:
The all-in-one on-air download throughput is 351Mbps + 747Mbps = 1098Mbps.
The separated on-air download throughput is 487Mbps = 974Mbps = 1461Mbps.
1461/1098 = 1.33 (separating the NICs shows a 33% increase).

In Test 3, channels 36 and 100:
The all-in-one on-air download throughput is 948Mbps + 966Mbps = 1914Mbps.
The separated on-air download throughput is 982Mbps + 974Mbps = 1956Mbps.
1956/1914 = 1.02 (separating the NICs shows a 2% increase).

In Test 4, channels 36 and 149:
The all-in-one on-air download throughput is 982Mbps + 980Mbps = 1962Mbps.
The separated on-air download throughput is 981Mbps + 967Mbps = 1948Mbps.
1962/1948 = 1.007 (separating the NICs shows a 0.7% decrease).

We hope that this post helps remind you that wifi is subject to RF interference from a variety of sources including active clients on a neighboring channel as well as the active wifi NIC in the next PCI-E slot.

If you would like more info on using LANforge for wireless testing, please contact us at or call +1-360-380-1618.

Redirecting Web Browsing Using Squid and DNSMasq

When testing a captive portal with a virtual station on a LANforge machine, you are faced with a problem of multiple gateways. When the station associates, it gains a DHCP address with a new gateway and DNS server. Your LANforge machine will already have a default gateway and DNS server assigned to it throught the management interface it has on boot. You need to use both. On top of that, you are connected via screen-sharing to a laptop and not to the LANforge machine directly.

When You Have a Serial Console

If you’re lucky, you have a working serial console connection between the screen-share laptop and the LANforge system. I mention working because sometimes the quality of USB serial adapters is spotty. When it works, you have a way to set new values into /etc/resolv.conf and a default gateway.

Getting the station information via the command-line is something you can do using the script:

/home/lanforge/scripts $ ./ --manager localhost \
 --card 1  --quiet yes \
 --port_name wlan0  --show_port
 >>RSLT: 0 Cmd: 'nc_show_port' '1' '1' 'wlan0'

Shelf: 1, Card: 1, Port: 3 Type: STA Alias: 
 IP: MASK: GW: VID: 0 ResetState: COMPLETE
 DNS Servers:
 IPv6-Global: DELETED
 IPv6-Link: fe80::a00:27ff:fe0a:973d/64
 IPv6-Gateway: DELETED
 MAC: 08:00:27:0a:97:3d DEV: wlan0 MTU: 1500 TX Queue Len: 1000

Note down your existing route information:

$ route -n > existing_route.txt

Set your route to the laptop so you don’t loose it. Then update your new route:

$ sudo route add -host $laptop-ip dev eth0
$ sudo route set default gw

Next, re-order the entries in your /etc/resolv.conf file:

# nameserver:

And that should allow your browser to work well enough. If you fat-finger the routes, you can renew the DHCP address on your management port (eth0) using the ifup/ifdown commands:

$ sudo ifdown eth0; sudo ip a flush eth0; sudo ifup eth0

And that should restore your management connection to what you had during boot up.

Often–No Serial Console

The problem is, everything your browser will want to do will use your default gateway, not the virtual stations. Changing the default gateway will drop your connection to the LANforge machine and you will be lost from it. When you are developing remotely, you can waste days waiting for your remote counterpart doing hands-on to reboot the LANforge when you lose the network settings. Luckily, LANforge boxes come with squid and dnsmasq installed.

Starting with dnsmasq

We can start by setting up dnsmasq, so that we can direct DNS requests through the stations DNS setting.

  1. Update /etc/resolv.conf to point at localhost:
    # old nameserver:
  2. Update the /etc/dnsmasq.conf to query the station’s DNS server:

Then start dnsmasq:

$ sudo systemctl start dnsmasq

Tail your system logs to check for typos:

$ journalctl -f

Test a DNS query:

$ host
Using domain server:
Aliases: is an alias for has address has address has address has address has IPv6 address 2a04:4e42::323 has IPv6 address 2a04:4e42:200::323 has IPv6 address 2a04:4e42:400::323 has IPv6 address 2a04:4e42:600::323

Do not enable dnsmasq using $ sudo systemctl enable dnsmasq; sudo systemctl daemon-reload. That will take effect after a reboot and you might not remember doing it. Please remember to comment out in /etc/resolv.conf.

Move onto squid

We want to set the tcp_outgoing_address setting in our /etc/squid/squid.conf file.


Start squid:

$ sudo systemctl start squid

Open a terminal and tail squid logs:

$ cd /var/log/squid; sudo tail -F *log

Next configure FireFox

The Network settings in FireFox are under Edit→Preferences and Advanced→Network. Edit the HTTP proxy to be and port 3128. Erase the values for No Proxy for because having and localhost in there will force FireFox to make no connections whatsoever.

Screenshot at 2017-11-10 13:35:08

Verify with tcpdump

Now the fun begins! We want to see web traffic and DNS queries flowing out the station interface and NOT the management interface. Open two windows, for two different tcpdump sessions. Monitor the management port:

$ sudo tcpdump -eni eth0 port 80 or port 53 or port 443

And monitor the station:

$ sudo tcpdump -eni wlan0 port 80 or port 53 or port 443

Using curl

to watch for redirects

Capturing a captive portal redirect is usually a matter of sending any HTTP request over the station and the response will be a 302 (or 307) redirect.

$ source ~/lanforge.profile # sets path
$ which curl # should be /home/lanforce/local/bin/curl
$ curl -svLki -4 
   --interface wlan0 \
   --localaddr \
   --dns-interface wlan0 \
   --dns-ipv4-addr \
   --dns-servers \

Use the script

In the /home/lanforge/scripts directory, you might have a script. This script performs most of the outlines steps discussed so far. It is useful for helping validate the environment and gathering data to use with portal-bot.bash.

A response looks like:

< Location:
< Content-Type: text/html; charset=UTF-8
< Date: Fri, 10 Nov 2017 22:05:01 GMT
< Expires: Sun, 10 Dec 2017 22:05:01 GMT
< Cache-Control: public, max-age=2592000
* Server gws is not blacklisted
< Server: gws
< Content-Length: 219
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<H1>301 Moved</H1>
The document has moved
<A HREF="">here</A>.


When diagnosing behavior of captive portals, often they are constructed to redirect to arbitrary high-level ports. Feel free to add more ports to your tcpdump filter. But at this point, curl is probably going to be your best friend.

Temporarily Changing the Default Gateway to a Virtual Station

When testing a virtual station and you need to verify with a browser that you can reach a captive portal page, or browse the Internet–you want to change the default gateway of the system to the gateway that the virtual station was assigned. This is a manual change, because when the virtual station associates, the default route of the system doesn’t change. If it did, you’d probably lose connection between the LANforge resource and your management station would probably be lost.

After checking your associated station, you can write a script to help set the new default gateway. Here’s an example with some fixed values:

echo "Please do this command in screen"
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - #
# Use this script to modify your routes to allow a station      #
# to become the default route. Should maintain route for your   #
# existing connection to management laptop through eth0.        #
# This script is an example to be modified.                     #
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - #
set -x
ip a show eth0
ip a show sta100
ip route show

# this adds address to station, referred to below
# ip a add dev sta100

# make static route to old gateway
ip route add dev eth0

# make static route to mgt laptop
ip route add dev eth0

# now update default route to go out sta100
ip route change default via dev sta100

sleep 1s
ping -nc2
ping -nc2

# adjust this to your work time window
read -t $((2 * 60)) -p "Press Enter to end and reset"

# and set the route back
ip route change default via dev eth0
ip route show
ping -nc2
ping -nc2

Notice that the script suggests using screen.  Why use screen? Because if your terminal session disconnects, the script will continue and reset the previous default gateway. If the script is interrupted (by session disconnection or control-c) your default route will remain running across your AP. (We add default link-local routes to our laptop to help mitigate this.)
There is a default .screenrc file in /home/lanforge. You want to run the script as root. This will work: sudo ./

This is an example script with some logic to help detect the default gateway, your ssh connection, and you tell it what station to use:

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - #
#  Use this script to modify your routes to allow a station     #
#  to become the default route. Should maintain route for your  #
#  existing connection to management laptop through eth0.       #
#  This script is an example to be modified.                    #
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - #

if [ -z "$3" ]; then
   echo "Use this script to assign a station on this system to be the new default gateway."
   echo "Usage: $0   "
   echo ""
   echo "Your manager-ip is the LANforge resource in mode 'manager' or 'both' and might"
   echo "not be the machine hosting the virtual station."
   echo "Example, sta3000 on resource 3, manager is"
   echo ""
   echo "./$0 3 sta3000"
   echo ""
   echo "Please do this command in screen"
   exit 1

def_gw_dev=`route -n | awk '/^ / {print \$NF}'`
def_gw_ip=`ip a show $def_gw_dev | awk '/inet /{print \$2}'`
echo "default gw device: $def_gw_dev $def_gw_ip "
sleep 3
manager_ip=`who | perl -ne '/\(([0-9.]+)\)$/ && print "$1\n";'`
#ip a show $def_gw_dev
echo "This is the station you requested:"
ip a show $3
sleep 3

#  make static route to old gateway
echo "ip route add $def_gw_ip dev $def_gw_dev"
ip route add $def_gw_ip dev $def_gw_dev

#  make static route to mgt laptop
echo "ip route add $manager_ip dev $def_gw_dev"
ip route add $manager_ip dev $def_gw_dev
sleep 3

# now update default route to go out sta100
cd /home/lanforge/scripts
sta_gw=`./ --manager $1 --card $2 --port_name $3 \
  --show_port --quiet yes \
  | awk '/ IP:/{print \$6}'`

echo "* Changing default gateway in 3 seconds *"
sleep 3s
echo "ip route change default via $sta_gw dev $3"
sleep 1s
ip route change default via $sta_gw dev $3
sleep 1s
ping -nc2 $def_gw_ip
ping -nc2 $manager_ip

# adjust this to your work time window (in seconds)
#  $(( 2 * 60)) == 120, 2 minutes
read -t $((2 * 60)) -p "Press Enter to end and reset"

# and set the route back
echo "changing route back to previous default gw..."
ip route change default via $def_gw_ip dev $def_gw_dev
ping -nc2 $def_gw_ip
ping -nc2 $manager_ip
echo "...done."

To use the script, you would say:

sudo ./ 3 sta3000

You are probably going to need to alter the script. If you hit Ctlc in the script you will need to reset the routes by hand.

For actually using Firefox, you need to operate this from the desktop of LANforge resource hosting the virtual station. So if, in this example, sta3000 is on resource 3, you need to get on the virtual desktop of resource 3. Open a terminal from the virtual desktop or ssh to resource 3 and start screen as user LANforge.

Modifying the Port Bringup Report

The LANforge Port Bringup plugin produces a highly detailed and verbose report including all the wifi events from the stations involved. This plugin can be used to record performance of station association and captive portal logins. When you save the Graphical Report screen, it saves an HTML document with pictures to a directory on your machine.




We see the report files created by saving the report below:


This report can be reformatted, and the log messages at the bottom of the output are a useful resource to recreate graphs of interest. With some accessory JavaScript, we can create a completely different report. I start off by writing a shell script that backs up the original index.html file of the report and concatenates it to the end of a new title block.

Example shell script below:
(please excuse my lazy [] brackets, because  &-lt; &-gt; takes longer to type)

if [ ! -f orig.html ]; then
   mv -v index.html orig.html
   dos2unix orig.html
cp -v "~/report-1s.js"      report.js
cp -v "~/chartist.min.css"  .
cp -v "~/chartist.min.js"   .
cp -v "~/moment.js"         .
cp -v "~/images/header.png" .
# now create a new header
cat ▶ 00-report.html ◀◀ EOF
     [title]Customized Report[/title]
     [link rel='stylesheet' type='text/css' href='chartist.min.css']
     [script src='moment.js'][/script]
     [script src='chartist.min.js'][/script]
     [script src='report.js'][/script]
     [h1]Station Latency Report[/h1]
     [img src='header.png' /]
     [div id="new_chart"][/div]
     [h2]Station event latency[/h2]
     [div class="ct-chart lf_chart" id="link_1"][/div]
     [pre id='log_messages' style='display:none;']
# and append old report like so:
cat orig.html ▶▶ 00-report.html
echo "please edit 00-report.html"

I have example and report.js examples posted for you to download and use.

The basis of the reformatting is to move the contents at the top of orig.html into the new report header. Delete content that you don’t want, but leave the Log Messages section, because it will be parsed by the javascript. The style tag keeps it from showing on the page. The first column of the messages is in Unix epoch seconds and milliseconds.

[pre id='log_messages' style='display:none;']
1499751346.692  Starting loop number: 1
1499751346.697  #[1] Bringing all selected ports down.
1499751346.841  #[1] Bringing up selected ports in batches of 100
1499751346.845  set_port 1 1 7 NA NA NA NA 0 NA NA NA NA 8388610
1499751346.854  set_port 1 1 8 NA NA NA NA 0 NA NA NA NA 8388610
1499751346.854  set_port 1 1 9 NA NA NA NA 0 NA NA NA NA 8388610

Data is collected in the report.js javascript by parsing each line for important events, using regular expressions:

matches = line.match(/^(\d+\.\d+) EVENT: .*? eventType:Link-Up details:Port (sta\d+) is Link UP.$/) || [];

Bar charts are created with the create_bar() function.

Line and point charts are created with the Chartist javascript library. Point charts are line charts with lines turned off. The moment.js library allows the Chartist library to process domain data on a time axis.

Below is an example of what a resulting report looks like:pbu-05

Captive Portal Testing

A popular feature of the LANforge WiFIRE feature set is captive portal testing portal-bot. If you’re considering doing captive portal testing, the first thing to know is that you are not going to be automating a bunch of web browsers (that would be madness). You will be writing a perl script. More specifically, a perl .pm extension to our script.

Portal testing can be complex because it normally involves re-creating whatever JSON/AJAX/REST call chain the web pages does with a perl script. This process looks like:

  1. Establish laptop with serial connection to LANforge system and with an Ethernet cable
  2. VNC to LANforge and change default gateway and use browser to document login HTTP process
  3. Write a portal-bot extension that mimics that behavior
  4. Incorporate a logout mechanism

It is very useful (and a requirement for long-running capacity testing) to have an API call to the captive portal controller that will log out a session. Otherwise you cannot repeat the test for each station without manually intervening at the captive portal controller to log out one/some/all logged in stations. This logout mechanism could be web-based or could be ssh based, so long as the mechanism can be unattended.

Please review some of our existing portal-testing documentation: