MiniWorld Documentation

Release

Nils Schmidt, Patrick Lampe

Jul 02, 2017

 Contents

1 Use MiniWorld 1
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 OpenWRT Custom Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6 Use a custom VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.7 Network Backends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2 Links 15
2.1 Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3 Contribute 17
3.1 Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4 Indices and tables 19

i
ii
 CHAPTER 1

Use MiniWorld

Introduction

Table of Contents

• About
• B.A.T.M.A.N Demo
– Preparation
– Connections/Links
– Create topology
– Stop the scenario
– Get a shell

About

MiniWorld is an acronym for Mobile Infrastructure’n’Network Integrated World. Its purpose is to help evaluat-
ing/developing any kind of software that relies on network. MiniWorld requires you to deploy the software-under-
test in a VM. A custom network topology can then be used to interconnect the VMs, additionally simulating link
impairment such as delay, loss etc.

B.A.T.M.A.N Demo

In the following, a short demo on how to use MiniWorld is given. For that purpose, we are going to start 3 OpenWRT
nodes running the B.A.T.M.A.N. advanced routing algorithm.

1
MiniWorld Documentation, Release

Note: All MiniWorld commands such as ‘mwserver’ and ‘mwcli’ have to be run wih docker-compose if MiniWorld
is installed via docker-compose. You can either get a shell with

docker-compose exec core bash

or run a single command like this

docker-compose exec core mwcli start

Preparation

We are going to use 2 shells. In the first, we are going to start MiniWorld’s server process. In the second we are using
a command-line tool to start the scenario.
In shell 1 start the server process:

$ mwserver

Note: If for any reasons, MiniWorld won’t stop correctly, you can force the server process to exit and perform the
necessary cleanup with ./cleanup.sh

In shell 2, start the scenario and wait until the command returns:

mwcli start examples/batman_adv.json

In shell 1 you should see the 3 OpenWRT VMs booting.

Ok, what have we done so far? We used a scenario file called batman_adv.json to boot 3 OpenWRT VMs with
B.A.T.M.A.N. advanced.
The nodes are not yet interconnected. For that purpose we have to perform a step:

Connections/Links

Before we do that we check the connections and links. You can see that only connections to ‘mgmt’ exist. This is due
do the management where nodes are connected internally to a virtual node.

$ mwcli info connections

{
"1": [
"mgmt"
],
"2": [
"mgmt"
],
"3": [
"mgmt"
]
}

2 Chapter 1. Use MiniWorld

 MiniWorld Documentation, Release

$ mwcli info links

{
"('1', 'mgmt')": null,
"('2', 'mgmt')": null,
"('3', 'mgmt')": null
}

Create topology

Let’s switch to the first topology and check the connections/links again.

mwcli step

You can see that the first topology is a chain: 1 <-> 2 <-> 3.

$ mwcli info connections

{
"1": [
"2",
"mgmt"
],
"2": [
"3",
"mgmt"
],
"3": [
"mgmt"
]
}

To both connections (1 <-> 2 and 2 <-> 3), a link impairment with 54000 bytes/s and a delay of 1s in each direction is
applied.

$ mwcli info links

{
"('1', '2')": {
"delay": "1.00ms 0.10ms 25%",
"reorder": null,
"loss": null,
"bandwidth": "54000.0",
"duplicate": null,
"limit": null,
"corrupt": null,
"rate": null
},
"('1', 'mgmt')": null,
"('2', '3')": {
"delay": "1.00ms 0.10ms 25%",
"reorder": null,
"loss": null,
"bandwidth": "54000.0",
"duplicate": null,
"limit": null,

1.1. Introduction 3
MiniWorld Documentation, Release

"corrupt": null,
"rate": null
},
"('2', 'mgmt')": null,
"('3', 'mgmt')": null
}

We can now check the neighbours of node 1:

$ mwcli exec --node-id 1 'batctl o'

[B.A.T.M.A.N. adv 2014.4.0, MainIF/MAC: eth0/02:01:00:00:00:01 (bat0 BATMAN_IV)]

Originator last-seen (#/255) Nexthop [outgoingIF]: Potential
˓→nexthops ...

02:01:00:00:00:02 0.840s (188) 02:01:00:00:00:02 [ eth0]: 02:01:00:00:00:02

˓→(188)

02:01:00:00:00:03 0.080s (122) 02:01:00:00:00:02 [ eth0]: 02:01:00:00:00:02

˓→(122)

Node 2 and node 3 are both reachable via node 2, hence the routing works since there is no direct connection between
1 <-> 3

$ mwcli exec --node-id 1 'batctl tr 02:01:00:00:00:03'

traceroute to 02:01:00:00:00:03 (02:01:00:00:00:03), 50 hops max, 20 byte packets

1: 02:01:00:00:00:02 2.648 ms 2.586 ms 2.644 ms
2: 02:01:00:00:00:03 5.840 ms 5.075 ms 5.412 ms

If we switch to the wheel topology where all nodes are connected with node 1, we can see that B.A.T.M.A.N. changed
the routes accordingly.

mwcli step

Node 3 is now reachable directly from node 1:

$ mwcli exec --node-id 1 'batctl tr 02:01:00:00:00:03'

traceroute to 02:01:00:00:00:03 (02:01:00:00:00:03), 50 hops max, 20 byte packets

1: 02:01:00:00:00:03 2.687 ms 2.803 ms 3.050 ms

Stop the scenario

Before a new scenario can be started, the currently running scenario has to be stopped. Further starts of the same
scenario use the snapshot boot mode which uses KVM snapshots to enhance boot times drastically.

mwcli stop

Note: You may need to kill the server process when switching between different scenarios.

Get a shell

Now you can further explore the VMs by getting shell access.

4 Chapter 1. Use MiniWorld

 MiniWorld Documentation, Release

mwcli shell 1

Install

Table of Contents

• Docker-compose
• Without Docker

Note: MiniWorld is tested on Ubuntu 16.04.2 LTS and Ubuntu 14.04.5 LTS (Travis CI). Probably the latest HWE
stack is required.

Docker-compose

For ease of use we decided to use docker-compose instead of “pure” Docker since the Docker commands can get very
long. If you decide to use MiniWorld with docker-compose which is highly recommended, just follow the steps at
https://docs.docker.com/compose/install/ to install docker-compose.
Then follow the instructions at the MiniWorld Playground repository on how to use docker-compose. Afterwards you
should visit the introduction page.

Without Docker

git clone https://github.com/miniworld-project/miniworld_core.git

git checkout <master|nightly>

Install system dependencies

# Mostly copied from the `MiniWorld Dockerfile <https://github.com/miniworld-project/

˓→miniworld_core>`_:

sudo apt-get update

sudo apt-get install ebtables iproute2 qemu-kvm bridge-utils bison flex libdb-dev
˓→psmisc curl wget kmod libdb5.3-dev libxtables11 iptables-dev pkg-config socat

You should install the python packages as root since advanced privileges are required for kvm, network switching
etc.

sudo su

Use a virtualenv for MiniWorld or set python3 as system default.

pip install virtualenv

virtualenv -p python3 mw
source mw/bin/activate

Install python dependencies:

1.2. Install 5
MiniWorld Documentation, Release

cd miniworld_core
pip install --upgrade .\[server,develop\]

Install iproute2:
./scripts/install_iproute2.sh

Concepts

Table of Contents

• Network Backend
• Link Quality Model
• Movement Pattern
• Scenario Config

Note: This is a very short introduction into the basic concepts of MiniWorld. Please read the master thesis for further
documentation .

Network Backend

The network backend controls the virtual network. The Bridged WiFi network backend e.g. uses tools such as iproute2
and ebtables to create the network topology which is specified by Movement Pattern. It takes care of creating new
links, taking them up and down, creating switches etc.

Link Quality Model

The link quality between links is specified by a Link Quality Model. Link quality models for the Bridged network
backends rely on Linux network shaping (tc command). Queuing disciplines such as HTB and netem are used to
control the link impairment between nodes. Currently, only loss and delay is implemented.

Movement Pattern

The Link Quality Model takes the distance between nodes as input to create the link impairment. The distance between
nodes is determined by the Movement Pattern. At the time of writing, the Core Mobility Pattern may be the best choice.
It allows to define network topologies with the CORE network emulation tool. Multiple of these core topologies can
be created and then MiniWorld can switch between the topologies via a mwcli step.

Scenario Config

A scenario config holds all information about the number of VMs, how they shall be interconnected, which commands
shall be run on the VM shells, defines the link quality model as well as the mobility pattern.
Let’s have a look at the scenario config we used in the introduction (examples/batman_adv.json):

6 Chapter 1. Use MiniWorld

 MiniWorld Documentation, Release

1 {
2 "scenario": "batman-adv",
3 "cnt_nodes": 3,
4 "walk_model": {
5 "name": "core"
6 },
7 "provisioning": {
8 "image": "examples/openwrt-15.05-x86-kvm_guest-combined-ext4_batman_adv.img",
9 "regex_shell_prompt": "root@OpenWrt:/#",
10 "shell": {
11 "pre_network_start": {
12 "shell_cmds": [
13 "until ifconfig eth0; do echo -n . && sleep 1; done",
14 "ifconfig eth0 0.0.0.0",
15 "modprobe batman-adv",
16 "batctl if add eth0",
17 "iperf -s &"
18 ]
19 }
20 }
21 },
22 "qemu": {
23 "nic": {
24 "model": "virtio-net-pci"
25 }
26 },
27 "network": {
28 "links": {
29 "configuration": {
30 "nic_prefix": "bat"
31 },
32 "model": "miniworld.model.network.linkqualitymodels.LinkQualityModelRange.
˓→LinkQualityModelWiFiExponential"

33 },
34 "core": {
35 "topologies": [
36 [
37 0,
38 "tests/core_topologies/chain5.xml"
39 ],
40 [
41 0,
42 "tests/core_topologies/wheel5.xml"
43 ]
44 ],
45 "mode": "lan"
46 }
47 }
48 }

Root section:
• 2: The scenario is named batman-adv
• 3: We want to start 3 VMs (nodes)
Walk Model
• 5: We use the Core Mobility Pattern

1.3. Concepts 7
MiniWorld Documentation, Release

Provisioning
• 8: Declares the image to use for the VM. Note that for each node a custom image can be used
• 9: The shell prompt is used to determine when a VM has finished booting and for shell provisioning
• 12: The commands to be executed on each VM before the network is set up.
Qemu
• 24: Use the virtio NIC model for best bandwidth
Network
• 30: Provision IPs on all bat prefixed NICs inside the VM
• 32: Use the LinkQualityModelWiFiExponential to simulate the link impairment between nodes

Examples

MiniWorld comes with a few examples such as B.A.T.M.A.N. advanced as well as a LAN and WiFi demo.
We created an extra playground for that purpose. Check out the README at MiniWorld Playground.

OpenWRT Custom Software

In the following, you will learn how to create a B.A.T.M.A.N. advanced topology with 3 nodes based on OpenWRT:
1. In the following you will learn how to deploy software in an OpenWRT image. We will use B.A.T.M.A.N
advanced for the routing on layer 2.
2. Afterwards you will see the basic unit MiniWorld uses, a scenario config.
3. Finally, you will start 3 VMs interconnected in a chain where node 1 routes to node 3 with the help of
B.A.T.M.A.N.

Table of Contents

• Deploy OpenWRT Image

• Prepare Scenario Config

Deploy OpenWRT Image

First, get OpenWRT:

wget https://downloads.openwrt.org/chaos_calmer/15.05/x86/kvm_guest/openwrt-15.05-x86-
˓→kvm_guest-combined-ext4.img.gz

gunzip openwrt-15.05-x86-kvm_guest-combined-ext4.img.gz

The image does not have batman installed yet, hence we need to boot the VM with KVM. We leverage the qemu user
network backend to get internet access. Note that ICMP is not working in the user network backend.

kvm -nographic -netdev user,id=net0 -device virtio-net-pci,netdev=net0 openwrt-15.05-

˓→x86-kvm_guest-combined-ext4.img

8 Chapter 1. Use MiniWorld

 MiniWorld Documentation, Release

You need to press enter until you see the following login banner:
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
CHAOS CALMER (15.05.1, r48532)
-----------------------------------------------------
* 1 1/2 oz Gin Shake with a glassful
* 1/4 oz Triple Sec of broken ice and pour
* 3/4 oz Lime Juice unstrained into a goblet.
* 1 1/2 oz Orange Juice
* 1 tsp. Grenadine Syrup
-----------------------------------------------------
root@OpenWrt:/#

Next, we need to get an IP on the eth0 network interface via DHCP.

Edit /etc/config/network such that the config block for interface lan looks like the following:
config interface 'lan'
option ifname 'eth0'
option proto 'dhcp'

Afterwards, apply the new network settings:

/etc/init.d/network restart

Now, verify eth0 has an ip address

root@OpenWrt:/# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56
inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:44 errors:0 dropped:0 overruns:0 frame:0
TX packets:66 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7028 (6.8 KiB) TX bytes:7416 (7.2 KiB)

We can proceed with our actual intention, to prepare batman advanced on the OpenWrt image:
opkg update
opkg install kmod-batman-adv
opkg install batctl

Now that batman is installed, power off the VM. We are going to use this image as a read-layer later.
poweroff

Prepare Scenario Config

For simplicity, we will work from the miniworld repo directory.

This is the scenario config we are going to use. Save in in the directory where you cloned MiniWorld as bat-
man_adv.json. Moreover, make sure that the openwrt image on line 8 is named correctly.

1.5. OpenWRT Custom Software 9

 MiniWorld Documentation, Release

1 {
2 "scenario": "batman-adv",
3 "cnt_nodes": 3,
4 "walk_model": {
5 "name": "core"
6 },
7 "provisioning": {
8 "image": "examples/openwrt-15.05-x86-kvm_guest-combined-ext4_batman_adv.img",
9 "regex_shell_prompt": "root@OpenWrt:/#",
10 "shell": {
11 "pre_network_start": {
12 "shell_cmds": [
13 "until ifconfig eth0; do echo -n . && sleep 1; done",
14 "ifconfig eth0 0.0.0.0",
15 "modprobe batman-adv",
16 "batctl if add eth0",
17 "iperf -s &"
18 ]
19 }
20 }
21 },
22 "qemu": {
23 "nic": {
24 "model": "virtio-net-pci"
25 }
26 },
27 "network": {
28 "links": {
29 "configuration": {
30 "nic_prefix": "bat"
31 },
32 "model": "miniworld.model.network.linkqualitymodels.LinkQualityModelRange.
˓→LinkQualityModelWiFiExponential"

33 },
34 "core": {
35 "topologies": [
36 [
37 0,
38 "tests/core_topologies/chain5.xml"
39 ],
40 [
41 0,
42 "tests/core_topologies/wheel5.xml"
43 ]
44 ],
45 "mode": "lan"
46 }
47 }
48 }

Use a custom VM

10 Chapter 1. Use MiniWorld

 MiniWorld Documentation, Release

Table of Contents

• Debian 8 (Systemd)
– Systemd
– Upstart

In general MiniWorld can run any OS/software that runs under KVM. For automation purposes however, MiniWorld
requires access to the serial console of a VM. This enables MiniWorld to provision the VM without network/SSH
access. The downside of this approach is that a shell has to be spawned on the serial console. Moreover, root has to be
logged in automatically. Note that ‘OpenWrt is usable out of the box with MiniWorld.
The required modifications depend on the init system.

Debian 8 (Systemd)

The following illustrates how VM images can be created and deployed. Moreover, it points out the required modi-
fications of a VM such that it works with MiniWorld. New images can be created with the commands shown in the
following.

1 wget http://saimei.acc.umu.se/debian-cd/8.6.0/amd64/iso-cd/ debian-8.6.0-amd64-

˓→netinst.iso

2 qemu-img create -f qcow2 debian_8.qcow2 5G

3 kvm -boot once=dc -vga qxl -spice port=5900,addr=127.0.0.1,disable-ticketing -redir :
˓→<host_port>::22 -cdrom debian-8.6.0-amd64-netinst.iso debian_8.qcow2

First, an image has to be downloaded. Debian 8 (Jessie) is used in the example (line 1). Then a QCOW2 image is
created which serves as the hard disk for the VM. The VM is booted from the live image (line 2). The user can then
install the OS to the harddisk of the VM. Note that starting KVM without the -vga switch does not work for images
which have a graphical installer. The UI can be accessed with spice compatible programs. After installing the VM, it
can be started without the live image by leaving out the –boot and the -cdrom command line switches. The -redir CLI
switch redirects the port 22 to localhost. This enables accessing the VM via ssh if the network is configured by means
of Dynamic Host Configuration Protocol (DHCP) in the VM. Moreover, the VM can access the internet for further
deployment. Note that ICMP does not work with the user network backend (SLIRP) of Qemu. The modifications
of the VM required by MiniWorld depend on the boot loader and the init system. For systems with grub, the serial
console can be enabled by modifying the /etc/default/grub config file.

Note: Installation hints for the application remote-viewer:

Linux: apt-get install libvirt-bin
Mac: brew cask install remoteviewer

Run the following command to connect via spice to the VM:

remote-viewer spice://0.0.0.0:5900

The following listing enables the serial console and disables the new NIC naming scheme for Ubuntu 16.04 systems.
The command update-grub has to be run after the file is modified.

GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0 net.ifnames=0 biosdevname=0"

GRUB_TIMEOUT=0
GRUB_TERMINAL=console

1.6. Use a custom VM 11

MiniWorld Documentation, Release

This allows the NetworkConfigurator to configure based on the ethX naming scheme. The timeout is not required,
but improves VM boot times. The modification of grub redirects the kernel boot log to the ‘serial console so that
MiniWorld can detect when the boot process is over. There is no autologin mechanism implemented. Therefore, the
root user is expected to be logged in on the serial console’s shell. The modification depends on the init system. Ubuntu
16.04 uses Systemd while older versions used the Upstart init system.
The modifications for both systems are depicted in the following listings respectively.

Systemd

mkdir /etc/systemd/system/serial-getty@.service.d
cat << EOF >
/etc/systemd/system/serial-getty@.service.d/override.conf
[Service]
ExecStart=
ExecStart=-/sbin/agetty --keep-baud 115200,38400,9600 -a root %I $TERM
EOF

Upstart

T0:23:respawn:/sbin/getty -L ttyS0 --autologin root 38400 vt100

Another reduction of VM boot times can be achieved by disabling any DHCP configuration.

Network Backends

A network backend is responsible to create and manage the network between VMs. Currently there are 3 implemented
network backends. One of them (VDE) may be removed in future releases and is not discussed in the following.
Basically there is one network backend for wired and one for wireless communication. The bridged network backends
are based on Linux Bridges.

Table of Contents

• Bridged LAN
– Example
– Details
• Bridged WiFi (pseudo)
– Example
– Details

Bridged LAN

Bridged LAN uses for each connection between two nodes one tap device. Link shaping is done on the tap device
since each tap device represents a connection. The number of connections have to be known beforehand, hence it is
only usable with the Core Mobility Pattern. TODO: link Two connected tap devices are put onto the same bridge.

12 Chapter 1. Use MiniWorld

 MiniWorld Documentation, Release

Example

mwcli start examples/nb_bridged_lan.json

Details

The following command shows how the nodes are connected to each other.There are 3 bridges: mgmt is the bridge
for the management network. No link shaping is done on this network. For each connection between two nodes are
bridge is created. The bridges are prefixed with br_. br_00001_00002 for example is the bridge for the connection
between node 1 and node 2.

$ brctl show
bridge name bridge id STP enabled interfaces
br_00001_00002 8000.8a97fd2704ee no tap_00001_1
tap_00002_1
br_00002_00003 8000.968ace0fbf60 no tap_00002_2
tap_00003_1
mgmt 8000.329fcad5b6da no tap_00001_5
tap_00002_4
tap_00003_4

Bridged WiFi (pseudo)

Example

The bridged WiFi network backend multiplexes connections via a single tap device. The different connections are
marked in the Linux kernel such that for each connection a different link impairment can be set. All connections are
put on the same hub (Linux bridge). For connected nodes ebtable rules allow communication.

mwcli start examples/nb_bridged_wifi.json

Details

On wifi1 links are shaped according to the link quality model.

$ brctl show
bridge name bridge id STP enabled interfaces
mgmt 8000.464c4db9ebb2 no tap_00001_2
tap_00002_2
tap_00003_2
wifi1 8000.5e9c09a15fb3 no tap_00001_1
tap_00002_1
tap_00003_1

The connection firewall is done with ebtables:

$ ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 1, policy: ACCEPT

1.7. Network Backends 13

MiniWorld Documentation, Release

--logical-in wifi1 -j wifi1

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

Bridge chain: wifi1, entries: 4, policy: DROP

-i tap_00003_1 -o tap_00002_1 -j mark --mark-set 0x2 --mark-target ACCEPT
-i tap_00002_1 -o tap_00003_1 -j mark --mark-set 0x2 --mark-target ACCEPT
-i tap_00002_1 -o tap_00001_1 -j mark --mark-set 0x1 --mark-target ACCEPT
-i tap_00001_1 -o tap_00002_1 -j mark --mark-set 0x1 --mark-target ACCEPT

14 Chapter 1. Use MiniWorld

 CHAPTER 2

Links

Links

This section summarizes a few links for further documentation:

• Youtube channel
• Master thesis
• Paper: Link following if paper is published
• MiniWorld readthedocs

15
MiniWorld Documentation, Release

16 Chapter 2. Links
 CHAPTER 3

Contribute

Documentation

Documentation should be edited and tested locally. If you push to a remote branch for which there are hooks on
readthedocs.org (currently master and nightly), the documentation will be build and updated accordingly.
The documentation is build with sphinx. We use reStructuredText sine it offers more possibilities compared to mark-
down. Either learn reStructuredText from the website’s quick reference or simply have a look at the source of existing
documentation (click on Show Source on the side bar).
If you have not installed sphinx, install it with pip.

pip install sphinx sphinx_rtd_theme

Afterwards go to the doc directory and create the documentation:

cd doc
make clean
make html

Open the html file:

open build/html/index.html

Note: If you encounter problems, try it within a virtualenv.

Testing

17
MiniWorld Documentation, Release

Table of Contents

• Run tests
– Docker-compose
– Locally
• Further Tips

Run tests

Docker-compose

./scripts/start.sh
./scripts/test.sh

Locally

mwserver
pytest -x -vvvvv -m "not examples" tests/

Further Tips

To drop to pdb and do not terminate QEMU processes add the –pdb flag to pytest or run ./scripts/test.sh –pdb

18 Chapter 3. Contribute
 CHAPTER 4

Indices and tables

• genindex
• modindex
• search

19