CoAP White Paper

Table of Content
1. I NTRODUCTION
1.1. 1.2. 1.3. 1.4.

Internet of Things Vision .................................................................................. 4 Need for interoperability testing event in IoT ......................................................... 4 What is CoAp? ............................................................................................... 5 Objective of 1 st CoAp plugtest ............................................................................ 5

2. TEST PROGRAMS
2.1. 2.2. 2.3. 2.4. 2.5.

What do you test? ........................................................................................... 6 Test specification process ................................................................................. 6 Test bed architecture ...................................................................................... 8 Test suite structure ........................................................................................ 9 Test tools used ............................................................................................. 11

3. TEST TOOL SUPPORTERS

12

3.1. IRISA ......................................................................................................... 12 3.2. BUPT ......................................................................................................... 13

4. TEST CAMPAIGN 5. E VENT ORGANIZERS

13 14

5.1. PROBE-IT .................................................................................................... 15 5.2. ETSI .......................................................................................................... 15 5.3. IPSO ALLIANCE .............................................................................................. 15

6. C ONCLUSION

16

PROBE-IT Page 2/16

CoAP White Paper

Executive summary
This interoperability event, organised in joint co-operation between the Probe-IT project, the IPSO Alliance and ETSI (European Telecommunication Standards Institute) was held in Paris, France 24-25th March and co-located with IETF#83. CoAP is a lightweight application-protocol for devices that are constrained in their resources such as computing power, RF range, memory, bandwidth, or network packet sizes. CoAP is an integral part of the ETSI M2M architecture, where global interoperability is a key issue. This landmark event allowed vendors to evaluate the interoperability of their products and to validate their understanding of the base specifications. It also provided useful feedback to enhance the ongoing ETSI and IETF standardisation. The features tested included the base CoAP specification, CoAP Block Transfer, CoAP Observation and the the CoRE Link Format. The interoperability test cases were jointly developed by IRISA/Universit de Rennes 1 (France), BUPT (China) on behalf of the Probe-IT project, IPSO Alliance and ETSI. This event was attended by 18 companies testing the worlds first CoAP client and server implementations from China, EU, Japan and Korea Tools used to execute test cases, to analyse test results and logs provided by IRISA/Universit de Rennes 1 and BUPT on behalf of the Probe-IT project were considered very useful for all participants. In the end, this event turned to be a successful first CoAp plugtest event by demonstrating need for Interoperability testing in IoT and need for such event in future which was proven by the number of companies participated for this event.

PROBE-IT Page 3/16

CoAP White Paper

1. Introduction
1.1. Internet of Things Vision
The challenges for Internet of Things "The Internet of Things has the potential to change the world, just as the Internet did. Maybe even more so." - Kevin Ashton, the inventor of the term Internet of Things ,2009 Internet of Things (IoT) is a vision towards Future Internet where things have enough intelligent to communicate with other things without intervention of human based on interoperable communication protocols. In near future, these things are expected to be active part in business, information technology, environmental studies, automotive etc. As the number of things interacting using IoT is expected to grow substantially, the heterogeneous nature of implementations demands the need for interoperability testing. The effective testing of IoT ensures proper deployment, interoperability and reliability of the IoT network infrastructure which will be a key factor for vendors to successfully launch their product in the market.

1.2.

Need for interoperability testing event in IoT

Vendor
Standard/Specification
Errors & ambiguities

Vendor

Design/developement # A

Design/Development #B

Human Error (Programming, interpretation of standards) Component #A Interoperability issues Component #B

Figure 1: Need for interoperability testing The objective of interoperability testing is that independent implementations of the same standard interoperate. It is a well-known fact that, even following the same standard, two different implementations might not be interoperable. The heterogonous nature of IoT technologies requires interoperability issues to be solved before the deployment of the product. Having a simple view on interoperability for different technologies may not be possible at this point of time. Since it is a complex topic and needs more research activities to face the challenges raised. To efficiently address this problem, it is necessary to see the interoperability addressing all components within the complete development chain (standards, products, tests, tests tool, etc) with different tools. Interoperability testing events are important and pragmatic tools to validate standards improve implementation and finally improve interoperability.

PROBE-IT Page 4/16

CoAP White Paper

1.3.
CoAp UDP 6LowPan
Figure 2: Protocol stack of CoAP

What is CoAp?
Constrained Application Protocol (CoAp) is a specialized web transfer protocol which is designed by CORE working group of IETF keeping in mind the various issues of constrained environment to realize interoperations with constrained networks and nodes for machine to machine (M2M) applications like smart energy, building automation, smart home etc. CoAP is one IoT (young) protocol of the Internet of Things as identified important by PROBE-IT and where useful activities such as interoperability can help to validate, to mature the specifications while helping on validating its implantation in interoperable products

There are many important features defined by CORE working group for resource constrained devices like Constrained application protocol, web linking for constrained web servers, implementation of datagram transport layer security, communication between sleepy node and intermediate proxy node, group communication (multi-cast in resource constrained networks) etc. For this CoAP plugtest interoperability event, four main features were selected from CORE: CoAP: CoAP can be seen as a complementary to HTTP which has a client/server interaction model similar to HTTP. Unlike HTTP, CoAP targets for resource constrained and lossy networks such as Wireless sensor networks. In the protocol stack CoAp lies over unreliable UDP layer. CoAP defines four types of messages: Confirmable, NonConfirmable, Acknowledgement, Reset. Like HTTP, method codes and response codes included in some of these messages make them carry requests or responses. Ref: http://tools.ietf.org/html/draft-ietf-core-coap-09 CoRE Link format: In this specification Web Linking is extended with specific constrained M2M attributes, links are carried as a message payload rather than in an HTTP Link Header, and a default interface is defined to discover resources hosted by a server. This specification also defines a new relation type "hosts", which indicates that the resource is hosted by the server from which the link document was requested. Ref: http://tools.ietf.org/html/draft-ietf-core-link-format-11 Observing Resources in CoAP: The protocol specified in this document extends the CoAP core protocol with a mechanism to push resource representations from servers to interested clients, while still keeping the properties of REST. Ref: http://tools.ietf.org/html/draft-ietf-core-observe-05 Block wise Transfers in CoAP: This feature is a complementary to CoAp base specification which defines a pair of CoAP options to enable block-wise access to resource representations. The Block options provide a means to transfer large resource from a server to client by fragmenting into smaller blocks. Ref: http://tools.ietf.org/html/draftietf-core-block-08

1.4.

Objective of 1 st CoAp plugtest

Plugtest is the place where vendors have opportunity to identify potential issues related to their product/implementation effectively. It also provides a means to improve the features of a technology standard for technology consortium and standardisation bodies. This CoAP plugtest was a two days long event carried out during IETF#83 meeting in Paris to motivate vendors to verify the interoperability of their equipment with others. Equipment is considered PROBE-IT Page 5/16

CoAP White Paper

interoperable once they have successfully showed that their implementation is able to communicate with implementation of other vendors without any issues. Before the testing session, participants must agree on a set of configurations to test their equipment/implementations. In all tests, at least two different vendor products must be available to conduct a suite of selected interoperability testing scenarios. The main objectives of this event are First opportunity to test their CoAP implementations/equipment in one place with test suites provided by worlds best test labs Verify the interoperability of your product with other major actors in the market Identify the issues and improve your CoAP implementation effectively with test suites provided by worlds best test labs (ETSI and IRISA) Share experience and improve interoperability of your product

2. Test programs
2.1. What do you test?
The idea of conducting first IoT CoAP plugtest is proposed by ETSI and PROBE-IT consortium to gettogether industry people to share their experiences, test their equipment in order to make their product successful in multi-vendor environment by achieving interoperability. The technical aspects of this program are managed mainly by Probe-IT consortium (IRISA and BUPT), ETSI and IPSO alliance. This test program contains four different groups which covers the following features of CoAp: CORE (Constrained Application Protocol (CoAP)): This group of test contains 15 tests which covers the basic transaction of the CoAp request/response model with or without options. All four message types and method codes were covered. LINK (CoRE Link Format): This group contains 2 tests which cover the basic functionality of web link format for constrained nodes. OBS (Observing Resources in CoAP): It contains 5 tests to check the main functionalities of CoRE observe specification. BLOCK (Blockwise transfers in CoAP): This group contains 4 tests which checks the main methods of CoAp to block-wise transfer of large resource. A total of 26 tests were defined, in that CORE group which contains 15 tests which were considered as mandatory tests and other groups which contains 11 tests [LINK (2 tests), OBS (4 tests), BLOCK (5 tests)] were taken as optional. These tests cover maximum functionalities of CoAp features which are selected for the plugtest. Here in this test program, instead of injecting stimuli actively in the network for testing which disturbs the normal operation of System Under Test (SUT), a method of observing the network transactions and post-analysing it for interoperability issues was chosen. This technique is called Passive interoperability testing The following part of this document introduces test specification process steps, test event methodology, test architecture, test suite structure.

2.2.

Test specification process

Test methodology and Test event organisation In order to derive the test cases which have maximum coverage of CoAp features, the methodology shown in the figure 3 was adopted. ETSI, IRISA and BUPT were involved in defining the objective and it was reviewed with help of Zach Shelby from IPSO alliance. Once test purposes were defined, the test scenarios

PROBE-IT Page 6/16

CoAP White Paper

were described and reviewed with IPSO alliance. Finally test specification draft was made available for eventual comments from all participants of this event.

CoAp specifications
Definition of test objective

Review

Test purposes
Validation of test objective

Review

Descriptions of test scenerios

Test descriptions
Validation of test scenerios

Tools
Test tool development

Figure 3: Test development cycle methodology

ONLINE

Test report

Test specifications

Test sessions
Captured packet

Packet analyse and Verdict logging

Arrangements for tests

Figure 4: Test event methodology The procedures followed during the testing event were demonstrated using the figure 4. Every session in the event includes test arrangement and plan of tests to be conducted. During test arrangements, the test

PROBE-IT Page 7/16

CoAP White Paper

architecture setup and initial configurations are made to start the tests. During the execution of each test suite, packets exchanged are captured using wireshark and pcap files were produced. In the end of test sessions, participants need to submit the pcap files in the web interface provided by IRISA to know the test reports/logs/verdicts.

2.3.

Test bed architecture

Two test configurations are defined and adopted in the IoT CoAp plugtest. Here, a System Under Test (SUT) is a network consisting of a server and a client with or without a gateway. Two test configurations/architectures were defined. The 1st configuration includes a client and server with packet sniffer and test operator represents an ideal situation whereas the second configuration represents a real scenario by simulating a lossy environment (replicates lossy environment of the constrained nodes in the network) by adding a gateway in-between client and server. SUT1: A client and a server with packet sniffer and test operator SUT2: A client and a server including lossy gateway with packet sniffer and test operator Devices in each SUT were connected with wired means.

2.3.1.

SUT1: Basic face to face configuration

Figure 5 below shows the test configuration in which, a client is connected to a server by means of wire. Client and server might from the same vendor or different. Generally, the initial configuration comes under test arrangement before engaging in interoperability testing session. Test operator provides preamble for the test suite before its execution. Since, it is a passive testing activity; the packet sniffer is used to capture the exchange between client and server to have a test report from the tool developed by IRISA.

Client

Server

Packet Sniffer Figure 5: SUT configuration 1

2.3.2.

SUT2: Basic face to face configuration in lossy context

Test operator

Figure 4 below shows the test configuration in which, a client is indirectly connected to a server by a gateway. This gateway was used to emulate a lossy medium. It does not implement the CoAP protocol itself (in other terms it is not a CoAP proxy), but works at the transport layer. It drops the packet randomly between Client and the server which goes nearly to 80 % packet loss as demanded by the participants. Like the SUT1, it also adopts the passive interoperability technique.

Client

Server

Packet Sniffer Figure 6: SUT configuration 2

PROBE-IT Page 8/16

CoAP White Paper

2.4.

Test suite structure

Test suites

CoRE

LINK

OBS

BLOCK

Test cases

Test cases

Test cases

Test cases

Figure 7: Test suite structure

Figure 7 shows the structure of the test suites found in the test descriptions document provided during the testing event. Test cases were developed to test different important functionalities needed to achieve interoperability among different vendo equipment. A total 27 test cases were developed, in that 16 vendors were considered as mandatory and 11 were considered as optional tests. These scenarios were developed considered to determine conformance against a standard and interoperability against other implementation by steps called Check and Verify. The steps in the test cases can be clearly seen in the following examp test example case. Example Test case # 1: There are four types of messages found in the CoAp core specification: Confirmable (CON), Non NonConfirmable (NON), Acknowledgement (ACK) and Reset (RST). Like HTTP, four different methods are defined in the specification: GET, PUT, POST, DELETE. : This example demonstrates the execution of POST transaction in CON mode between a Client and a Server with SUT configuration 1. Both these devices could be from the same vendor or different. With a pre pre-test condition as server accepts creation of new resource on /test (resource does not exists yet), the test cepts sequence starts by sending a POST request from client to server

PROBE-IT Page 9/16

CoAP White Paper

Identifier: Objective: Configuration: References: Pre-test conditions: Test Sequence:

Interoperability Test Description TD_COAP_CORE_02 Perform POST transaction (CON mode) CoAP_CFG_01 [1] 4.4.1, 4.4.3, 5.8.2 Server accepts creation of new resource on /test (resource does not exists yet)

Step 1

Type stimulus

2 3 4

check verify check

Description Client is requested to send a POST request with: Type = 0(CON) Code = 2(POST) An arbitrary payload Sent request contains Type value indicating 0 and Code value indicating 2 Server displays received information Server sends response containing: Code = 65(2.01 Created) The same Message ID as that of the previous request Client displays the received response

verify

Client

Server
PASS/FAIL

Sent POST request

Check : Type = 0 (CON) Code = 2 (POST)

PASS/FAIL

Check : Code = 65 (2.01 created)

Response

Client displays the received info

PASS/FAIL Figure 8: Message sequence chart

PROBE-IT Page 10/16

CoAP White Paper

2.5.
2.5.1.

Test tools used

Packet sniffer

Wireshark was used as a tool to capture the packets exchange in the network. It produces the pcap file which contains the transactions found in the network during the test session.

2.5.2.

Gateway for lossy context

The Gateway emulates a lossy medium between the client and the server. It does not implement the CoAP and protocol itself (in other terms it is not a CoAP proxy), but works at the transport layer. It provides two features: It performs NAT-style UDP port redirections towards the server (thus the client contacts the style gateway and is transparently redirected towards the server) ateway It randomly drops packets that are forwarded between the client and the server The idea of developing this tool was proposed by IRISA to verify the interoperability in a real scenario where random packet loss occurs. It was developed by BUPT.

2.5.3.

Post analysis of pcap files

IRISA team with support of PROBE-IT project provides a passive validation tool for the CoAP plugtests. This IT tool is based on the CoAP Test specification edited for the interoperability event. Figure 9 demonstrates interoperability the test tool working methodology adopted and tool developed by IRISA to provide testing verdicts and information on results. As shown in the figure 9, a web interface was developed to accept pcap files from . the vendors after the testing session to provide test log on interoperability testing. Once the pcap file was submitted, a UPD/CoAP filtering was made using source IP address and destination IP address to filter only the conversation made between particular implementations under test. When the transactions are en isolated (or grouped as CoAp conversation), then test cases found in the test description documents are conversation), executed.

Figure 9: Passive Test tool working methodology Such Passive Validation Tool For Co CoAP can be downloaded here: http://www.irisa.fr/tipi/wiki/doku.php/Passive_validation_tool_for_CoAP Figure 10 demonstrates the structure of result presentation in the end of the event. These results could be visually seen from different point of view. For example, It allows to see the verdicts corresponds to each frame, each transaction, and each test case. Three types of verdicts given in the end were PASS, FAIL and INCONCLUSIVE. The best thing about this passive validation tool is that it allows validating the CoAP . conversation anytime through the HTTP interface.

PROBE-IT Page 11/16

CoAP White Paper

Test case #1 IUT 1 vs IUT2 IUT 1 vs IUT3 IUT 1 vs IUT4 Test case #2 #4 Test case #3 Test case #5 Test case #6

Conversation1 Conversation2

Frame#1

Match :: PASS Mismatch :: FAIL/INCON

Figure 10: Result presentation methodology The figure 11 shows the web interface where pcap files should be updated, the test results found after updating the pcap files of the test session and information on the results obtained. It embeds the result presentation methodology shown in the figure 10.

Figure 11: Web interface of Passive test tool

3. Test tool supporters

3.1. IRISA

IRISA is a research institute in which Inria (the French National Institute for Research in Computer Science and Control), CNRS (the French National Center for Scientific Research), The University of Rennes I and INSA (the National Institute of Applied Science, an engineering school in Rennes) are partners. IRISA's activities are at the heart of Information and Communication Science and Technology (ICST) and contribute to achieve scientific and technological breakthroughs in telecommunications, multimedia information processing, software engineering, medicine and particularly bioinformatics. IRISA participates in many technology transfer activities with major industrial partners, and the participation in numerous European projects. IRISA's activities are at the heart of Information and

PROBE-IT Page 12/16

CoAP White Paper

Communication Science and Technology (ICST) and contribute to achieve scientific and technological cience breakthroughs in telecommunications, multimedia information processing, software engineering, medicine and particularly bioinformatics. IRISAs Dionysos team is dedicated to Computer Networks (Internet protocols and architectures, wireless, performance evaluation, security, traffic engineering, pricing, QoS, multicast, etc.) and Conformance and Interoperability Testing. The Dionysos team has a critical activity in the deployment of IPv6 and is activity involved in several IPv6 related programs (test generation, methodology and tools, interoperability events, the IPv6 Ready Logo program, standardization, participation to working groups of the IETF, training, etc.). working-groups Since 2000, our engineers bring expertise to ETSI Plugtest service and take part since 2001 in the annual sessions of interoperability organized by Japanese project TAHI.

3.2.

BUPT

Beijing University of Posts and Telecommunications (BUPT) is one of the 73 nat national key universities in China.BUPT has distinguished itself by its excellent teaching and research in the field of Information Technology and Telecommunications, with engineering as its main focus and multidisciplinary combination of Engineering, Managem Management, Humanities and Science as its main pursuit. BUPT is one of the leading research and development centre in the field of IoT in China. In order to promote IoT technologies, BUPT set up a new organization, Institute of Sensing Technology and Industry, in Wuxi City in Dec. 2009, with the registered capital of 20 million Yuan, which provides a common service platform to convert advanced new IoT technological achievements into productivity. BUPT will soon start a Chinese National funded project on 6lowpan te tests and tool.

4. Test campaign
A total of 16 test slots were allocated for two days event and the network setup was made available by ETSI. A test tool to analyse the CoAP transaction for verdict logging was made available online by IRISA 2 weeks before the event to have review from participants with a motivation of making the test event successful. Thanks to ETSI for the dynamic hosting and IRISA for pragmatic approach for the test tool development which made the event successful. The table below shows the name of the companies participated and components they brought for the event event.
Company name ACTILITY/Watteco ETH Zurich Hitachi Huawei Intecs KoanLogic Srl Patavina technologies ROSAND Technologies Sensinode ltd Toyota ITC USA TZI Uni bremen Vitaverna Watteco Uni Rostock RTX IBBT Consorzio Ferrara Ricerche Client X X X X X X X X X X X X X X X Server X X X X X X X X X X X X X X

PROBE-IT Page 13/16

CoAP White Paper

5. Overall tests results

A total of 3141 tests were executed during the test event and 94% of executed tests gave pass verdict. ETSI and PROBE-IT test experts say that getting 90% and above of pass verdict in a first Plugtests event for a new technology is a success showing that the tested components are almost near to be fully compliant and interoperable. There were a total number of 234 test sessions during this two days event. In each session 27 tests were executed and 388 tests were not executed due to time constraints and/or because some CoAP features (mostly from BLOCK and OBSERVE) were not yet implemented in tested components. Figure 12 shows the overall results obtained during the event. Figure 13 shows the result obtained for the CORE group of tests, in other words it is the overall results for mandatory tests. In a total of 2843 tests executed, 2679 (94.2%) of the tests passed. This confirms that most of tested components support necessary features of CoAP base specification.

Fig 12 : Overall results for executed tests

Fig 13 : Overall results for CORE group

Figure 14 shows the overall results of optional tests that are also near 91% of pass verdict, showing that most of implementations successfully support optional features. Even though there is not much difference in the optional test results, BLOCK group (86.3% of pass verdict) seems to have a little bit more remaining implementation issues.

Fig 14 : Overall results for LINK, BLOCK and OBSERVE respectively To conclude, this first CoAP Plugtests event was a success as 93% of tests executed were pass, which reveals that almost all implementations from the participants are almost mature enough to be interoperable with each other. These results also show the need for more such events in the future to improve both CoAP protocol specification and corresponding implementations paving by this way the road for successful deployment of this technology.

PROBE-IT Page 14/16

CoAP White Paper

6. Event organizers
6.1. PROBE-IT IT
PROBE-IT is a two years European project with international partners from IT China, Brazil and Africa that aims at supporting exploitation of European research advances in IoT deployments. It is indeed needed to ensure interoperability and acceptance of validated IoT solutions in a global context to avoid unnecessary competitions and overlaps. For that, PROBE PROBE-IT focuses on analysing existing and on going worldwide deployments in different on-going perspectives filling the needs o policy makers, deployments drivers, technology of providers and users. PROBE-IT is addressing three different areas: IT benchmarking of IoT deployments to provide stakeholders with decision tools aimed at identifying the best options when deploying or using IoT Guidelines for stakeholders to plan IoT roll roll-out testing roadmap and solutions to provide stakeholders with elements to validate technologies conformance and interoperability.

For each area, implementation is done in three phases: the first one focuses on developing tools and frameworks, the second one focuses on testing the tools and the third one focuses on analysing the results of testing and provide lesson learnt from existing deployments and final version of frameworks and tools. The work is supported by organizing regular workshops and interoperability test events. Visit: www.probeinteroperability it.eu.

6.2.

ETSI

ETSI, is an independent, non for-prot organization whose mission is to , nont produce Information and Communication Technologies standards. ETSI unites nearly 700 Members from ve continents, and brings together manufacturers, network operators, service providers, administrations, regulators, research bodies and users providing a forum in which all the key players can contribute. The ETSI TC M2M is currently examining how CoAP and the RESTful interfaces can be used to de dene horizontal Service Capabilities that can support multiple M2M applications over multiple core and access networks. visit: www.etsi.org As a complement to its core standards-making task, ETSI specializes in standards-making running interoperability test events for a wide range of telecommunication, interoperability Internet, broadcasting and multimedia converging standards. Plugtests events are open to all companies, organizations, working and study groups implementing a standard. For information about ETSI Plugtests and upcoming events, please visit: www.etsi.org/plugtests www.etsi.org/plugtests.

6.3.

IPSO ALLIANCE
The IPSO Alliance is the primary advocate for IP for smart objects for use in energy, consumer, healthcare and industrial applications. The Alliance, a non non-prot organization whose members include leading technology, t communications and energy companies, is providing the fo foundation for a

PROBE-IT Page 15/16

CoAP White Paper

network that will allow any sensor-enabled physical object to communicate to another as individuals do over the Internet. The IPSO Alliance membership is open to any organization supporting an IP-based approach to connecting smart objects. For more information, visit: www.ipso-alliance.org.

7. Conclusion
This edition of white paper describes about the first successful CoAP plugtest which organised by Probe-IT, ETSI and IPSO Alliance. The success of this event motivates to conduct more such events in near future to help the products to be successfully launched in the market. The participants were quite interested in the event right from the announcement of this event and made it a successful event. Future events are now scheduled also expanding to 6lowpan protocol. Probe-IT is planning to organise another event in October 2012 in China , Wuxi collocated with the www.iot2012.org

PROBE-IT Page 16/16