You are on page 1of 17

1

A Practical Approach towards Automatic Testing of


Web Services Interoperability
Ivano Alessandro Elia, Nuno Laranjeiro, and Marco Vieira

AbstractWeb Services are a technology designed to support the invocation of remote elements by
client applications, with the goal of providing interoperable application-to-application interaction while
assuring vendor and platform independence. In business-critical environments, the occurrence of interoperability issues can have disastrous consequences, including direct financial costs, reputation, and
client fidelity losses. Despite this, experience shows that services interoperability is still quite difficult
to achieve. The goal of this paper is to propose a practical testing process to understand the real level of
interoperability provided by web services platforms. An extensible tool, that implements the proposed
approach, has been used to run a large campaign during which we have tested the interoperability of a
large number of web services, comprising both home-implemented and publicly available services, deployed on top of several web service platforms and against 11 client-side web service platforms. Numerous issues have been disclosed, showing the effectiveness of the proposed approach and the importance of having an automatic tool for testing web services for interoperability.
Index Terms Interoperability, Web service, Testing, Testing Tools

1 INTRODUCTION
Web services are being increasingly used to integrate enterprise systems, linking complex applications
in large-scale organizations and supporting critical business-to-business interactions, where the typically heterogeneous environments frequently demand a strict definition of service interfaces and interaction patterns. In such heterogeneous scenarios, systems designers frequently opt for using web services
technology, as it was created with the key goal of supporting the inter-operation between different systems, including systems built using different programming languages (Erl, 2005).
In a typical services environment a service provider announces its interface using a WSDL document
that describes, in a programming language agnostic way, the operations available to the clients. In turn,
clients make use of the interface description to understand how the service operations can be invoked
(Curbera et al., 2002). Nowadays, developers do not need to implement all supporting code for generating services or invoking operations, as most web service frameworks (platforms for creating, deploying,
and invoking web services) provide automatic code generation facilities and runtime support for invoking service operations (Hewitt, 2009). The problem is that, although web services are supported by machine independent protocols and have been designed with interoperability in mind, practice shows that
it still is quite difficult to link distinct applications. This is especially true in complex and more heterogeneous scenarios, such as the ones frequently found in enterprise environments.
The interoperability of web services is an issue in which the Web Services Interoperability Organization (WS-I) (Web Services Interoperabily Organization (WS-I), 2002) has been working for several
years. However, practice shows that developers still struggle to deploy services that can fully interoperate with client applications. In fact, developers many times create and deploy their web services
expecting the underlying framework to provide full interoperability, which is not always the case. Our
tests, presented later in the paper, confirm this scenario, showing that the inter-operation between different frameworks is not yet fully achieved and highlighting many cases where inter-operation is not
possible.
A key problem is that, although efforts have been undertaken towards creating tools to test services
(Eviware, 2011, WS-I, CrossCheck Networks), they appear to be quite limited, when it comes to interoperability testing. Among all tools, those provided by the WS-I Testing Tools Working Group (WSI,) are definitely a step forward, but experience suggests that even WS-I compliant services may show
interoperability issues. Thus, there is a clear need for a specialized approach for interoperability testing
that helps developers conducting broader interoperability assessments, from a practical perspective.

I. A. Elia, N. Laranjeiro, and M. Vieira are with the CISUC, Department of Informatics Engineering University of Coimbra, 3030-290, Portugal.
E-mail: ivanoe@dei.uc.pt, cnl@dei.uc.pt, mvieira@dei.uc.pt

We presented a tool for interoperability testing of web services in (Elia, Laranjeiro, & Vieira, 2014b).
In (Elia, Laranjeiro, & Vieira, 2014c) and (Elia, Laranjeiro, & Vieira, 2014a) we have conducted experiments regarding part of the full inter-operation process (which in the end involves communication). In
this paper we greatly extend the previous work, by having refined the approach and formalized key aspects, extended the approach to now include the exchange of messages between client and server, and
by augmenting the tests and tool implementation to cover the approach extension and thus covering the
whole inter-operation process.
We propose, in this paper, a practical approach to support the evaluation of all the steps of the interoperation process in web services environments. We also present ITWS (Interoperability Testing for
Web Services), an open-source web services testing tool available at (Elia, Laranjeiro, & Vieira) that
supports the execution of the proposed approach. Developers can use ITWS to evaluate the level of interoperability attained by their services (running in the context of a given framework, i.e., a platform
that provides the deployment and runtime support for web services) with regard to one or more clientside frameworks (that offer client-side support for invoking the web services operations). ITWS currently supports testing a set of well-known web service frameworks, but the tool was designed to be extensible and adding more frameworks requires only a simple configuration effort.
The ITWS tool was used to run a massive test campaign involving about 8 thousand web services,
comprising over seven thousand home developed services deployed on top of three major WS frameworks and over one thousand web services publicly available on the Internet and also deployed on top
of multiple frameworks leading to a total of 174547 interoperability tests.
The outline of this paper is as follows. The next section presents the web services inter-operation
problem and discusses existing techniques and tools for interoperability testing. Section 3 describes our
approach for testing interoperability of web services and Section 4 provides an overview of the ITWS
tool. Section 5 presents the experimental evaluation carried out using the tool and Section 6 summarizes
the key findings. Finally, Section 7 concludes this paper.

2 THE WEB SERVICES INTER-OPERATION PROBLEM


In a typical web services environment, the provider (i.e., the server) offers a well-defined interface to
consumers (i.e., clients), which includes a set of operations and typed input/output parameters. Clients
and servers interact by using a web service framework that provides a set of mechanisms to support the
execution of the following key stages: 1) the service is deployed at the server, along with a service interface description (i.e., a WSDL is published); 2) a client developer generates (and compiles, when required) client-side artifacts to easily invoke the service operations; 3) both client and server applications
communicate by exchanging SOAP messages that are generated by the underlying frameworks, on behalf of the client and the server applications (Curbera et al., 2002).

2.1 The five steps of the inter-operation process

Fig. 1 represents a typical web services inter-operation scenario using two different frameworks at the
client (on the left-hand side of Fig. 1) and server sides (in Fig. 1, at the right-hand side), which are
placed at each endpoint of the interaction. In the scenario represented, the client uses the client-side
subsystem of framework A, while the server uses the server-side subsystem of framework B (however
these could also be two subsystems of the
same framework). As shown, a set of 5 key
(1)'Service'
steps needs to be performed for a client to inDescrip*on''
Framework'A'
Framework''B'
voke a remote web service operation. These
Genera*on'
steps represent the critical points where platAr*fact'Generator'
form-level interoperability issues can arise.
(2)'Client''
The Service Description Generation step
Ar*fact''
WSDL'
Genera*on'
(1) typically occurs automatically when the
Service'
service is deployed at the server, although it
Client'Ar*facts'
can also be executed manually, for instance by
(4)'Client''
Compiler'
using directly the WS framework tools and the
(5)'Service'
Communica*on'
Processing'
service code as input. The result of this step is
(3)'Client''
a WSDL document that provides the inforSOAP'
Ar*fact''
Client'
mation needed by a client that uses framework
Compila*on'
A to invoke the service provided by framework
B.
Client'side'
Server'side'
During the Client Artifact Generation step
Fig. 1. A typical web services environment.
(2) the clients artifacts are produced using the

artifact generation tool provided by framework A. These artifacts are pieces of code that, at runtime,
will translate application-level calls at the client to SOAP messages that will be delivered to the server.
Some frameworks do not produce source files; instead they generate these artifacts dynamically, at
runtime.
The Client Artifact Compilation step (3) is necessary only for platforms that are implemented in
programming languages that require compilation of the code before execution (e.g., Java, C#), and in
some languages/frameworks (e.g., Python) it may not be required (the client-side artifacts are dynamically generated at runtime). Obviously, the artifacts (that after this step may be in a compiled form) only
provide the methods for a client application to invoke the remote service, but do not actually invoke it.
It is up to the developer to create the client code to invoke the artifact methods that allow communicating with the server.
Communication is represented by the Client Communication step (4) in Fig. 1, which in practice results in sending a SOAP message to the server and ends with the delivery of that message to the service
application. This message is processed at the server during the Service Processing step (5) and a response SOAP message is produced and sent back to the client, carrying the result of the operation. The
entire process ends when this message arrives at the client application.

2.2 Approaches and Tools for Interoperability Assessment

A set of interoperability issues in the web services technology is analyzed in (Senthil Kumar, Das, &
Padmanabhuni, 2004). These include aspects related with the representation of sequences of elements
and namespaces in WSDL files. The authors also analyze how the WS-I Basic Profile 1.0 (Keith Ballinger, Ehnebuske, Gudgin, Nottingham, & Yendluri, 2004; Web Services Interoperabily Organization
(WS-I), 2002) tries to address the raised issues and highlights the limitations of the WS-I set of interoperability recommendations. Conclusions include the fact that adhering to the WS-I recommendations
helps reducing the web services interoperability issues. However, a more extensive and pragmatic view
is still needed, as the WS-I Basic Profile 1.0 does not cover the practical support for interoperability
provided by current web service frameworks.
The authors of (Ramsokul & Sowmya, 2006) trace back the interoperability issues to the use of a
non-formal language in the specifications, but also recognize that problems can occur much later with
improper implementations of the protocols. A key aspect is that no practical insight is provided on the
interoperability of different frameworks currently being used in the field. Obviously, when deploying an
interoperability-critical system, a practical assessment is of utmost importance.
Interoperability problems can occur when, for instance, native data types or specific language constructs of the service implementation are present at the interface. The problem is that, to the best of our
knowledge, there is no standard indicating which types are adequate to be used in a service interface. In
(Pautasso, Zimmermann, & Leymann, 2008) authors confirm precisely that it is difficult to identify the
right constructs to express the data types that are fully supported by all service frameworks.
The WS-I Testing Tools Working Group has developed tools that are able to check the compliance of
a given web service with various documents, named Profiles (K. Ballinger et al., 2006; Web Services
Interoperabily Organization (WS-I), 2002, 2014). These Profiles are a set of non-proprietary web services specifications, including clarifications, refinements, interpretations and their amplifications,
which promote interoperability. Since our focus is inter-operation, the WS-I Basic Profile and respective tools are closely related to our work (K. Ballinger et al., 2006; Web Services Interoperabily Organization (WS-I), 2014). As previously referred, although the WS-I effort represents a first step, experience
suggests that even WS-I compliant services may present interoperability issues, and obviously a more
practical and realistic insight of a given platforms interoperability is many times preferred to conceptual results. This is especially true in systems where interoperability is critical (e.g., systems supporting
business-to-business interactions).
SOAPSonar (CrossCheck Networks, ) is a web services testing tool, described by its development
company as the global leader in service and API testing. In what concerns interoperability, the tool is
able to check for WS-I Basic Profile 1.1 compliance. SOAPSonars website claims: Adhering to WS-I
Basic Profile 1.1 ensures that your components are interoperable and that your WSDL can be consumed
by any .NET or Java client. In this paper, we show that this is not the case, as our tool outperforms
SOAPSonar in WSDL checking for interoperability, by being able to disclose issues that the referred
WS-I Basic Profile does not cover.
SoapUI (Eviware, 2011) is a well-known open-source tool for testing web services that is able to create various types of tests, including functional, load, and security tests. SoapUI supports compliance
testing by including automated execution of the WS-I Basic Profile 1.1 test suites, for which we already
highlighted the limitations. SoapUI also supports the generation of client-side artifacts with a limited set
of frameworks. However, developers do not have a way of adding a new framework (or in some cases,

not even a new version of an existing one) without changing SoapUIs code, which obviously is not
practical. In ITWS, plugging a new framework has been simplified to the execution of a uniform configuration effort. Also, the analysis of the results can be conducted in a uniform way for all the different
frameworks being used.

3 INTEROPERABILITY TESTING APPROACH


In this section we present our approach for testing the interoperability of web services. The approach is
based on tests that target the 5 key inter-operation steps presented in Section 2 (see Fig. 1 for details).
We describe how each step is tested and how we evaluate the result of each test. A key characteristic of
the approach is that tests are performed in a realistic environment by actually putting to work real-world
tools and frameworks currently being used by service developers and providers. Fig. 2 presents an outline of the proposed testing approach, in which we grouped the testing of the 5 inter-operation steps in
two evaluation phases.
As we can see in Fig. 2, during the Generation Phase the tests target the first three inter-operation
steps, which represent the generation of all interfaces and supporting code needed to support the actual
communication between client and server (with the exception of the actual client implementation,
which at the moment must be provided by the client developer). During the Execution Phase, the code
produced is actually executed and communication between client and server is carried out. Each step in
the approach is followed by a classification procedure that can result in one of the following three cases: an Error is detected (i.e., a fatal event that prevents further inter-operation occurs); a Warning is detected (the presence of one or more non-fatal events, which do not prevent further inter-operation steps,
although they might be an indication of some problem); the outcome of the step is Correct (no fatal or
non-fatal issues observed).

3.1 Generation Phase

As mentioned before, the inter-operation process begins with the Service Description Generation step,
which includes the deployment of a web service and the generation of an interface description (i.e., a
WSDL file) by the server-side framework. After the step concludes, we carry out a classification procedure, which consists of checking the service interface for WS-I Basic Profile compliance (Web Services
Interoperabily Organization (WS-I), 2014). In practice, the classification procedure can result in one of
the following cases:
Error: When the WSDL is absent, i.e., when the framework has failed to produce the WSDL we
classify the event to be an Error as it is not possible to continue to the next inter-operation steps.
Warning: When the WS-I toolset raises at least one interoperability issue we mark the outcome
of the step as a Warning.
Correct: All other cases where no problematic events (fatal or non-fatal) are observed.

Classica&on(

(5)(Service(
Processing(

Execu&on(Phase(Evalua&on(
Fig. 2. Interoperability testing phases.

Classica&on(

(4)(Client(
Communica&on(

Classica&on(

The goal of this step is to uncover potential


artifact generation issues that can prevent developers from creating a functional web service client. Note that if the WSDL is correct,

Classica&on(

Classica&on(

The next step is the Client Artifact Generation during which we test the capability of the client-side
framework to produce client-side artifacts. In practice, the step consists of using one or more client-side
artifact generation tools (e.g., Metros wsimport (wsimport), JBossWS wsconsume (jBoss: wsconsume))
to try to generate client-side code from the WSDL produced in the previous step. Again, the step is followed by a classification procedure in which we verify the occurrence of one of the following cases:
Error: The client artifact generation tool failed to produce any output, which might be signaled
by the tool being used, or not.
Warning: The artifact generation tool
produces output but also points out
Genera&on(Phase(Evalua&on(
some issue during its execution. For
(2)(Client(
(3)(Client(
(1)(Service(
instance, a tool can inform that altDescrip&on(
Ar&fact(
Ar&fact(
hough it produced some client-side
Genera&on(
Compila&on(
Genera&on(
code, it was also unable to generate
Client((
code for some operations.
Implementa&on((
Correct: No observation of an Error
or Warning.

the client artifact tools should generate artifacts without any issue. On the other hand, if the WSDL
document is problematic (e.g., which is obvious when the previous step results in a Warning), the artifact generation tool should detect the presence of a problem and provide useful information to the service developer. Erratic artifact generation tools might silently produce the artifacts without signaling a
problem (and this problem may only prevent inter-operation in a later step).
A relevant aspect in this step is that some frameworks do not support generation of the usual static artifacts (which exist in a physical file); instead, they provide support for in-memory artifacts, which are
generated at runtime. In these cases, the classification procedure should verify if the in-memory artifacts, or any other support structure, can be instantiated. It should be possible to create them without
triggering any fatal event (i.e., an Error that prevents further execution of the client application and,
thus prevents further inter-operation, such as an Exception being raised at the client) or non-fatal (i.e., a
Warning, such as an informative warning produced at the client-side). The goal is to verify, only to the
possible extent in this step (i.e., without proceeding to the next steps), if it is possible to generate artifacts without visible issues being observed.
The third step is the Client Artifact Compilation, in which we try to compile the artifacts generated
in the previous step (e.g., using Javas compiler). If the artifacts cannot be compiled, it is a clear sign
that something is not correct and anyway no further inter-operation steps can be taken. The classification procedure carried out after this step consists precisely of evaluating the output of the compilation
tools looking for one of the following cases:
Error: No compiled form of the code is produced.
Warning: The compilation tool is able to compile the generated code, but displays some warning, suggesting some problem (e.g., unsafe operations being used).
Correct: No issue is observed.
This step concludes the Generation Phase. The next phase involves the exchange of SOAP messages
and is described in the next Section.

3.2 Execution Phase

The client artifacts produced in the previous phase are not a fully functional client, but rather provide
the developer with the support to invoke the operations provided by the service. To effectively test the
two final steps (Client Communication and Service Processing), a complete client must be implemented
at this point (starting with the generated artifacts) and compiled and executed. Executing the client begins the Client Communication step, which finishes with the delivery of a message to the web service
application. Before reaching the service, the message must pass from the client application through the
client-side framework (i.e., in which it is converted to SOAP), the network, then pass through the server-side framework (during which it is converted from SOAP to a programming language level object),
and finally is delivered to the service application. For interoperability testing purposes, the service implementation should be empty of any business logic extraneous to the tests (as we are testing the correctness of the underlying middleware, and not specific service implementations).
The classification procedure in the Client Communication step consists of observing the delivery of
the client message to the service implementation. In practice we analyze the message contents at two
observation points: immediately before leaving the client application (and before being transformed into
SOAP by the client-side framework); and immediately after reaching the service implementation (i.e.,
after passing through the server-side framework). These two points are used for message content comparison and can be achieved, for instance, by using logs. The outcome of the classification is:
Correct: When the message content is the same at the two observation points.
Error: The message is not delivered to the service application (e.g., due to the client failing to
send the message, or due to the server failing to parse it). This is a fatal event, as it prevents the
execution of the last inter-operation step (Service Processing).
Warning: A message is delivered to the service application, but is holding a different content
from the one created at the client (including missing content). This is obviously a problem, yet it
does not prevent the execution of the next step, the Service Processing (i.e., it represents a nonfatal event).
In the final step, Service Processing, the services business logic is executed and the outcome is the
delivery of a response back to the client. This involves having the message going back from the service
application to the client application (and again passing through the server-side and client-side frameworks). Thus, our two message observation points are now the service application (immediately before
the message leaves the service code) and the client application (immediately after the message arrives at
the client application).

In the same manner as before, a classification procedure takes place and the outcome is:
Correct: When the messages content is the same at the observation points.
Error: When no message is delivered to the client application, for instance, due to problems in
building a response at the server-side framework or parsing the response at the client-side.
Warning: A response that is delivered to the client application but is holding incorrect or incomplete information.
Note that, in all steps, we assume correctness from the environment external to the web service frameworks at the server and client sides (e.g., during the Execution Phase we assume there are no network
failures during the SOAP message exchange).
4

THE ITWS TOOL

In this section we present the ITWS, a web services interoperability testing tool that implements the approach described in the previous section. ITWS provides an easy way to test web services for interoperability and was designed to overcome the limitations of current tools, including the limited extensibility
and reduced interoperability analysis, as previously referred.
ITWS provides support for the inter-operation process presented in Section 3. Currently, the tool automatically handles testing all steps of the Generation Phase. It also provides partially automated support for testing the Execution Phase. The limitation in this phase is that the tool requires client code to
be provided in order to complete the inter-operation process. Although the tool could also generate this
code automatically, we consider this to be a mere technical aspect and not part of the core of our proposal. Anyway, this is an aspect that we intend to develop in future versions of the tool. In the following
sections we describe ITWSs architecture with respect to the approach steps and tool usage.

4.1 ITWS components and testing steps

Fig. 3 presents the application-level architecture of ITWS, also including client framework tools (at the
top of the left-hand side in Fig. 3) and deployed services used (at the bottom left in Fig. 3). The following paragraphs describe the tools main components and their role in the execution of the testing steps.
The Startup component (see Fig. 3) is responsible for initiating (and performing a clean shutdown) of
ITWS. It also sets up the application by instructing the AppConfig component to load application-level
configuration from configuration files. This information includes environment configuration, application-level paths, and a list of service URIs to be tested (refer to Section 4.2 for details on all required
configuration).
The Test Manager is triggered by the Startup component (after the AppConfig finishes its tasks) and
is the core part of the application. The Test Manager supervises a test cycle (each test cycle is executed
once for each service to be tested) that comprises the 5 key inter-operation steps previously referred.
Note that a user may also configure ITWS to execute only part of this cycle (e.g., just the first two
steps). Despite this option, we describe the case for a typical full-cycle execution in the next paragraphs.
For each service to be tested, the Test Manager will start by executing the assessment of the first step
of this testing cycle: the Service Description
Generation. This consists of contacting the WS-I
Client+Tools++
Compliance Tester, which in turn will test the
Axis2.conf+ + Congura8on+
ITWS%Applica,on%
Axis2.conf
Axis2.conf
+
Axis2.conf+ +
Axis2.conf
service for WS-I Basic Profile compliance [16].
Problems++
Tool++
Results+
Results+
The WS-I Basic Profile is the current standard for
Results+
Iden8er++
Cong+
Results+
Results+
understanding the interoperability of a particular
Deployed+Client+Tools+
web service. Many tools provide support for the
Client+ Client+ Client+
Client+ Client+ Client+
Results+
Tool+
+Tool+
+Tool+ Client++
Client++
+Tool+ Client++
Client++
+Tool+
+Tool+ Client++
+Tool+ Client++
older version of Basic Profile (v1.1) (CrossCheck
Tool++
Tool++
Tool++ Tool+
Manager+ handler+
Tool+
Tool+
Networks),(Eviware, 2011). The newer versions
WS@I++
Test+
of Basic Profile (v1.2 used for SOAP 1.1 and v2.0
WS@I+
Test+
compliance+
compliance+
used for SOAP 1.2) appear to be not very comreport+
Manager+
Tester+
Deployed+Services+
mon yet (we found only one application providService+
Service+
Service+
Service+
ing direct support for these tests (yaronn)). DeApp+
Service+
Service+
WSDL+
WSDL+
Service+
Service+
WSDL+
WSDL+
Startup+
Service+
Service+
WSDL+
WSDL+
spite this, for the sake of completeness, ITWS
Cong+
WSDL+
WSDL+
WSDL+
WSDL+
uses both versions of the Basic Profile in our
Applica8on+
tests.
ITWS.conf+
List+of+Services+ Services.conf+
+Congura8on+
In short, ITWS evaluates the interoperability of
the WSDL document produced during service
Fig. 3. ITWS architecture and testing environment.
deployment, according to the WS-I guidelines.

The user always has the option to analyze the documents produced by the WS-I tool, but ITWS clearly
indicates if the WSDL passes WS-I Basic Profile or not (i.e., to mark the presence of a Warning, or not).
This latter task is executed by the Results Handler component, which basically aggregates the WS-I
tool output (i.e., the identification of problems is carried out by the WS-I tool).
After the previous step, ITWS proceeds to the second step of the testing cycle: the Client Artifact
Generation. For this, the Test Manager contacts the Tool Manager, which will instruct the Tool Configuration component to load the configuration of all deployed client-side tools for artifact generation. After configuration finishes, the Test Manager will ask the Tool Manager to execute the next available client artifact generation tool. When the client tool finishes execution, the Tool Manager invokes the Results Handler component.
The Results Handler receives the output produced by each client tool and, when applicable, delegates
the analysis on the Problems Identifier component, which uses regular expressions to identify different
types of inter-operation problems (such as Errors and Warnings) that may have been found during the
tool execution. Obviously, this classification of problems is largely dependent on the client tool being
executed; therefore, this information is also part of the client tool configuration and is provided by the
Tool Manager. The user is also able to tune the default classification procedure (Section 4.2.4 describes
how to customize the configuration of the Problem Identifiers). In the case of frameworks that only
support dynamic generation of artifacts (i.e., no files are created), such as PHP or Python frameworks,
ITWS includes scripts to verify the instantiation of client-side objects.
The tool can now proceed to the third step of the testing cycle - Client Artifact Compilation. This
step is also framework-specific and therefore is executed in the same manner as in the previous step.
The difference now is that a compilation tool is being executed (instead of a client artifact generation
tool) and the identification of problems may be carried out in a different manner (e.g., using different
regular expressions), but in ITWS all of these aspects are simply configuration details. ITWS automatically compiles the client code generated in the previous step. Obviously this step is not executed when
the framework/language does not support compilation. As in the previous step, a default (but configurable) classification procedure based on regular expressions is used to identify Errors and Warnings.
The artifacts generated and compiled up until this point do not compose a fully functional client. The
client creation requires the intervention of the client developer to implement the actual code. This code
will then use the support functions provided by the client artifacts to consume the service. As discussed
in Section 3, to proceed to the two final steps of the testing we need the client code to be placed in the
artifacts directory (see Section 4.2.3 for details on this procedure). After the compilation is carried out
ITWS can be used to execute the two final steps.
The Client Communication step is also initiated by the Test Manager component that orchestrates
the execution of this step by using the Tool Manager component, which in turn starts by recompiling
the full set of the client code to generate an executable client and then executes the actual client. The
identification of problems is performed by configurable Problem Identifiers and assumes the presence
of a log (or redirected console output) produced by the service from which ITWS can read and analyze
the output parameters (which should be the same as the ones sent by the client).
The Service Processing step again assumes the presence of an information log (or redirected console
output) in which is possible to read the response of the service as perceived by the client. This response
should be equal to the one produced by the service and is again classified by the Problems Identifier.

4.2 How to use ITWS to test Web Services

In this section we illustrate how a web services developer can use ITWS to test the inter-operation of a
given set of services against several different client-side frameworks. This section includes: a) how to
setup the testing environment; b) how to use ITWS to test the generation phase of the inter-operation
process; c) how to test the execution phase of the inter-operation process; d) how to extend ITWS to use
more client-side frameworks.

4.2.1 Preparing the Testing Environment


The ITWS application is quite modular and independent from the client artifact generation tools (and
obviously from the services to be tested). In practice, this means that the application will be made aware
of the location of the tools and web services via configuration files (which can also be deployed independently of the applications location). The following steps describe how to setup the environment to
use ITWS.
1. Unzip the ITWS package: unzip the archive on a given location on the system. This results in
the creation of a main working directory (which we will refer to, from this point onwards, as
%ITWS_HOME%), which contains the following three subdirectories: one named tools will host
the client tools; a second named logs will store the logs of the tests results; the third one is

TABLE 1
named testing and will store the generated client artifacts.
CLIENT SIDE FRAMEWORKS
2. Deploy the desired client artifact generTools
ation tools: any client code generation
Framework
Generation
Compilation
Language
tools must be copied to
Tool
Tool
%ITWS_HOME%/tools directory (each
Oracle Metro 2.3 (Oracle,
wsimport
javac / ant
Java
tool under its own subdirectory). Table
2013b)
1 presents the list of pre-configured
Apache Axis1 1.4 (Apache
wsdl2java
javac
Java
Software Foundation, 2008)
tools and describes the name of the
framework, the executable code genera- Apache Axis2 1.6.2 (Apache
wsdl2java
javac / ant
Java
tion tool (for simplicity, code interpret- Axis2 Web Service Framework)
Apache CXF 2.7.6 (Apache
wsdl2java
javac / ant
Java
ers are also included in here), a compiSoftware Foundation, 2013)
lation tool, and the target language of
JBossWS CXF 4.2.3 (Red Hat
wsconsume
javac / ant
Java
the generated code. The user might
Middleware)
need to setup environmental variables
csc
C#
Microsoft WCF .NET Framerequired by specific tools (e.g., setting
work 4.0.30319.17929 (Chapwsdl.exe
vbc
VB .NET
pell, 2009)
JAVA_HOME).
jsc
JScript .NET
gSOAP Toolkit 2.8.16
wsdl2h.exe and
3. Deploy the desired client artifact comgcc
C++
(Engelen, 2013)
soapcpp2.exe
pilation tools: This means installing
Zend Framework 1.9 (Zend
PHP interpreter
N/A
PHP
compilers, such as the ones in Table 1.
Technologies, 2013)
Note that these tools are many times alsuds Python 0.4 (Ortel)
Python interpreter
N/A
Python
ready present in a typical development a) The tool does not compile automatically. A script was added to perform the
environment. As in the previous step,
task; b) Compilation via a tool-generated ant task; c) Compilation is not possible
and depending on the tool, the user
with these langauges; client object instantiation is checked.
might need to do tool-specific configurations.
4. Deploy the services to be tested: as in the client-side, also the server-side platforms are deployed
independently from the ITWS application. The server-side web service framework(s) must be
installed and the services that need to be tested must be deployed. This is a developers task that
should not be managed by ITWS (otherwise we would be interfering with the inter-operation
process, and thus potentially tainting the results). The ITWS user simply needs to provide a
plain-text list of the services URIs that will undergo testing in %ITWS_HOME%/services.conf.
a)

b)

a)

a)

a)

a)

c)

c)

4.2.2 Using ITWS to test the Generation Phase


The execution of interoperability tests with ITWS is quite simple and it consists of the following two
steps:
1. Execute ITWS: The application can be launched from a command line by simply executing
java -jar ITWS.jar. Without further options, the application will be able to find the path to:
%ITWS_HOME%; the path to the services list file (services.conf); and all client generation
tools to be used. If customization is required for
TABLE 2
these aspects, the user can provide command line
INFORMATION IN THE RESULT FILE
switches: respectively, -w, -s, and -t (see the next
Field
Description
section for details on tool configuration).
Test Identifier
A unique numeric identifier of the test.
Tool Name
An identifier of the client tool used.
2.
Examine the results: After the apTool Version
The version of the tool used.
plication
finishes
performing the tests, the results
Service URI
URI of the service used in the test.
will
be
stored
in
%ITWS_HOME%/logs.
Each exResult of the WS-I Basic Profile test. The value can be
WS-I Validation
OK or ERROR.
perimental campaign will generate one result file
Type of problem found during the Client Artifact Gen(in csv or XML format). The results will include
Artifact
eration step. When no problem is found, the value will
Generation Prob- be OK. When a problem is detected, the default conone entry for each performed test cycle (i.e., one
lem
figuration (for the pre-configured tools) sets an ERROR
line per each service/client framework pair). Table
tag or a WARNING, depending on the tool output.
2 describes the information present in each line of
Artifact Generation Full output of the generation phase. Also includes the
Output
invoked command.
the tests results file.
Type of problem found during the Client Artifact CompiITWS identifies problems by defining two cateArtifact
lation step. If no problem is found, the value is OK.
gories for test executions: Errors and Warnings, as
Compilation Prob- When a problem is detected, the default configuration
lem
sets an ERROR tag or a WARNING, depending on the
discussed in Section 3. ITWS already provides
tool output.
configuration for the complete set of default clientArtifact Compila- Full output of the compilation phase. Also includes the
side tools (refer to Table 1 for the list of all pretion Output
invoked command.

configured tools). If needed, the user can customize the problem identification scheme (see point 3 in
Section 4.2.4). Obviously, in some complex cases the user might need to investigate the nature of the
problems by manually analyzing the produced client artifacts (in %ITWS_HOME%/testing). The artifacts of each test cycle have their own numbered separate subdirectory; this number is the same present
in the results output file. Thus, identifying a specific test is quite easy.

4.2.3 Using ITWS to test the Execution Phase


As mentioned, to test the Execution Phase the client developer must provide the client code. The full
procedure consists of the following steps:
1. Add the client code alongside with client artifacts to %ITWS_HOME%/testing.
2. Create a client.properties configuration file holding three properties: path.client.exec the path
of the newly added client; path.client.log the path to the client logs; path.server.log the path
to the server logs.
3. Execute ITWS: The application can be launched from the command line with java -jar
ITWS.jar. This time give the e option to tell the application to test the client execution and service invocations steps.

4.2.4 Extending ITWS


Although we provide configuration for multiple well-known tools, in some cases a developer may want
to add a new client framework to the pool of client artifact generator or compiler tools. From the ITWS
point-of-view, this is a quite simple task that involves three steps:
1. Deploy the new client tool: the new tool must be deployed in %ITWS_HOME%/tools directory.
2. Create a tool properties configuration file: the configuration file must include the configuration
for the Client Artifact Generation and Client Artifact Compilation steps. All properties are described in Table 3.
3. Configure the problem identifiers: The problem identification process is configured on a pertool basis, as the syntax of the output varies a lot among the different tools. The identification of
the problems is configured using the GenProblems and CompProblems properties in the tool
configuration file (also referenced in Table 3). This configuration is quite simple (configuration
examples are provided with the tool at (Elia et al.)), and consists of defining one or more categories of problems (e.g., Errors, Warnings). To define a category the user must provide a category
name and a regular expression that will be used by the application to analyze each client tool
output. If the user defines multiple categories, the tool might also signal multiple matches (depending on that definition).
After these 3 steps the client framework tool can be used in tests just as any of the other preconfigured tools.
TABLE 3
CONFIGURATION OF A CLIENT-SIDE TOOL
Field
Name
Version

Env

GenScript

GenOptions

GenProblems
CompScript
Comp
Options
Comp
Problems

Description
An identifier of the client tool used
A version for the tool being used
Comma separated environmental variables needed to call the
generation (or compilation) tool.
Example: AXIS2_HOME=%ITWS_HOME%/tools/axis2-1.6.2. Optional,
when the developer has a properly configured system.
Relative path for the location of the artifact generation tool.
Command-line options to be applied to the artifact generation
tools. The %OUTPUT_DIR% and %URI% variables can be
used to dynamically refer to the default output directory and
service to be tested (values will be set by ITWS at runtime and,
as such, will be part of the command that will be executed).
Categories of artifact generation problems to be identified.
Relative path for the location of the compilation tool.
Options for the compilation tool. The %OUTPUT_DIR% variable can be used as in the GenOptions property.
Categories of compilation problems to be identified.

EXPERIMENTAL EVALUATION

In this section we present and discuss the results


of a large set of inter-operation experiments conducted using the ITWS tool. In the experiments
we tested two sets of services (adding up to 8263
services), deployed in multiple frameworks,
against 11 different client-side frameworks and
resulting in a total of 174547 tests.
The first set of services includes 7239 homedeveloped services, built with the specific purpose of being used for testing. The second set is
composed of 1024 public services freely available
on-line. In the experiments for the homedeveloped services, we address all steps of the
Generation and Execution Phases. However, for
the public services we can only address the Generation Phase, as since the service is not under our
control, we do not have access to the message observation points (as discussed in Section 3) at the
server-side. The next sections describe the tests

10

with each set of services and explain the results.

5.1 Home-developed Services

In this section we present the results of an experimental evaluation using 7239 home developed web
services deployed on top of the three major (server-side) WS frameworks: Oracle Metro 2.3, JBossWS
CXF 4.2.3, Microsoft WCF .NET 4.0.30319.17929 (C#) (Chappell, 2009; Oracle, 2013b; Red Hat Middleware). The services were hosted in major application servers, respectively GlassFish 4, JBoss AS
7.2, and Microsoft IIS 8.0.8418.0 (Express) (GlassFish; Microsoft Corporation, 2013; Red Hat Middleware, 2013).
As mentioned, for this set of services we tested both the Generation Phase and the Execution Phase.
The results for the Generation Phase involve the complete set of server-side resources (i.e., the services
interfaces, generated by the three frameworks on behalf of the services), and the complete set of clientside resources (the eleven client-side frameworks). The results for the Execution Phase, due to the intensive manual work required in creating each client, involve a subset of the services (2489 services
supported by Metro and Glassfish) and one client-side framework (also Metro) supporting each client.
It is important to emphasize that in the tests for the Execution Phase we have the server and client-side
of the same framework, for which we were expecting to see none, or very few interoperability issues,
which was not the case.

5.1.1 Creation of test services and clients


Since our aim is to test inter-operation, the focus is on the input and output parameters of the services
the critical inter-operation points in this technology. Therefore, the creation of services required, for
each one, developing a single operation with one input and one output variable of the same type (no
business logic is required, as we are testing the correctness of middleware and not the service application). In practice, the operation simply returns the provided input without further processing.
A code generation script (Elia et al.) has been used to automatically create thousands of services,
each of them having as input/output all the different native data types of the programming language
supported by the framework. This process led to the initial generation of 3971 Java services for GlassFish, 3971 Java services for JBoss AS, and 14082 C# services for the .NET framework (one for each of
the data types supported by the corresponding programming language). To gather the list of native classes in the two languages we used automated scripts (Elia et al.) to crawl the online documentation (Oracle, 2013a), (.NET Framework Class Library (C#), ).
We tried to deploy all generated services but ended up having to exclude many, since some data types
cannot be used as input or output of a service (e.g., java.awt.Desktop, System.TypedReference) and thus
the frameworks were unable to deploy them. For the sake of fairness, we assume that if the platform is
unable to deploy the service it is due to the fact that the data type cannot be used as part of the service
interface, meaning that the server platform is unable to handle the specific data type and cannot bind it
to any XML Schema data type. In the end, we were able to deploy 2489 services in Glassfish, 2248 in
JBoss AS, and 2502 services in IIS.
All services were used first for the Generation Phase inter-operation tests and, as described, the Metro services then also proceeded to the Execution Phase. For each of the Metro services that reached the
Execution Phase (2488 services), we used a set of custom scripts for creating client implementations.
The scripts automatically generate a client (see Section 3.2) for each of the successfully compiled client
artifacts that result from the Generation Phase. Each generated client implementation calls the service
invocation method made available by the corresponding artifacts, using random test values as input.

5.1.2 Results overview


Our tests disclosed a large number of Errors and Warnings in all five steps of the inter-operation process. Table 4 presents a detailed view of the number and types of issues detected in each step for each
client tool against each server platform. Full details can be found in (Elia et al.).
The Errors in Table 4 that correspond to the Service Description Generation step account for the services for which the frameworks were unable to produce a WSDL. Among the remaining services, we
have encountered a total of 86 warnings (see Table 4), corresponding to services that do not pass the
WS-I compliance test (Nezhad, Benatallah, Casati, & Toumani, 2006). From these 86 services, 4 services (.NET based) will conclude the Generation Phase without showing an Error with any of the client-side frameworks, even though they are not WS-I compliant.

11

In the Client Artifact Generation Step, 4763 tests produced at least one Warning and 287 resulted in
Errors (out of the total 152019). The JScript .NET clients are responsible for about 99.8% of all Warnings due to an incompatibility with the Java platforms (Metro and JBossWS) that generates warning
messages at every execution of the tool. The remaining Warnings correspond to other tools trying to
generate artifacts from the WSDL documents that failed the WS-I check. About 97% of the Errors in
this step are produced when using WSDL documents that failed the WS-I check. The remaining 3% are
actually the result of processing WS-I compliant services.
During the Client Artifact Compilation Step the tools produced 14478 Warnings. All of these are due
to compiling the artifacts produced by Axis1 and Axis2 (see also Table 4) and are non-disruptive warnings, which refer to the use of unchecked or unsafe operations. Regarding the Errors in this step, we
detected 1301 tests resulting in compilation Errors that were, in most cases, due to generation of code
carrying naming collisions or missing parameters and even crashes of the compilation tool (the JScript
.NET compilation tool crashed in some tests, with the message 131 INTERNAL COMPILER
CRASH). The compilation of the Axis1 and JScript .NET artifacts account for the vast majority of the
compilation step Errors (approximately 68% and 30%, respectively). The remaining Errors occurred
when compiling the Visual Basic .NET and Axis2 artifacts.
The tests for the Execution Phase identified a quite high number of inter-operation problems. The
Client Communication step failed with an Error in 1303 services (52% of the 2488 Metro services) and
inter-operation was not possible. No Warning was detected in this step. Regarding the Service Processing step, we did not detect any Warning or Error. Thus, once the information arrived at the service
it was able to travel back to the client without any observable issue.

5.1.3 Detailed analysis


In this section we detail the main findings observed during the experiments with the home-developed
services. In the Service Description Generation step we noticed that, under certain conditions, the server platforms actually publish service descriptions, even when the service interfaces do not pass WS-I
compliance. In our view, this should not occur as the servers are publishing a potentially problematic
WSDL. This is confirmed by the fact that about 95.3% of the services that did not pass the WS-I compliance check also did not complete the Generation Phase without showing some kind of Error.
Despite the utility of the WS-I compliance check, it is still not able to detect all types of problematic
WSDL documents. As an example, for two services JBossWS published WSDL documents that passed
the WS-I tests and still were unusable, as they did not describe any operation to be invoked by clients.
A WSDL generation tool should fail, or at least show a warning, when a service description document is
created and misses critical information (such as operations). This is precisely what happened with Metro, which refused to publish the WSDL document for these two problematic services.
TABLE 4
EXPERIMENTAL RESULTS FOR THE HOME DEVELOPED SERVICES
Metro
JBossWS CXF
WCF .NET
1482 out of 3971 services
1723 out of 3971 services
11580 out of 14082 services
2 out of 2489 services
4 out of 2248 services
80 out of 2502 services
Artifact Genera- Artifact Compila- Artifact Genera- Artifact Compila- Artifact Genera- Artifact CompilaClient-side FW Warn-tionErrors Warn-tionErrors Warn-tionErrors Warn-tionErrors Warn-tionErrors Warn-tionErrors
a)
e)
c),d)
f) g)
ings
ings
ings
ings
ings
ings
Metro
0
1
0
0
1
3
0
0
0
77 3
0
0
a)
d)
3g)
Apache Axis1
0
1
2489
477
0
1
2248
412
0
3
2502
0
a)
c)
h) g)
Apache Axis2
0
1
2489
1
0
2
2248
1
0
0
2502
1 2
a)
d)
f) g)
Apache-CXF
0
1
0
0
0
1
0
0
0
77 3
0
0
a)
d)
f) g)
JBoss
0
1
0
0
0
1
0
0
0
77 ) 3
0
0
a),b)
c),d),e)
f)
3g)
0
.NET C#:
0
2
0
0
0
4
0
0
1
0
0
a),b)
c),d),e)
f)
.NET VB
0
2
0
1
0
4
0
1
1
0
0
4
a),b)
c),d),e)
f)
.NET JS
2489
2
0
50
2248
4
0
50
1
0
0
301
3g)f)
b)
gSOAP
0
1
0
0
2c
0
0
0
0
13
0
0
Zend Frame0
0
2c
0
0
0
0
0
a)
d)
work
Suds
Python
0
1
2c
1
0
0
1
0
a) Service based on javax.xml.ws.wsaddressing.W3CEndpointReference fails WS-I; b) Service based on java.text.SimpleDateFormat
fails WS-I; c) Services based on java.util.concurrent.Future/javax.xml.ws.Response are WS-I compliant but have no callable methods;
d) Service based on javax.xml.ws.wsaddressing.W3CEndpointReference fails the WS-I check; e) Service based on java.text.SimpleDateFormat fails the WS-I check; f) 77.NET services fail the WS-I check; g) WS-I compliant services based on System.Data.DataTable and System.Data.DataTableCollection; h) WS-I compliant services based on System.Net.Sockets.SocketError.
Errors
Warnings

12

The Client Artifact Generation Step is not very prone to Errors, as visible in Table 4. As referred, the
vast majority of the Errors were caused by the unusable WSDL documents mentioned in the previous
paragraph. This is probably due to the fact that, in most cases, the generation of the clients is a very automated task, where each statement is translated to a specific piece of code, and this will obviously fail
when something unexpected is being processed (such as a WSDL without operations). Despite this, Axis1, Apache CXF and JBossWS did not signal any problem when using the WSDL documents without
operations, which is obviously not the right behavior for the users of the tools (since the tools are silently failing to produce usable code), and also shows the silent propagation of a severe issue to the client
side.
Client Artifact Compilation appears to be a more error prone step, where we found the majority of the
errors (refer to Table 4). On the other hand, if we exclude the warnings in the Axis platforms (which are
all of the same type and refer to the use of unchecked operations), the total number of Warnings found
in this step drops to zero. This shows that either the generation process goes perfectly (and also without
warnings), or it fails in result of serious problems in the artifacts code. Most of the compilation failures
detected are due to wrong variables names, naming collisions or duplicate variables in within specific
groups of services (detailed results are available at (Elia et al.)).
We would also like to emphasize the huge behavior differences shown by the different tools. The client artifact generation tools provided with the tested versions of Metro, JBossWS, Apache CXF,
gSOAP, and C# .NET appear to be quite mature as they fail almost only in presence of non WS-I compliant WSDL documents. Moreover, Errors are always in the Client Artifact Generation step and these
tools never produced code that later results in compilation errors or warnings. In addition, the two client
artifact generation tools for PHP and Python show a similar performance, although they lack the compilation step.
Results for the Execution Phase have uncovered a total of 1303 Errors (52% of the 2488 Metro services). All Errors were detected in the Client Communication Step and are essentially of two types: a)
the client-side framework fails to translate the input object into a SOAP message, which results in an
exception on the client application; b) the server-side framework fails to translate the incoming SOAP
message into an application-level object causing the server to respond with 500 Internal Server Error. The former type of Error accounts for 34% (446 Errors) of the total number of Errors, the majority of which related with clients trying to manipulate abstract data types. The latter type of Error accounts for the remaining 66% (857 Errors) and is more common in clients managing data types that
have no placeholders for content (e.g., empty classes).
The high number of issues detected in the Client Communication step is quite surprising, considering
that we are using the same framework on each of the sides and once the framework has published ta
given WSDL, it should offer no later inter-operation issue.
Regarding the Service Processing step, we observed no issues. In fact, once the message arrives at
the service without any kind of issue, it is somehow expectable that it can perform the reverse path. In
future work, we plan to use more combinations of frameworks to understand if this scenario still applies.
As a final note, we have observed of the 1185 cases in which it was possible to complete the interoperation cycle (48% of the 2488 Metro services), 767 cases where client and server are exchanging an
empty message. This can be due to many factors, however what is important is that the developer
should at least be informed with some warning, which was never the case.

5.2 Public Services

For the second set of services we have chosen to analyze a set of 1024 publicly available services. The
idea is to use more complex services that are currently being used in a real environment so to identify
different patterns of behavior with respect to the home-developed services analyzed before.

5.2.1 Collection of the public web services


We searched the web for publicly available web services (i.e., WSDL documents) using Googles search
engine and searching for web service related keywords (e.g., WSDL, web service, SOAP). This resulted
in a list of 3800 URLs pointing to WSDL files, over which we performed a preliminary validation procedure. In particular, we verified that the collected URLs pointed to the expected WSDL document and
we checked the validity of the WSDL document against its XML Schema (WSDL XML Schema, ),
which essentially guarantees that the document is syntactically correct, according to the specification.
This validation procedure resulted in a total of 1024 valid services. Obviously, as our results show (see
next section), even with syntactically correct documents, interoperability problems can arise.

13

5.2.2 Results Overview

"

No"WS1I"Warnings"
57"
19" WS1I" 38"
WS1I"Warnings"
Warnings"
Errors"(549)"

Regarding the tests executed over the


WS1I"
WS1I"Warnings"
BP"
1024 public services, we were able to disclose Errors in about 7% of the executed
Client"
tests, from which about 83% (1258 errors)
No"Errors"(475)"
57"
Tools"
were disclosed during the Client Artifact
Generation step, and about 17% (260 erCompila8on""
Genera8on""
0"
1024"
Errors"(243)"
Errors"(363)""
Services"
rors) were uncovered in the Client Artifact
"
Compilation step. Despite being a small
percentage, these errors involve more than Fig. 4. General view of the results for the public services
half (53.61%) of the services. Due to
space constraints we present only the main findings here (detailed results are available at (Elia et al.)).
Fig. 4 presents a summary of the results obtained and including the three key-steps studied: Service
Description Generation (WS-I Basic Profile assessment (K. Ballinger et al., 2006), (Basic Profile Version 1.2 (Final),), (Basic Profile - Version 2.0 (Final),)); Client Artifact Generation; and Client
Artifact Compilation. As we can see in the top part of the figure, only a very small number of services
(57) are WS-I compliant (considering the union of the results produced by the different versions of the
WS-I tests 1.0/1.2/2.0). In fact, the WS-I Basic Profile tests produce a very high number of warnings
(on a total of 967 services), suggesting that services are being created and deployed without taking into
account the WS-I recommendations, which might potentially result in future interoperability issues.
More than half (549) of the 1024 services tested showed at least one Error (no possible interoperation) with at least one of the client artifact generation/compilation tools either during the Client
Artifact Generation step (363) or during the Artifact Compilation (243) step (57 services show some
problem in both steps). All Artifact Generation errors were observed in non WS-I compliant services
and, of the 243 Artifact Compilation errors, 92% were also observed in non WS-I compliant services.
The remaining 8% refer to errors found with 19 WS-I compliant services, which actually account for
about one third of the total 57 WS-I compliant services.
Regarding the 475 services that reach the end of the tests without the occurrence of any Error (at the
bottom-right in Fig. 4), only 8% of these (38 services) were marked as WS-I compliant, indicating the
strictness of the WS-I tests. Note that these 38 services represent only 3.7% of all tested services. Thus,
the vast majority of the tested services, which are meant to be interoperable, are being deployed in the
field with either severe issues (described in this section) or holding potential for interoperability problems (marked by the WS-I tests or by client-side framework warnings). In the following paragraphs we
describe the main interoperability issues found, in each of the three key-steps of the interoperability
tests.

5.2.3 Detailed Analysis


For the public web services the Service Description Generation step had been already carried out (as
the service interface was already published), which in practice means that we are not able to mark any
Error resulting from the execution of the step. Table 5 presents the results of assessing the Service Description Generation step with the WS-I Basic Profile tools (Web Services Interoperabily Organization
(WS-I), 2014), including a detailed view of the results obtained with each individual test assertion that
composes the WS-I tests. We also provide the union of the results obtained with the two BP versions (in
column Union in Table 5), pessimistically considering that a warning indicated by just one version is
potential ground for an interoperability problem. Finally, Table 5 also presents the number of non WS-I
compliant services for which actual interoperability errors have been found in later phases (i.e., during
artifact generation or compilation). We include not only the partial results for the Client Artifact Generation and Client Artifact Compilation steps, but also the total value under column Errors. Note that this
total value is not equal to the sum of the values obtained in the two steps (generation and compilation),
as some services display errors in both steps. From the analysis of the global results in Table 5 (see the
Total row) we can highlight the following key results:
The WS-I compliance tests are quite strict: 967 (over 94%) of the services where considered
non-compliant and thus prone to interoperability issues;
Of the 967 non-compliant services, 55% (530/967) are involved in a later fatal interoperability
error. In 363 services the errors are detected during the Client Artifact Generation step, and in
224 services errors are detected in the Client Artifact Compilation step;
Out of the WS-I Basic Profile compliant services (57), one third (19 services) are involved result
in errors in a later phase of the inter-operation tests (100% of these 19 services result in compilation errors, 0% result in generation errors.

14

TABLE 5

For these reasons, considering only the compliant or non-compliant nature of a given service proTest
Generation Compilation
vides very little information to early identify inErrors
Assertion
Errors
Errors
V.1.1 V1.2/2 Union
teroperability issues. Thus, we have analyzed the
Total
951
959
967 363/967(37.5%) 224/967 (23.2%) 530/967(54.8%)
result of the (failed) individual test assertions that
BP2402
697
706
713 123/713 (17.3%) 183/713 (25.7%) 281 / 713 (39.4%)
compose the WS-I Basic Profile and tried to unBP2114
706
0
706 122/706 (17.3%) 181/706 (25.6%) 279 / 706 (39.5%)
BP2122
333
0
333 120/333 (36.0%) 48/333 (14.4%) 137 / 333(41.1 %)
derstand how each individual warning relates with
BP2406
236
236
242 242/242 (100%) 34/242 (14.1%)
242/242 (100%)
the interoperability issues found in our tests in subBP2017
1
236
236 236/236 (100%) 29/236 (12.3%)
236/236 (100%)
BP2102
218
0
218
7/218 (3.2%)
17/218 (7.8%)
23/218 (10.6%)
sequent steps.
BP2201
40
0
40
37/40 (92.5%)
19/40 (47.5%)
38 / 40 (95%)
We have observed that some specific test asserBP2108
35
30
37
36/37 (97.3%)
28/37 (75.7%)
37/37 (100 %)
BP2202
0
20
20
8/20 (40%)
15/20 (75%)
17/20 (85%)
tions are good indicators of specific types of genBP2106
0
20
20
8/20 (40%)
15/20 (75%)
17/20 (85%)
eration or compilation errors. In fact, some test asBP2115
14
4
14
4/14 (28.6%)
8/14 (57.1%)
11/14 (78.6 %)
BP2019
14
14
14
3/14 (21.4%)
7/14 (50%)
9/14 (64.3%)
sertions (e.g., BP2402, BP2114, BP2122) are very
BP2703
8
0
8
8/8 (100%)
4/8 (50%)
8/8 (100%)
loosely associated with Errors (at least in the CliBP2417
3
8
8
8/8 (100%)
2/8 (25%)
8/8 (100%)
BP2101
3
7
7
7/7(100%)
5/7(71.4%)
7/7(100%)
ent Artifact Generation and Compilation steps)
BP2113
4
0
4
4/4 (100%)
1/4 (75%)
4/4 (100%)
while other assertions are strongly correlated with
BP2701
3
0
3
3/3 (100%)
1/3 (33.3%)
3/3 (100%)
BP2010
3
0
3
3/3 (100%)
1/3 (33.3%)
3/3 (100%)
generation Errors (e.g., BP2406, BP2017,
BP2208
1
1
1
1/1 (100%)
0/1 (0%)
1/1 (100%)
BP2703) or compilation Errors (e.g. BP2202,
BP2118
1
0
1
1/1 (100%)
1/1 (100%)
1/1 (100%)
BP2106, BP2101). An interesting example is the
case of test assertion BP2406 and its prerequisite BP2017. 100% of the services where these test assertions were signaled resulted in an Error in a later step (with at least one framework). In most of the cases (around 99%) the artifact generation error is caused by the fact that the WSDL requires the use of
RPC/Encoded SOAP messages, which are not supported by some of the mainstream frameworks (e.g.,
Metro, Apache CXF and JBoss) and forbidden by WS-I Basic Profile (BP2406) but appear still quite
common in web services found in production scenarios.
Finally, another aspect is that we could observe is that although the newer versions (BP 1.2/2.0) of
the WS-I tests should be retro-compatible (Basic Profile - Version 1.2 (Final),),(Basic Profile - Version 2.0 (Final),) with the older version (BP1.1), we have verified that this is not the case in many situations. As an example, the Basic Profile Test Assertion BP2122 (which has the same goal in both Basic
Profile versions) fails in 333 services for BP version 1.1 but does not fail in any service in BP 1.2/2.0
(please refer to Table 5 for the details on the remaining test assertions).
In Table 6 we present the detailed results for each of the client-side tools, including Warnings and
Errors. As we can see in Table 6, the most of the errors were disclosed when executing the Client Artifact Generation. We can notice that Metro, Apache CXF and JBoss (widely used and quite popular
frameworks) are responsible for the majority of the generation Errors observed, with around 340 generation errors each.
Most of the detected generation errors (in about 238 services) are related with the WSDL requiring
the use of RPC/encoded messages, which is not supported by these platforms. A large part of the remaining problems found (in 79 services), refer to Undefined Element Declaration errors, which relate to
the use of some XML tags (s:schema, s:lang, any) that the client frameworks are unable to handle. Note
that this can be solved by manually customizing the
data type bindings (Vivek), however the client develTABLE 6
oper has to know precisely which binding to define,
EXPERIMENTAL RESULTS FOR THE PUBLIC SERVICES
which may again result in an interoperability issue.
The remaining detected generation Errors (about
967 out of 1024 services
WS-I Warnings
25) when using Metro, JBoss, and Apache CXF are
Artifact Generation
Artifact Compilation
Client Tools
mostly due to the use of unknown elements in the
Warnings
Errors
Warnings
Errors
XML Schema; missing or duplicate (replicated) defiMetro
717
344
0
0
nitions of types in the WSDL; and collision of attribApache-CXF
1
343
0
1
utes due to problems of case insensitiveness of the
JBoss
0
341
0
1
target platform. This latter is a very interesting case of
Apache Axis1
0
15
1010
17
artifact generation error: it is due to the use of two
Apache Axis2
0
38
1010
9
elements in the WSDL that have the same name, but
gSOAP
71
12
6
6
with different cases (e.g., IP2PROXY and IP2Proxy).
.NET C#:
259
20
0
3
When a client generation tool tries to generate the cli.NET Visual
259
20
1
28
Basic
ent artifacts it needs to create two Java classes with
.NET
JScript
518
20
1
195
the same name, but different case, which results in an
Zend PHP
12
12
error under Windows (which does not support caseSuds Python
0
93
sensitive filenames), but not under Linux. The probWS-I BASIC PROFILE RESULTS FOR THE PUBLIC SERVICES
WS-I BP Warnings

15

lem can be solved by some manual configuration, but still it is a very interesting case where the interoperability issue actually extends from the framework to the Operating System. Obviously, this signals that there is a huge space for improvement. On one hand, when generating, defining, or deploying
the WSDL a developer should be warned that his WSDL file carries potential problems. On the other
hand, some of the client-side tools (in this case, the frameworks showing this problem are mainstream)
could also be improved and be prepared to automatically handle these problematic cases.
Metro is also responsible for a huge number (717) of generation warnings. These occur mostly in
services (703) that offer SOAP 1.1 and 1.2 bindings, where the wsimport tool selects version 1.1 over
version 1.2. The remaining 14 wsimport warnings are related to a basic WS-I check performed by the
tool itself (the .NET framework also performs this type of tests) which reports that the service is not
WS-I Basic Profile 1.0 compliant. Axis1 and Axis2 account respectively for 15 and 38 errors in this
step. The errors are due to missing elements or type definitions (i.e., due to the use of unsupported or
missing schemas), duplicate or missing operations (operations not defined or with the same name).
gSOAP uses two tools to generate artifacts. The first one (wsdl2h) generates a C++ header file from
the service description document and produced 5 generation errors while parsing the WSDL file. The
second one (soapcpp2), which creates the cpp files, accounts for the remaining 7 generation errors (due
to incorrect syntax in the previously generated header file). This framework is also involved in several
generation warnings: non compliance with WS-I Basic Profile version 1.0 (31 services), use of reserved
character sequences in the generated code (27 services), missing bindings or messages (10 services),
duplicate schemas (4 services), unexpected elements (2 services), and undefined QName (3 services).
The three .NET tools fail to generate clients for the same 20 services. Out of these, 15 are due to
binding problems related to missing or not supported element definitions, and 5 represent failures when
processing the WSDL file. In addition, the .NET framework produced a high number of warnings. Most
of these (in 257 services) are due to a preliminary failed check of compliance to WS-I Basic Profile version 1.1 (this check is executed by the client-side framework itself). The other warnings include failed
schema validation (4 services) and optional WSDL extension ignored (2 services). The client generation
tool for JScript.Net produced an additional set of 501 warnings for skipped operations that were not
used in the code generation. In 65% of these 501 cases, the generated code did not have methods that
could be used to invoke the service, which obviously renders the code useless.
The Zend PHP framework produced 12 Generation errors, with 5 occurring when parsing the WSDL
document and 7 when parsing the XML Schema. The tool also produces an internal warning related to
the improper use of system's timezone settings (for the same 12 services).
Finally, the Suds Python platform accounts for 93 generation errors, of which 96% are type not found
exceptions, caused by the inability to process some XML types (mostly s:schema and soapenc:Array).
The remaining errors include internal error messages, which require further analysis by the service or
framework developers. All individual detailed results are available at (Elia et al.).
The Client Artifact Compilation step accounts for a smaller number of Errors when compared to the
previous step (260 compilation Errors against 1256 generation Errors). Metro is the only platform with
no compilation Errors, but also JBoss and CXF performed well, as the only Error reported is related to
the use of an uncommon encoding in one WSDL file (which can be solved, provided that the right encoding is specified when creating the WSDL file or when generating and compiling artifacts).
Problems with unmappable characters cause 3 compilation errors in Axis1 and Axis2. Other compilation problems for these two frameworks include severe issues, such as: package names generated with
invalid characters; variables used without previous instantiation (due to wrong naming); and duplicate
variable names. The Axis frameworks also produce a very high number of compilation warnings (in all
services for which the compilation is possible); however, these are all non-disruptive warnings, which
refer to the use of unchecked or unsafe operations.
The gSoap platform reports a compilation Error related to duplicated namespace definitions for 4
services, while for other 2 it fails to generate the required source files.
Finally, the .NET clients are quite prone to compilation Errors, especially Visual Basic and JScript
clients. The C# client code results in 3 compilation Errors caused by duplicate variables (more than one
variable defined with the same name) and missing type of a variable (which is generated as a generic
Object). The compilation of the VB code results in 25 Errors referring to the presence of duplicate variables. The compilation of the JS .NET client code is responsible for almost 3/4 of the total compilation
errors (190/260). These include 89 errors due to wrong code (e.g., duplicate variables, missing types,
types mismatch, more the one constructor with the same parameters, and code trying to extend nonextendable classes). Moreover, 105 errors are compiler crashes reporting the message 131 INTERNAL
COMPILER CRASH, emphasizing the need for interoperability assessment and improvement.

16

6 KEY FINDINGS
In this section we summarize the main findings and trends obtained from our huge test campaign.
Code generation tools show quite disparate performance. Some appear to be immature (although
they are being used in production environments) and crash during the generation process, others
provide detailed information when in presence of problems, while others provide insufficient information.
Mature generation tools (e.g., Metro, CXF, JBoss, .NET C#) tend to identify issues when generating artifacts. Once the code is produced the likelihood of containing errors is small. Less mature tools (e.g., .NET Jscript and Axis 1) often produce problems that are identified only at compilation time. Most errors identified during compilation are due to wrong naming of namespaces, packages and variables.
Over half of the 1024 public services were involved in some fatal issue and 23% still require
RPC messages, which is prohibited by WS-I Basic Profile and no longer supported by some
mainstream tools. Despite this, each of the 1024 services was able to inter-operate with at least 4
of the client-side frameworks, indicating that there is space for improvements.
Fatal inter-operation errors occur even when the same framework supports client and server.
Interoperability issues are not limited to the framework and programming language domain and
can extend to the operating system. In some cases, a framework was able to generate code in
Linux, but not in Windows due to the case-insensitiveness nature of this operating system.
The WS-I Basic Profile tests do not help much to understand how interoperable a service is.
Their capability of detecting services that in alter steps are involved in inter-operation issues
was very high for our home developed services (where the WSDL documents are quite simple).
On the other hand, this analysis was almost useless for the public services (that can have more
complex WSDL documents), as almost all the services were considered non-compliant (including services that actually created an inter-operation problem and services that did not).
First results from the testing Execution Phase suggest that problematic services might reach the
this phase without showing a single warning, and that in some cases, the server platforms silently publish services, even when the usability of such services is limited.

7 CONCLUSION
Inter-operation of different web services providers and consumers revealed to be a difficult task, resulting in efforts to promote interoperability, such as the ones carried out by the WS-I organization. Despite
this, still developers have limited tools and resources to understand how interoperable their services are.
For this reason we have proposed a well-defined approach for interoperability testing from a practical
perspective, and a tool, named ITWS, for the automatic implementation of such testing. We have also
performed a massive testing campaign providing insights about the interoperability of more than 8
thousands services by detecting a huge number of interoperability issues and being able to disclose
problems even in WS-I compliant services, thus outperforming competitor tools in interoperability assessment.
ACKNOWLEDGMENT

This work has been partially supported by the Project ICIS - Intelligent Computing in the Internet of
Services, number 4774, of the program CENTRO-SCT-2011-02, and by the projects CErtification of
CRItical Systems (CECRIS), Marie Curie Industry-Academia Partnerships and Pathways (IAPP) number 324334; and DEsign, Verification and VAlidation of large-scale, dynamic Service SystEmS (DEVASSES), Marie Curie International Research Staff Exchange Scheme (IRSES) number 612569, both
within the context of the EU Seventh Framework Programme (FP7).
References

Apache Axis2 Web Service Framework. (n.d.). Retrieved from http://axis.apache.org/axis2/java/core/


Apache Software Foundation. (2008). Apache Axis. Retrieved September 4, 2008, from http://ws.apache.org/axis/
Apache Software Foundation. (2013). Apache CXF. Retrieved December 7, 2013, from http://cxf.apache.org/
Ballinger, K., Ehnebuske, D., Ferris, C., Gudgin, M., Liu, C. K., Nottingham, M., & Yendluri, P. (2006). WS-I Basic Profile - Version 1.1. Profiles/BasicProfile-1.1. Html, April.
Ballinger, K., Ehnebuske, D., Gudgin, M., Nottingham, M., & Yendluri, P. (2004). WS-I Basic Profile - Version 1.0. Retrieved December 6, 2013,
from http://www.ws-i.org/Profiles/BasicProfile-1.0.html
Basic Profile - Version 1.2 (Final). (n.d.). Retrieved February 13, 2014, from http://ws-i.org/Profiles/BasicProfile-1.2-2010-11-09.html
Basic Profile - Version 2.0 (Final). (n.d.). Retrieved February 13, 2014, from http://ws-i.org/Profiles/BasicProfile-2.0-2010-11-09.html
Chappell, D. (2009). Introducing Windows Communication Foundation in .NET Framework 4. Retrieved December 7, 2013, from
http://msdn.microsoft.com/en-us/library/ee958158.aspx

17

CrossCheck Networks. (n.d.). SOAPSonar - Advanced Client Simulation for Service Testing. Retrieved January 13, 2014, from
http://www.crosschecknet.com/products/soapsonar.php
Curbera, F., Duftler, M., Khalaf, R., Nagy, W., Mukhi, N., & Weerawarana, S. (2002). Unraveling the Web Services Web: An Introduction to
SOAP, WSDL, and UDDI. IEEE Internet Computing, 6(2), 8693.
Elia, I. A., Laranjeiro, N., & Vieira, M. (2014a). A Field Perspective on the Interoperability of Web Services. In 2014 IEEE International Conference on Services Computing (SCC) (pp. 7582). doi:10.1109/SCC.2014.19
Elia, I. A., Laranjeiro, N., & Vieira, M. (2014b). ITWS: An Extensible Tool for Interoperability Testing of Web Services. In 2014 IEEE International Conference on Web Services (ICWS).
Elia, I. A., Laranjeiro, N., & Vieira, M. (2014c). Understanding Interoperability Issues of Web Service Frameworks. In Dependable Systems and
Networks, 2014. DSN 2014. IEEE International Conference on. Atlanta, Georgia USA.
Elia, I. A., Laranjeiro, N., & Vieira, M. (n.d.). ITWS (Interoperability Testing for Web Services) tool and Experimental Results. Retrieved from
http://eden.dei.uc.pt/~ivanoe/TSC/TSC.zip
Engelen, R. van. (2013). gSOAP: SOAP C++ Web Services. Retrieved December 7, 2013, from http://www.cs.fsu.edu/~engelen/soap.html
Erl, T. (2005). Service-Oriented Architecture: Concepts, Technology, and Design. Upper Saddle River, NJ, USA: Prentice Hall PTR.
Eviware. (2011). soapUI. Retrieved July 4, 2011, from http://www.soapui.org/
GlassFish open-source application server for Java EE. (n.d.). Retrieved from https://glassfish.java.net
Hewitt, E. (2009). Java SOA Cookbook. OReilly Media, Inc.
jBoss: wsconsume. (n.d.). Retrieved from https://community.jboss.org/wiki/JBossWS-wsconsume
Microsoft Corporation. (2013). IIS 8.0 Express. Retrieved December 9, 2013, from http://www.microsoft.com/enus/download/details.aspx?id=34679
.NET Framework Class Library (C#). (n.d.). Retrieved from http://msdn.microsoft.com/en-us/library/gg145045(v=vs.100).aspx
Nezhad, H. R. M., Benatallah, B., Casati, F., & Toumani, F. (2006). Web services interoperability specifications. Computer, 39(5), 2432.
doi:10.1109/MC.2006.181
Oracle. (2013a). Java SE 7 API Specification. Retrieved December 7, 2013, from http://docs.oracle.com/javase/7/docs/api/
Oracle. (2013b). Metro. Retrieved December 7, 2013, from https://metro.java.net/
Ortel, J. (n.d.). Suds - Lightweight SOAP client. Retrieved December 7, 2013, from https://fedorahosted.org/suds
Pautasso, C., Zimmermann, O., & Leymann, F. (2008). Restful Web Services vs. Big Web Services: Making the Right Architectural Decision.
In Proceedings of the 17th International Conference on World Wide Web (pp. 805814). New York, NY, USA: ACM.
doi:10.1145/1367497.1367606
Ramsokul, P., & Sowmya, A. (2006). A Sniffer Based Approach to WS Protocols Conformance Checking. In The Fifth International Symposium
on Parallel and Distributed Computing, 2006. ISPDC 06 (pp. 5865). doi:10.1109/ISPDC.2006.5
Red Hat Middleware. (2013). JBoss Application Server. Retrieved December 8, 2013, from http://www.jboss.org/jbossas/
Red Hat Middleware. (n.d.). JBoss Web Services (JBossWS). Retrieved December 7, 2013, from http://www.jboss.org/jbossws
Senthil Kumar, K. M., Das, A. S., & Padmanabhuni, S. (2004). WS-I Basic Profile: a practitioners view. In IEEE International Conference on Web
Services, 2004. Proceedings (pp. 1724). doi:10.1109/ICWS.2004.1314719
Vivek, P. (n.d.). How to deal with unresolved xs:schema references in WSDL. Retrieved from
https://weblogs.java.net/blog/vivekp/archive/2007/05/how_to_deal_wit.html
Web Services Interoperabily Organization (WS-I). (2002). Web Services Interoperability Organization (WS-I). Retrieved September 16, 2008,
from http://www.ws-i.org/
Web Services Interoperabily Organization (WS-I). (2014). Deliverables - Basic Profile Working Group. Retrieved January 13, 2014, from
http://www.ws-i.org/deliverables/workinggroup.aspx?wg=basicprofile
WSDL XML Schema. (n.d.). Retrieved from http://schemas.xmlsoap.org/wsdl/
WS-I. (n.d.). Retrieved December 5, 2013, from http://www.ws-i.org/deliverables/workinggroup.aspx?wg=testingtools
wsimport. (n.d.). Retrieved from http://docs.oracle.com/javase/6/docs/technotes/tools/share/wsimport.html
yaronn. (n.d.). WsiHero. Retrieved February 13, 2014, from https://github.com/yaronn/WsiHero
Zend Technologies. (2013). Zend Framework. Retrieved December 7, 2013, from http://framework.zend.com/
Ivano Alessandro Elia received the PhD degree from the
University of Naples Parthenope. Currently he is a Postdoc research fellow at the University of Coimbra, Portugal.
His research focuses on web application security as well as
experimental dependability evaluation.
Nuno Laranjeiro is an assistant professor at the University of
Coimbra, Portugal. His research focuses on robust software
services as well as experimental dependability evaluation,
web services interoperability, services security, and enterprise application integration. He is a member of the IEEE
Computer Society.
Marco Vieira is an assistant professor at the University of
Coimbra, Portugal. His interests include dependability and
security benchmarking, experimental dependability evaluation, fault injection, software development processes, and
software quality assurance. Vieira received the PhD degree
from the University of Coimbra. He is a member of the IEEE
Computer Society.

You might also like