Generic API Test tool

By
Moshe Sapir
Almog Masika

Agenda
Testing

practice in the industry

The Automatic API test tool concept
Attributes supplied to the developer
Test coverage
Design and implementation
Tool demonstration

Testing practice in the industry

Testing software module requires deep

understanding of the modules inner logic and its
interfaces semantics
Currently, dedicated tests are written to each
software component
Usually the tests do not try to retrieve functional
information from the modules code
Changes in the software logic are resulted in
changes in the test automation logic

Automatic API Test Tool Concept

Enable the developer to add

metadata to the modules code
Use .NET attributes to represent
the metadata
The metadata should contain
information which will enable
automatic testing of the module
Create automatic test tool that
given the module and attributes
can generate tests

Tested
Module

Tested
Code

Generic API Test Tool

Test Results

Attributes supplied to the developer

Function level attributes

Input-Output Assertion:

Assertion is defined by three expressions

<P, Q, L>

If P is true before the function is run, then Q is true after the

function is run, or one of the exceptions in L is thrown
during execution
For double Sqrt(double x) the developer will define:
P: x > 0
Q: ret ^ 2 = x
L = {DivisionByZero }

Attributes supplied to the developer

Function level attributes (cont)
Input

Parameters attribute

Provide set of interesting assignments to the

methods parameters
For double tan(double x) the developer will
define the following assignments:
X

= pi
X = 0
X = pi/2
X = -1234.45*pi

Attributes supplied to the developer

Class level attributes

Class Instances

Provide set of interesting instances of the class

For class Int32 the developer will define the following
instances:
0
1
-1
2^31
(-2^31) + 1
12345
-780

Attributes supplied to the developer

Class level attributes (cont)

Class Invariants

Define set of Boolean expressions that should be true after and

before executing any member function on an instance of the class
For Account class:
class Account {
string Owner;
int Balance;
}
The developer will define:

Owner != string.Empty
Balance >= 0

Test Coverage
Method level coverage

General test

Run the method as many times as possible, using

parameters supplied by the developer. If no parameters
were supplied, try to generate the them

Input Output Assertion test

For each <P, Q, L> defined on the method, try to find an

assignment that satisfies P, run the method and validate Q
and L.

Test Coverage
Assembly level coverage

Function call composition

Try to build a call graph of the functions in

the assembly and generate the call paths.
Validate input-output assertions and class
invariants during the calls:
For the classes:
A, B, C, D
And the functions:
M1: A x B -> A
M2: B x C -> D
M3: A x D -> A
Build the following graph:

M1

M2

M3

Design and implementation

Functional decomposition

Test Attributes- Attributes that are available to the assembly

developer.
User Interface- Retrieve from the user the input required to run
the test and to present him the test results.
Test Engine- Responsible to run the tests and produce the test
results.
Results Reporting: Responsible to save the results in
persistent file and redirect them to the graphic user interface.

Design and implementation

Functional decomposition
Data

flow between the different components:

Test Engine Design: The Algorithm design

pattern

Test Attributes Hierarchy

Design and implementation

User Interface Structure

Test execution sequence diagram

Demonstration
. . .

Conclusions

.NET attributes can be used to keep information

that will enable automatic testing of software
The technique we have shown is limited to
platforms that allow introduction of Meta data in the
binary code (.NET, Java)
The Syntax of defining expressive test attributes
can be error prone (mainly due to the use of strings
as function identifiers)
The ability of such tools to verify large scale
software is still to be investigated. Currently is
seems can be used as first tire testing (BVT).