Software Testing & Test-Driven Development

JAMS Workshop Makerere University September 2010

Agenda
Intro to Software Testing Test-Driven Development Writing a Test Plan Test Frameworks
JUnit Visual Studio

Software Testing
Software testing is the process of verifying that a software program works as expected.

Without testing, there s no proof that the software does what it is intended to do
Testing == Quality

Testing should be incorporated into the development process from the beginning
The need to test your software will impact how you develop it The earlier you find a problem, the easier it is to fix
Cost of fixing a defect depending on the stage it was found
Time Introduced Requirements Design Implementation Time Detected Requirements Design Implementation QA Post-Release

1x -

3x 1x -

5-10x 10x 1x

10x 15x 10x

10-100x 25-100x 10-25x

Who is Involved with Testing?

Developer
Writes implementation spec that enables tester to complete a test plan Reviews the test plan Implements unit tests

Program Manager
Writes functional spec that enables tester to write a test plan Reviews the test plan

Test Engineer
Writes test plan Writes and executes test cases, files bugs Signs off on final product

Business Stakeholder
Participates in internal alpha testing before the software is released

Customer
The person who is ultimately affected if the code was not well-tested May participate in beta testing or perform useracceptance testing

Testing Methods
The box analogy describes the point of view the engineer takes when testing
Tester has access to internal data structures and algorithms
White Box
API testing Code coverage Fault injection

Black Box

Tester treats the code as a black box implementation

Boundary-value analysis Alpha/beta testing, dogfooding

no knowledge of

Grey Box

Tester has knowledge of internal data structures and algorithms, but tests at the user/black-box level
Integration testing (between two modules of code at the interface level) Reverse engineering

Many Levels of Testing

Unit Testing Tests that verify a specific section of code, e.g. classes or methods Usually white box tests written by developers Integration Testing Verifying interfaces between components Iteratively add elements until the whole system is verified System Testing Testing a completely integrated system to verify that it meets requirements Regression Testing Tests that serve to prevent old bugs from coming back Usually written as part of the bug fix process Acceptance Testing Also known as User Acceptance Testing Testing performed by the customer, in their own lab environment Alpha Testing Testing by bleeding edge internal users or external customers Could also refer to internal dogfood efforts Beta Testing A broader test deployment, usually happens after alpha testing Usually involves an external audience

System Integration Testing Testing that a system is integrated to an external or 3rd party system Each system has already passed system testing

Test-Driven Development
A development technique that relies on the concept of writing the test cases before the product code
Validates that spec and requirements are well-understood Test cases will initially fail Developer writes code to make them pass

The test is the proof that the code works

Developer must clearly understand user requirements in order to write tests There should be no functionality in product code that isn t tested

Encourages simple designs and inspires confidence

Clean code that works

TDD Workflow
1. (Re)write a test Test succeeds

Repeat

2. Run all tests & see if the new one fails 3. Write some code Test(s) fail

Test fails

Clean minimal writing code Write up the code as necessary Always start byproduct a testto If all test fail pass, alland Write some code to make run to achieve production tested make the testbecause quality Test mustcasescompile the the requirements test duplication, test pass that are met Focus on Validates removingharness works feature isn t implemented (if it succeeds,test itself: for the If they fail,the need between May the duplicationsuboptimal; including keep iterating Testsbe inelegant ormake sure it feature improve it later code Assuming the code we willt is obviated) new test & pass without doesn producttests are comprehensive,cases to to To write a test passthe onfully Make the test, for must no the Re-run the fail move expected Test shouldtest you only,ensure final refactoring functionality understand the featureis testing extra ensure that it s that stage reasonuntested isn t breaking specification and requirements any functionality the right thing

4. Run the automated tests & see them pass

All tests succeed

5. Refactor code

Writing a Test Plan

Like a functional specification for your test code Test Plan Template
Test Plan Objectives Scope
Features to be tested Features NOT to be tested

Test Strategy Test Cases Open Issues

Test Plan Objectives & Scope

Similar to functional spec
What are we trying to accomplish with our test cases? What is in and out of scope for testing?
Test Plan Objectives [P1] Test functional correctness of the game s underlying methods using white-box unit testing [P1] Test that only valid moves are allowed [P1] Validate that the game can correctly switch between players and declare the end of the game [P2] Test advanced features, like the statistics table, play timer, and computer player, if implemented [P3] Performance testing Scope Features to be tested Basic game play o The game alternates between players o Players can place moves o Only legal moves are allowed Ending the game o The game can declare a winner o The game can declare a tie Player statistics table (if implemented) Timer (if implemented) Machine player (if implemented) Features NOT to be tested User interface rendering Networked play

Test Strategy
Describe the testing methodology you plan to use
E.g. white box, black box, grey box, or a combination
Test Strategy We plan to employ mostly white-box unit tests in order to test the application, using Visual Studio s built-in unit-testing framework. Each of the following major classes will have at least one unit test: Game GameBoard Player GameStatistics

Explain which testing frameworks you ll use, if any

E.g. JUnit, Visual Studio

Will your tests require any sophisticated infrastructure, setup, or tools?

E.g. mock objects, load simulation, test bridges

Test Case Detail

Feature 2: Validate Basic Moves/Allow Legal Moves Only
Test Case ID 2.1 Test Case Name Test basic moves Steps Expected Result Markers are present at expected positions (assertions pass) 1. Place an X at TopLeft 2. Assert that the board records an X at TopLeft 3. Place an O at Center 4. Assert that the board records an O at Center Disallow player from 1. Place an X at TopLeft placing his marker on a 2. Place an O at TopLeft square occupied by his opponent Disallow player from 1. Place an X at TopRight playing on his own 2. Place an O at Center occupied square 3. Place an X at TopRight

2.2

Move 2 fails

2.3

Move 3 fails

Testing Frameworks
JUnit is a unit-testing framework for Java
Developed by the same people who pioneered TDD Uses source code annotations to decorate special methods to be run by the test harness Integrated with Java IDEs like Eclipse and JCreator

Visual Studio has a test framework for any .NET language

Based on the same ideas as JUnit Supports unit tests, database unit tests, generic tests, manual tests, load tests, web tests

Test Methods
JUnit
Annotate test case methods with @Test Use methods from org.junit.Assert to check your test conditions or fail the test case
import org.junit.Test;

public class AdditionTest { private int x = 1; private int y = 1; @Test public void testAdd() { int z = this.x + this.y; assertEquals(2, z); } }
Imports Microsoft.VisualStudio.TestTools.UnitTesting

Visual Studio
Annotate test case methods with <TestMethod()> Use methods from the Assert class to check your test conditions or fail the test case

<TestClass()> Public Class AdditionTest Private x As Integer = 1 Private y As Integer = 1 <TestMethod()> Public Sub TestAdd () Dim z As Integer = Me.x + Me.y Assert.AreEqual(2, z) End Sub End Class

Initialization & Cleanup Methods

Common code that runs before/after each test case also known as Fixture methods
Add a field for each part of the fixture Annotate a method with Before/TestInitialize and initialize the variables in that method Annotate a method with After/TestCleanup to clean up before the next test
@Before protected void setUp() { this.x = 1; this.y = 1; } @After protected void tearDown() { this.x = 0; this.y = 0; } <TestInitialize()> Public Sub SetUp() Me.x = 1 Me.y = 1 End Sub <TestCleanup()> Public Sub TearDown() Me.x = 0 Me.y = 0 End Sub

Running Tests in the IDE

Visual Studio & Eclipse both enable running tests inside the IDE
Select specific tests to run or run the whole suite IDE reports which cases passed/failed Run selected tests again

Easily switch between test code and product code

Change each as needed and rerun tests Easy to debug a test case, set breakpoints in test or product code

Further Reading
TDD
Test-Driven Development: By Example (Google Books) testdriven.com

JUnit
JUnit Cookbook JUnit Javadoc An early look at JUnit 4

APPENDIX

Suite Initialization Methods

Similar to setUp and tearDown, but they run before & after the entire test suite Useful for expensive config operations that don t need to be run for each unit test
E.g. setting up a DB or network connection, redirecting System.err when testing 3rd-party libraries Be careful that your unit tests don t make changes to static state that will impact other unit tests later in the suite
// This class tests a lot of error conditions, which Xalan annoyingly logs // to System.err. This hides System.err before the run, restores it after. private PrintStream systemErr; @BeforeClass protected void redirectStderr() { systemErr = System.err; // Hold on to the original value System.setErr(new PrintStream(new ByteArrayOutputStream())); } @AfterClass protected void tearDown() { // restore the original value System.setErr(systemErr); }

Testing Exceptions
It is easy to test for expected exceptions
Annotate your test with the expected exception If the exception isn t thrown (or a different one is), the test will fail
@Test(expected=ArithmeticException.class) public void divideByZero() { int n = 2 / 0; }

Limitation: if you need to test the exception s message or other properties, use a different structure:
@Test public void divideByZero() { try { int n = 2 / 0; fail("Divided by zero"); } catch (ArithmeticException(success) { assertNotNull(success.GetMessage()); } }

Timed Tests
Simple performance bench-marking:
@Test(timeout=500) public void retrieveAllElementsInDocument() { doc.query("//*"); }

Network testing:
@Test(timeout=2000) public void remoteBaseRelativeResolutionWithDirectory() throws IOException, ParsingException { builder.build("http://www.ibiblio.org/xml"); }

Creating JUnit Tests with Eclipse

Using JUnit with Eclipse is easy Create new JUnit tests using File -> New -> JUnit
Specify JUnit 4 Select location, class you want to test, method stubs to create Then select which methods you want to test, Eclipse will generate test stubs