SOFTWARE TESTING METHODLOGIES

Chapter - 1 INTRODUCTION

Purpose of Testing

The purpose of Testing is to show that a program has bugs. The purpose of testing is to show that the software works. The purpose of testing is to show that the software doesnt works.

Productivity and Quality in Software

In Production of consumer goods and other products, every manufacturing stage is subjected to quality control and testing from component to final stage. If flaws are discovered at any stage, the product is will either be discarded or cycled back for rework and correction. Productivity is measured by the sum of the costs of the materials, the rework, and the discarded components and the cost of quality-assurance and testing.

Productivity and Quality in Software-continued.,

There is a trade off between quality assurance costs and manufacturing costs.

If insufficient effort is spent in quality assurance, the reject rate will be high and so will the net cost. If inspection is so good that all faults are caught as they occur, inspection costs will dominate, and again net cost will suffer.

Productivity and Quality in Software-continued.,

The biggest part of software cost is the cost of bugs: the cost of detecting them, the cost of correcting them, the cost of designing tests that discover them and the cost of running those tests. For software, quality and productivity are almost indistinguishable because the cost of a software copy is trivial.

Goals for Testing

Primary Goal: Bug Prevention Secondary Goal: Bug Discovery Unfortunately the primary goal we cant achieve because we are human, so it must reach its secondary goal. Bug: A Bug is manifested in deviations from Expected behavior. Test Design: a test design must document expectations, the test procedures in details and the results of the actual test.

Phases in a Testers Mental Life

Theres an attitudinal progression characterized by the following five phases: PHASE 0: There is no difference between testing and debugging, other than in support of debugging, testing has no purpose. PHASE 1:The purpose of testing is to show that the software works. PHASE 2:The purpose of testing is to show that the software doesnt works. PHASE 3: The purpose of testing is not to prove anything but to reduce the risk of not working to an acceptable value. PHASE 4: Testing is not an act, it is a mental discipline that results in low-risk software without much testing effort.

Test Design

Tests themselves are need to be designed as like how the code must be designed and tested. The test designing phase of programming should be explicitly identified. Programming Process should be described as: design->test design->code->test code-> program inspection-> test inspection-> test debugging -> test execution-> program debugging-> testing.

Testing isnt Every thing

Inspection methods Design style Languages Design methodologies and development environment

Inspection Methods

These include walkthroughs, desk checking, formal inspections and code reading. These methods appear to be as effective as testing, but the bugs caught do not completely overlap.

Design style

Design style means the stylistic criteria used by programmers to define what they mean by a good program. Sticking to outmoded style like tight code or optimizing for performance destroys quality. Adopting stylistic objectives such as testability, openness, and clarity can do much to prevent bugs.

Languages

The source language can help reduce certain kinds of bugs. Prevention of Bugs is a driving force in the evolution of new Languages. Curiously, though, programmers find new kinds of bugs in new languages, so the bug rate seems to be independent of the language used.

Design methodologies and Development Environment

The design methodology(the development process used and the environment in which that methodology is embedded) can prevent many kinds of bugs.

The Pesticide Paradox and the Complexity Barrier

First Law: the Pesticide Paradox Every method you use to prevent or find to bugs leave a residue of subtler bugs a g a i n st w h i c h t h o s e m e t h o d s are i n e f f e c t u a l Second Law: the Complexity BarrierSoftware Complexity grows to the limits of our ability to manage that complexity.

Some Dichotomies

Testing Versus Debugging Function Versus Structure The Designer Versus the Tester Modularity Versus Efficiency Small Versus Large The Builder Versus the Buyer

Testing Versus Debugging

Testing and Debugging are often lumped under the same heading, and its no wonder that their roles are often confused; The purpose of testing is to show that a program has bugs. The purpose of debugging is find the error or misconceptions that led to the programs failure and to design and implement the program changes that correct the error. Debugging usually follows testing. But they differ as to goals, methods and psychology.

Testing Versus Debuggingcontinued.,

Testing starts with known conditions and has predictable outcomes testing is a demonstration of error or apparent correctness. Testing proves programmers failure Much of testing can be done without design knowledge Testing can often be done by an outsider. Much of test execution and design can be automated.

Debugging starts from possibly unknown initial conditions. Debugging is a deductive process. Debugging is the programmers vindication Debugging is impossible without detailed design knowledge Debugging must be done by an insider Automated debugging is still a dream.

Function Versus Structure

Tests can be designed from a functional or a structural point of view.

In functional testing the program or system is treated as a black box. It is subjected to inputs and its outputs are verified for conformance to specified behavior. Structural testing does look at the implementation details. Things as programming style, control method, source of language, database design, and coding details dominate structural testing.

The Designer Versus the Tester

If testing were wholly based on functional specifications and independent of implementation details, then the designer and the tester could be completely separated. Conversely, to design a test plan based only on a systems structural details would require the software designers knowledge, and hence only he could design the tests. As one goes from unit testing to unit integration, to component testing and integration, to system testing, and finally to formal system feature testing, it is increasingly more effective if the tester and the programmer are different persons.

Modularity Versus Efficiency

A module is a discrete, well defined, small component of a system. There is a trade off between modularity and efficiency. The overall complexity minimization can be achieved by the balance between internal complexity and interface complexity. As with system design, artistry comes into test design in setting the scope of each test and groups of tests so that test design, test debugging, and test execution labor are minimized without compromising effectiveness.

Small versus Large

Programming in large means constructing programs that consists of many components written by many different persons, Programming in the small is what we do for ourselves in the privacy of our own offices. Qualitative and quantitative changes occur with size and so must testing methods and quality criteria.

The Builder Versus the Buyer

The builder, who designs for and is accountable to The buyer, who pays for the system in the hope of profits from providing services to The user, the ultimate beneficiary or victim of the system. The users interests are guarded by The tester, who is dedicated to the builders destruction and The operator, who has to live with the builders mistakes, the buyers murky specifications, the testers oversights, and the users complaints.

A Model for Testing

Un expected

The environment

Environment
model

The program

Program model

Tests

outcome

Nature& psychology Bug model expected

The Environment

A programs environment includes the hardware and software required to make it run. For example., for online systems the environment may include communication lines, other systems, terminals, and operators. The environment also includes all programs that interact with and used to create the program under test such as operating system, loader, linkage editor, compiler and utility routines. Because the hardware and firmware are stable, it is not smart to blame the environment for bugs.

The program

Most programs are too complicated to understand in detail. The concept of the program is to be simplified in order to test it. If simple model of the program does not explain the unexpected behavior, we may have to modify that model to include more facts and details. And if that fails, we may have to modify the program.

Bugs

Bugs are more insidious than ever we expect them to be. An unexpected test result may lead us to change our notion of what a bug is and our model of bugs. Some optimistic notions that many programmers or testers have about bugs unable to test effectively and unable to justify the dirty tests most programs need.

Optimistic notions about bugs.

Benign Bug Hypothesis Bug Locality Hypothesis Control Bug Dominance Lingua Salator Est Corrections Abide Silver Bullets Angelic Testers

Benign Bug Hypothesis: The belief that bugs are nice, tame and logical. Bug Locality Hypothesis: the belief that a bug discovered with in a component affects only that components behavior. Control Bug Dominance: the belief that errors in the control structures of programs dominate the bugs. Lingua Salator Est: the hopeful belief that language syntax and semantics eliminate most bugs. Corrections Abide: the mistaken belief that a corrected bug remains corrected. Silver Bullets: the mistaken belief that language, design method, representation, environment grants immunity from bugs. Angelic Testers: the belief that testers are better at test design than programmers are at code design.

Optimistic notions about bugscontinued.,

Tests

Tests are formal procedures, Inputs must be prepared, outcomes predicted, tests documented, commands executed, and the results observed; all these steps are subject to error.

Testing and Levels

We do many distinct kinds of testing on a typical software system.

Unit testing Component testing Integration testing System testing

Unit, Unit Testing

Unit : A Unit is the smallest testable piece of software, by which It mean that it can be compiled, assembled, linked,loaded and put under the control of a test harness or driver. A unit is usually the work of one programmer and consists of several hundred or fewer lines of code. Unit Testing: Unit Testing is the testing we do to show that the unit does not satisfy its functional specification or that its implementation structure does not match the intended design structure.

Component, Component Testing

Component: a component is an integrated aggregate of one or more units. Component Testing: Component Testing is the testing we do to show that the component does not satisfy its functional specification or that its implementation structure does not match the intended design structure.

Integration, Integration Testing

Integration: It is a process by which components are aggregated to create larger components. Integration Testing: Integration Testing is testing done to show that even though the components were individually satisfactory, as demonstrated by successful passage of component tests, the combination of components are incorrect or inconsistent.

System, System Testing

System: a system is a big component. System Testing: system testing is aimed at revealing bugs that cannot be attributed to components as such, to the inconsistencies between components, or to the planned interactions of components and other objects. It includes testing for performance, security, accountability, configuration sensitivity, start up and recovery.

The role of Models

The art of testing consists of creating, selecting, exploring, and revising models. Our ability to go through this process depends on the number of different models we have at hand and their ability to express a programs behavior.