Software testing is the process used to assess the quality of computer software.

This
includes, but is not limited to, the process of executing a program or application with the
intent of finding software bugs.

Over its existence, computer software has continued to grow in complexity and size.
Every software product has a target audience. For example, a video game software has its
audience completely different from banking software. Therefore, when an organization
develops or otherwise invests in a software product, it presumably must assess whether
the software product will be acceptable to its end users, its target audience, its purchasers,
and other stakeholders. Software testing is the process of attempting to make this
assessment.

.[2Scope
The software faults occur through the following process. A programmer
makes an error (mistake), which results in a defect (fault, bug) in the
software source code. If this defect is executed, in certain situations the
system will produce wrong results, causing a failure.[3] Not all defects will
necessarily result in failures. For example, defects in a dead code will never
result in failures. A defect can turn into a failure when the environment is
changed. Examples of these changes in environment include the software
being run on a new hardware platform, alterations in source data or
interacting with different software.[3]

A problem with software testing is that testing all combinations of inputs

and preconditions is not feasible when testing anything other than a simple
product.[4] This means that the number of defects in a software product can
be very large and defects that occur infrequently are difficult to find in
testing. More significantly, parafunctional dimensions of quality--for
example, usability, scalability, performance, compatibility, reliability--can
be highly subjective; something that constitutes sufficient value to one
person may be intolerable to another.

There are many approaches to software testing. Reviews, walkthroughs or

inspections are considered as static testing, whereas actually running the
program with a given set of test cases in a given development stage is
referred to as dynamic testing.

Software testing is used in association with verification and validation:[5]

• Verification: Have we built the software right (i.e., does it match the
specification)?
 • Validation: Have we built the right software (i.e., is this what the
customer wants)?

Software testing can be done by software testers. Until the 1950s the term
software tester was used generally, but later it was also seen as a separate
profession. Regarding the periods and the different goals in software
testing[6] there have been established different roles: test lead/manager,
tester, test designer, test automater/automation developer, and test
administrator.

Testing methods

Software testing methods are traditionally divided into black box testing and
white box testing. These two approaches are used to describe the point of
view that a test engineer takes when designing test cases.

Black box testing treats the software as a black-box without any

understanding of internal behavior. It aims to test the functionality according
to the requirements.[14] Thus, the tester inputs data and only sees the output
from the test object. This level of testing usually requires thorough test cases
to be provided to the tester who then can simply verify that for a given input,
the output value (or behavior), is the same as the expected value specified in
the test case. Black box testing methods include: equivalence partitioning,
boundary value analysis, all-pairs testing, fuzz testing, model-based testing,
traceability matrix etc.

White box testing, however, is when the tester has access to the internal
data structures, code, and algorithms. White box testing methods include
creating tests to satisfy some code coverage criteria. For example, the test
designer can create tests to cause all statements in the program to be
executed at least once. Other examples of white box testing are mutation
testing and fault injection methods. White box testing includes all static
testing.

In recent years the term grey box testing has come into common usage. This
involves having access to internal data structures and algorithms for
purposes of designing the test cases, but testing at the user, or black-box
level.. Grey box testing may also include reverse engineering to determine,
for instance, boundary values.
Special methods exist to test non-functional aspects of software.
Performance testing checks to see if the software can handle large quantities
of data or users. Usability testing is needed to check if the user interface is
easy to use and understand. Security testing is essential for software which
processes confidential data and to prevent system intrusion by hackers.

[edit] Testing process

A common practice of software testing is that it is performed by an

independent group of testers after the functionality is developed but before it
is shipped to the customer.[16] This practice often results in the testing phase
being used as project buffer to compensate for project delays, thereby
compromising the time devoted to testing.[17] Another practice is to start
software testing at the same moment the project starts and it is a continuous
process until the project finishes.[18]

In counterpoint, some emerging software disciplines such as extreme

programming and the agile software development movement, adhere to a
"test-driven software development" model. In this process unit tests are
written first, by the software engineers (often with pair programming in the
extreme programming methodology). Of course these tests fail initially; as
they are expected to. Then as code is written it passes incrementally larger
portions of the test suites. The test suites are continuously updated as new
failure conditions and corner cases are discovered, and they are integrated
with any regression tests that are developed. Unit tests are maintained along
with the rest of the software source code and generally integrated into the
build process (with inherently interactive tests being relegated to a partially
manual build acceptance process).

Testing can be done on the following levels:

• Unit testing
tests the minimal
software
component, or
module. Each unit
(basic
component) of the
software is tested
to verify that the
detailed design
for the unit has
been correctly
implemented. In
an object-oriented
environment, this
is usually at the
class level, and
the minimal unit
tests include the
constructors and
destructors.[19]
• Integration
testing exposes
defects in the
interfaces and
interaction
between
integrated
components
(modules).
Progressively
larger groups of
tested software
components
corresponding to
elements of the
architectural
design are
integrated and
tested until the
software works as
a system.[citation
needed]

• System
testing tests a
completely
integrated system
to verify that it
 meets its
requirements.[20]
• System
integration testing
verifies that a
system is
integrated to any
external or third
party systems
defined in the
system
requirements.
[citation needed]

Before shipping the final version of software, alpha and beta testing are
often done additionally:

• Alpha
testing is
simulated or
actual operational
testing by
potential
users/customers
or an independent
test team at the
developers' site.
Alpha testing is
often employed
for off-the-shelf
software as a
form of internal
acceptance
testing, before the
software goes to
beta testing.[citation
needed]

• Beta testing
comes after alpha
testing. Versions
 of the software,
known as beta
versions, are
released to a
limited audience
outside of the
programming
team. The
software is
released to groups
of people so that
further testing can
ensure the
product has few
faults or bugs.
Sometimes, beta
versions are made
available to the
open public to
increase the
feedback field to
a maximal
number of future
users.[citation needed]

Finally, acceptance testing can be conducted by the end-user, customer, or

client to validate whether or not to accept the product. Acceptance testing
may be performed as part of the hand-off process between any two phases of
development.[citation needed]

[edit] Regression testing

Main article: Regression testing

After modifying software, either for a change in functionality or to fix

defects, a regression test re-runs previously passing tests on the modified
software to ensure that the modifications haven't unintentionally caused a
regression of previous functionality. Regression testing can be performed at
any or all of the above test levels. These regression tests are often
automated.
More specific forms of regression testing are known as sanity testing, when
quickly checking for bizarre behaviour, and smoke testing when testing for
basic functionality.

[edit] Measuring software testing

Usually, quality is constrained to such topics as correctness, completeness,

security,[citation needed] but can also include more technical requirements as
described under the ISO standard ISO 9126, such as capability, reliability,
efficiency, portability, maintainability, compatibility, and usability.

There are a number of common software measures, often called "metrics",

which are used to measure the state of the software or the adequacy of the
testing.

[edit] Testing artifacts

Software testing process can produce several artifacts. A test case is a
software testing document, which consists of event, action, input, output,
expected result, and actual result. Clinically defined a test case is an input
and an expected result.[23] This can be as pragmatic as 'for condition x your
derived result is y', whereas other test cases described in more detail the
input scenario and what results might be expected. It can occasionally be a
series of steps (but often steps are contained in a separate test procedure that
can be exercised against multiple test cases, as a matter of economy) but
with one expected result or expected outcome. The optional fields are a test
case ID, test step or order of execution number, related requirement(s),
depth, test category, author, and check boxes for whether the test is
automatable and has been automated. Larger test cases may also contain
prerequisite states or steps, and descriptions. A test case should also contain
a place for the actual result. These steps can be stored in a word processor
document, spreadsheet, database, or other common repository. In a database
system, you may also be able to see past test results and who generated the
results and the system configuration used to generate those results. These
past results would usually be stored in a separate table.

The test script is the combination of a test case, test procedure, and test data.
Initially the term was derived from the product of work created by
automated regression test tools. Today, test scripts can be manual,
automated, or a combination of both.

The most common term for a collection of test cases is a test suite. The test
suite often also contains more detailed instructions or goals for each
collection of test cases. It definitely contains a section where the tester
identifies the system configuration used during testing. A group of test cases
may also contain prerequisite states or steps, and descriptions of the
following tests.

A test specification is called a test plan. The developers are well aware what
test plans will be executed and this information is made available to the
developers. This makes the developers more cautious when developing their
code. This ensures that the developers code is not passed through any
surprise test case or test plans.

The software, tools, samples of data input and output, and configurations are
all referred to collectively as a test harness.

Although testing varies between organizations, there is a cycle to testing:

• Requiremen
ts analysis:
Testing should
begin in the
requirements
phase of the
software
development life
cycle. During the
design phase,
testers work with
developers in
determining what
aspects of a
design are
testable and with
what parameters
those tests work.
 • Test
planning: Test
strategy, test plan,
testbed creation.
A lot of activities
will be carried out
during testing, so
that a plan is
needed.
• Test
development:
Test procedures,
test scenarios, test
cases, test scripts
to use in testing
software.
• Test
execution: Testers
execute the
software based on
the plans and tests
and report any
errors found to
the development
team.
• Test
reporting: Once
testing is
completed, testers
generate metrics
and make final
reports on their
test effort and
whether or not the
software tested is
ready for release.
• Retesting
the defects. Not
all errors or
defects reported
 must be fixed by
a software
development
team. Some may
be caused by
errors in
configuring the
test software to
match the
development or
production
environment.
Some defects can
be handled by a
workaround in the
production
environment.
Others might be
deferred to future
releases of the
software, or the
deficiency might
be accepted by
the business user.
There are yet
other defects that
may be rejected
by the
development team
(of course, with
due reason) if
they deem it

Software Quality Assurance (SQA) consists of a means of monitoring the

software engineering processes and methods used to ensure quality. It does
this by means of audits of the quality management system under which the
software system is created. These audits are backed by one or more
standards, usually ISO 9000.
It is distinct from software quality control which includes reviewing
requirements documents, and software testing. SQA encompasses the entire
software development process, which includes processes such as software
design, coding, source code control, code reviews, change management,
configuration management, and release management. Whereas software
quality control is a control of products, software quality assurance is a
control of processes.

The Benefits of SQA

An SQA plan can take a number of paths, testing for different capabilities
and performing different analyses, depending on the demands of project, the
users, and the software itself. But any rigorous SQA plan carried out
scrupulously by seasoned QA professionals will confer certain benefits:

Improved Customer Satisfaction Improved customer satisfaction means

longer, more profitable customer relationships, positive customer
testimonials, and waves of referral business generated from positive word of
mouth.

If customers are dissatisfied with a product they have purchased from a

particular software vendor, they're likely never to recommend that product
nor buy from that software vendor again. Bugs and defects, in addition to
seriously hampering an application's functionality, look sloppy and
unprofessional, and reflect poorly on a company's reputation.

What's more, without proper testing, it's virtually impossible to know how
new users will respond to an application's functions, options, and usability
features. Unbiased software quality assurance (SQA) specialists come to a
project fresh, with a clear outlook, and so serve as the first line of defense
against unintuitive user interfaces and broken application functionality. A
quality application is guaranteed to result in enhanced customer satisfaction.

Reduced Cost of Development Because the process of software quality

assurance (SQA) is designed to prevent software defects and inefficiencies,
projects that incorporate rigorous, objective testing will find that
development costs are significantly reduced since all later stages of the
development life cycle become streamlined and simplified. With SQA, all
further testing and development including user testing and customer
deployments will go more smoothly, and of course more quickly -- which
means your software development project will consistently reach completion
on time and within budget, release after release.

Reduced Cost of Maintenance Bug-infested applications are troublesome

to support. The combined cost of unnecessary recalls, returns, and patches
can be frightful. And that says nothing of what will have to be spent on
ongoing customer support, be it by telephone, email, or in person. All these
costs and more can be dramatically reduced by releasing only rigorously
quality-assured products. Software vendors that invest in quality now can
avoid big losses in the future.

Validation Testing

Validation testing is the act of entering data that the tester knows to be
erroneous into an application. For instance, typing "Hello" into an edit box
that is expecting to receive a numeric entry.

Data Comparison

Comparing the output of an application with specific parameters to a

previously created set of data with the same parameters that is known to be
accurate.

Stress testing

A stress test is when the software is used as heavily as possible for a period
of time to see whether it copes with high levels of load. Often used for
server software that will have multiple users connected to it simultaneously.
Also known as Destruction testing.

Usability testing

Sometimes getting users who are unfamiliar with the software to try it for a
while and offer feedback to the developers about what they found difficult to
do is the best way of making improvements to a user interface.

Static testing is a form of software testing where the software isn't actually
used. This is in contrast to dynamic testing. It is generally not detailed
testing, but checks mainly for the sanity of the code, algorithm, or document.
It is primarily syntax checking of the code or and manually reading of the
code or document to find errors. This type of testing can be used by the
developer who wrote the code, in isolation. Code reviews, inspections and
walkthroughs are also used.

From the black box testing point of view, static testing involves review of
requirements or specifications. This is done with an eye toward
completeness or appropriateness for the task at hand. This is the verification
portion of Verification and Validation.

Bugs discovered at this stage of development are less expensive to fix than
later in the development cycle.

Dynamic testing (or dynamic analysis) is a term used in software

engineering to describe the testing of the dynamic behavior of code. That is,
dynamic analysis refers to the examination of the physical response from the
system to variables that are not constant and change with time. In dynamic
testing the software must actually be compiled and run; this is in contrast to
static testing. Dynamic testing is the validation portion of Verification and
Validation.

Some of dynamic testing methodologies include unit testing, integration

testing, system testing and acceptance testing.

Dynamic Testing involves working with the software, giving input values
and checking if the output is as expected. These are the Validation activities.
Unit Tests, Integration Tests, System Tests and Acceptance Tests are few of
the Dynamic Testing methodologies.

Equivalence partitioning is a software testing related technique with the

goal:

1. To reduce
the number of test
cases to a
necessary
minimum.
2. To select the
right test cases to
cover all possible
scenarios.
Although in rare cases equivalence partitioning is also applied to outputs of
a software component, typically it is applied to the inputs of a tested
component. The equivalence partitions are usually derived from the
specification of the component's behaviour. An input has certain ranges
which are valid and other ranges which are invalid. This may be best
explained at the following example of a function which has the pass
parameter "month" of a date. The valid range for the month is 1 to 12,
standing for January to December. This valid range is called a partition. In
this example there are two further partitions of invalid ranges. The first
invalid partition would be <= 0 and the second invalid partition would be >=
13.

... -2 -1 0 1 .............. 12 13 14 15
.....
--------------|-------------------|----------
-----------
invalid partition 1 valid partition invalid
partition 2

The testing theory related to equivalence partitioning says that only one test
case of each partition is needed to evaluate the behaviour of the program for
the related partition. In other words it is sufficient to select one test case out
of each partition to check the behaviour of the program. To use more or even
all test cases of a partition will not find new faults in the program. The
values within one partition are considered to be "equivalent". Thus the
number of test cases can be reduced considerably.

An additional effect by applying this technique is that you also find the so
called "dirty" test cases. An inexperienced tester may be tempted to use as
test cases the input data 1 to 12 for the month and forget to select some out
of the invalid partitions. This would lead to a huge number of unnecessary
test cases on the one hand, and a lack of test cases for the dirty ranges on the
other hand.

Boundary value analysis is a software testing design technique to

determine test cases covering off-by-one errors. The boundaries of software
component input ranges are areas of frequent problems.

[edit] Introduction
Testing experience has shown that especially the boundaries of input ranges
to a software component are liable to defects. A programmer implement e.g.
the range 1 to 12 at an input, which e.g. stands for the month January to
December in a date, has in his code a line checking for this range. This may
look like:

if (month > 0 && month < 13)

But a common programming error may check a wrong range e.g. starting the
range at 0 by writing:

if (month >= 0 && month < 13)

For more complex range checks in a program this may be a problem which
is not so easily spotted as in the above simple example.

[edit] Applying boundary value analysis

To set up boundary value analysis test cases you first have to determine
which boundaries you have at the interface of a software component. This
has to be done by applying the equivalence partitioning technique. Boundary
value analysis and equivalence partitioning are inevitably linked together.
For the example of the month in a date you would have the following
partitions:

... -2 -1 0 1 .............. 12 13 14 15
.....
--------------|-------------------|----------
-----------
invalid partition 1 valid partition invalid
partition 2

Applying boundary value analysis you have to select now a test case at each
side of the boundary between two partitions. In the above example this
would be 0 and 1 for the lower boundary as well as 12 and 13 for the upper
boundary. Each of these pairs consists of a "clean" and a "dirty" test case. A
"clean" test case should give you a valid operation result of your program. A
"dirty" test case should lead to a correct and specified input error treatment
such as the limiting of values, the usage of a substitute value, or in case of a
program with a user interface, it has to lead to warning and request to enter
correct data. The boundary value analysis can have 6 testcases.n, n-1,n+1 for
the upper limit and n, n-1,n+1 for the lower limit.

Fuzz testing or fuzzing is a software testing technique that provides random

data ("fuzz") to the inputs of a program. If the program fails (for example,
by crashing, or by failing built-in code assertions), the defects can be noted.

An important distinction between "fuzz testing" and for example "mutation

analysis" is that random data is not necessarily going to find software
weaknesses in the shortest amount of time. Fuzz testing transmits data that
does not necessarily map to anything dangerous within the target, whereas
other, better-chosen, data might do more damage. Mutation analysis
carefully models the state, structure, and semantics of the protocol and
maximizes the damage that can be caused in the shortest possible time.
Another aspect of mutation analysis is that test cases can be more specific
than a fuzzer (which might send a packet that causes a crash, but the packet
may have multiple problems which may make the fault(s) perhaps less
reproducible, thus making remediation more difficult; mutation analysis test
cases try to invalidate only one aspect of a protocol at a time). For the
purposes of this page, the term "fuzz testing" is used as a generic description
for this category of testing. Mutation analysis is a specific, highly optimized
kind of fuzz testing.

The great advantage of fuzz testing is that the test design is extremely
simple, and free of preconceptions about system behavior

White box testing

Mutation testing (sometimes also called mutation analysis) is a method of

software testing, which involves modifying program's source code in small
ways.[1] These, so-called mutations, are based on well-defined mutation
operators that either mimic typical user mistakes (such as using the wrong
operator or variable name) or force the creation of valuable tests (such as
driving each expression to zero). The purpose is to help the tester develop
effective tests or locate weaknesses in the test data used for the program or
in sections of the code that are seldom or never accessed during execution.

Mutation testing is done by selecting a set of mutation operators and then

applying them to the source program one at a time for each applicable piece
of the source code. The result of applying of one mutation operator to the
program is called a mutant. If the test suite is able to detect the change (i.e.
one of tests fails), then mutant is said to be killed.

For example, let's consider the following C++ code fragment:

if (a && b)
c = 1;
else
c = 0;

The condition mutation operator would replace '&&' with '||' and produce the
following mutant:

if (a || b)
c = 1;
else
c = 0;

Now, for the test to kill this mutant, the following condition should be met:

• Test input
data should cause
different program
states for mutant
and original
program. For
example, a test
with a=1 and b=0
would do this.
• The value of
'c' should be
propagated to the
program's output
and checked by
the test.

Weak mutation testing (or weak mutation coverage) requires that only the
first condition is satisfied. Strong mutation testing requires that both
conditions are satisfied. Strong mutation is more poweful, since it ensures
that the test suite can really catch the problems. Weak mutation is closely
related to code coverage methods. It requires much less computing power to
ensure that the test suite satisfies weak mutation testing than strong mutation
testing.

In software testing, fault injection is a technique for improving the coverage

of a test by introducing faults in order to test code paths, in particular error
handling code paths, that might otherwise rarely be followed. It is often used
with stress testing and is widely considered to be an important part of
developing robust software[1].

The propagation of a fault through to an observable failure follows a well

defined cycle. When executed, a fault may cause an error, which is an
invalid state within a system boundary. An error may cause further errors
within the system boundary, therefore each new error acts as a fault, or it
may propagate to the system boundary and be observable. When error states
are observed at the system boundary they are termed failures. This
mechanism is termed the fault-error-failure cycle [2] and is a key mechanism
in dependability.

A function point is a unit of measurement to express the amount of business

functionality an information system provides to a user.

Regression Testing

A sanity test or sanity check is a basic test to quickly evaluate the validity
of a claim or calculation. In mathematics, for example, when multiplying by
three or nine, verifying that the sum of the digits of the result is a multiple of
3 or 9 respectively is a sanity test.

In computer science it is a very brief run-through of the functionality of a

computer program, system, calculation, or other analysis, to assure that the
system or methodology works as expected, often prior to a more exhaustive
round of testing.

Smoke testing is done by developers before the build is released or by testers

before accepting a build for further testing.

In software engineering, a smoke test generally consists of a collection of

tests that can be applied to a newly created or repaired computer program.
Sometimes the tests are performed by the automated system that builds the
final software. In this sense a smoke test is the process of validating code
changes before the changes are checked into the larger product’s official
source code collection. Next after code reviews, smoke testing is the most
cost effective method for identifying and fixing defects in software; some
even believe that it is the most effective of all.[citation needed]

In software testing, a smoke test is a collection of written tests that are

performed on a system prior to being accepted for further testing. This is
also known as a build verification test. This is a "shallow and wide"
approach to the application. The tester "touches" all areas of the application
without getting too deep, looking for answers to basic questions like, "Can I
launch the test item at all?", "Does it open to a window?", "Do the buttons
on the window do things?". There is no need to get down to field validation
or business flows. If you get a "No" answer to basic questions like these,
then the application is so badly broken, there's effectively nothing there to
allow further testing. These written tests can either be performed manually
or using an automated tool. When automated tools are used, the tests are
often initiated by the same process that generates the build itself.