Non-Functional Testing

In order to ensure your system is ready to go live it is necessary to go beyond just functional testing.
Non-Functional Testing is designed to evaluate the readiness of your system according to several
criteria not covered by functional testing. These criteria include:

Performance: We test a system's performance to ensure that it provides acceptable response times and
user load capabilities. Performance testing will:

Test applications against predicted future volumes

Identify break points to assess the applications ability to cope with a given number of users
Undertake performance based load testing to assess applications performance when being accessed by
large numbers of users simultaneously
Test over an extended period of time to ensure, for example, that an application is not running out of
memory

Disaster Recovery: We will ensure that you are prepared in case of a disaster. Disaster recovery plans
are checked and back-up systems are tested. Proving the effectiveness of your back-up systems will
prevent damage to your business’ reputation and minimize the costs involved if the main system
malfunctions.

Security: We will test the security of your system and assess its vulnerability to hacking, etc.

Plan for Non-functional testing of evolution -:

The non-functional testing of evolution will include the following types of testing:

Reliability Testing -:

Under this category, evolution client will be tested to evaluate the ability to perform its required
functions under stated conditions and set of operations for a specified period of time or number of
iterations.

Scope -:

The scope of reliability testing is to execute basic functionalities provided by evolution client i.e.
Mailer, Calendar, Task and Address Book operations. The type of account in which these operations
will be performed will be Groupwise, Exchange and IMAP. LDAP address book also needs to be
considered, with account of any type.

Setup -:

6 evolution clients will be installed on a SLP 10 test machine and will be configured in the following
manner:

• 2 evolution clients will be configured for a groupwise account

• 2 clients will be configured for an exchange account
• 2 clients will be configured for an IMAP account
Test Scenarios :-

All the 6 clients running on different test machines will be running simultaneously performing the
following basic operations (test scenarios) :

1. Mailer :-

• Send a mail to different contacts randomly selecting from a list of email-ids.

• Send a mail with or without attachments
• Send a plain text mail with or without attachments
• Send an HTML mail with or without attachments
• Send a mail with huge text/HTML content spanning multiple pages
• Open a large text/html mail with or without attachment
• Reply to an existing mail
• Forward an existing mail

2. Calendar :-

• Create a new meeting request

• Add 1 or multiple invitees to a meeting request and send the appointment.
• Add 1 or multiple attachments (different file formats) in the meeting request and send the
appointment.
• Accept an appointment present in the inbox.
• Discard an appointment present in the inbox.

3. Address book :-

• Send a mail/meeting request autocompleting the ids from the address books.
• Send a mail/meeting request autocompleting the ids to distribution lists.
• Search for the contacts in remote address books and large local books.

NOTE:-

• The attachments mentioned in the above set of operations will be selected randomly from a
directory containing various sizes and types (file formats) of attachments and there can be
more than one attachment present in the mail.
• At this point of time only Mailer and Calendar are being considered to execute for reliability
testing, later Tasks and Address book can be added.
• There are more test scenarios but for the purpose of reliability testing we are considering only a
small set of operations that are performed by a normal user quite frequently.

Execution :-

The execution of the above scenarios will be automated by using a shell script and the input data files.
The shell script will perform the operations in the evolution client using a GUI automation tool, ltfx.
The currently available shell script handles the following tasks so it needs to be modified in order to
add other test scenarios explained above:

• Launch evolution after setting the debug environment variables on command line
 • Automates the selection of the operation to be performed i.e. sending mail, calendar OR task.
• Adds the email-ids and performs the auto-completion.
• Adds the text in the various text fields.
• Attaches files automatically.
• Send a mail (with/without attachments to different people)
• Send a meeting request (with/without attachments)
• Create a new task (with/without attachments) and assign it.

The script will be started once and will be left to run continuously for 5 days. At the end of 5 days we
will be observing the state of the system and the evolution application.

Results :-

The result of reliability testing of evolution will consist of the system statistics (memory and CPU
usage of all evolution processes) and the statistics of the operations performed (this will include the
number of operations of each type that are performed by evolution client program)
There can be various circumstances under which the statistics should be collected. Some of these are
listed below (Think of some more situations):
1. If the execution of reliability test script has stopped before the specified end time we need to collect
the following data:

• Number of mails/appointments sent/received by evolution through the script.

• Time for which the test was running.
• Memory snapshots of evolution and e-d-s processes at specific intervals when its performing
the various operations in mailer, calendar or addressbook.
• Reason for the termination OR hang.
• Stack trace at the time of termination OR hang.

2. If the execution has completed without any unexpected errors and problems we need to collect:

• Number of mails/appointments sent/received in the specified time period.

• Time taken for sending these mails/appointments.
• System state at the end of the test execution (will include the memory snapshots and CPU
utilization of all evolution processes and load average of the system.

Performance Testing :-

This testing will be conducted for evolution to evaluate the time taken or response time of evolution to
perform it's required functions (in mailer, calendar, address book and tasks) under stated conditions in
comparison with different versions of evolution.

Scope :-

The scope of performance testing of evolution is to execute the basic functionalities provided by each
of its component i.e. Mailer, Calendar, Address Book and Tasks and record the time it takes to execute
each of the functions. Execution will involve creating a specific set of input data for the operations and
then measuring the performance of the system. These performance tests will be executed on 2 different
versions of evolution, 2.2 (the last stable version) and 2.4 (the latest stable version). The output of
performace test will be the comparison of results between the 2 versions of evolution.
Setup :-

There will be 6 evolution clients installed on different test machines. Out of the 6 clients 3 will be
evolution-2.2 running on SuSE 9.3 and 3 will be evolution 2.4 running on SuSE Linux Professional
10. Each of the 3 evolution clients (2.2 and 2.4) will be configured with any one of the IMAP,
Groupwise OR Exchange accounts. 3 Evolution-2.2 clients will be configured with IMAP, Groupwise
and Exchange accounts 3 Evolution-2.4 clients will be configured with IMAP, Groupwise and
Exchange accounts. And also LDAP address books. (on SSL and non-SSL connections).
For the performance testing it is better to have SSL enabled connections. Also the servers should
contain huge number of realistic objects (typical organization like). I mean we should not have user
objects with names user1, user2... or only first names.

Data/Load Profile :-

We will be starting the tests with an existing set of data (mails, appointments, contacts and tasks) in
each of the different kinds of email accounts (configured in evolution). We also need to create some
amount of data with which the tests will be done. The generation of test data will include the various
scenarios OR situations in which this data will be handled by evolution client. So basically we will be
dividing the inputs for performance test into the following:
1. Different types of users of evolution client.
EXAMPLE :
Who is actually using the client. Is it a Home user, a student, a software developer, etc.. Each of these
users will use the evolution mailer in a different pattern, so we will have different input sizes.

2. Different operations that a user can perform.

EXAMPLE :

• Send Mail
• Receive mail
• Searching for mails
• Sorting of mails
• Syncing up with the server
• Delete/Move/Copy a mail

• Having message filters.

• marking a folder (mail/contacts/calendar) for offline usage, this involves buidling the local
cache.
• with autocompletion and without autocompletion (By disabling autocompletion => not
selecting any folder for autocompletion)
• searching for a contact.
• ... Can add more ...

3. Different scenarios when the number of operations will be more OR less

EXAMPLE :
When is the mailer being used. If its during some festival season, the number of mails flowing to and
from the evolution client will be very HIGH and if its a vacation period it might be very LOW. Based
on this we can generate the amount of input that we need to use.

4. Number of operations performed by a particular user.

EXAMPLE :
How many mails in a typical scenario (#3 above) are being sent, received, deleted, moved, searched,
sorted by a particular user (#1 above)

Conclusion :-

Based on the above 4 categories we can come up with some set of data that has following
characteristics:

• Email size (it will include attachments also)

• Number of emails present in folders
• Number of emails being received
• Rate of retrieval of mails
• Mail filtering

• Number of contacts.
• Number of contacts with images.
• Number of distribution lists with large number of members.
• Number of contacts with all the fields filled. (not only name and e-mail id)
• ... there can be some more ...

And this data can be used for measuring the performance of evolution client with respect to the
number of operations performed in a particular scenario.

Test Scenarios :-

Mailer :-

The following measurements will be taken on evolution-2.2 and 2.4:

1. Time taken for initial loading of evolution with different number and types of mails present in the
INBOX [Test with 10, 100, 500, 1000, 5000, 10000, 50000 mails in the inbox]
2. Total time taken to compose and send a new mail [with OR without attachments]
Total time taken = Time taken for (Open compose mail window + To: + Cc: [with OR without
autocompletion] + write mail text + Send the mail)
3. Time taken to open an existing mail [with OR without attachments]
4. Time taken to move a mail from one folder to another. Time taken to move a set of mails (say 100)
to another folder.
5. Time taken to save an attachment [with size of attachments varying from 1KB, 500KB, 1MB, 5MB,
10MB, 50MB, 100MB, 500MB and with different types of attachments]
6. Time taken to reply to an existing mail [with OR without attachments]
7. Time taken to reply to all to an existing mail.

Calendar :-

Some calendar performance testing scenarios are explained in CalendarPerformanceTesting .

NEED COMMENTS :- I am not sure if the above mentioned scenarios can also be included in the
scope of this Non-functional Test Plan.
In my opinion the following measurements can be taken on evolution-2.2 and 2.4:
1. Time taken for initial loading of calendar component with different number and types of
appointments present in the calendar [Test with 10, 100, 500, 1000, 5000, 10000 appointments]
2. Time taken to create a new meeting by filling in all the details in the create meeting window.
3. Time taken to accept an existing appointment in the inbox.
4. Time taken to open an existing appointment in the calendar.
5. Time taken to add new invitees to a meeting request [with OR without autocompletion of mail ids]
6. Time taken to create a new meeting with 1 or more attachments [test with attachments of different
sizes and types]
7. Time taken to display all the appointments in list view.
8. Time taken to randomly open an appointment by specifying a date/ date range,

Address Book :-

NEED COMMENTS : I am not sure if the above address book performance scenarios can be
included in the scope of this Non-Functional Test Plan.

Execution :-

The execution of the above scenarios will be automated by using a shell script and the input data files.
The shell script will perform the operations in the evolution client using a GUI automation tool, ltfx.
The script for performing the above Mailer/Calendar/Address Book operations is not currently
available so we need to write it in order to run these tests.

Results :-

When the various operations are being performed with the evolution client, the values of the following
2 system parameters will be captured: 1. Memory usage of all evolution-* processes. 2. CPU
utilization of all evolution-* processes.

Scalability Testing :-

Scalability testing of the evolution client will be done together with the performance testing and will
mainly involve the following:
1. Measuring the time it takes for evolution to load when the number of mails in the inbox are 100,
1000, 5000, 10000, 50000, 100000.
2. Measuring the time it takes for the address book to load when the number of contacts are gradually
increased from 50 to 10000 and beyond
3. Measuring the time it takes to perform a search when there are many number of mails present in the
inbox.
4. Measuring the time it takes to perform a search in the address book
5. Measuring the time it takes to load the calendar entries from a remote server.
6. Measuring the time when the number of entries in the calendar is very high.
7. Measuring the time it takes to perform auto-completion.

As we are mainly concerned with the performance of evolution client, the scalability tests will be
limited to scaling the amount of input data, so it can be performed in conjunction with the
Performance testing.