Professional Documents
Culture Documents
Science or Fiction?
Author:
Pronil Sengupta (pronil@adobe.com)
Dinesh Kukreja (dinesh@adobe.com)
Adobe Systems India Pvt. Ltd. I-1A, City Center, Sector – 25A,Noida (U.P.) - 201301
1
Robots are science but Robots which has Emotional Intelligence or those who can reproduce, is a fiction. We all can
easily make out what is a fiction and which is science. Today in testing industry, automation is somehow believed to
be a science in which many of its part is actually a fiction. We should automate but automate correctly. Here in this
paper we are trying to segregate reality from imagination, science from fiction by discussing the correct ways to
automate, the importance of manual testing and the territory of manual testing where automation should not enter.
After all
Automation + Manual testing = Science
but
Automation + Automation (Manual testing) = Fiction.
Preface
In 1985 government created a microcomputer physically indistinguishable from a ten years old boy, the boy has
been designed for an artificial intelligence experiment. The boy somehow escapes and got adopted by a couple. He
baffles everyone with his superhuman capabilities. One day government finds him and takes him back. But the boy
develops emotions and attachment to his parents. Government scraps the project and tries to terminate him, but with
all the superhuman thrills at the end the boy reaches his parents and stays with them ever after. A boy, a robot but
has artificial intelligence and emotion? Does it sound like science or fiction?
Lot of you would agree that this is a nice fiction. Yes, indeed, it is the story of a Hollywood movie Daryl, Data
analyzing Robot youth Lifeform written by David Ambrose, Allan Scott and Jeffrey Ellis. Directed by Simon
Wincer.
So, what is a fiction? A science which has broken the bounds of imagination or some patentable ideas which does
not look real today but looks nice or a kind of hyped science is a fiction.
In the world of testing, the scientific approach has given birth to automation. As automation gained pace over the
time, it is now breaking the bounds of imagination and developing a perception which is making automation a
fiction. Perception like, automation can replace manual testing completely, or automation takes 100% control over
testing are the perception which makes automation a Fiction. Automation and Manual testing have their own
strengths and limitations. Automation and Manual testing actually complements each other. We should understand
what science of automation is and what kind of perceptions can make it fiction. We should focus our effort on right
strength of Automation and Manual testing.
Scope
In this paper we tried to put forward the Idea that neither the Automation nor the manual testing is an over head.
Among automation and manual testing, neither automation can replace manual testing nor can manual testing
replace automation. Automation and manual testing each have their own set of advantages and limitations. One
needs to understand the benefit of each to leverage best out of both the methodologies of testing. Through this paper
we are trying to build an understanding of actual benefits of automation by following topics:
It is important to build an understanding around Automation so as to keep it away from it being fiction, the
innovation in Automation should be seen in vertical rather than horizontal. By vertical we mean, we should start
innovating and expanding automation to different areas like development, coding, management rather than only
testing (horizontal)
2
What is Science and what is Fiction?
Robotics is Science. Artificial Intelligence is an open research area, and Artificial Intelligence to an extent where
Robots can imitate true human emotions and think by their own is a Fiction. If we need some examples of Fiction in
robotics we would probably list down some statements like:
Robots can Feel
Robots can Reproduce
Robots can Suspect
Robots can Inspect
Robots can Adapt
May be someone can go into long argument to prove these fiction a science, maybe he win at some point with
logics, but the basic is, robot may be able to do a part or each, but would not be able to imitate 100% as human.
Robotics is science and the scientific statements around robotics would be:
Robots can Jump
Robots can Run
Robots can Work
Robots can execute commands what they are programmed for
Here lies the difference between Robots and Human. Robots no doubt would increase efficiency, increase accuracy
and help in giving space to human to think more. We should not forget the real strength of human, and we should
remember that, robots can’t feel, reproduce, suspect, inspect or adapt but human can
At the same time one should be aware of limitations of automation. What automation cannot do which human can.
Automation cannot feel, adapt, evolve, suspect and inspect but a manual tester can. During manual testing when a
3
tester finds a bug, he investigates, he hunts for the root cause, he then suspects that there could be more bugs.
Automation system cannot have this suspicion. Automation system cannot make itself aware of surrounding
environment and generate ideas which can bring innovations. However, during manual testing, testers can have
ideas which can evolve and bring innovation. These are some of the limitations of automation where manual testing
adds value.
If we think that manual testing is prone to human error, then this should not be the only reason to automate. Testing
got its place not because of the only reason of syntactical or semantic error in code. Thus, automation and manual
testing goes hand in hand.
It is important to understand that, automation should not be a goal of testing. The purpose of automation is to
facilitate us in achieving testing goal.
Plan:
Once the goal is set, automation needs to be planned. Automation project many a times face problems with deciding
the right time to start automation, defining the right scope of the automation and selecting the right approach for the
automation. While planning automation we should remember the following things:
4
- Availability of test data (files) at proper places
- Specify the special test requirement for the test cases.
- Verify the test specification (Priority, Language, platform etc) for test cases
Execution ->Inspection->Correction:
Once the detailed automation plan has been created, on the basis of various best practices, now it’s time to start
writing the automation scripts using the identified automation tool. The various best practices like reusability of
objects, proper log generation, error handling mechanism, extensibility etc. which were defined in the test plan
needs to be taken care while writing the automation scripts.
Once the automation scripts are ready then trial run should be performed on the actual builds of the product. Result
of the trial run needs to be analyzed over few builds. The reason for analyzing the result over few build is to make
sure that automation is not giving any false positives. There are occasion when automation scripts itself are working
fine but due to some other glitches it gives incorrect result. Once we start seeing the static result over few builds
then we need to dig into the actual automation failures and start analyzing them manually. There could be three main
reasons for the failure of an automation scripts i.e
5
It could be an actual bug which is found by automation.
A buggy script itself.
Known/Unknown limitation of automation tool.
It’s important to take the Corrective actions as soon as the root cause of the failure has been analyzed. If corrective
action against failure is not taken in its first instance then it will keep bugging you again and again each time the
automation is executed. This will result in unnecessary wastage of time. The various corrective actions which can be
taken as:
If it’s an actual bug then log the bug and specify the defect against its result so that next time you saw the
same failure then you can look at the previous result and ensure that it’s an actual defect.
If it’s an automation script failure then script needs to be corrected for logical, syntax error
If it’s happening due to unknown/known limitation of the automation tool then it’s better to keep such test
case as Manual test case.
Retrospection:
In the due course of action once the automation has gone through several passes on daily builds now it’s time to give
a pause; look back and analyze whether we are on the right track of achieving the defined goal ? It is important to
answers of the following question for measuring the effectiveness of the automation.
Is automation is giving the desired result which we have aimed for?
What are the things which went good?
What are the things which went bad?
What are the backlog items?
What is the learning from the project?
Where is the Scope of improvement?
Maintenance:
Maintenance of automation suite is an integral part of the overall automation project. At this stage automation has
become our asset now and it’s become important to keep our assets updated as and when it is required. It’s necessary
to regularly monitor for the instance when automation scripts needs to be modifies. Several possible instances could
be:
- Major change in functionality.
- Change in UI of the application.
- Baselines need to be updated.
- Added Support for languages.
- Added support for platforms.
Process improvement:
Manually lookout for gaps in process and do continuous process improvement. Process improvement should a focus
of every organization. A correct process can stop some bugs being introduced. Automation process as suggested
above in retrospection phase can be assessed for gaps and then the process can be enhanced to avoid the gap.
6
Review:
Involve testers at the initial phase of the project, and have them review requirements. Review of any document
created should be carried out as a practice throughout the project. Any bug caught at review phase can save cost if
those bugs discovered at later phase.
Manually test:
Finally even after implementing automation, what you should manually do is manually test.
New Features:
Newly developed features needs to be tested manually, since these feature are not stable and there will be many bugs
in the initial stages of the feature. Also changes in the feature will keeps on coming build over build as and when
bugs gets fixed or there is some change in functionality.
Adhoc/Exploratory Testing:
Adhoc/Exploratory testing is the one which gives us lots of bug in the feature. Even if a particular feature is 100%
automated we should perform the adhoc/exploratory testing at regular interval. This is type of testing where we test
a feature in integration with various other feature and try to create the complex situation. This testing helps us in
finding real bugs which could be very annoying to our customer if not discovered.
Unstructured Testing:
Apart from Adhoc/Exploratory testing there are couple of other unstructured testing techniques like Bug hunts,
Feature swap testing, workflow testing, customer site testing which we should keep on doing for our feature even if
they are automated. These testing techniques continuously keeps on improving the quality of our feature by finding
the potential bugs which are not possible to catch through automation.
7
Conclusion & Summary
Daryl had a happy ending and it was fiction, fictions do open the doors for a new line of thought and imagination
and may research results in some science, which makes Daryl a reality. But we would say it is witty to use available
science on your project, rather than doing research with a live project. Concluding, we only want to say that
automation and manual testing both have their own benefits. Use automation where it is most effective and use
human where they are most effective.
References:
http://www.imdb.com/title/tt0088979/
http://www.fast-rewind.com/daryl.htm
Biography:
Pronil Sengupta is working as Lead Quality Engineer for Adobe since more than 4 years. He has a total experience
of 8 years in software testing. Pronil has varied experience in testing application, application installer and online
travel portal.
Academically Pronil is a management graduate. He has also co-authored and presented a paper, Product Vaccination
and Quality Upbringing in STC2009
Email: pronil@adobe.com
Dinesh Kukreja is Lead Quality Engineer at Adobe and has 5years or experience. He has a total experience of 6
years in software testing.
Academically Dinesh holds Bachelor degree in IT. He has also co-authored and presented a paper, “Continuous
Quality improvement through unstructured testing”, in STC2009.
Email: dinesh@adobe.com
Appendix
Explained inline