Combining Your Scripted, Exploratory and Automated Mobile Testing

Contents
Introduction.............................................................................................3 Why Testing Mobile Apps Is Different Than Any Other Platform............ 4
Rule #1 - Make sure your mobile app does what the user is expecting it to do. .............................................................................................. 5 Rule #2 - Concentrate on the devices where the majority of your target market will be and camp out there................................................... 5

Asking the Right Questions....................................................................6

GPS................................................................................................................... 6 Data................................................................................................................... 6 Gestures............................................................................................................ 7 GUI.................................................................................................................... 7 Processing Power.............................................................................................. 8 Memory.............................................................................................................. 8 Battery............................................................................................................... 8 Fragmentation of devices.................................................................................. 8

Scripted, Exploratory and Automated Testing in the Mobile Space........ 9

Automated Testing........................................................................................... 10 Scripted Testing............................................................................................... 12 Exploratory Testing.......................................................................................... 13

The Mobile Testing Balancing Act.........................................................14 Conclusion............................................................................................15

by SMARTBEAR

TestComplete

Introduction
SoLoMo. You know the mantra. Social, local, mobile. Its the way the Internet is heading. Its the direction that the world is heading. We have been hearing it for years. The difference with that clarion cry is that we are way past the hype phase. No longer do we wonder if this is the Year of Mobile. Mobile has arrived and it is past the point of novelty. Whether it is business or personal use, mobile is getting more and more ingrained in all aspects of our lives each and every day. As a result, users are more savvy than ever before and their patience with poorly designed and delivered mobile apps is very limited. This shift in how the world accesses information and uses data has forced the mobile world to grow up faster than it may have been ready for. No longer can an app be almost there and be released into the market. Today, if your mobile app is even the slightest bit buggy or crashes from time to time or doesnt work on as many platforms as possible, you stand to lose big. Its for this reason alone that the practice of mobile app testing is a critical step in the life cycle of an app. Very few companies are allowed any leeway in how their apps perform. A perfect example of this is Facebook. Its hard to get more ubiquitous than Facebook, yet when the company went public the doubts about their mobile capabilities hampered the stocks performance. Now, if Facebooks mobile presence doesnt meet the high expectations of the marketplace the entire brand suffers. Granted, not all businesses face being put under this kind of harsh interrogation, but that day is not far off if we continue to progress at the current pace. A mobile app that has even the smallest imperfection will be quickly pushed aside in search for another to serve an end users needs.
by SMARTBEAR

TestComplete

That said, it is critical to address the testing aspect of your app development with the utmost concern. In this eBook, we will take a look at the different ways mobile app testers need to test.. Scripted testing alone will not be enough. Exploratory testing alone will not be enough. Automated testing alone will not be enough. Instead, todays testers of mobile apps will be required to find the blend of all three types of testing in order to feel confident that their apps are ready for a rigorous marketplace. We will explore the unique elements of the mobile world and how software testers will need to find the right mix of testing so that, when their app gets to the end user, they wont immediately be cast aside.

Why Testing Mobile Apps Is Different Than Any Other Platform

Mobile is like the wild west of the digital era. While in some ways its not much different than other areas, what separates it is that the wide range and array of users and available devices create a myriad of combinations. Because of this never ending series of combinations of user types, platforms and other variables, it is likely to be an impossibility to test for everything. Accepting this fact from the onset will be important just so any work that has been done can actually see the light of day in the marketplace. To start, well first take a look at a few of the elements that make the mobile environment so challenging especially from a testing perspective, and some general rules to help you deal with them.

by SMARTBEAR

TestComplete

Rule #1 - Make sure your mobile app does what the user is expecting it to do.
Before even getting near the intricacies and complexities of the mobile environment, it has to be determined whether the app itself simply accomplishes what it was originally intended to do. Why is this more difficult in the mobile world? Its quite simple as to why, but its a whole other issue as to what to do about it. The why its hard is due to how end users behave in the mobile space. It is a very new area of exploration for everyone and almost everything that is seen is some variation of first use case, which means that there is little or no historical data or experience to fall back on for validation. With this in mind, before you go through the hard work of putting an app through its paces with a variety of tests, you better make sure it is accomplishing the desired business task. Before you go through the hard work of putting an app through its paces with a variety of tests, you better make sure it is accomplishing the desired business task.

Rule #2 - Concentrate on the devices where the majority of your target market will be and camp out there.
There may not be a more difficult hurdle to overcome in the mobile app world than the issue of device acceptance. iOS, Android, Windows, and BlackBerry are difficult enough to test for on their own. Then, get into the various versions of each mobile OS and you can quickly feel like the better alternative is to simply scream and hope it goes away. But add on to this the many and various operating systems of Android, and the mobile landscape begins to seem infinite so it is rarely enough to only develop for iOS anymore. Certainly the demographics of the iOS mobile device user are more attractive than any other, but even that metric is becoming less dependable with the
by SMARTBEAR

TestComplete

advances that Samsung and HTC have made with their Android models. Many kids these days are walking around with a Galaxy S4 and not an iPhone 5S. Since its difficult to be all things to all people, if you think that most of your users are going to be a combination of iOS and the most popular flavor of Android / device, you may need to be developing and testing just for those targets.

Asking the Right Questions

We are going to make a quick run through a variety of elements that will be critical to consider as you test your mobile apps. Take note of some of the questions you should be asking of each element.

GPS How important are geolocation elements to your app? The mere
fact that you are working in the mobile environment implies that you know the location of the user will be a critical piece of your mobile app puzzle.

Does your app play with the major map platforms? Will your app integrate properly with Google Maps? Apple
Maps?

Who do you use? Whom do you partner with? Who gives the best data and user experience? How are the various APIs? What are the limitations and costs? Data How much data? What data? What are your privacy policies? What are you accessing on their device? What is your end users tolerance for privacy encroachment?
by SMARTBEAR

TestComplete

 Network Availability How does your app perform on various carrier networks as
opposed to Wifi?

Will your app be a data hog that causes users to watch their
bandwidth usage as a result?

Is the performance on Wifi so markedly improved as to

discourage the end user from using the app in any other environment?

Gestures Does your app give the user gesture options that are confusing
or that create conflict with other apps?

If they are not willing to adapt to the gestures you have

incorporated in your app is the app navigable without them?

GUI Lets get down to brass tacks here. Is your app simply butt-ugly? Does your app have a command line feel to it or is it appealing
to even the most non-technical of users (who will be a vast majority of most app audiences anyway)?

Portrait and Landscape Views What happens to your app when your end user turns his or her
device on its side or on its head?

Does it respond quickly? Is it responsive at all? Are there any fields that suddenly become impossible to
navigate because of the limited space due to the keyboard expanding for use?

by SMARTBEAR

TestComplete

Processing Power The last thing you want your app to be associated with is
making your users device so slow because it is a tremendous resource hog. If within those first few hours of downloading your app your customer notices that nothing is working as well on their device as it did before they downloaded your app guess what happens.

Memory Lets be honest. Most mobile device users are pushing their
memory limits to the nth degree on any given day.

How will your app impact device memory? Battery What impact will your app have on the battery life of a mobile
device? If an end user downloads your app then suddenly sees their battery life fall through the floor you may be viewed as a troublemaker. Not good.

Fragmentation of devices We alluded to this earlier, but its the simple truth about the
mobile space. There are an ever growing list of devices, screen sizes, form factors, etc. that make up the mobile marketplace.

Its not cheap to make design alterations for every make and
model. Also, once you make accommodations for one you may end up breaking something with another.

Will you need to pick and choose which devices are going to
represent the largest portion of your target market then go from there? This is a critical decision that must be taken seriously.

by SMARTBEAR

TestComplete

Scripted, Exploratory and Automated Testing in the Mobile Space

There is one more important element in this process. The element is time. Everyone gets the same amount of it every day and it is a resource that cannot be grown or added to. The results, as it relates to your mobile app development and testing, is that you have to place time limits and parameters on the testing environment. Therefor, it becomes even more critical to use the right type and combination of testing methodologies to ensure that your mobile app is ready for its introduction to the marketplace. Now that we have explored the various elements of the mobile environment in at least a cursory fashion, its time to look at what types of app testing will help you put your best mobile app foot forward. To set the stage lets take to heart this quote from a Mobile Marketer article Mobile is different, and when companies dont treat mobile applications as a key part of their business by investing in a great user experience, consumers are disappointed in the applications, said Peter Galvin, senior vice president of marketing at Soasta, Mountain View, CA. Our research has shown that many mobile applications are not properly tested before they are delivered to consumers, he said. In the physical world marketers test their advertising, store concepts and messaging to ensure they resonate with consumers. If that is the case, then why is testing in the mobile space not prioritized properly by many? Simply put, testing effectively can be a daunting task. Effective app testing in much more stable and less variable environments is hard enough, but add the reality of testing in a world of constant change and it becomes a challenge that has to be faced with appropriate expectations.

by SMARTBEAR

TestComplete

What are those expectations? Well, with a fair amount of planning, a concentrated effort and a little luck, one should expect that they could cover most of the bases when testing mobile apps. What is probably unrealistic is the expectation of testing for every situation, under every potential scenario and uncovering every little ailment of an app before it gets to market. Especially since the fluid nature of the mobile space changes the rules so often, you can never take care of every angle. That doesnt mean that it cant get close though. The best way to get result that are going to set your efforts apart from your competitions is by utilizing various testing methods in the right mix. As with any software or app testing, the process is as much art as it is science. Flexibility will win the day, so loosen your collar as we delve into the testing options that are available and help you decide what is the right formula for your needs.

Automated Testing
We are going to start off this conversation about automated testing with this caveat. Automated testing in the mobile environment should not be an end unto itself. In other words, the majority of mobile app testing should occur in the manual (scripted) and exploratory space, while automated testing serves as a support system for those two approaches. Now that we have established that, we can look at automated testing of mobile apps in the proper light. Just the mere mention of automation in the mobile space can make many queasy because automation implies process. Process implies repetition. Repetition implies that the mobile experience is something that is rote and repeatable and doesnt necessarily require human intelligence or creativity. Well, its not really true, especially given the strategy required in understanding how and when to use automated testing.

by SMARTBEAR

TestComplete

10

So, what can automated tests actually cover? The best-case scenario is to test aspects that require no user interaction. That might include:

Did this API function return the correct value? Did this function calculate the correct result? Did we successfully send a file over the network with the correct
checksum?

Did this callback function actually fire?

(Source: Corona Labs) The automated piece of mobile app testing is more about the backend functionality of the app. In other words, if something were to be triggered by a user (regardless of how that trigger was activated) how does the app respond? When looking into automated testing for a mobile app, it is critical to remember that anything regarding UI and UX requires some form of human interaction in order to truly see what happens. Relegating this interaction of end user and app to an automated process exposes the test process to the risk of missing all the nuance of having a real person generate an action in the app. Automation will likely only investigate a few options as to how something may occur in the app. Its these limitations in the automated testing process that keep this form of testing in a supporting role of the other testing methods. It is important to realize the limitations of automated testing and making sure it stays within those limits. Now that we have established where automated testing plays best in the mobile space, lets turn our attention to the methods that are going to produce the most important feedback.

by SMARTBEAR

TestComplete

11

Scripted Testing
In manual testing, scripted tests are the next step up from automation. There is much more flexibility in the testing, but the fact that it is scripted implies limitations. Thats fine. Developers, business stakeholders, and other critical input sources have areas of concern that need to be vettedand scripted tests allow for those areas of concern to be addressed. Using Personas Developing user personas (different profiles of those who the app is really being built for) is a key component in increasing the effectiveness of a mobile app. Hopefully, the various personas that make up the target market of any mobile app were considered in the design and development of the app from the onset. There can be scripted testing that is built around personas. This will help to uncover potential issues that a specific subset of customers might encounter based on:

Who they are How they function What they like Why they would use the app, etc.
The number of personas tested will depend on how many there are for a particular app and what kind of resources are available (read: time and money) to test as many variations of a theme as possible. The use of scripted tests from that personas point of view can do well to uncover potential issues.

by SMARTBEAR

TestComplete

12

Exploratory Testing
Its at this point, however, that we quickly reach the limitations of scripted testing. No matter how much an end user fits the persona, a scripted test will never allow for the inevitable variations on a theme. In other words, human beings rarely, if ever, truly follow a script. With that said, it becomes imperative for those conducting the tests of an app to start to go off script and into an area that might more accurately represent reality. The means to get there is through exploratory testing. Exploratory testing is simply a more free flowing methodology of testing that, through experimentation, builds on itself and allows testers to get into real world situations that may never be accounted for in automated or scripted tests. This mindset becomes even more valuable in the testing of mobile apps because the situations an end user finds him or herself in change constantly due to the fact that they are mobile. They are on the move. Performing a task on a mobile device while sitting at a desk may look very different than performing the same task while walking down a crowded sidewalk. Or at a traffic light. Exploratory testing fits perfectly with mobile app development because the very nature of many mobile apps is exploratory in nature. People are doing things on the move and are often in discovery mode themselves. Each time the app is openeven though the function that is being called upon in the app is the samethe reason for its use and its ultimate utility could be different almost every time. In other words, in the mobile world things that might look like they are repeated events are only repeated at a surface level. The intent and
by SMARTBEAR

Exploratory testing fits perfectly with mobile app development because the very nature of many mobile apps is exploratory in nature. People are doing things on the move and are often in discovery mode themselves.

TestComplete

13

ultimate goal of the user might be completely different each time that particular function is called upon, which may create a different interaction with the same function based on the external variables the end user is experiencing. To test how an app responds to such a dynamic environment requires freedom. Exploratory testing that builds on previous experience or on a new experience can allow the test to flex and move in directions that best mimic that of a real life end user. As a result, the real discovery regarding the strength and weakness of an app can occur. It is critical to make sure that in this exploratory process there is accurate data collection. This is a difficulty of the exploratory model that is compounded when used in the mobile space. How you require testers to report their findings could mean the difference between a jumble of interesting tidbits and a result-set that impacts the performance of an app.

The Mobile Testing Balancing Act

Now that we have looked at the three areas of testing, their limitations and their place individually, it is critical to make sure that we use the right blend of all the available techniques to produce the strongest possible testing environment and the ensuing results. To accomplish this takes much more than just crossed fingers and a few Hail Marys. There needs to be real intentionality in the blending of different testing options so that the entire scope of the mobile app can be tested thoroughly enough to give greater go-to-market confidence to the business stakeholders. How this is accomplished is strategic planning on the upfront along with the willingness to be truly agile during the process. This agility and sensitivity to the very nature of mobile computing will make the difference in how your tests perform.

by SMARTBEAR

TestComplete

14

Obviously, this kind of balancing act is best performed by those with experience in these matters. If they are not available, or affordable, this might be the time to consider outside help in the form of products and services designed to get you most of the way there.

Conclusion
In the end, testing mobile apps is something that needs special attention and a renewed approach from traditional software testing. Over time, it is likely that new methodologies will be developed that are more appropriate for the mobile space. Right now though, at this early stage in the game, we need to work with the existing testing types like automated and scripted testing but allow them to be flavored by a sense of adventure, so to speak. The mobile space is new and exciting. It is also relatively unmapped as compared to other places in the digital geography. With that in mind, effective mobile app testing will require a spirit of true exploration. Testing mobile applications so that you may be able to create the best possible experience for your end users is a true work in progress. As you move through it, mixing and matching testing procedures and techniques will be the key to giving your efforts the best chance to truly make an impact in the mobile world. Are you ready? Are you flexible? Are you adventurous?

by SMARTBEAR

TestComplete

15

About SmartBear Software

More than one million developers, testers and operations professionals use SmartBear tools to ensure the quality and performance of their APIs, desktop, mobile, Web and cloud-based applications. SmartBear products are easy to use and deploy, are affordable and available for trial at the website. Learn more about the companys award-winning tools or join the active user community at http://www.smartbear.com, on Facebook or follow us on Twitter @smartbear and Google+.

SmartBear Software, Inc. 100 Cummings Center, Suite 234N Beverly, MA 01915 +1 978.236.7900 www.smartbear.com 2013 by SmartBear Software, Inc. Specifications subject to change. EBOK_TC_20130913_V1_Exploritory-Automated-Testing