PRESENTATION OVERVIEW

Why to test at all? Richness of Android devices a real opportunity Android success story in numbers Urban myths about mobile testing Options for professional grade testing Typical application implementations pitfalls Some automated testing frameworks for Android

1
info@bitbar.com

WHY TO TEST AT ALL?

Beta test approach
Pre launch activities: Light manual testing for catching most obvious bugs. Usually done on one or max 2-3 devices. Beta phase: App is distributed to 10-100 beta users who will report bugs and any other issues Launch: Based on the feedback from beta testing a new version is created and launched to the app stores.

Systematic testing
Testing criteria: Some level of basic acceptance criteria needs to be created: What devices are supported, what use cases are important, what is the expected performance Setting up test automation: Test cases are converted to test scripts, that are executed every time when the app changes Testing throughout the development cycle: Testing starts when first features are implemented and continues even when the app is released

WHY TO TEST AT ALL?

Some quotes:
Summary: Developers need to avoid treating end-users as beta testers and conduct prelaunch app testing to avoid critical reviews due to poor usability once released to market If there is a major bug that is not caught before the launch, then a lot of expensive development and marketing work is quickly wasted "The first wave of users gives the app bad ratings, which then undermine [the number of] downloads and the app would [eventually] sink into oblivion."

Source: http://www.zdnet.com/mobile-app-testing-imperative-but-not-foolproof7000000787/

WHY TO TEST AT ALL?

2012 Cap Gemini Quality Report (Mobile Testing)
Mobile design and development fits nicely with iterative development. Looping around the cycle of requirements, design, build and test in smaller chucks means that testing need not simply be a cost tacked-on to the end of a project. There are some types of such as penetration and security testing that should only be done towards the end of the project, but for functionality, performance and ease-of-use testing we recommend that this be done iteratively during the mobile project.

Source: http://www.capgemini.com/technology-blog/2012/09/quality-report-2012mobile-testing/

WHY TO TEST AT ALL?

2012 Cap Gemini Quality Report (Mobile Testing)
Notable from the survey is that 18% of responding companies say that they do not have enough time to test mobile apps, and 65% do not have the right tools. Also 52% cite lack of devices as a reason not to do mobile testing

Source: http://www.capgemini.com/technology-blog/2012/09/quality-report-2012mobile-testing/

THE RICHNESS OF ANDROID ECOSYSTEM

Fragmentation is innovation

Samsung Galaxy S2
< 10% market share

Graphical representation of fragmentation in the Android ecosystem. Each rectangle represents the market share of one Android device. Samsung Galaxy S2, the most successful Android to date, has less than 10% market share. Source: Open Signal Maps, May 2012

ANDROID SUCCESS STORY IN NUMBERS

Over 500m Android devices shipped since Android 1.0 About 220m iOS devices have been shipped since 2007

OS

Q2/2012 shipments

Q2/2012 market share

Q2/2011 shipments

Q2/2011 market share

YoY change

Android
iOS

104.8
26

68.1%
16.9%

50.8
20.4

46.9%
18.8%

106.5%
27.5%

Blackberry
Symbian

7.4
6.8

4.8%
4.4%

12.5
18.3

11.5%
16.9%

-40.9%
-62.9%

Windows

5.4

3.5%

2.5

2.3%

115.3%

7
info@bitbar.com Source: IDC, 8.8.2012

URBAN MYTHS ABOUT ANDROID TESTING

Myth #1: Testing on emulators is enough
This app works perfectly on Android 2.3 and 4.0 emulators but on some of the most popular devices looks like this:

8
info@bitbar.com

URBAN MYTHS ABOUT ANDROID TESTING

Myth #2: Testing on 5-8 most common devices is enough
Yes, if reaching every every fourth Android user is ok for your CEO:

9
info@bitbar.com

URBAN MYTHS ABOUT ANDROID TESTING

Myth #3: Exploratory testing just before launch is enough
Exploratory testing is not very good for catching any regressions Exploratory testing is not very repeatable

Exploratory testing (any manual testing) does not scale very well
However, exploratory testing has its own merits it is not enough just by itself

10
info@bitbar.com

URBAN MYTHS ABOUT ANDROID TESTING

Myth #4: Users will understand if there are some issues
Application not working and having bugs is #1 reason for uninstalling the app Users dont update their apps very frequently

Quality issues are #1 reason for bad reviews in Google Play (about 80%), Application content related issues are only (20%)

11
info@bitbar.com

TYPICAL APPLICATION IMPLEMENTATION PITFALLS

Mix different technologies and frameworks Extensive use of web views
Webkits differ from one device to another All sorts of timeout due to most content coming from server Issues in http headers across devices CSS&Javascript cause their own problems

Use cross platform tools

Especially the ones that generate the code on server side are very difficult to debug Wrapping HTML5 to an app is quite ok The time you save on development, you loose on debugging

12
info@bitbar.com

HOW TO SET UP A PROFESSIONAL GRADE TESTING

Set up Continuous Integration and version control Automate most common use cases
Address the basics first Establish a repeatable baseline Keep adding new tests as your functionality grows Do not try to automate everything there is still time and place for manual checks as well Maintain your test library as your code evolves (store both in version control)

Define your target devices:

Depending your ambition level anything from 20 to 50 will do. Test automatically and often with these devices To get a reference point run occasionally tests with all possible devices (especially before any release)

13
info@bitbar.com

TESTDROID PRODUCTS
Testdroid Cloud
Device cloud for developers to run their tests on 100+ real Android devices, in parallel Provides detailed test results and analytics

Testdroid Cloud

Testdroid Private Cloud

Private on-premise device cloud for Enterprises to run their tests on Android devices in their own development lab Provides detailed test results and analytics

Testdroid Recorder
Developer tool to simplify and accelerate the creation of test scripts

14
info@bitbar.com

INTRODUCING TESTDROID APP CRAWLER

Intelligent systematic app exerciser
Provides detailed test results and analytics

See video: http://youtu.be/eYvhyjf7HoY

Device cloud for developers to run their tests on 100+ real Android devices, in parallel

Visual easy to read results

Visual browser for inspecting the crawler's path through the app Screenshots, logs and performance info

Android testing will never be the same!

15
info@bitbar.com

TEST AUTOMATION FRAMEWORKS FOR ANDROID OVERVIEW

This is an overview of OPEN SOURCE Android testing frameworks there are tens of closed proprietary testing tools/frameworks that are not covered here

Selected frameworks:
- Monkey exerciser - Monkey Runner

- Robotium
- Roboelectric - Calabash

- Monkey Talk

16
info@bitbar.com

AUTOMATION FRAMEWORKS FOR ANDROID OVERVIEW

Two main approaches:

Android Instrumentation Framework

Instrumentation wrapped to the app itself

17
info@bitbar.com

AUTOMATION FRAMEWORKS FOR ANDROID MONKEY EXERCISER

What:
- A UI/Application exerciser monkey for creating stream of random user events such as clicks, touches, gestures and some system level events

Usage: $ adb shell monkey -p your.package.name -v 500 Pros: - Useful way to stress test your application to find non-obvious usage patterns and corner cases Cons:
- No way to create any other inputs than clicks which make for instance logging in to an app impossible
info@bitbar.com

18

AUTOMATION FRAMEWORKS FOR ANDROID MONKEY RUNNER

What:
- An API toolkit for writing programs that control an Android device or emulator from outside of Android code

Usage: device.press('KEYCODE_MENU','DOWN_AND_UP') Pros: - Only way to create tests that span across application boundaries Cons:
- UI elements can only be clicked with x,y clicks that do not scale => you have to create one script for each resolution. There is no validation mechanism for any events created by clicks
info@bitbar.com

19

AUTOMATION FRAMEWORKS FOR ANDROID ROBOTIUM

What:
- It's like Selenium for Android. A JUnit based helper library/framework

Usage:
solo.clickOnText("Configuration");

Pros:
- Powerful UI, functional and system testing capabilities (because it's based on JUnit) Cons:
- Cannot cross the application boundaries in test scripts (because it is using Android Instrumentation Framework)

20
info@bitbar.com

AUTOMATION FRAMEWORKS FOR ANDROID CALABASH

What:
- Cucumber based test automation framework for Android and iOS

Usage:
And I select Options" from the menu

Pros:
- Creating scripts does not require any programming skills, Same scripts work across platforms Cons:
- Wraps the application under test with an http server so you will be testing a different binary that the one you are going to ship.

21
info@bitbar.com

AUTOMATION FRAMEWORKS FOR ANDROID MONKEY TALK

What:
- Cross platform testing tool/framework for Android and iOS

Usage: Input address Verify 123 Main St. %timeout=500 Pros:

- Same script works for your Android and iOS app (if the apps are identical) Cons:
- Uses proprietary scripting language. Requires you to wrap your app with a testing library so you will be testing a different binary that the one you are going to ship.

22
info@bitbar.com

AUTOMATION FRAMEWORKS FOR ANDROID ROBOELECTRIC

What:
- Running unit tests on virtual machine instead of an emulator or real devices

Usage:
activity.findViewById(R.id.press_me_button);

Pros:
- Ultra fast! Can be run in every code commit Cons:
- It is just for unit testing it doesn't give you any info on how your app runs on target devices

23
info@bitbar.com

FURTHER INFORMATION
Additional information on web:
Company and Products: http://www.bitbar.com Testdroid Cloud: http://cloud.testdroid.com Testdroid App Crawler: http://youtu.be/eYvhyjf7HoY

Demo video (3min): http://youtu.be/zj11lzUmScg?hd=1

24
info@bitbar.com