Ricky Jones

Movement classification using Android sensor data

Computer Science Tripos Jesus College 2012

Proforma
Name College Project Title Examination Approximate Word Count Project Originator Project Supervisor Ricky Jones Jesus College Movement classification using Android sensor data Computer Science Tripos Part II July 2012 9,200 Kiran Rachuri Kiran Rachuri

Original Aims The aim of the project was to investigate, implement and evaluate various algorithms for movement classification using Android accelerometer data. Movement will be classed as stationary, walking, cycling etc. This would be done by developing an application that would allow multiple algorithms to be used and tested. The algorithms will be evaluated by accuracy and energy usage. The project aimed to investigate efficient methods such as adaptive sampling, where a low power classifier triggers a more energy intensive classifier. Work Completed I have achieved the original aims and have met the success criteria. Four main classification algorithms are investigated, and their accuracy and energy usage evaluated. Special Difficulties None

Declaration of Originality
I, Ricky Jones of Jesus College, being a candidate for Part II of the Computer Science Tripos, hereby declare that this dissertation and the work described in it are my own work, unaided except as may be specified below, and that the dissertation does not contain material that has already been used to any substantial extent for a comparable purpose.

Signed:

Date:

Table of contents
1 Introduction ......................................................................................................................................... 1 1.1 Related Work ................................................................................................................................ 1 1.1 Motivation..................................................................................................................................... 1 1.1.1 Exercise monitoring ............................................................................................................... 1 1.2.2 General context awareness ................................................................................................... 2 1.2 Results ........................................................................................................................................... 2 2 Preparation .......................................................................................................................................... 4 2.1 Overview ....................................................................................................................................... 4 2.2 Background research .................................................................................................................... 4 2.3 Project management..................................................................................................................... 5 2.3.1 Requirements analysis ........................................................................................................... 5 2.3.2 Planning and management .................................................................................................... 5 2.4 Setting up the development environment ................................................................................... 5 2.4.1 Android development ............................................................................................................ 5 2.4.2 Version control and backup ................................................................................................... 6 2.5 Android development ................................................................................................................... 6 2.5.1 Activity lifecycle ..................................................................................................................... 6 2.5.2 Android manifest ................................................................................................................... 6 2.5.3 Data storage ........................................................................................................................... 7 2.6 MATLAB ......................................................................................................................................... 7 3 Implementation ................................................................................................................................... 8 3.1 Overview ....................................................................................................................................... 8 3.2 Ground Truth Data Application..................................................................................................... 8 3.2.1 Requirements ......................................................................................................................... 8 3.2.2 Implementation ..................................................................................................................... 9 3.3 Classification ............................................................................................................................... 10 3.3.1 Classification methodology .................................................................................................. 10 3.3.2 Feature extraction................................................................................................................ 11 3.3 MATLAB scripts ........................................................................................................................... 13 3.3.1 Loading the data .................................................................................................................. 13 3.3.2 Cleaning the data ................................................................................................................. 14 3.3.3 Feature extraction................................................................................................................ 15

3.3.4 Appling the functions to the collected data......................................................................... 15 3.4 Classification Application ............................................................................................................ 16 3.4.1 Requirements analysis ......................................................................................................... 16 3.4.2 Overview .............................................................................................................................. 16 3.4.2 Application structure ........................................................................................................... 16 3.4.3 Simple classifier .................................................................................................................... 17 3.4.5 Nave Bayes classifier ........................................................................................................... 18 3.4.6 k-NN classifier ...................................................................................................................... 19 3.4.7 Voting classifier .................................................................................................................... 20 3.4.8 Adaptive classifier ................................................................................................................ 20 3.4.9 Additional features .............................................................................................................. 20 4 Evaluation .......................................................................................................................................... 22 4.1 Overview ..................................................................................................................................... 22 4.2 Offline evaluation of classification algorithms............................................................................ 22 4.3 On device testing ........................................................................................................................ 23 4.4 User testing ................................................................................................................................. 24 4.5 Energy usage ............................................................................................................................... 26 5 Conclusions ........................................................................................................................................ 29 5.1 Project success ............................................................................................................................ 29 5.2 Further work ............................................................................................................................... 29 Bibliography .......................................................................................................................................... 31 Appendices ............................................................................................................................................ 32

1 Introduction
This project aims to develop a mobile phone application that can determine what type of movement the user (who is carrying the device) is doing. Most smartphones today contain a variety of sensors that collect useful information, such as location (GPS), the orientation of the phone and the amount of light falling on the phone. This dissertation uses one of these sensors, the accelerometer. All Android phones have a triaxial accelerometer embedded within them. The sensor returns a real valued estimate of the acceleration along the x, y and z-axis. The goal is to convert this data into semantic information about the uses activity. The activities focused on are Walking, Cycling and Running, although any could be added with extra time. This process of converting input data into a semantic result is called classification. An overview of classification is presented in 2.2. Multiple classification algorithms are implemented, described in 3.4 and evaluated in 4.

1.1 Related Work

There are many examples of accelerometer data being used for useful applications. Last year a student did a Part II project on inferring a users transportation mode (Train, Bus, Car) using this data (Piggott, 2011). Apples Fall Detection sensor, which uses an embedded accelerometer in a laptop, can detect falls or shocks. Making an application able to detect the users movement fits in to the general problem of context awareness. Context awareness determining the users current activity and location is a key functionality in many pervasive computing applications. The motivation for context awareness is highlighted in many papers; Dey A K (2001) presents a brief summary at the start of his book. Movement inference is a major component of context awareness. While there has been a lot of research in determining a users location (Balakrishnan et al, 2000), interest in motion classifcation has been growing. Bao & Intille (2004) used five wearable acceleromters on different locations on the body, getting subjects to perform a variety of everyday tasks. More specifically work has been done testing various classification algorithms (Ravi, 2005), using similar classification algorithms to this dissertation, but with the accelerometer in a fixed postion on the user. Rather than using multiple sensors, this project uses an accelerometer built into Android phones. No special equipment is required. However this means the the acclerometer is not in a fixed position, it is free to be anywhere on the user, just as they would carry their phone. For this reason, analysis techniques not effected by the orientation of the accelerometer are used.

1.2 Motivation
There are many ubiquitous computing applications that could benefit from knowledge of the users movement. I have listed a couple of applications below. 1.2.1 Exercise monitoring

Perhaps the most obvious application would be to monitor and record all the movement of a user at all times. This data could be used to provide the user with feedback on the type and duration of their activities over the day or week, encouraging the user to do more exercise. This data could be coupled with information such as the calorific impact of various exercises to present the user with an estimate of calorie loss over the day. A social aspect could be introduced, where users can share or compare exercise activity, further incentivising the user. 1.2.2 General context awareness As described in the previous section context awareness has numerous benefits for application developers. For instance if a user is cycling, and receives a call, their phone could automatically pass on a preprogramed message to the caller. The music player could be adapted to change choose tracks depending on what the user is doing, changing to some upbeat lively music if the user is going for a jog.

1.4 Summary of results

The finished application is evaluated extensively in 4, this subsection presents a summary of the findings. The aim of this project is to compare different algorithms, in terms of accuracy and energy usage. Accuracy testing is performed in three separate ways. Firstly the accuracy of the classifiers are tested offline (on the desktop), using labelled data reserved for testing (4.2).
100 95 90 85 80 75 70 Nave Bayes k-NN k=3 k-NN k=5 Voting Stationary Walking Cycling Running Overall

Table 1 - % Accuracy of classification algorithms in offline testing

Next the algorithms are tested online (on the device) by myself (4.3). As I was involved in collecting training data the accuracy was similar to the offline testing above. Finally the accuracy was tested in the most interesting case, with people who had not been involved in the training (4.4).

100 95 90 85 80 75 70 Nave Bayes k-NN k=3 k-NN k=5 Voting Stationary Walking Cycling Running Overall

Table 2 - % Accuracy of classification algorithms in online testing with new users

Overall the accuracy was very good. As is expected the algorithms performed worse when tested on new users. This could be improved by having more participants involved in the training, to create a larger training set. This is explained in more detail later. The energy usage of the various classifiers was also measured (4.5), it was found that the more accurate classifiers were, the more energy they required. Choosing between higher accuracy or energy efficiency would depend on the application domain. A number of techniques are presented and tested to reduce energy usage. Within this dissertation some explanation may be duplicated. I have tried to keep this to a minimum and in the cases where it does occur; it is intended to help the reader maintain understanding of the algorithms and methods used.

2 Preparation
2.1 Overview
This chapter describes the work that was done before any implementation of the project began. Initially time was spent learning about classification, which I describe briefly, along with background reading that was done. I describe the foundations for a professional implementation, such as planning, setting up a robust development environment, and progress monitoring. Finally I touch on some key technologies that I learnt about, namely Android and MATLAB.

2.2 Background research

This chapter describes the work that was done before any implementation of the project began. Initially time was spent learning about classification, which I describe briefly, along with background reading that was done. I touch on some technologies that I learnt about, such as Android development, and MATLAB. The chapter then describes how I set about monitoring my progress, and preparing a professional development environment. After discussing with my supervisor he recommended I read the Wikipedia pages about classification, in particular Nave Bayes as this was the first classification algorithm I was going to implement. I also read a couple of papers in the field (DeVaul & Dunn 2001), and (Ravi et al. 2005), the latter of which being particularly on topic. In the next subsection Im going to briefly describe how classification works as this knowledge will help general understanding of this dissertation. 2.2.3 Classification Like many problems in AI, the problem of classification can be seen as the challenge of creating a signal-to-symbol converter. We have some data as input, and we want to infer some semantic information about the source of the data, or what it represents. An example is optical character recognition (OCR), where the input data is an image of a character, possibly handwritten. The classifier attempts to discern what underlying letter actually is. To convert the messy input signal, into a symbol readily understood. In this project, the input data is a series of accelerometer readings. A single reading is a float array of length 4, the beginning contains a timestamp, followed by the acceleration in the x, y and z axis. These readings are clumped together in a 2 dimensional float array, representing a sample of acceleration over time. This data will be recorded from the mobile phone, and the classification task here is to discern what movement the user was doing during that time. One expects that if they were walking, the acceleration should be somewhat different to them standing around. However initially, a classification algorithm is just a piece of code, it has no knowledge of how Walking, or Running, or Cycling or any movement is likely to be represented by acceleration. This is where the first main task of this project comes in, training. In order to train a classifier you need to have correctly labelled input data. It is vital that the training data is labelled accurately as if it is not 4

the classifiers will learn incorrectly, reducing the end accuracy. How exactly the classifiers learn is implementation dependent and will be discussed more in the following chapter.

2.3 Project management

This section covers the requirements analysis, the planning and the overall administration that was done in order to ensure the project would be completely correctly and on time. 2.3.1 Requirements analysis In the project proposal a number of aims are presented, using those follow success criteria were identified: Create and test an application for gathering labelled training data. Use the application to gather a large amount of accurate training data. Implement a classification application for simple motion detection Implement a more advanced classification application for more classes, such as Walking, Cycling and Running

As an extension I aimed to look at the energy use of the application and try to minimise it using adaptive sampling techniques. 2.3.2 Planning and management For the project proposal a detailed schedule for completed work was produced, this outlined what should be completed on a fortnightly basis. It also included succinct milestones, and when they needed to be achieved. The benefits of such planning are enormous, something I have learnt from past experience. Regular meetings with other people active in the project were also extremely important. My supervisor and I organised twice weekly meetings in the computer laboratory. The purpose was to ensure I was making progress and not having any problems. Throughout the project emphasis was placed on planning for evaluation. The requirements for specific pieces of code include the need to build in methods for testing.

2.4 Setting up the development environment

2.4.1 Android development In order to develop for Android a number of technologies have to be installed. Android applications are written in java so one has to install the JDK. As I use Java regularly I had this and Eclipse running on my computer. Android offers a custom plugin for development in Eclipse, Android Development Tools. The tool is designed to give you an environment where it is easy to deploy and debug Android applications much like you would standard java applications. In general, developing with the ADT plugin in Eclipse is a highly recommended approach. It extends the abilities of Eclipse to let the developer quickly set

up new Android projects, develop an UI, debug applications and export signed APKs (Application package files) to distribute your application. Another handy tool in the SDK is the Dalvik Debug Monitor Server (DDMS). This provides lots of realtime information about the device, such as thread and heap information, along with the logs. It also provides port-forwarding services, screen capture, incoming call and SMS spoofing. Lastly as I was using a real device rather than the emulator, the USB drivers needed to be installed. 2.4.2 Version control and backup I used Subversion (SVN) for version control, this allowed me to track changes and undo changes if they caused a problem on the device. I used Unfuddle as a host, which provides free SVN and GIT hosting. This set up also serves the purpose of a conventional backup, if something went wrong with my development machine.

2.5 Android development

In the next two sections I describe the two key technologies I learnt about whilst preparing for the project. The first piece of preparation required was getting up to speed with how applications are developed on Android. Although I used Android during an internship in 2010, I had not used it since so I spent the first couple of weeks reading the developer website developer.android.com. In the following sections I touch briefly on the main Android specific things I learnt about. 2.5.1 Activity lifecycle All applications that a user can interact with have an Activity component. Every Activity is given a window in which to draw content that is displayed on the screen. Activities need to be managed with respect to what happens at different stages of their lifecycle. This is done by implementing callback methods. An activity can be in three different sates: Stopped: this is when the activity is completed obscured by another activity, when it is in the background; Paused: this is when another activity is in the foreground and has focus but is not completely covering the main activity; Resumed: the activity is running in the foreground and has user focus. What happens when an activity transitions between these states needs to be controlled by the developer, otherwise data can be lost, and the UI can crash. For example when an Activity is paused, its entire stack is popped, destroying all of the variables. When the application is resumed, if these have not been stored correctly then it is very likely the program will crash due to a null pointer. The solution to this is to use the saved instance state paradigm. An android construct that allows the developer to easily save variable data when a pause event is triggered, and then recover it on then the activity resumes. 2.5.2 Android manifest

Every application has a manifest file: AndroidManifest.xml. This contains meta-data about the application, mainly a list of components that make up the application. Quite often if some piece of code that is obviously correct isnt functioning correctly; it is probably due to an incorrect declaration in the manifest file. An activity is declared with information such as its name, and an icon. Within an application there are components such as Activities and Services. The most important component of the application declaration are permissions:
<uses-permission android:name="android.permission.INTERNET"/> <uses-permission android:name="android.permission.WAKE_LOCK"/>

Nearly any interaction with the rest of the phone by an application requires a uses-permission. The first example above is required for an application to use the internet, for http, ftp, or socket programming. The second is used to prevent an application from losing CPU resources when the phone is locked, or the application is in the background. 2.5.3 Data storage There are a variety of ways to store persistent information in an application, which is best depends on exactly what the developer is aiming to achieve. Apart from using the internet to store data on a server, the key methods of data storage are SharedPreferences, and using the internal file system. The SharedPreferences class provides a framework that allows the programmer to store key-value pairs in any primitive data type. This data is persistent across user sessions, even when the application is killed. For larger amounts of data, such as accelerometer traces, using the internal storage is required. Each subclass of Activity has access to some simple methods for using the file system. The openFileOutput() is used to open a file for writing, and openFileInput() for reading. The other key methods are deleteFile(), and fileList(), the latter of which returns an array of files that the application has created in the past.

2.6 MATLAB
MATLAB is a fantastic tool for prototyping ideas and algorithms quickly. I spent an afternoon with one of the researches from the Systems group who uses MATLAB in his work. He gave me a quick overview of how the software works, and I was able to get up to speed quite fast. In the implementation chapter there is more on MATLAB and how it was used in this particular project.

3 Implementation
3.1 Overview
The project is made up of three separate pieces of work, but which are all needed to create a successful application. The first is an Android application designed to record accelerometer readings. It records the data, along with a label submitted by the user. This is the basis for the training data described earlier in 2.3. The second are a series of MATLAB functions and scripts that are used to process the data collected by the first application into a form suitable for the classification algorithms. The final, and most significant piece implemented is the classification application. This is the application that will run on the phone and infer in real time what activity a user is performing. Although this is the most interesting part of the project, without the first two pieces, it wouldnt be able to function.

Figure 1 - Classification application, user is stationary then cycles for a period in the middle

To make the reading more interesting, it may be preferable to skip the section describing classification.

3.2 Ground Truth Data Application

3.2.1 Requirements The motivation for this application was to collect a large amount of sampled training data. It was important to make the application simple and reliable. One of the worst things would be if someone had volunteered to collect some data and the data had been lost. The key requirements of the application are listed below:

Allow the user to label a sample of accelerometer data with the movement they are performing Automatic uploading of data over Wi-Fi (if there is no Wi-Fi the application will store the sample for later upload). Ability to cancel a sample if it has been ruined (for example if the user went for a cycle then have to stop because of a phone call).

Initially the application had other features such as the ability to tag the data with extra information, but after some use I decided it was best to make the application as simple as possible.

Figure 2 - GTD Application

3.2.2 Implementation Android allows an application to record accelerometer readings at various sample frequencies, FASTEST, GAME, UI and NORMAL. At the beginning of the project neither myself or my supervisor were sure of what was best, so it was decided to allow the GTD Application to record all the frequencies concurrently. The main class is the GTDActivity class which contains all the UI Event handlers. If the user starts collecting this class creates instances of the AccelerometerListener class to capture the accelerometer readings at various sample rates. The class also starts a separate data upload thread that continually monitors the applications internal file space for new sample files. When new files are discovered they are then compressed before being uploaded over Wi-Fi. I decided to use FTP to upload the data to a webserver. The key benefit was having a guaranteed server 24/7, also the Apache FTP libraries are very easy to use.

Figure 3 - General structure of the GTD Application

3.2.3 Initial analysis of the collected data The first version of the application was distributed, so that my supervisor and some of his colleagues could collect information. One of the concerns my supervisor had was that having four concurrent threads reading values from the accelerometer might reduce the frequency each was receiving. To test for this I ran the GTD application with all four sample frequencies being recorded, and then just with the FASTEST. On my device there was little difference, when collecting FASTEST in parallel with the others the frequency was 94 Hz, and on its own it was 96 Hz. However for my supervisor and his colleagues the difference was much greater, with the frequency dropping to 16 Hz when collected concurrently. This turns out to be a difference in how Android 2.1 and 2.2 operate. This was the first major problem in the project, should I just collected FASTEST and forget about the other frequencies. The motivation behind wanting to get all sample rates was that it would be potentially useful information to help save energy later in the classification application. For instance if one can classify simply MOVING/STATIONARY using a low frequency classifier, then switch to a higher frequency for more accurate classification, energy would be saved as opposed to always using the latter. After some experimentation and analysis of the data it was apparent that one could generate all the sampling frequencies from the FASTEST rate, by offline resampling. For this reason I decided to further simplify the GTD application such that it would only collect at the fastest rate, removing the problem with Android 2.2, and reducing the amount of data to needed to be sent over the network.

3.3 Classification
3.3.1 Classification methodology Before diving into the exact implementation of the classification algorithms, this section will describe classification in general and how it is applied in this project. In order to improve the understanding of the reader some points will be made again even though they were included in the Preparation section. Classification is the problem of identifying which of a set of classes a new observation belongs, based on information from a training set. The previous section, regarding the GTD Application explains the source of this training set. In the following sections some terms will be used frequently and it a good idea to define them now.

10

Class A class is a category that an observation can belong to. In this project each class is a type of movement, such as walking, cycling or running. Accelerometer reading An accelerometer reading is a single measurement of the accelerometer at a point in time, stored as a float array [timestamp, x-acceleration, y-acceleration, z-acceleration]

Figure 4 - Acceleration axis

Observation / Sample An observation or sample is what is being classified. In this project is it a set of 256 consecutive accelerometer readings, representing the acceleration of the phone over some period of time. Before the sample is classified, it is processed into what is called a feature vector. This is designed to represent the sample more informatively. The feature vector contains feature values. In general features can be anything about the data, but in this project I used the following: Mean, Standard Deviation, Energy1, Energy2, Energy3 and Energy4 to create feature values of length 6. The mean and standard deviation are obvious choices, the idea of energy values was used by (Ravi et al. 2005) to good effect. 3.3.2 Feature extraction This section describes in detail how the features are extracted from an observation. These equations are used in both the following chapters, for the MATLAB scripts and for the classification application.

11

Figure 5 - Feature extraction

Step 1: Removing gravity Recall an observation is an array of accelerometer readings. As the phone is always experiencing the force of gravity, that needs to accounted for. For this reason when the device is resting on a table, the accelerometer reads a magnitude of g = 9.81 m/s. In order to measure the real acceleration of the device, the contribution by the gravity must be eliminated. The recommended method on the Android developer site is to apply a high-pass filter. Let represent gravity in the x-axis at time t, and be the reading from the accelerometer in the x-axis. The actual acceleration (without gravity) in the x-axis is then given by:

where

is approximated by:
1

Step 2: Calculating the magnitude Once the gravity has been removed, the magnitude of the acceleration is calculated for each point in time. This generates a 256 length magnitude vector, representing the sample. Calculating the features on the magnitude vector rather than on individual axis means the features are independent of the devices orientation. That is vital for this project as people can have their phones on them in any position. Step 3: Calculating the features The mean and standard deviaton are calculated as you would expect. The energy values are calculated by taking the fourier transform of the sample, to produce an array representing the frequency components of the data. Each energy value is the energy from a frequency band. The frequency spectrum is devided into four, the lower quartile for energy1, up to the upper quartile for energy4.

is the value used in the Android documentation (Google, 2012)

12

Let

be the fourier tranform of the magnitude vector. Let .

be energy feature of band

The energy of a given frequency is simply the square of that frequency component; the sum and the denominator give the average energy over a band of frequencies. The mean, standard deviation and the energy values are put together to form a feature vector for the sample.
4.8915860 3.4314873 55.478584 0.86720960 0.68431590 0.07791250 4.7732573 3.3957403 49.822770 0.86712134 0.45566043 0.14041372
Figure 6 - Example feature vectors taken from labelled walking data

3.3 describes a concrete implementation of this process in MATLAB, for processing the labelled training data. The same process is also used on the device when classifying a new observation.

3.3 MATLAB scripts

MATLAB was used to process the collected labelled data into a form suitable for the on-device classification algorithms to use as training parameters. The best way to explain what was implemented using MATLAB is to describe each function in turn. 3.3.1 Loading the data
function [a_m, a_x, a_y, a_z, freq] = loadAccel( filename )

This function loads the raw accelerometer values from a .csv file generated by the ground truth data application. Below is the layout of one of those .csv files. The columns are: timestamp, x-axis acceleration, y-axis acceleration and z-axis acceleration.
10839906912000, -2.030283, 6.129156, 10.91756 10839938332000, -0.9576807, 5.3247046, 9.423578 10840365885000, -2.528277, 7.3166804, 7.469909
Figure 7 - Raw accelerometer readings

The function loads the values from the disk into a matrix within MATLAB, a high-pass filter is applied as described in the previous section to remove the gravity component. There is a built in function for this, filter.
a_x_temp = filter([alpha -alpha], [1 -alpha], accel_x);

The next part generates a vector representing the magnitude of acceleration at reach point:
a_m_temp = sqrt(a_x_temp.^2 + accel_y_temp.^2 + accel_z_temp.^2);

13

The data in the .csv file is not split up into separate 256 element observations, so this has to be done before features can be calculated. The vector is converted into a two dimensional array of observations.
a_m[i][j] = a_m_temp[i*256 + j]

Figure 8 A flattened version of a_m from a[0] to a[10000].

The figure above shows the result of the processing so far, this is the magnitude vector of all the windows flattened out. This data was labelled as walking, however you can see that there is a section where the acceleration drops down to almost zero. The next function is designed to remove such areas. 3.3.2 Cleaning the data
function [a_m, a_x, a_y, a_z] = cleanData(a_m, a_x, a_y, a_z)

One of the problems identified in the data are anomalous sections, for example in the trace above from 0.4 0.5 the acceleration drops to zero. As this is a trace of someone walking there should always be some acceleration. This problem occurs when the user collecting stops to do something forgetting that the application is building a trace. The purpose of this function, cleanData, is to remove portions of the trace that are too far from the median. After some experimentation, an accepting range of median +/- 1*standard deviation was applied.
% Accepted if (mean_F1 - std_F1) < f1 < (mean_F1 + std_F1) for i = (w_count:-1:1) if(F(i, 1) < (median_F1 - std_F1) || F(i, 1) > (median_F1 + std_F1) ) % delete window i a_m(i, :) = []; end end

14

After this function has been applied the section from 0.4-0.5 is removed, along with some trimming from the start and end of the trace. This is expected as when the user is just starting the data collection he will be moving the device into a pocket, or removing at the end.

Figure 9 The result of the cleanData function

3.3.3 Feature extraction function [ F, N ] = calculateFeatures( a_m, a_x, a_y, a_z ) Once the data has been loaded and filtered, the next process is to calculate features. At this point it will be helpful to the reader if the read 3.3.2 to understand the motivation for feature extraction. The main body of this function is a for loop that runs through each window calculating the feature values, mean, std, energy1, energy2, energy3, and energy4. These are returned in a feature matrix F, where F[i][j] is the result of feature j calculation on window i.
for i = (1:w_count) S = real(fft(a_m(i,:))); F(i, 1) = mean(abs(a_m(i,:))); F(i, 2) = std(abs(a_m(i,:))); F(i, 3) = energy1(a_m(i,:), S); F(i, 4) = energy2(a_m(i,:), S); F(i, 5) = energy3(a_m(i,:), S); F(i, 6) = energy4(a_m(i,:), S); end

This function also contains the procedures for each feature calculation.

3.3.4 Appling the functions to the collected data

15

All the functions are used together to convert a raw accelerometer data file to a list of feature vectors, one for each observation in the file. These functions are then applied to multiple input files, generating all the feature vectors for a given class.

3.4 Classification Application

At this stage in the project training data had been collected by the Ground Truth Data application, and processed into feature vectors using MATLAB. This section of the dissertation describes the final component of the project, the Classification Application. The phone application classifies accelerometer data in real time using one of the selected algorithms. This chapter will describe the requirements of the application, the general implementation, the various classification algorithms and finally any additional features that the application had. In total there were four main classification algorithms, Simple, Nave Bayes, k-NN and Voting, these will be described individually later in this chapter. In the evaluation chapter the accuracy of the various classifiers are presented. 3.4.1 Requirements analysis The key requirements of the application are listed below: Record and classify accelerometer data in real time, presenting the results on screen Allow the user to choose between the various classification algorithms Update the parameters for the algorithms dynamically Code for easy monitoring and debugging, to aid evaluation

3.4.2 Overview The classifier application is an android application that shows the real time results of classification on accelerometer data. The application records readings from the accelerometer, bundling them into samples of 256 readings for classification. The sample is passed into one of the algorithms and then the result is displayed to the user. The application updates the classifiers dynamically by checking a webserver for the most up to date parameters. This means once the application has been distributed it can be updated without requiring the user to install a new application. Another feature implemented is automatic log uploading. All debug messages produced by the application pass through a logger class that uploads them to a database running on a webserver. This feature was implemented because it greatly improves testing; one can go out with the phone for a cycle, then come back and look at the webserver for an exact trace of what was happening. 3.4.2 Application structure

16

Figure 10 - Movement classifier class structure. *in the Android SDK

As is common with Android applications that interact with a user, there is a subclass of Activity, the MovementClassifierActivity, abbreviated MC Activity in the class diagram above. This class is responsible for setting up the layout and controlling user interaction. It is also the entry point for the application. The DownloadParamatersTask will be explained in more detail in 3.4.8, but it is used to keep the algorithm parameters up to date. The AccelerometerListener class the same as in the GTD Application. It simply records accelerometer readings into a 2 dimensional float array, when 256 readings are reached; the array is wrapped in a Sample class, then passed to the current classifier instance for classification. The Logger class extends the normal functionality of the Android Log class, it allows for automatic uploading of log messages to a server, to aid in the debugging process. The abstract class Classifier is instantiated as one of the concrete classification classes depending on the user selection. These classification classes will be explained individually in the following sections.

3.4.3 Simple classifier The key method that all the instances of the classification algorithm have to implement is the following:
public abstract int classify(Sample sample);

What is passed to the classify function is an instance of Sample, representing the 256 accelerometer readings recently collected. The sample class wraps the float array with functionality and state. The functionality is used to calculate the features as described in 3.3.2. Firstly gravity is removed, and then the features are calculated. The state is used to store these feature values. This is very important as it prevents the same features from being calculated repeatedly. In some cases a sample may be classified by multiple algorithms. This class reduces the marginal cost significantly. 17

The simple classifier classifies between STATIONARY and MOVING. It does this using only the mean, comparing it with a decision value. The result is returned as an int. The abstract classifier class contains a method to convert this into a string for the UI. 3.4.5 Nave Bayes classifier 3.4.5.1 Theory This subsection describes the mathematics behind Nave Bayes. The probability model for the | classifier is a conditional model over a random variable C, which can take class values, conditional on a number of feature variables to . Using Bayes theorem, we write: | |

We are aiming to find the maximal probability, i.e. given the feature values to what the class is most likely. The denominator of the above fraction is irrelevant to the classifier as its constant for all possible class values. The numerator can be split up using the chain rule: | | | | We assume that each feature | | | | | |

is conditionally independent of each other meaning: ( | ) |

This allows one to form the standard Nave Bayes result: | |

| we assume that for a given class C, each feature is normally distributed, with To calculate a mean and standard deviation . For each feature the mean and standard deviation is calculated from the training data. Once we have the mean and standard deviation for each feature for each class, calculating the conditional probability is simple:
( )

18

So finally to classify, one calculates classification.

for each class, taking the maximum to be the

[ ] 3.4.5.2 Implementation

The first part of the constructor loads the parameters (mean and standard deviation values for each feature and class) from a stored file. The values are parsed and converted in Features. A Feature class contains the featureID, its mean, and its standard deviation. For each movement type, a FeatureSet is created, storing the six Features that make up that type. The classifier maintains a list of FeatureSets, one for each class of movement.

Figure 11 - Naive Bayes class structure

When a sample is passed to the classifier, the probability of it being from each class is computed in the method described in the previous section. The source code for this class can be found in Appendix A. 3.4.6 k-NN classifier 3.4.6.1 Theory The k-Nearest Neighbours algorithm (k-NN) uses more memory than Nave Bayes, but unlike that algorithm, it does not require any prior probability knowledge. The algorithm is simple. The training data consists of a series of feature vectors, labelled with their source classification. All of these vectors are used as is in the k-NN algorithm, rather than just the mean and standard deviation of their values as in Nave Bayes. When a new sample is recorded, the k-NN algorithm searches for the closest matches from the training data. If you imagine the vectors in a 6-dimensional vector space, and the sample vector a point in that space, the algorithm searches for the k nearest neighbours to that point. The final result is simply a vote between the k neighbours. 3.4.6.2 Implementation In the concrete implementation I used 800 labelled feature vectors for each class. As there are 4 classes (Walking, Stationary, Driving and Cycling), a total of 3200 vectors are stored on the device. When a new observation is collected, it is processed into a feature vector. The k-k-NN algorithm classifies this vector by searching the vectors stored for the k nearest neighbours. The new observation is classified by adding up the labels, and returning the most common.

19

The implementation is quite simple, when a new feature vector is being classified, the Euclidian distances from that point to all the other points in the vector space are calculated, they are then sorted in reverse, and the top k-elements are the nearest neighbours. For instance if k = 5, and the top five vectors are labelled W, W, R, W, C, then Walking is returned. In the instance where there is a tie the result with the closest vector is returned. In order for one feature not to have too much influence, the vectors are normalised prior to searching. 3.4.7 Voting classifier A voting classifier was recommended in Ravi et al. This classifier uses a combination of Nave Bayes and k-NN with various k values and then performs a vote on the result. As described earlier, the float array of the sample is wrapped in the Sample class, which maintains values for calculated features. The k-NN classifier maintains a record of the previous sample it classified, if it is the same, the nearest neighbours are not recalculated meaning classifying the same sample for another k value is free. These design choices mean the processing required for the Voting classifier is less than the sum of the three components run separately. 3.4.8 Adaptive classifier There are actually three versions of the adaptive classifier, Adaptive Nave Bayes, Adaptive k-NN and Adaptive Voting. The adaptive classifier is designed to reduce processing power and thus save energy. It classifies a given sample using the Simple classifier, if that returns Stationary it returns Stationary, however if the sample is classified as Moving, the sample is then classified with a more complex classifier. This saves processing power in the instances when the phone is stationary. 3.4.9 Additional features The application contains a number of additional features touched on earlier. These are described below. 3.4.9.1 Dynamic parameter updating Every time the application is started it checks for an internet connection, if this is available it downloads a file from a webserver containing all the available parameter files and their associated hash values. It then compares the filename-hash pairs with the files currently stored on the device. If any differ it downloads the new file from the webserver. The dynamic updating is done by the DownloadParamatersTask class, an extension of AyncTask. AsyncTask is an Android SDK class which enables background work to be completed while facilitating proper communication with the UI Thread.

20

<!-- <Filename> <SHA-1Hash> --> naivebayes.csv 0c13791ee21654f67c8357c24b159592d59e91f7 KNNParameters_walking.csv 576b754df3a82f3364313e4a2fb6916f5ee81da4 ...

Figure 12 - Server file containing filename - hash pairs, used for dynamic updating

The filename-hash pairs are stored using shared preferences, described in 2.5.3. The SharedPreferences class provides a general framework that allows you to save and retrieve persistent key-value pairs of primitive data types. The parameter hash file is downloaded from the server using an URL connection. Once any modified parameter files have been downloaded the key-hash value pairs are updated in the applications shared preferences. This is done using an instance of shared preferences Editor class. 3.4.9.2 Log uploading Uploading log messages is done using the message uploader thread. When a log message is written using logger.write(tag, msg) the message is added to a queue ready for upload. Its also passed on to the standard Log.d(tag, msg) android logging function. The thread starts by creating a FTPClient to connect to the webserver. The thread then polls the message queue for a new message. The standard JDK Linked Blocking Queue was used to allow blocking.

21

4 Evaluation
4.1 Overview
Evaluation of the project is done in two ways. Firstly I evaluate how the algorithms perform offline by testing them against some labelled data that was reserved for this use rather than for training. Secondly the performance of the program running on the device is evaluated. The first approach can tell us how the algorithms are performing for data gathered using the Ground Truth Data application. If the algorithms are accurate in the offline testing then that gives a baseline of confidence in their efficacy. However there are a number of factors that may lead to a lower rate of accuracy for on device testing, compared with offline testing. There could be bugs in the android application that lead to inaccurate classification. The device, having to now collecting and analysing samples simultaneously, may collect samples at a lower frequency than the training data, affecting the feature values. The algorithms may be suffering from over fitting, where they are accurate with the offline test sample (most of which was collected by the same couple of people), but not with new data from new test subjects. These factors are the motivation behind doing online testing on the device.

4.2 Offline evaluation of classification algorithms

Initially I planned to do this stage in MATLAB. However it was possible there could be subtle variations in how the calculations were done between Java and MATLAB, in particular with the Fourier Transform libraries. In order to check that the algorithms that the device application uses are working, it seemed appropriate to test hose algorithms directly. I developed a small application in java that took unused labelled training data (around 30% of the total), imported the exact algorithms the device was using, and computed their accuracy on the labelled test data. This also allowed me to experiment with the k number for the k-NN algorithm. Throughout the next section a common representation for results will be used, this representation is a confusion matrix. The leftmost column represents the label of the test samples in that row. The next four columns, S, W, C, R, indicate the classification results for those samples. The final column is the accuracy for that class. For the offline testing 1044 samples were used. Below are the results for each classifier. Naive Bayes S 122 0 0 0 W 0 275 10 22 C 3 24 247 4 R 0 2 1 334 Accuracy 97.6 91.4 95.7 92.8 94.8

Stationary Walking Cycling Running

22

Table 3 Nave Bayes, offline testing confusion matrix

k-NN (K = 3) S 124 0 4 0 W 0 292 10 25 C 1 5 243 0 R 0 4 1 335 Accuracy 99.2 97.0 94.2 93.0 95.8

Stationary Walking Cycling Running

Table 4 k-NN k = 3, offline testing confusion matrix

k-NN (K = 5) S 124 0 3 0 W 0 295 8 26 C 1 5 246 0 R 0 1 1 334 Accuracy 99.2 98.0 95.3 92.7 96.3

Stationary Walking Cycling Running

Table 5 k-NN k = 5, offline testing confusion matrix

Voting (Nave Bayes, K = 3, K = 5) S 124 0 3 0 W 0 295 7 26 C 1 5 247 0 R 0 1 1 334 Accuracy 99.2 98.0 95.7 92.8 96.4

Stationary Walking Cycling Running

Table 6 Voting classifier, offline testing confusion matrix

The results are very good. All the classifiers are accurate to at least 92% for any class. This is expected as the test data used here was collected by the same users as the training data on the same device so in the data is very similar. Although it does give confidence that there are no major bugs or errors in the classification algorithms. Also the other key point is all the algorithms are performing to a very similar level. The Voting classifier is only marginally better than the others for twice the processing power. However the difference increases in the user testing.

4.3 On device testing

The next stage of the evaluation procedure was to evaluate the performance of the classifiers running on the device. This was done by recording the results of the real-time classification application, while maintaining knowledge of the real movement type. As in the previous section, a confusion matrix is used to represent the results. However the matrices for Nave Bayes and K-NN have been omitted for brevity. 23

Voting (Nave Bayes, K = 3, K = 5) S 34 0 0 0 W 0 94 3 5 C 0 0 86 0 R 0 2 0 72 Accuracy 100 97.9 96.7 93.5 97.0

Stationary Walking Cycling Running

Table 7 Voting classifier, on device testing confusion matrix

Below is a table showing the accuracy of all the classifiers: NB 100 93.8 96.7 93.0 97.0 K = 32 100 91.7 96.7 93.0 95.4 K=5 100 97.9 96.7 96.1 97.7 Voting 100 97.9 96.7 97.4 98.0

Stationary Walking Cycling Running Overall

Table 7 Overall on device accuracy of the classifiers, using personal data

This testing was done by me, and as I made up a large component of the training data, is it unsurprising that the classifiers show a high level of accuracy. Again there is little to no improvement using the voting classifier as opposed to the simpler methods. However as in the previous section this shows that the classification application is working as intended. In the next section, the classifiers are evaluated using new users that were not involved in the training or testing so far.

4.4 User testing

In this part of the evaluation I organised a number of users (my housemates) to use the application while I recorded what activity they were doing. The purpose of this was to test the accuracy of the classifiers with users who had not been involved in the training data. Overall the results were lower than in the offline testing, as to be expected. NB User1 Stationary Walking Cycling Running Overall User2 Stationary
2

K=3 94.0 84.4 92.0 86.6 89.2

K=5 94.0 87.5 90.0 88.4 90.0

Voting 94.6 84.4 90.0 90.0 90.0

93.4 84.4 82.0 82.2 85.7

100.0

100.0

100.0

100.0

K=3, and K=5, represent a k-NN classifier as described in the previous section, with the given k value

24

Walking Cycling3 Running Overall User3 Stationary Walking Cycling Running Overall User4 Stationary Walking Cycling Running Overall

48.6 74.3

74.3 78.2

71.4 87.5

74.3 87.2

88.6 83.3 85.2 85.7 85.7

88.6 86.7 88.5 86.5 87.6

89.2 86.7 88.5 86.5 87.7

89.2 86.7 90.1 88.7 88.7

92.2 82.1 100.0 90.4 91.2

93.4 85.7 100.0 92.3 92.8

92.2 85.7 100.0 92.3 92.5

94.0 85.7 100.0 96.0 93.9

Table 8 - User testing

There are a couple of notable points with regards to the results. Firstly, they are generally very high which leads me to conclude that the classification application is successful in what it was designed for. All the classification algorithms performed badly on user2. This I suspect is because User2 was female and all the training data had been gathered by males. More investigation is needed. If this was the case, in order to improve accuracy you could have the user answer a few questions about their sex, possibly age, and use these factors along with the accelerometer data for classification. To conclude this section I have collated the accuracies above, to give an idea of the total accuracy of each classifier. NB 84.2 K=3 87.0 K=5 89.4 Voting 90.0

Accuracy

Table 7 Overall on device accuracy of the classifiers, with housemates

Accuracy

NB 87.5

K=3 89.9

K=5 90.1

Voting 90.9

Table 8 Overall on device accuracy of the classifiers, with housemates excluding user2

Overall the voting classifier is no better than using k-NN with k set to 3. I had collected the processing time of each of the classifiers, as I was going to investigate the trade-off between

Running and Cycling are omitted as this user was unavailable to test the application

25

accuracy and power for each of the classifiers, but as the Voting classifier isnt any better than the simple k-NN algorithm it wouldnt be a valuable addition to this evaluation.

4.5 Energy usage

To analyse the energy usage, I used an application called PowerTutor. It allows you to profile the energy used by the system, and filter by individual applications. The testing was performed with the dynamic updating, and log uploading turned off, to remove the energy required to use the internet. The screen was also turned off, leaving the CPU to be the only component of energy usage. The energy used when applying different algorithms varies greatly as would be expected. When using the simple classifier the energy is minimised. In the simple case, the application is performing three main tasks: collecting a sample, removing gravity and calculating the mean. When Nave Bayes is used, the algorithm is doing all the above, but also calculating more complex features, and then performing some calculations to get a maximal probability. For k-NN the features are calculated, then the distance from that feature vector to all the others is calculated, and finally these distances are sorted.

Classifier Simple NB k-NN Voting

Time(S) 328 565 365 372

Energy (mJ) 990 2250 2100 2200

Power (mW) 3.02 3.98 5.75 5.92

Table 9 - Rate of energy usage (final column) for each classification algorithm

Using the k-NN classifier is significantly more energy intensive than NB, a 45% increase. However the voting classifier is not much more. This is because if you are calculating k-NN, an extra k-NN with a different k value is free. Plus NB is not very expensive because all the features will have been calculated, leaving just the probability calculation, which is likely to be what is causing the extra 0.17mW above k-NN alone. The next chart shows the classifiers when plotted against accuracy and energy usage.

26

91 90 89 Accuracy % 88 87 86 85 84 83 3 3.5 4 4.5 5 5.5 6 6.5 Energy usage (mW)

4

Nave Bayes k-NN Voting

Table 10 - Accuracy / Energy trade off

From this one can see that in order to have higher accuracy classification, more energy is needed. If one was developing a commercial application the need for accuracy would have to be balanced with the need for low power. The adaptive classifier, described in 3.4.8, is another technique for reducing power. In the case when the user is stationary, the adaptive classifier would only use the power required for the simple classifier, however if motion is detected it would trigger a more complex classifier. If the application were running constantly to provide context awareness, it can be assumed that a large amount of time will be spent stationary. The simple classifier uses 49% less power than voting. Assuming the device is stationary for just half of the time the application is running, the energy usage overall would drop by 25%.

k value of 5 used for k-NN. Accuracy taken as overall accuracy for user testing, as in table 7

27

28

5 Conclusions
5.1 Project success
Looking back at the requirements analysis, 2.3.1, the core aims of the project were to: Create and test an application for gathering labelled training data. Use the application to gather a large amount of accurate training data. Implement a classification application for simple motion detection Implement a more advanced classification application for more classes, such as Walking, Cycling and Running

All of this was completed. Also the energy usage of the application was evaluated. I had planned to go further and investigate varying the accelerometer sample rates to reduce power but I wasnt able to get that finished. In hindsight I didnt give enough time for the collection of accurate ground truth data. This part of the project took up almost the whole of first term. There were a lot of problems with people uploading inaccurate data by mistake. I think overall not enough people were used for the training data which lead to problems such as a low accuracy when tested with woman rather than men. That said all of the pieces are there to make the application more accurate. The software for collecting and processing new training data is complete, and the algorithms on the device automatically update without having to be reinstalled.

5.2 Further work

There are many ways the project could be extended, something I hope to do over the summer. More activity classes could be added, such as driving, (Ravi, 2005) even used vacuum cleaning as an activity type. Adding an activity would not involve any development because one could use the software developed in this project. However it would require management, time and a lot of volunteers. Other methods for reducing the energy usage of classification could be investigated. One idea would be to collect sensor readings at one of the slower rates, such as NORMAL, when the user is not moving, and then have user movement trigger a faster sampling frequency.

29

30

Bibliography
DeVaul R, D. S. (2001). Real-Time Motion Classification for Wearable Computing Applications. Technical report, MIT Media Laboratory. Dey, A. K. (2001). Understand and Using Context. Personal and Ubiquitous Computing. In A. K. Dey, Understand and Using Context. Personal and Ubiquitous Computing (pp. 4 - 7). Google. (2012). Retrieved from developer.android.com: http://developer.android.com Intille, L. B. (2004). Activity Recognition from User-Annotated. Lee S, M. K. (2002). Activity and Location Recognition Using Wearable Sensors. IEE. Nissanka B. Priyantha, A. C. (2000). The Cricket Location-Support System. Piggott, D. (2011). Inferring Transportation Mode using Smartphone Sensor Data. Fitzwilliam College. Ravi, D. M. (2005). Activity Recognition from Accelerometer Data.

31

Appendices

32

33