GoldenEye: A Face Recognition Based Authentication System for Smartphone Applications

Shashank Hegde
Department of Computer Science University of California Santa Barbara Santa Barbara, CA

Kowshik Prakasam
Department of Computer Science University of California Santa Barbara Santa Barbara, CA

hegde@cs.ucsb.edu
ABSTRACT The ubiquity of internet enabled smartphones enables us to always remain connected. Applications on mobile phones such as Facebook, Twitter, etc provide us a convenient way of keeping in touch with our family and friends. But at the same time, these very applications are also a source of a wealth of private information. Previous studies [1] have shown that sharing of mobile phones is very prevalent. This raises a number of privacy concerns. In order to protect users privacy concerns, we propose GoldenEye, a face recognition based authentication system for smartphone applications. GoldenEye is an additional layer between the application and the users login credentials which recognizes the current phone user by taking their picture and the application is allowed to use the credentials only if the current user is authorized by the phone owner. General Terms Authentication, Privacy, Smartphones, Design, Sharing Keywords GoldenEye, Android, Service, Face recognition 1. INTRODUCTION Mobile web access on smartphones and netbooks is very popular due to the high availability of wireless internet and improved user interfaces. Advancements in social networking and location sensing have led to a large number of native smartphone applications for the web. An increasing need to share content through social networking services has led to occasional sharing of smartphone and netbook access among users. Instant messaging and quick access to emails are other reasons for sharing access. Native smartphone applications that connect to the web require authentication through username and password. When applications are installed, the owner of the phone usually configures them to work with his/her account permanently. The application saves the login credentials for convenience and enables an always-logged-in mode for the owner. This poses a serious threat to privacy when multiple 1

kowshik@cs.ucsb.edu
users access the same device. Due to the relatively small size of smartphone screens, it is hard for the onlooker to notice people sniffing on private data stored on the smartphone during their occasional access. Just like other portable electronic devices, smartphones are attractive targets for thieves. In any case, if the worst happens it is important that the user's private information remains well protected. Switching off always-logged-in modes to safegaurd privacy introduces the annoyance of having to repeatedly type usernames and obscure passwords on the smartphone. Unfortunately, the OpenID [2] standard of authentication has been adopted only by a minor subset of web-based services. The effectiveness of touchscreen typing is improving rapidly due to technological advancements. Yet, they may never match the ergonomics of a keyboard or typewriter due to the small size of smartphone screens. There is clearly a rising need to accommodate a central system on mobile platforms that can offer services to ease multiple user authentication on the same device. Biometric identification mechanisms that can cleverly handle false negatives and false positives are a best fit for this requirement. Modern smartphones come equipped with supremely effective cameras that offer solid support for using a facial recognition system to solve the above problem. 2. CHALLENGES Any system can be made arbitrarily secure by adding more and more layers of authentication systems. But the challenge is to find the fine balance between security and usability. So, any authentication layers that are added must be as user friendly as possible. As the number of applications increases, having different authentication systems for each application would confuse the user, prompting him to reduce or remove authentication layers or in the worst case, he may not remember the required credentials leading to application lockout.

Biometric authentication systems are secure and are relatively unobtrusive which can solve the challenges of usability, security and reliability. Modern smartphones come equipped with extremely good cameras and a majority of them even have front facing cameras. This gives us an opportunity to implement an authentication system which uses face recognition. Even though smartphones are becoming smarter, they still have severe resource constraints. Hence, face-recognition using limited resources presents a great challenge. Resource constraints on smartphones exist in the form of reduced disk space and processing power. Studies [3] show that as of 2009, ARM processors account for approximately 90% of all embedded 32-bit RISC processors. ARM processors are used extensively in consumer electronics, including PDAs, mobile phones, digital media and music players, hand-held game consoles, calculators and computer peripherals such as hard drives and routers. Though ARM processors are fast enough to power traditional smartphone applications, they havent been benchmarked against a full-fledged biometric system.

3. SYSTEM DESIGN The goal of GoldenEye is to come up with a secure, central and easy-to-use authentication system which can be used by the application developers to provide convenient and consistent authentication process. Hence, we designed [Figure 1] the system around a central service which takes care of authentication. Applications can make use of this service and use the results.

Figure 1 : High-level design of GoldenEye Persistent storage space of the order of a few tens of megabytes are available in a smartphones inbuilt SD card. Though external SD cards offer a few gigabytes of extra storage, the I/O access is slow. Hence they are not the ideal choice for persistence for a face-recognition application. Based on the above facts, we present the key technical challenges of GoldenEye : (1) How can a face recognition system be built on top of a processor incapable of handling complex mathematical computations in a biometric system ? (2) Given the disk space constraints on a smartphone, how can the system be made to work with images at a reduced resolution level, and how many number of different users can the system handle on a single smartphone ? (3) How can the system interface efficiently with numerous applications that will rely on the system for authentication ? (4) How can the system be made to work under poor lighting conditions, mediocre camera quality ? How can it handle the possibility of production of blurred images when users take pictures of themselves in non-front facing cameras ? (5) How can the system be made to not produce false negative and positive results ?

3.1 Pluggable Authentication System GoldenEye is deployed as a platform service and it exposes an Identify User interface. Applications can use this interface for authentication. The AuthenticationService of GoldenEye is designed so that different user authentication modules like face recognition and gesture recognition can be plugged in. Gesture-based recognition is particularly useful in smartphones that dont ship with cameras. 3.2 Face Recognition Module In order to improve the speed of facial recognition, the system is designed to be contained in a shared object written in C++. Each call to the GoldenEye authentication service delegates to a C++ class inside the shared object. 3.3 Face Recognition Algorithm We have used the Eigenfaces technique for face recongition in GoldenEye. Eigenfaces refers to an appearance-based approach to face recognition that seeks to capture the variation in a collection of face images and use this information to encode and compare images of individual faces in a holistic (as opposed to a parts-based or featurebased) manner. Specifically, the Eigenfaces are the principal components of a distribution of faces, or equivalently, the eigenvectors of the covariance matrix of 2

the set of face images, where an image with pixels is considered a point (or vector) in -dimensional space. The motivation of Eigenfaces is twofold: (1) Extract the relevant facial information, which may or may not be directly related to human intuition of face features such as the eyes, nose, and lips. One way to do so is to capture the statistical variation between face images. (2) Represent face images efficiently. To reduce the computation and space complexity, each face image can be represented using a small number of parameters. The Eigenfaces may be considered as a set of features which characterize the global variation among face images. Then each face image is approximated using a subset of the Eigenfaces, those associated with the largest eigen values. These features account for the most variance in the training set. 3.4 Cropped, Resized Images Before passing images to the shared object for processing, they are converted into greyscale, resized and cropped to remove unwanted features other than the face of the person [Figure 2]. This brings down the size of a single image to about 15KB. Thus, the system can persist hundreds of images with a few megabytes of disk space.

3.5 Camera Positioning Though a majority of contemporary smart phones come equipped with front facing cameras, those that dont cannot be ignored. Rear-facing cameras make it difficult for the user to shoot clear images of themselves as it is hard to reach out for the snap button during shooting of self pictures. We have designed a timer-based user interface that allows the user to wait after pressing the snap button on the smartphone for a period of five seconds before the snap is taken. This effectively reduces the chances of blurred input images. 4. IMPLEMENTATION 4.1 Face-recognition implementation The face recognition system is powered by the OpenCV library. OpenCV is a library of programming functions mainly aimed at real time computer vision, developed by Intel and now supported by Willow Garage [4]. It is free for use under the open source BSD license. The library is cross-platform. It focuses mainly on real-time image processing. If the library finds Intel's Integrated Performance Primitives on the system, it will use these commercial optimized routines to accelerate itself. Since Android phones dont ship with Intel hardware, we were unable to make use of these optimizations. OpenCV also provides a stable implementation of Eigenfaces. However, the OpenCV API is tied closely to the complex mathematical computations required for face recognition. To provide a clean interface to the Android service that consumes the compiled shared object, we wrote custom C++ wrappers around the OpenCV API. The official android Native Development Kit does not support C++ concepts fully like RTTI, exceptions, and most of the Standard Template Library. As a workaround, we compiled the wrapper code and the OpenCV library using a customized Android NDK with enabled support of C++ exceptions, RTTI and Standard C++ Library developed by Dmitry Moskalchuk [5].

Crop to Face

Training

Recognition We interfaced the C++ wrappers with the Android service through the Java Native Interface (JNI) and SWIG (Simplified Wrapper and Interface Generator) [6]. Face Database 4.2 Authentication Service Android operating system has several features on its lifecycle and the Service is one of them. We can simply define the Service in Android system as a class by which can run an independent process (thread) in background without 3

Figure 2 : Image resizing and cropping

displaying it through any kind of window. Basically the Service does not have an UI element and it has an independent memory region. Because of this feature, services are mostly used for background service such as music streaming, notification, and so on. For example, services running in background update Content Providers, generate Intents, and create Alarms. Service also has higher priority than any other activities. An interface to a service can be defined through AIDL (Android Interface Description Language) [7] which enables intercommunication between services and activities. Android has two types of services: Local and Remote. A local service is closely coupled with an application and its interface is not exposed to any other applications. Whereas a remote service exposes its interface to all the applications. In order to provide a central authentication system, we implemented GoldenEye as an Android Service. By doing so, any application can communicate with the GoldenEye service and request for user authentication. The service uses face-recognition module for user authentication. The Authentication Service exposes an interface called IdentifyUser() which can be called by user applications to identify the current user. The Authentication Service delegates the identification to the active authentication module, which in this case is face recognition. The face recognition module captures the current users photo through the camera and searches for a match in the database. If a match is found, it returns the users name. The application then uses the mapping between the phone users name and his login credentials to sends the user name and password to the service. 4.3 API We expose a simple API to trigger the GoldenEye service from the developers own application. The face recognition service is hidden behind a class named AuthenticationHandler, whose constructor accepts a callback parameter to which the service should return the result of face recognition. The authentication itself is triggered through a method called authenticate() in class AuthenticationHandler. Figure 3 shows a code snippet consisting of the application android activity called SampleApp. class SampleApp implements the IVerificationResult callback interface, which is used by the AuthenticationHander object to communicate the result asynchronously.

public class Application extends Activity implements IVerificationResult { public void someAuthenticationMethod( ) { // Start the login process if(authHandler == null) authHandler= new AuthenticationHandler(Application.this); assert(authHandler != null); authHandler.authenticate(); } // callback from AuthenticationHandler public void VerificationResult(String aUserName) { } }

Figure 3:Android application consuming GoldenEye service

5. EVALUATION 5.1 Training Before any user can be authenticated with GoldenEye, the users face has to be trained to work with Eigenfaces. The process of taking pictures consumes approximately 30 seconds of time for a single user. The time taken to merge the new face images with the existing image database consumes 2-3 seconds. However, this merging process may increase to 10 seconds [Figure 4] as the number of images in the database increases with usage. However, in an average case scenario, we can assume that a phone may not be shared by more than 10 unique persons. Under such circumstances, GoldenEye has an acceptable response time of 40 seconds. 5.2 Face Recognition The facial recognition process consumes much lesser time when compared to the training process. For any given person whose images have been trained beforehand, facial recognition consumes approximately 4-5 seconds [Figure 5]. Using eigenfaces is very fast, and able to functionally operate on lots of faces in very little time. Unfortunately, this type of facial recognition does have a drawback to consider: trouble recognizing faces when they are viewed with different levels of light or angles. For the system to work well, the faces need to be seen from a frontal view under similar lighting. Face recognition using Eigenfaces 4

users, it is important to implement any face recognition system that is foolproof as well as fast and efficient. The work on face recognition via sparse representation [9] presents an interesting prospect. However no stable, opensource implementation of the technique exists presently. This will be a good area to work on as an extension to our project. In an ideal scenario, there will be multiple applications queueing authentication requests to GoldenEye. One may argue that this may not be much of a concern as multiprocessing is not supported presently on smartphones. However, each process can go to a Pause state in Android and be reactivated when the system is about to resume it again. Hence, it is imperative that the system maintains distinct sessions across clients. It would be appropriate then to experiment with SQLite, an ACID-compliant embedded relational database management system which is fully supported in the Android SDK. Though the likelihood of two users with the same name sharing a smartphone is remote, to make the system complete it is necessary that this case be handled in the underlying image database that stamps images with user names. Currently, GoldenEye doesnt offer support for such a case. It will be interesting to add features that can warn the user if a user name exists already. Presently, the underlying image database is a flat file database containing information as plaintext. To secure the system against malicious users, it is neccessary that all persisted data be encrypted by the system. The implementation of quick encryption techniques for GoldenEye is a good area to work on in the future. With the advent of 3G and wifi, the computational power required for quick and reliable face recognition can always be obtained from the cloud. It will be interesting to attempt offloading of the face recognition system to the cloud.

Figure 4 : Time taken to train faces. TS refers to Training Set.

8 7 6 5 4 3 2 1 0 TS 1 - 5 images TS 2 -10 images TS 3 - 20 TS 4 - 40 images images

Training Time (s)

Figure 5 : Time taken to recognize faces. TS refers to Training Set.

has been shown to be quite accurate. By experimenting with the system to test it under variations of certain conditions, the following correct recognitions were found: an average of 96% with light variation, 85% with orientation variation, and 64% with size variation. [8] 6. FUTURE WORK Usage of Eigenfaces technique doesnt result in a foolproof identification system due to the limited capabilities of the algorithm. However, Eigenfaces guarantees speed and efficiency which is a necessary in a resource-constrained environment. Before this system can be deployed to end 5

7. CONCLUSION In this project we proposed a central authentication system for Smartphones. We were primarily motivated by the fact that smartphone sharing is very prevalent among users and there is a need for easy and convenient user authentication. We used face recognition to provide secure and simple user authentication. We implemented an android service which runs in the background and exposes an interface which can be used by any third party application to authenticate the current user.

Our implementation of face recognition on android smartphones using eigen faces showed that face recognition is possible in a reasonable amount of time even with the limited resources available on a smartphone. 8. REFERENCES
[1] Yunxin Liu, Ahmad Rahmati, Yuanhe Huang, Hyukjae

Jang, Lin Zhong, Yongguang Zhang, Shensheng Zhang "xShare: Supporting Impromptu Sharing of Mobile Phones", in Proceedings of the 7th international conference on Mobile systems, applications, and services, Krakw, Poland: MobiSys, 2009
[2] OpenID standard : http://openid.net [3] ARM Architecture :

http://en.wikipedia.org/wiki/ARM_architecture
[4] OpenCV : http://opencv.willowgarage.com/wiki/ [5] Android CrystaX NDK :

http://www.crystax.net/android/ndk-r4.php
[6] SWIG : http://www.swig.org/ [7] Android Interface Definition Language (AIDL) :

http://developer.android.com/guide/developing/tools/ai dl.html
[8] Eigenfaces : http://en.wikipedia.org/wiki/Eigenface [9] Face recognition via sparse representation :

http://perception.csl.uiuc.edu/recognition/References.ht ml