How to Beat Facial Recognition Technology

February 25th 2013

How to Beat Facial Recognition Software without Looking Like a Bank Robber

On the record, though, no agency in the US has been forthcoming with why its using those specific facial scanners or even confirming its in their arsenal of ever expanding surveillance tools. Over the last decade, computers have become better at seeing faces. Software can tell if a camera has a face in its frame of vision, and law enforcement has been testing facialrecognition programs that can supposedly pick out suspects in a crowd. That's prompted an arms race between the people who build facial-recognition systems and those seeking ways to defeat them. Facial-recognition software is becoming a bigger issue for privacy advocates as well. Surveillance cameras are already ubiquitous in the U.K., are showing up in more places in the U.S. and may increasingly be connected to facial-recognition systems. "I went to a Kinko's a while ago," said Alex Kilpatrick, chief technology officer and co-founder of Tactical Information Systems, a company in Austin, Texas, that sells facial-recognition software to law enforcement and the military. "I saw three cameras just while I was standing in line. You see them in all kinds of places now." The American Civil Liberties Union (ACLU) has said it is deeply concerned with the way facialrecognition systems are used. Police use such systems to flag criminals in public places, the ACLU says, but it argues that the Transportation Security Administration's (TSA) use of the technology in Boston's Logan Airport and in T.F. Green Airport near Providence, R.I., doesn't seem to have helped catch any criminals or terrorists.

Beating the machine

Kilpatrick gives presentations about the capabilities of facial-recognition systems, but is concerned about privacy as well. To him, it's every citizen's duty to throw the occasional monkey wrench into what he sees as Big Brother-like surveillance systems. So Kilpatrick started investigating how facial-recognition systems can be fooled. It turns out that in order to do it, you have to know how machines think. "People don't appreciate that humans are really good at certain types of face recognition," Kilpatrick said. "You can compress a picture a lot, and a person will recognize it but a computer won't." 1

On the other hand, Kilpatrick said, people are not so good at identifying someone from an ID picture. That's why the TSA officers at the airport often have to look at you and your ID twice. There is also a big difference between determining that what's in front of you is a face finding a face and linking it to a particular person, or recognizing one. For both humans and machines, the first is almost trivial, but the second is much harder. Kilpatrick did a set of experiments with a face-matching program called Neurotechnology, which used an algorithm called Eigenfaces. He found that in some cases, sunglasses that cover a big part of the eye regions and cheekbones such as the ones Audrey Hepburn wore in "Breakfast at Tiffany's" would fool the software enough that it couldn't match the image to a reference picture of the same person. On the other hand, a baseball cap or shadows falling on the face would not fool the software. A critical region, as it turns out, is the space around the bridge of the nose and between the eyes. So San Francisco Giants relief pitcher Brian Wilson's signature bushy black beard, for example, wouldn't make much difference. But hairstyles that cover half the face, or asymmetrical makeup, will. But that's the rub: Different face-finding and recognition programs rely on different characteristics to pick out features that identify the picture in front of them as a face. Matching that face to another picture uses yet another kind of algorithm. Even the most sophisticated computer programs can still be fooled but you have to know what each program is looking for. For example, software that focuses on the distance between the nose and ear won't even notice if you obscure the central part of the face. In other cases, software relies on the deviation from an idealized face, or from an "average" face generated from several people's images, to find a face in a picture. Matching one face to another recognition might use several points and check those against a reference image. But even that, Kilpatrick notes, isn't as easy as it is often portrayed in Hollywood movies.

Wham, bam, thank you glam

Web designer Adam Harvey approached the problem of fooling Big Brother as something of an art project while he was studying at New York University. He made it part of his thesis project, and came up with something called CV Dazzle. It's named after OpenCV, a form of open-source facial-recognition software, and the "dazzle ships" of World War I and World War II, which used abstract, high-contrast "dazzle" paint schemes to foil enemy range-finders. "There's all this money for the military and corporate applications," Harvey said. "I wanted to do something to balance that." Harvey's plan was to not only find out what would fool facial-recognition software into not even finding a face, but also to explore norms about what's socially acceptable on a face. He said many of the designs that successfully fooled the software were pretty outr, with dramatic, high-contrast glam-rock makeup and artsy-looking hair. "People have mentioned David Bowie and Adam Ant," Harvey said. Harvey found that symmetry seems to be what face-finding software looks for. In one case, the model's face was not obscured, but the bridge of the nose was, and that confused the symmetry-seeking part of the program. Covering half the face seemed to work as well. 2

With the help of some artistically inclined friends, he came up with a set of hairstyles and makeup patterns that made the software unable to identify what it saw as a face. For the CV Dazzle project, Harvey mainly used OpenCV. But now he's working on an application that will be able to determine what any facial-recognition algorithm, even one never before seen by the user, is looking for. Harvey's new software presents a baseline photo over and over again to a face-finding program, with small alterations to the photo in each instance. Sometimes the program being tested will see a face, and sometimes it won't, but eventually Harvey's software will be able to tell you what the face-finding software is zeroing in on. And once you know that, you can fool it. The advantage to this method, time-consuming as it may be, is that it doesn't require investigating the tested software's code.

Face in the crowd

However, Kilpatrick and Harvey both investigate facial recognition in controlled settings. What happens when such research is taken out into the real world? Alessandro Acquisti, associate professor of information technology and public policy at Carnegie Mellon University in Pittsburgh, recently studied the issue of facial recognition when applied to social-networking websites, as well as how it applied to the difference between the real and online worlds. Acquisti found that it wasn't all that hard to match public photos on an online-dating site, where many members used pseudonyms, to public photos on Facebook, where people have to use their real names. Nor was it difficult for Acquisti to match the faces of people walking across the Carnegie Mellon campus, which he shot with a simple webcam, to their Facebook profiles. That was also a relatively controlled situation, because the number of possible faces on campus was limited, and the computing power necessary to recognize random faces is still quite large. "Computer face recognition cannot yet allow the identification of every person all the time everywhere: even using cloud services, it still would take too much time for real-time application, and the false positives would be a huge obstacle," Acquisti told SecurityNewsDaily via email. "But 10 or 15 years from now? I am not so sure." Acquisti added that it isn't likely that people will be wearing masks any time soon, or that social norms will allow for people to wear masks, or even some of Harvey's weirder makeup designs, in public. (Acquisti did note that a baseball cap adorned with glowing LEDs will sometimes confuse face-finding algorithms). Ralph Gross, who worked on the facial-recognition studies with Acquisti, noted that false positives are indeed a problem for most facial-recognition software. But Gross added that the chance of identifying an unknown subject against a database of even 1.6 million criminal records is 92 percent using the best algorithms available. That percentage drops as the number of people goes up, but it is still quite good. Poor success rate? 3

Yet the kinds of deployments Kilpatrick, Harvey and Acquisti have experimented with aren't how law enforcement and the military envision using facial-recognition programs. That's why the ACLU is worried. At the Super Bowl XXXV, played in 2001 in Tampa, Fla., facial-recognition software connected to security cameras in Raymond James Stadium flagged 19 people. None were more than petty criminals, and the major offense they had committed was ticket scalping. (Following the Sept. 11, 2001, terrorist attacks, Super Bowl security was taken over by the Secret Service, which hasn't disclosed its methods.) Since then, the Pinellas County Sheriff's Department has deployed a newer system, which is used to match arrestees up with a database of mug shots. Even so, a human being has to make the final call on whether there's a match. The facial-recognition software that's been deployed at airports by the TSA over the past decade has failed to catch any terrorists. And the ACLU says it's far from clear how the TSA compiles its database of suspect faces if digital cameras simply take pictures on public streets, then potentially everyone could end up in the database. Acquisti said that while many people may want to fool facial-recognition software, it may one day be impossible to do so. The more researchers come up with techniques to hide or mask faces to avoid computer face recognition, the more other researchers will come up with techniques able to bypass those protections, he said. The conditions under which a human face will not be recognizable by a computer will be the conditions under which also humans cannot recognize each other. In the Orwellian Police State that is our world, facial recognition software is cropping up more and more. From the ever-encroaching Facebook to local police departments to airport checkpoints, its becoming harder to be anonymous and protect your personal privacy. RT reports: The deployment of FaceFirst in the United States still begs questions concerning the relationship between security and privacy, though, and is likely to remain an issue of contention until agencies in San Diego and elsewhere explain what exactly theyre up to. According to a report in Southern Californias News 10 published this week, an unnamed law enforcement agency in San Diego County has been testing a handheld version of FaceFirst for about five months now. If they spot someone who doesnt have identification, they can take their picture with their phone and immediately get a result, Joseph Saad, business development director for FaceFirst, tells News 10. Saad says his company predicts that facial recognition will be in every day society soon, perhaps before many Americans want to admit. According to filings available online, Airborne Biometrics was already cleared by the Government Services Administration (GSA) last year to have FaceFirst sold to any federal agency in the country.

The ability to apply our technology for the advancement of our country has always been my number one goal, Rosenkrantz said in April 2011 when Airborne was awarded an IT 70 Schedule contract for FaceFirst by the GSA. Because that contact has since been signed with Uncle Sam, Rosenkrantz and company can see that goal through, at least until its up for renewal in 2017, through a deal that lets them sell FaceFirst to all federal agencies and other specified activities and agencies. This means that coming soon, to a law enforcement officer near you, a simple photo snapped with a device that looks similar to a cell phone will enable you to be identified. According to New Scientist, facial recognition systems have reached the point where they can match a single face from a pool of 1.6 million mug shots/passport photos with 92% accuracy, in under 1.2 seconds. (source) Not only that, but by linking through information posted on social media networks and photo sharing applications, the agency identifying you will have personal information such as your marital status, where you like to hang out (think about those Facebook check-ins), your gym, your workplace, your childrenand its nearly all information that we put out there ourselves. To make matters worse, its not only government employees that can delve into your privacy. A garden variety stalker with off-the-shelf software and some minor data mining skills can get just as much information. If Joe Nutcase at the coffee shop can snap your picture, he can find out your name and a variety of other personal nuggets within seconds of downloading the photo. Carnegie Mellon University researcher Alessandro Acquisti says that he has proven that most people can be identified through one photograph. Acquisti found that the convergence of facial recognition software with social networks like Facebook tilt those odds wildly in favor of the would-be exposer, or stalker. Acquisti searched for dating site users within 50 miles of a zip code, found about 6,000, and then found 110,000 Facebook profiles where users said they lived near that same zip code. After eliminating some profiles that didnt match his criteria, he instructed computers to churn through about 500 million pairs of possibilities. It would take a human about 2 million hours to compete such a task, but Carnegie Mellons cloud computing cluster got results in about 15 hours. One in 10 members of the dating site were positively outed by the database search. A bit of fine-tuning limiting the geographic area further or allowing approximate matches produced even better results. And one sobering reminder: The researchers didnt even need to log in to Facebook to get these results. Acquistis team enjoyed even better results when they could obtain photographs themselves for matching purposes. Random students who agreed to be photographed on the Pittsburgh campus of Carnegie Mellon could be positively identified at three times the initial rate or more than 30 percent. The researchers didnt stop there. Next, they linked the photos and names to student likes and dislikes gleaned from their profiles, with about 75 percent accuracy. Then, they combined this effort with work Acquisti had done in 2009 on predicting Social Security numbers, and found that they could predict the Social Security number for 28 percent of the subjects within four 5

guesses. Finally, they built a mobile phone application that could achieve the same results while wandering around campus. To refresh: Starting from a mere photo, they were able to determine name, friends, even SSNs. Source Sohow can you protect your privacy from the surveillance cameras that seem to be everywhere? First, you need to understand how the technology works. There are certain points on the face that are targeted by the software: particularly eyes, nose, and mouth; where they meet, and the distance between them. The trick is to obscure these points without drawing undue attention to yourself through your efforts to camouflage them. Black geometric shapes painted on the face seem to fool the devices into not recognizing the photo as a human face. However, it isnt exactly subtle to walk around with black triangles and squares painted randomly on ones face. One researcher from NYU, Adam Harvey, has been working on reverse engineering the technology in order to protect privacy. In the photo below, the faces with the red squares around them were identifiable by the software.

The places you want to obscure are: Distance between the eyes Width of the nose Depth of the eye sockets The shape of the cheekbones The length of the jaw line A spike of hair proved to be effective if it covered the area where the eyes, forehead and nose come together. Computer-vision software often looks for that spot, said Harvey, and will not detect a face without it. (source) 6

Hats, hoodies and big sunglasses can help obscure some of the targeted facial areas but can also make you look suspicious, especially if the weather conditions dont support the wearing of those items. The hacktivist group Anonymous offers some suggestions too. The video below outlines a simple hat with LED lights that are invisible to people around you but complete obscure the face of the wearer. The lights will be relatively invisible to the naked eye, but cameras looking to identify you will pick up the light and the resulting bloom will cover your face completely when viewed by cameras.

Anonymous releases how-to instructions on fooling facial recognition

http://www.youtube.com/watch?feature=player_embedded&v=mxgOn8ikQuQ If you are identified, take steps to limit the amount of information that is available about you. Limit your use of photographs on photosharing websites like Picasa and Photobucket. Dont provide personal information about yourself and your family on social media sites like Facebook. Check your privacy settings on Facebook to be sure that others cannot tag you in photographs. Dont use a photograph of yourself as a profile picture on Facebook use a pet, a cup of coffee or something unidentifiable. Profile pictures cannot be made private. Resist the urge to have your work place, your Alma Mater, and your family members linked to your Facebook profile. Do some Google searches of yourself and see what information comes up. Take steps to remove as much information as possible.

Dont fall for the adage, If you dont have anything to hide, you dont have anything to worry about. In a world that is looking more like an authoritarian science fiction movie every day, so many new laws are being passed that people formerly known as law abiding citizens are breaking them without even realizing that the regulations exist.

In the surveillance state that is America, we are all criminals waiting to be caught. (And never forget the NDAA you can be indefinitely detained just because the government said so). Privacy is liberty guard it carefully. Additional Articles: 10 Ways the Government Watches You How Your Cellphone Lets the Government Track You