Source: http://www.cio.com.au/article/398067/security_google_apps/?

fp=16&fpid=1#closeme

Security and Google Apps

An interview with Google Enterprise director of security, Eran Feigenbaum Georgina Swan (CIO) 22 August, 2011 22:36

How does Googles approach to security differ from traditional models? Its a bit of a different model than trying to protect the end point devices, the laptop, the desktop. We focus on protecting the data, and have very limited data at the endpoint. We are an internet company born and raised on the internet and we have built security as part of the core DNA of our products. I find it a model that is really scalable to millions and millions of users, both from a technology and operations perspective. We rely heavily on data replication and no single point of failure. We have learned that IT systems fail thats their nature so how do you build something that heals itself and doesnt depend on any single failure? Where do you think CIOs and CISOs are spending most of their time, money, their effort, in terms of their infrastructure? One of things they still tell me and Im a little surprised to be honest is that patching is still a huge problem. Most organisations have a very heterogenous environment. They have multiple operation systems, with multiple versions of the operating systems, multiple applications, user stores, and with different versions. And we all know software vendors issue security patches on a regular basis. It is [the CISOs] job to know whether those patches are applicable and get them deployed all before the bad guys reverse engineer them. According to Microsoft, companies take between 25-56 days to deploy an OS patch after it has been released. Thats a pretty scary number to have an open, known vulnerability that long. And most of the CIOs I meet with say that they wish they were in that category; they feel theyre at least one or two deltas beyond that. So its still a bit of a problem, and its a problem that can go away if you move into the Cloud. Because now you have no more servers to patch. So its very important that your Cloud provider doesnt have this problem and that youre not just handing the same problem over to somebody else. The other issue CISOs tell me is around the data and where the data resides. Sixty per cent of the data still resides on unprotected laptops and desktops. One out of every 12 laptops is lost or stolen within the first 12 months of purchase. Those USB keys that we find so convenient to use 66 per cent of us admit to losing them with 60 per cent of those having corporate private data on them. Now they arent, typically, malicious users; these are users that are really trying to get the job done, so theyre taking the data with them. What CISOs found is that theyre not making it easy for users to do the right thing. By putting the data in the Cloud and making it available anytime, anywhere, you dont have to worry about these issues and user behaviour starts to change. Even the FBI, one of the most conscious organisations around security, has admitted to

losing close to 150 laptops in the last four years. Two million laptops are lost or stolen every year. And really when you think about the Cloud, you almost need to start thinking about a different paradigm. Its changing that mindset. How does Google approach the security issue internally? We actually have one of the largest security practices I know of; it is over 250 people dedicated to security. I dont know of too many organisations, other than perhaps intelligence agencies, that have that kind of security. I cant guarantee it, but when you move into the Google Cloud, you are getting 250 people looking over the security for your data. And Im very lucky to work alongside some of the worlds leading experts in things like drive-by downloads and malware. And that doesnt count our internal audit and engineering compliance teams, our physical security teams and the security people who actually sit within the product teams. Do you find Google is a honeypot for attacks because of your size and who you are? I think were being challenged on a regular basis. People asked Jesse James why he robbed banks, and he said, thats where the money is. So I believe Cloud providers, or anybody who has large amounts of data, are going to be potential targets. Its something that we realized and we manage that threat. We monitor new threats and we react accordingly. But I think also the scale on which we operate gives us an advantage. For example, we process about 2 billion emails every day, and with that comes a tremendous amount of knowledge about new attacks and new viruses. [It means] we can protect our users from that without them installing any software or any updating signatures files. So yes, there is obviously the target, but with that also comes some good knowledge of what is going on. From a physical security perspective, most of our data centres are undisclosed and unmarked. We have publicly announced the locations of some of our data centres. Id say we have the typical security you expect from a world class data centre with 24x7 guard coverage, access logs, video surveillance, thermal imaging cameras, multiple backups et cetera. So regardless of which data centre it is, it has to meet our own minimum security standard that we audit our data centres against. One of the sticking points for Cloud computing for many CIOs is around the location of data, particularly for government and risk adverse organisations. How do you tackle that issue? My personal opinion is that it is not as important where the data is located, but how it is protected and who has access. So it doesnt matter if the data is physically stored in the United States or the Netherlands or in Ireland et cetera. I can understand the discomfort associated with moving away from a server that I know exactly where it is, to being in the Cloud, but does it really make a difference? How do you overcome any regulatory issues with Cloud computing? I am finding it more in the government sector because of some laws not specific to Australia, but in general. And I can see why government feels that by keeping it in country it helps more with economics. I think the banking industry is changing. We have banking industry customers all over the world that are using our services. I think its more important that you have access to the data in the country so if you have a

regulatory compliance request you can respond to it from that country. I also think there is a little bit of a misconception that if the data is not stored in a country then authorities would not have access to it. Thats not the case; Googles a US company and regardless of where the data is stored it is our responsibility to comply with US laws and all other applicable laws from other jurisdictions. Do you make use of Safe Harbor provisions? We are Safe Harbor compliant. It is also our policy to notify customers wherever legally possible that theres a request for their data. And specifically in the Google Apps for business area, we prefer customers respond to those requests themselves, and we have given them the tools to do so. How often would you have that kind of interaction? We have been very public with our data transparency report[2], where we show by which country how many requests we get from data. They typically fall into two different types of buckets; one is the request to remove data and one is the request to produce data. While the numbers are public, what a lot of people dont do is reverse calculate the percentages. If you consider Gmail has a couple of hundred million users and look at the number of requests for data, you realise its a tiny percentage. What are the benefits of Cloud computing from a security standpoint? When we think about security, we think about it from a people, process and technology perspective. One of the core benefits of moving specifically to the Google Cloud is the way in which we store data. I use mail as an example but it is true of all the applications in the Google enterprise suite. If I was a typical on-premise environment, all my mail would be sitting on a single mail server along with everybody elses mail. And if I compromised that single mail server, I got everybodys mail. And you might replicate that server for a redundancy and an availability perspective. Weve taken a different approach. Weve taken all of my mail, for example, and weve broken it into small pieces, which we have spread across our environment. We have done the same with all our consumers and business users. Now, I dont have a dedicated server but I have a series of servers. Rather than having just one copy, I have multiple copies, within a single data centre and within a secondary data centre. Now Im not only not dependent on a single server, Im not even dependent on a single data centre. We built our infrastructure to accept servers to go down, to accept entire racks to go down and even entire data centres and hopefully without users ever knowing. We have zero scheduled downtime for our services. And in fact in 2010 Gmail was available 99.984 per cent of the time thats less than five minutes of downtime a month and so far in 2011 it has been 99.999. I think if you compare that to an on-premise solution, which you have to take down at least once a month for patching and so on, it really blows that out. What this also means is we have a recovery point objective and a recovery time objective of zero and one minute; if we recover from one data centre to another, our recovery point is to lose zero data. And to recover from one data centre to another data centre within one minute. This is not a primary data centre with a backup. It is a primary and a secondary data centre. And they may change to balance the load and see where users are coming from. There is another added benefit to the traditional on-premises environment. Almost every customer I have spoken to, the data on the mail server is sitting in the clear and rather

than doing that we obfuscate not encrypt but obfuscate that data so that it is not humanly readable. So if I do get access somehow to a disk, I do not know how to read the data, nor do I know which data belongs to which user. In a typical environment it would be simple to understand; it would probably be called something like Eranmail.db. At Google, each of these files is given a truly random file name so I cant go and map it back, nor can I read the contents. Its very different to the traditional storage model. Why obfuscation rather than encryption? Encryption is a valuable, great technology and it has its purpose. And we do encrypt data where its appropriate; we were revolutionary in encrypting all transactions with Gmail and with Docs. And we are still one of the major Cloud providers that, by default, turns encryption on for all communications using SSL between Google and the users. But we didnt see much of a security benefit from encrypting the data centre. IT was going to require a lot of key management, and going back and forth to get those keys, so every time you want to do something simple like search your inbox or search a keyword, I would have to decrypt each of those small files. And I would have to go to the key store, get the key, decrypt it, hand it back and go to the next one. It was going to introduce a lot more complexity and latency and we really felt that obfuscation gave us the risk mitigation that we needed while balancing those other aspects. How does Google deal with patching then? One of the reasons patching exists and takes so long is that people have very heterogeneous enviroments. We are, according to Gartner, the fourth largest server manufacturer in the world. It is kind of amazing because we dont really sell servers. Were only producing them for ourselves, but were fourth in the market after IBM, HP and Dell. We designed our own chips, we wrote our own operating system, we completely created our own infrastructure on a heavily, modified Linux stack. One of the advantages of this is that everything looks the same. It makes it really easy to manage the infrastructure so that when it is time to update we can do it in a rapid, uniform fashion. Now if you are going to put all your eggs in one basket, you need to guard that basket really, really well. Thats what having a single, custom-built, hardened solution allows you to do. Because you now know everything about that system as opposed to having to know about 10 different systems. How do keep that homogeny given the sheer numbers of servers involved? Its tough. Its part of our core strengths for example, another technology that we have written and used is something that sits on every server and asks: Do I look like the Google Standard Gold Image? And if it doesnt it tries to correct itself. And if it cant, it sends an alert to somebody. It turns out to have not only management advantages, but also security advantages, because one of the first things hackers tend to do when they get access to a machine is put a rootkit on which changes how that machine is going to look. As long as you have a Google Gold Image and that changes with time everything else knows to phone home and look like that image; its a very manageable problem. Any plans to take those servers and technology to the market? I cant comment on any future plans, but you can see a lot of lessons we have learned from managing this infrastructure in moving to things like Chrome OS and having a very different model to Chrome OS. Its a definite maybe. We typically try not to comment

about any future release stuff. So what storage technologies do you use? We really just use consumer-grade hard drivesand just lots of them. The same things you have in your PCs at home. It could become an operational nightmare but we turn that into an advantage in how we manage those disks and the life of the disk. I know in any given point of time where a disk is they each have a serial number who put it there, what was on it when it was decommissioned, when it went back to inventory, when it was deleted. I have not met too many organisations that have any idea where their disks are, let alone given them serial numbers. When a disk goes bad, most organisations have an outside vendor come in and replace it. Now when I talk to CIOs their eyes light up and they ask, How did I miss this? Do you use tape technologies at all? We do use tape for some things like Gmail. We do tape backups on top of the multiple copies we talked about online. Some products like Gmail take a snapshot of everything every so often and go offline. If there was a huge catastrophic failure, and we had to recover everybodys data, we have that ability. You can imagine the Hollywood-type of scene where a group of Google engineers collude to delete everybodys Gmail unless theyre given $3 million. Well, we have backup tapes in an offsite location that these people dont have access to. It sounds as if youve War Gamed just that scenario... We play a lot of games here. Part of our disaster recovery plan is to assume the worst has happened. Last years scenario was Google got attacked by aliens and Californias off the map. What do we do? How do we run our infrastructure? From a network security perspective, we build our own Google front ends which are custom-built firewalls that use the same concepts a homogeneous environment where we can learn a lot of lessons. I have been in the security field for about 20 years and in my mind the measure of a good security organisation is how it reacts to an incident. People dont like to talk about we never want to think about getting into a car accident but the reality is security incidents happen for various reasons. They may not even be your fault, but how do you react to that? Having a 24/7 security team is part of that and having our major security operations in California and Zurich so we can work through time zones. When there is a security incident, we assign an incident coordinator whose job is to triage that incident. And I think a big misnomer about this is if there is a security incident that affects customer data, we believe and contractually commit that it is our responsibility to notify those customers. Theres an idea that if something happens to your data, you wont know. For sure we will tell you. Can you give us an example? Docs oversharing. It affected 0.01 per cent of documents and we notified the people affected with as much information as possible who, what documents, when and let them make their decisions. Its important to put it in perspective. We make headlines because we are Google but the reality is worse stuff is happening in the traditional environment every day. Is Cloud computing perfect security? No. Its not. Ill be the first one to say that. I was in an intelligence community where we proved we could find out information about a computer that was not connected to a network and was in a secure room using various

technologies. But I think Cloud computing is as secure, if not more secure, than what most organisations are doing today. What kind of verification does Google undertake in relation to security? Penetration tests and various audit reports SAS 70[3] is the new one, SSAE16 and the new international version of that, the ISAE 3402[4]. And were glad to give those reports to customers so they can see what the auditor is looking at regarding confidentiality, integrity and availability of data. I think also one of the great things of the Cloud is the ability to innovate. About a year ago we gathered all the top professionals in the company and we asked: If we were going to make one change that would have the biggest impact across the board, what would it be? And the reality is most customers still get compromised because of a password. Its something that security professionals dont like to admit; we know all this wonderful cryptography and all these great systems we pay millions of dollars for and it comes down to a password. We knew that if we could fix that problem, it would address a lot. And what we came up with was a two-step verification system. Its free for enterprise and for consumers and its a one-time token on top of my password. I have to enter a six digit number that changes on a regular basis so if somebody steals my password, its not enough. They need a number that is generated on my smartphone and that number will change. You can also have it sent to you as an SMS or as an automated number and we have now released that in 150 different countries and 40 languages. Its easy to use and it really improves security at the customer level. Whats the take up like at the enterprise level? Some enterprises are using their own system, so we support SAML[5] [Security Assertion Markup Language] for single sign-on also. Its got good takeup, Id like to see more.
References
1. http://www.cio.com.auhttp://www.cio.com.au/article/381146/google_apps_vs_microsoft_bpos_office_365__part_2/ 2. http://www.google.com/transparencyreport/governmentrequests/ 3. http://sas70.com/ 4. http://isae3402.com/ 5. http://code.google.com/googleapps/domain/sso/saml_reference_implementation.html