A SSH Key Management White Paper

SSH User Key Remediation

Getting Control of One of the Most Signicant Hidden Threats to Your Enterprise Security

SSH User Key Remediation

TABLE OF CONTENTS: Executive Summary...........................................................................................................1 Virus and Cyber Weapon Threat.......................................................................................2 Exposure to Substantial Insider Risks.............................................................................3 Most Enterprises Are Non-Compliant with Mandatory Security Regulations..........................3 SSH Key Remediation as a Process........................................................................................5 Conclusion......................................................................................................................6

About SSH Communications Security:

Founded in 1995, SSH Communications Security is the company that invented the SSH protocol - the gold standard protocol for data-in-transit security solutions. Today, over 3,000 customers across the globe, including 7 of the Fortune 10, trust our Information Assurance Platform to secure the path to their information assets. Our platform enables businesses of all types and sizes to protect their information assets by providing the gold standard data-in-transit security solutions that prevents data loss in both internal and external environments, hardened perimeter security through our multi-channel two-factor authentication and internal security control management solutions that enables organizations to more easily manage user keys and monitor administrator trafc across your networks.

2012 SSH Communications Security

SSH User Key Remediation

Executive Summary
Sloppy management of authentication keys for SSH, an encryption protocol used for automation in IT systems, risks catastrophic IT failure in banks, government and industry. Most organizations have no process for managing, removing, and changing access-granting keys. This violates SOX, FISMA, PCI, and HIPAA, all which require proper control of access to servers and proper termination of access. A cyber weapon or virus using SSH keys to spread could destroy IT infrastructure and online data of an enterprise or government agency within minutes. Similar attacks were rst used by the Morris worm, which took down the Internet in 1988. IT staff can use poorly managed SSH keys to create permanent backdoors to production servers, bypassing all normal privileged access management and session auditing systems. Risky insiders include anyone who has ever had access to a server, including former system administrators, consultants, and anyone with access to backups or decommissioned hardware. The problem has been largely overlooked by IT security auditors, because the it is highly technical and hidden inside the system administrator domain in a distributed environment where no single person has had full visibility of the problem. Every organization should audit their SSH keys and other automated access mechanisms (including Kerberos). Furthermore, they should set up and enforce policies for managing automated access and remedy existing risks or face an existential threat when viruses using SSH keys to spread inside an organization emerge. Remediation includes removing unused keys, enforcing controlled key setup processes, automating key setups to reduce number of administrators who can set up keys (and to save costs), changing all keys regularly, and controlling what can be done with each key. This white paper focuses on SSH user key remediation as a process which all organizations utilizing SSH should be aware of and consider implementing. It will outline a basic process and set of tools which can be utilized to identify the existing trust relationships in your environment, bring legacy keys under control, and automate the creation, deployment, rotation and removal of keys.

SSH User Key Remediation

Virus and Cyber Weapon Threat

A cyber weapon or virus can be engineered to use SSH keys to spread to nearly all servers within an organization once it has penetrated one server. This ruins layered defense, and experience has shown that viruses frequently manage to penetrate individual servers in an organization. A cyber weapon would likely use multiple attack vectors to penetrate an organization, and could use SSH keys (or other trust relations such as Kerberos) to spread throughout the server organizations server infrastructure in minutes after penetrating the rst server. In 1988 the Morris Worm utilized trust relationships for automated access based on what was called rhosts authentication. In nearly all modern systems this has been replaced by trust relationships based on SSH user keys. A recent IBM X-Force study found most attacks against Linux/Unix servers utilize SSH. Many computer forensics experts are aware of cases were SSH keys have been used to spread an attack from one server to another. Experience has shown that most organizations have on the average 8 to 100+ SSH keys congured granting access to each Unix/Linux server. Some keys also grant high-level administrative access. The mesh of key-based access is so dense that it is highly like that an attack can spread to nearly all servers in an organization, especially if the virus also utilizes other attack vectors to escalate privileges to root (high-level administrator) after penetrating a server. Implementing SSH keys as an attack vector in a virus is quite easy, requiring only a few hundred lines of code. There is no doubt this will be exploited, soon. Once inside a server, a virus may open a backdoor, steal, alter or corrupt data, subvert encryption systems or databases or outright destroy the server and any data (including backup media) accessible to the server. In the worst case, a virus or cyber weapon using multiple attack vectors could spread globally, Internet-wide, enterprise-wide in a matter of minutes. Combined with destruction technologies it could wipe out on-line data, hot/warm stand-by disaster recovery systems and backups accessible via tape robots and render most servers in enterprises inoperable by wiping their operating systems, storage (including network drives), and reprogramming ash memories, device rmware, and conguration memories. The worst case could mean life with no functioning banks, retail chains unable to process payments or supply inventory, logistics companies unable to operate effectively, most government agencies down, gas stations without gas with reneries having shut down and logistics being very inefcient, no or limited telecommunications and signicant parts of power generation and distribution networks down for weeks or even months.

SSH User Key Remediation

Exposure to Substantial Insider Risks

Improper management of SSH user keys opens a substantial insider risk to the primary information artery of many large enterprises. After studying the SSH user key management operations of a substantial number of the worlds largest enterprises numerous concerning trends have appeared. The following trends appear to be common: Insufcient (sometimes completely missing) controls for who can enable permanent keybased access to a system account (also called functional or process accounts) on a production server Keys are stored and SSH servers congured in a manner that permits anyone logging into a user account to congure new authorized keys for login, effectively granting permanent access even if original access was only limited to a particular administrative task No documentation what each key is used for and why it was created No systematic removal of keys when they are no longer needed (because organizations do not know what each key is used for, and removing needed keys could cause critical applications to fail) No systematic rotation (change) of SSH keys. Keys may have been copied by any administrator having had access to an account holding the private key. They are small enough to be even written on the back of ones hand, or may leak from backups or inadequately wiped decommissioned hardware - and continue to provide access until the authorized key is removed or keys are changed No visibility as to how many SSH keys are granting access to how many servers No visibility as to who can access what servers using key-based access Normal privileged access management / auditing systems can be bypassed using keybased access, opening an effectively uncontrolled, unaudited backdoor into production systems. The insider risks are substantial and involve in some cases hundreds of system administrators, people who can access to backups, people with access to decommissioned hardware, and include personnel and contractors who may have had access to systems in the past in addition to current employees in relevant roles.

Most Enterprises Are Non-Compliant with Mandatory Security Regulations

Most enterprises, government agencies, and health care providers appear to be non-compliant with mandatory security regulations and laws, as well as internal security policies (including in some cases policies mandated by their customers), because they do not properly control, audit, and terminate key-based access to their IT systems and data.

SSH User Key Remediation

All major security regulations - SOX (Sarbanes-Oxley for public companies), FISMA (US government), HIPAA (Health information), PCI (Credit card processing) - as well as NIST standards and equivalent international standards - require controlling access to servers and proper termination of access. These standards are being violated by improper management of SSH keys that allows permanent backdoors to be left on critical servers, bypassing normal controls for privileged access. Key-based access is extremely widespread, and it is the best practice for automated and scripted access to servers. While as safe as anything when properly managed, it turns out nearly all organizations have been sloppy in managing key-based access. Similar problems also abound with other automated access mechanisms, including Kerberos and certicate-based access. The situation is not a result of any vulnerabilities or aws in the SSH protocol itself or in the most commonly used implementations of the protocol (including commercial Tectia SSH and the open source OpenSSH products). Rather, it is a result of years of lack of clear guidelines or policies relating to SSH keys, lack of understanding of the scope and implications of the problem, insufcient time and resources to dig into the issue to gain understanding or develop solutions, earlier lack of good tools and guidelines for solving key management issues, and perhaps a general reluctance of auditors to ag issues for which they dont know effective solutions exist. In addition to this, identity and access management as a eld has focused primarily on interactive user access with little consideration of automated access. The fact of the matter seems to be actually that there are signicantly more permanent trust relationships for automated access than interactive user accounts in many environments. The problem has probably gone under radar because it is deeply technical and obscure, in the domain of system administrators. Each system administrator typically only sees a small corner of the IT environment, and does not have a full picture. Administrators and their managers are often so busy that while they may recognize that there is a problem, they simply have not had time to analyze the scope or possible implications of the problem. Finally, to date, many IT security auditors do not have SSH user key management on their checklist in relation to its role in identity access governance, and the issue has not been sufciently highlighted in implementation and auditing guidelines for SOX, PCI, FISMA, or HIPAA connecting access control to authentication methods. Even many CISOs (Chief Information Security Ofcers) are only vaguely aware of the problem, and many CIOs and IT risk management professionals have never heard of it. Yet at the same time it represents an existential risk for any major bank or enterprise and a systemic risk for the banking system or a developed country.

SSH User Key Remediation

SSH Key Remediation as a Process

Before the problem can be remedied, its existence must be recognized. Since remediation often involves a substantial amount of effort and affects operational processes in IT, obtaining funding, approvals and urgency for change, a remediation project may require the co-operation of IT security managers, IT risk management (including IT auditors), IT/Security architects, IT operations managers, and even higher-level general management (especially since in many cases a remediation project should be started without waiting for the next budget cycle). The problem generally involves all Unix/Linux servers, many Windows servers (especially those using SSH/SFTP for le transfers or scripting, which is increasingly common and recommended), IBM mainframes (commonly using SSH/SFTP based le transfers and remote command execution), and even Mac servers in heterogeneous environments. Thus several teams within IT operations may need to be involved in the remediation project and proper support and endorsement within the organization is needed. Remediation of the problem involves several steps: Discovering all existing trust-relationships (who can access what). This requires discovering of all user keys that are authorized for login (authorized keys) and all private keys (identity keys). This must be done for all users on all servers (and preferably also desktops), and involves several special issues. Monitoring the environment in order to determine which keys are actually used and where the keys are used from (one or multiple sources) and removing keys that are no longer in use Enforcing proper processes for all key setups and other key operations by relocating all authorized keys to a root-owned location and changing SSH conguration accordingly, so that only the key manager (or root) can remove or install new authorized keys, and detecting unauthorized key operations occurring outside the key manager (e.g. keys set up by a root user) and generating alerts about such key activities Automating key setups and key removals, eliminating manual work, human errors, and reducing the number of administrators who need to have sufcient privileges to do key setups from possibly several hundred to essentially nobody (though root can do it, but such setups are detected, and the few administrators with high-level access to key manager can also cause it to create new keys) Rotating keys, i.e., changing every authorized key (and corresponding identity keys) regularly, so that any compromised (copied) keys cease to work and proper termination of access can be ensured Controlling where each key can be used from and what commands can be executed using the key.

SSH User Key Remediation

A further risk mitigation technique is to dene certain internal boundaries within the organization, and strictly control what key-based trust relationships can cross which boundaries and in which direction, and enforce strict IP address restrictions and forced command restrictions at least for authorized keys involving trust relationships crossing such boundaries. It should be understood that while a key manager can reliably discover all authorized keys on the servers that it has access to, it is not possible to guarantee nding all copies of private keys. Some copies of private keys could be on desktops and laptops that are not scanned, on off-line USB memory sticks, backups, powered off machines or even printed on paper for later manual entry. Thus one can never say with absolute certainty who can currently log into an account with an authorized key. Only after rotating all keys can one be sure that previously copied keys cannot be used.

Conclusion
Nearly all Fortune 500 companies and major government agencies appear to be non-compliant in terms of how SSH user keys are managed in relation to access control and authentication. In general the major security issues which organizations consider on a day to day basis are primarily focused in relation to virus risks, hacking risks, and rogue employee risks. Nonetheless, the identity access governance around automated accounts directly correlates to these obvious risks in the most simple terms of cause and effect. Without the proper management of SSH user keys and proper remediation of legacy environments, organizations face signicant increased risks to the above mentioned threat vectors, even existential threats. This is however not an issue unique to large enterprises. Organizations of all sizes face challenges in managing SSH user keys. In fact, most of the next 10,000 largest companies in the world are also exposed to the same issues. On the other side of the coin, even customers with less than 20 servers in smaller security-sensitive organizations can benet from automated systematic key management practices. It all depends on the risk tolerance in terms of managing the access control of the critical servers. In summary, while SSH is considered the gold-standard for data-in-transit security, the current threat landscape requires organizations to rethink how they are managing access to their SSH networks. The SSH protocol has done a superb job in protecting data-at-transit, but effective management and oversight of the access to the SSH environment is now becoming a major concern. With Universal SSH Key Manager, organizations of all types and sizes can quickly and easily take control of their secure shell environment, remediate the usage of their SSH user keys, and meet todays modern security challenges while fullling or exceeding compliance mandates such as SOX, PCI, FISMA, and HIPAA. To get started with SSH key remediation in your environment, please visit: www.ssh.com/remediation or contact sales@ssh.com.

www.ssh.com