There are many ways attackers can try to infiltrate an enterprise, but many times enterprises make it so easy that the attackers don’t have to try too hard. Consider the current state of orphan SSH (Secure Shell) keys and how these keys represent one of the biggest risks in the enterprise.
These keys are a cryptographic network protocol for operating network services and are used for system to system automation and authentication, application integration, system management and other common functions. Should an attacker get ahold of these keys, they could find it very easy to burrow their way deeper into the network.
To better understand the state of SSH security, or insecurity, in the enterprise, we turned to the inventor of SSH, Tatu Ylonen chief executive officer at SSH Communications Security, and author of US National Institute of Standards and Technology Internal Report 7966, Security of Interactive and Automated Access Management Using Secure Shell (SSH), and several Internet Engineering Task Force standards.
Here’s our conversation.
CSO: As you see it, what is the general state of SSH key security within enterprises today?
Ylonen: We’ve worked for five years with half of the top 10 banks in the U.S., U.K., Germany and several stock exchanges; some central banks. And we are finding anywhere from several hundred thousand to several million SSH keys granting access to their servers. I don't know how familiar your readers are with SSH. Basically SSH Keys operate like passwords and provide access, but without the need to enter anything. And we are finding, typically, anywhere from 50 to 200 keys per server. This is in traditional enterprise environments with tens of thousands of servers.
For instance, in one bank we found 3 million keys. Another bank found 4.5 million keys. The fewest we've seen in the Fortune 500 companies that we’ve looked at has been in the hundreds of thousands of keys. It turns out these keys are almost always unused. In fact, something like 90 percent is unused, in most cases. Something that was from years ago but never removed when the people left.
And SSH keys are the only credential that users can still provision in a default configuration.
Are they mostly server and system admins who are creating the keys? Do you see poor implementations in regulated industries, such as healthcare?
It's mostly server and system admins. But also developers and database admins who want a quick, easy single sign-in access to of all their database servers. They create this convenience to do their jobs more easily, but it violates policy. And it creates a lot of risk because these keys accumulate and remain valid forever.
In fact, in one of our engagements at a healthcare organization, we tried an SSH key that was created 10 years ago and it still worked. We are finding keys that are 16 years old, in some of the biggest banks and some of the very highest profile technology companies in the U.S.
Considering all of that, how can SSH keys be leveraged by attackers?
These are effective to spread laterally and more deeply into an organization. When you get to one machine, you can use that to jump to another machine, to another, and so on. And given that there are typically about 50 to 200 keys granting access to each server, the probability that if you break into a server that you would be able to find private keys from that server and use those to log into a few other servers is high.
These keys are used for many critical business functions, such as backup recovery for data centers, server to server communications, system integration automation, and crucial business transactions. That’s why enterprises basically have to close their doors if SSH stops working. There are too many dependencies. At one bank we had a case where they authorized the removal of an SSH key, and suddenly the phone started ringing and some major transaction wasn't going through. So this isn’t as easy as just starting to remove keys.
Why do you think SSH Key security has gone unmanaged for so long?
This is about awareness. If you think of access management, it's been about usernames, passwords, and two-factor authentication for people. And access between systems and servers is an automated process and has been under the radar. It's not really been something that CSOs or security architects, generally, have been aware of. Or, they haven't understood the scope of the risk. So it's been accumulating for the last 20 years.
Still, isn’t this something you’d think internal security teams, or external auditors would know to look for?
We've been trying to educate them for the last several years, but still: most people don't really seem to understand the extent of the problem. Or what you can do to an enterprise with loose SSH keys, and we have hackers coming in all the time. And with these keys they can continue to attack silently, and unnoticed to other parts of infrastructure. The initial attack could be an insider. It could be something penetrating through using malware, or through e-mail using malware. And they’ll find a key and move deeper from there.
We have seen attackers buy keys. We have seen cases where SSH keys were being sold by insiders.
This seems like a tremendous risk for those organizations that don’t work to actively manage these keys. It’s something that could potentially deny access to many crucial IT services, or be used to clandestinely siphon data.
If you can infiltrate and get a hold of the keys, for instance, you can not only access the database but potentially modify the database, bypassing the actual database software to modify the files on disk. Or you can start reprogramming firmware and destroying backup volumes, and bring the whole enterprise down. And if you think of cyberwarfare, the whole idea is that you find access, you penetrate and you maintain that access. Remain secret. And then, if and when a decision is made to really disrupt something then you are able to do that very quickly.
There’s no silver bullet to remediate this. It's not really a technical problem. It's not a vulnerability in itself. It's a management issue. Enterprises can establish processes for terminating or removing keys when people leave. And they can establish processes for revalidating or reauthorizing access between applications on a regular basis.
That's pretty much the essence of it. There needs to be a controlled provisioning process, which involves taking the configuration so that users can still provision some tools to automate that process of monitoring and revalidation. And scanning the infrastructure, and removing all the unused keys. It all starts with creating a policy to manage the life cycle of SSH keys, and then enforcing that policy. Unfortunately, many enterprises don’t even have that policy.