SSH User Keys: The Unavoidable Audit Point

Matthew McKenna, Chief Strategy Officer, SSH Communications Security
Matthew McKenna, CSO, SSH Communications Security

By Matthew McKenna, Chief Strategy Officer, SSH Communications Security

True story: A developer and auditor are sitting together and the auditor asks the developer, “How do you gain access to your development environment to do your day-to-day work?”

The developer responds, “Well, I am supposed to go through this jump host, which allows me to access the servers I need and it oversees the commands I use and actions I take during my sessions.”

As the auditor steps back, pleased with the response, the developer quietly adds with a smile, “But I don’t go through the jump at all. I use SSH keys and it makes my life a whole lot easier!”

“You do what?” the auditor asks.

“I use SSH keys, so I don’t need to constantly retype my password, can access the hosts I need to, get my work down quickly and move on to the next task I have. Here, let me show you how easy I can set them up myself.”

And the audit flag is red!

The SSH protocol is one of the security industry’s greatest known unknowns. Used as the tool of choice for administrators around the world to remotely access servers and network devices as well as securely transfer data between applications, SSH has been quietly and efficiently doing its job providing encrypted, trusted access for the last two decades. Yet the power it actually harnesses is widely unknown and, consequently, creating a major gap in our identity access postures and risk to the resilience of our enterprises.

There a few things that stand out when it comes to SSH user keys. First, it is the only form of access that can be provisioned without oversight. Second, SSH user keys, unlike SSL certificates, have no expiration dates. So they continue to provide access until they are eliminated from the systems on which they reside. And finally, SSH user keys provide access not only for user-based access but also for service accounts, which move data between applications for critical transactions.

From a compliance and risk governance perspective, there are three major concerns that can easily come back to haunt us:

Unauthorized Root Access

Root access is one of our necessary evils in the world of IT administration. However, when not controlled correctly, it represents the greatest resilience risk. When it comes to SSH keys, here’s the blind spot that exists. When setting up any SSH user key, it is possible to put what is known as an IP Source restriction on a key, which will determine from what IP source the key may authenticate from and what destination server it has access to. Unfortunately, in over 90 percent of the cases, when root keys are being generated, the IP source restriction is simply forgotten. What does this mean? In the worst case, if we have public-facing servers or services and that private key is acquired maliciously or accidentally, it can now access that asset from anywhere.

Preservation of the Segregation of Duties

There are three components related to the preservation of segregation of duties that needs to be considered with SSH use keys.

Bypass of jump host architecture

It’s been said on numerous occasions, when it comes to interactive access related to SSH user keys, “We have this under control. All the users are required to go through our privileged access management solution and are controlled from there.” The issue that is becoming more evident to the security operations is outlined by the developer example above.

As in that example, very frequently jump server architectures are used to control users’ access to a destination server. The challenge here is that a user can generate themselves an SSH user key on that destination server and bypass the jump server in the future. That in itself is would bypass any monitoring capabilities of that jump server and would potentially make actions taken by the user more difficult to track. More concerning than this is if that user has the possibility to elevate privilege from that server and from there deploy new keys.

Non-production environment to production environment connectivity

The second aspect of segregation of duties that is problematic when it comes to SSH user keys is the connectivity from non-production environments to production environments. This happens for both interactive access and non-interactive access. With interactive users, the challenge is closely related to the bypassing of jump server architecture. The user comes through the jump host to the non-production environments as usual, uses tunneling (a sub-channel of the SSH protocol) to tunnel across to a production server and from there may drop in a new key pair, which will permit access again directly from their desktop to a production environment. Now, this is a real red flag in most regulated IT environments, but it happens much more frequently than you may expect.

Misuse of transitive trust and agent forwarding capabilities of SSH

The concept of transitive trust is a question of how deeply a single key or user may be able to penetrate the infrastructure utilizing key-based authentication. Again, when SSH user keys do not have IP source restrictions or forced commands dictating what a user may do during a session, it becomes a question of how far that user can continue to gain access within the environment with their privileges. So in this case, the user may access one server, elevate privilege drop in new keys and gain access to another server. From there, they may use the sub-channels of the SSH protocol to create connectivity back to their home desktops or other servers, bypass corporate firewall policies and exfiltrate critical data without being noticed. 

Monitoring of keys for trusted activities

One of the most important aspects related to solid SSH user key and access management relates to the monitoring of key-based authentication. All privileged access must be controlled and monitored according to the majority of compliance mandates; however, key-based authentication is ignored, as is where SSH user keys are authenticated from – and how frequently. Fortunately, this information sits at our fingertips and as long as VERBOSE logging is enabled on the SSH server, the logs for key-based authentication can be easily captured. This information is essential in us gaining control of our environments, whereas without key-based activity monitoring, we are unable to assess whether keys are obsolete or not. Remember, SSH user keys do not have expiration dates, so you may find up to 90 percent of the keys in your environment are no longer being used.

ShapeShiftIf you are still not 100 percent sure that SSH keys will become a burning audit point sooner than later, then please read Looting of the Fox: The Story of Sabotage at Shapeshift, by CEO Erik Vorhees. It is a great down-to-earth, practical application of everything in this article regarding how SSH user keys were misused to steal from his company. These keys represent one of the soft underbellies of infrastructure and can no longer afford to be ignored if organizations want to keep their data safe and remain compliant. 

About the author

Matthew brings over 10 years of high technology sales, marketing and management experience to SSH Communications Security and is responsible for all revenue-generating operations. His expertise in strategically delivering technology solutions that anticipate the marketplace has helped the company become a market leader.

Prior to joining the company, Matthew served as a member of the executive management team of Automaster Oyj, which was successfully acquired by ADP Dealer Services Nordic. Before this, Matthew played professional soccer in Germany and Finland.

Matthew holds a BA in German from the University of South Carolina and an MBA from the Helsinki School of Economics and Business Administration.