Darktrace: Developer tools under constant attack

Skills gap and the ephemeral cloud

A previous Darktrace survey found that skills shortages are one of the biggest cloud security challenges faced by organisations. “If you think it’s difficult to find a good security analyst, good luck finding a security analyst that can also look at the cloud environment and understand cloud attacks and cloud engineering,” said Heinemeyer.

Whether organisations want to hire skilled staff or outsource to a managed security service provider (MSSP), they are facing a huge war for talent, driving the increasing use of automation. “Our tooling and technology should make it so easy that any generalist or any junior person in the SOC [security operations centre] can make the cloud environment secure,” said Heinemeyer.

Part of the challenge is the dynamic nature of a cloud environment. In a traditional on-premise environment, an architecture is designed, implemented, and largely fixed. Changes require a formal request process.

“Cloud is way more dynamic, so things will change whether or not you want them to,” he said. “We’ve got elasticity and new features being rolled out by cloud vendors all the time. We turn them on, we want to play with them and get the benefits, sometimes before we really understand best practices around them and lock them down.”

This means that if you produce a topology diagram, it is out of date within five minutes as resources spin up and down. “That leads to gaps in visibility and gaps in knowledge, so it’s much more difficult to secure,” Jarvis added.

The ephemeral nature of cloud environments also makes investigations extremely difficult, said Heinemeyer. If something potentially malicious happens in a container that disappears within 30 seconds, it is hard for the SOC to understand what occurred.

“You might say, ‘The device is reset so nothing bad can come out of it,’ but if you are in a regulated industry and an auditor asks you, ‘Can you tell me if you had any incidents? What about this alert on your container? Was that an incident? Was that an attacker? Did any personal information get touched?’, your company will be in hot water if it cannot show it had the ability to investigate,” he warned. “That's why we started automating a lot of this plumbing to get the forensic-level data.”

Beyond ‘shift left’

Such issues were supposed to be addressed by DevSecOps, but Heinemeyer argued that most of the attention has gone towards securing the coding environment and the continuous integration pipeline – the ‘shift left’ concept.

“If we look further to the right, where it’s more about alerting, logging, investigation, workflows, you’d usually have different skill sets,” he said. “It’s not so much about reading and understanding code and the Docker pipeline. It’s much more about the SOC investigative workflow, and that’s where things get difficult and the disconnect usually happens. I wouldn’t say DevSecOps is not important, but I think as an industry, we focused a lot on that ‘shift left’ for a long time.”

He suggested that simple team-building activities, such as ‘lunch and learn’ sessions and virtual hangouts, can help bridge the gap between security and operations. Then, when an incident occurs, the wheels are already greased, making it easier for them to work together.

Cloud security should be treated holistically, he advised. During the 1990s and 2000s, IT security added layer upon layer: firewalls, anti-malware, phishing awareness training and monitoring. Organisations should not repeat these slow iterations in the cloud. “We know what best practice is, so implement it from the outset,” he said, adding that cloud security should not be treated as something separate from IT security in general. “Don’t create more silos.”

The SaaS security gap

While much of the discussion focuses on defending cloud infrastructure, software as a service (SaaS) has its own challenges. Heinemeyer pointed out that almost every organisation has adopted products like Microsoft 365, Salesforce, SAP or Dropbox.

“That feels more secure, but it does bring a different set of challenges,” he warned. Instead of worrying about an attacker using BloodHound to compromise a domain controller, the threat is an unexpected login from an unusual location using an administrator’s credentials. He noted a recent uptick in organisations wanting to monitor their Salesforce environments in response to advanced persistent threat (APT) groups targeting the platform.

SaaS makes it easier to consume IT, Jarvis observed, and people tend to trust the vendor’s security efforts. “But we see countless cases where there are vulnerabilities that don’t get picked up until it’s too late,” he said. “Somebody’s in your environment, they've been poking around, and they might have exported data. There is a distinct gap between consuming and feeling safe and actually putting proper protections in place.”

Those protections include not just best practices around passwords, multifactor authentication and passkeys, but also watching for unusual activity. An example would be a Microsoft 365 account login from a country where the company has no presence, especially if the user behaves atypically by changing email rules and creating new folders.

“You’ve got to do the checks and balances and not just try to design it as securely as you can,” said Jarvis. “You can’t protect what you can’t see, so it all starts with getting visibility.”

For SaaS, visibility comes via application programming interfaces (APIs). “We need to get the information out of these systems to be able to check what’s happening inside them,” he said. “The more we can extract, the more we can feed through our machine learning and AI [artificial intelligence] algorithms to start getting an idea of what is normal for a user, account or device.” Any abnormal behaviour can then be examined to determine whether it is an attack.

Security is always about people, processes and technology, in that order, Heinemeyer said, noting that a lot of automation has come to market recently using various machine learning techniques to make life easier for security practitioners. It is not just a matter of bringing more talent to bear, but also of using the right tools, he added.