Perfect storm for data science in security

Endpoint detection and response systems

Now, it’s finally possible to see into networks at high-resolution and at large scale, as well as capture the data using endpoint detection and response (EDR) systems, he said.

“That has given us the visibility and data we needed, and now we also have the cloud infrastructure needed to analyse that data. All these enabling technologies had to become available in parallel to enable us to be effective [with data science] in ways that were not possible in the early 2000s.”

The intervening years, said Neil, were difficult for those who had realised what data science could achieve if applied to cyber security, but who had no means of doing so yet.

“It took a lot of patience for those of us who had seen where we needed to get to, to wait for that technology to come along and mature enough, but now it is so exciting,” he said.

Joining Microsoft 18 months ago, said Neil, was an important step in his data science career. “Finally, after 20 years I am at a company with the scale, enterprise visibility and cloud computing that I need to do my job and have the impact that I knew would be possible [with the right tools].”

As a result, Neil said there is much more recognition among enterprises of the potential of data-driven approaches to security, but this is not equally well understood by all.

“This approach is best understood by high risk industries, such as the financial services industry, where some of the largest firms have even started setting up their own cyber data science teams that are collecting and analysing the data to help improve cyber protections.”

However, in many cases, Neil said these organisations realise that this is a challenging and expensive thing to undertake on their own or they are not realising the full potential of data science for cyber security because their data scientists often new to cyber security and tend to work in isolation.

Applied correctly, he said, data science is helping to improve cyber security capabilities both pre- and post-breach.

“In the pre-breach context, data science is making a huge contribution in enabling organisations to identify malicious code by using supervised machine learning technology to extract behavioural features from suspicious executables run in isolated environments.”

This approach means organisations are no longer relying on signature-based malware detection, making it much more difficult for malware authors to evade detection.

Anomaly detection

Post-breach, anomaly detection is among the most successful applications of data science, said Neil. “Once attackers are inside the enterprise, they look like users. They are using valid credentials to access systems and data, and they are stealing that data using built in system tools, making it difficult to detect.”

Anomaly detection, he said, uses self-learning models designed by data scientists to understand “normal” behaviour inside the enterprise.

“This can be very high-resolution. Every user’s credential behaviour is modelled and the behaviour between every communicating computer on a network is modelled, creating hundreds of millions of models for any given enterprise to identify anomalous – and potentially malicious – activity.

“Supervised machine learning pre-breach and anomaly detection or statistical methods post-breach are the two big areas where I think we have made the best contributions in terms of detection,” said Neil.

Automated methods

Another key contribution by data science is in describing the extent of an attack as well as possible through automated methods. “Detection and response go hand in hand, and so the more we can detail the extent of an attack in terms of detection, the more we can accelerate the response.”

Data scientists are also working in the field of automated response, but Neil said in this regard, it is “still early days” and automated response remains highly dependent on detection capability.

“You need to be very sure of your detection before you start shutting machines down because a False positive here is quite expensive for the enterprise, so this is a real challenge.

“However, progress is being made, and Microsoft has some of these automated response systems deployed. But we are very careful about this. Automated response is a very long-term goal. Regardless of the hype, it is going to take us years to realise this fully.”

That said, Neil believes a lot of the manual, human-driven cyber attacks by teams of well-funded attackers will start to be replaced. “I think we are going to start seeing attackers using automated decision making.”

This in turn will create the opportunity for defenders to write their own attack bots that can be used to fine-tune their automated defences. “We can play this game of attack versus defence before the adversary does.”

Eventually, Neil believes artificial intelligence (AI) bots will be used in both defence and offence without a lot of human involvement. “This could be a blessing or a curse for defence, but one thing for certain is that the state of things is changing and it’s changing very fast,” he said.