Buyers guide: network security at Patelco Credit Union

Patelco Credit Union has been refining its intrusion protection and network access control capabilities to avoid its cash machine network getting infected again.

Patelco Credit Union has been refining its intrusion protection and network access control capabilities to avoid its cash machine network getting infected again.

John Shields didn't want to go through the same problem that his organisation experienced in 2003. The senior vice president and chief technology officer at Patelco Credit Union remembers the chaos that ensued when an employee bought an unauthorised laptop into one of his branch offices and plugged it into the network. The worm that had been hiding on the hard drive spread rapidly to other machines on the network. The most visible sign of damage? Several automated teller machines (ATMs) in the company's network of 46 branches became infected, bringing customer service to a standstill.

 More guides to network security


"The tech guys had to come out and totally reformat the systems," he says. "Since then, the suppliers learned a lot, and they put firewalls on the ATMs themselves, helping to make them more secure."

Combined systems

Nevertheless, he has been looking for a combined network access control (NAC) and intrusion prevention system (IPS) ever since that would help to secure his countrywide network of branches and datacentres. The company recently upgraded to a newer version of CounterACT, an appliance from ForeScout that combines both functions, in a bid to better secure its systems.

The IPS capability in the CounterACT device complements the network access control capabilities in the appliance by blocking attacks based on an analysis of device intent. Called Active Response, it detects reconnaissance performed by self-propagating attacks and responds with doctored information. When the self-propagating attack tries to use this fake Network reconnaissance data, the device takes it as proof of malicious intent, and blocks attackers before they gain access to the network.

Patelco didn't have much in the way of computing technology when it first set up. It was 1936, and it had just $500 in initial assets. Today, it has assets totalling $3.7bn, and more than 290,000 members. It serves the employees of over 1,000 businesses throughout California and the rest of the US. With stakes so high, Shields learned from the 2003 incident, and took on ForeScout's product a couple of years afterwards to protect its systems.

Even after installing its original CounterACT appliances five years ago, Patelco had been using another product to try and assess the patch levels on machines that were plugged into the network, because this was difficult to do with the previous version of CounterACT. However, the other product was too intrusive, and had caused some endpoint devices to crash, he adds.

"We have a lot of different products that we attach to our corporate assets, but we were looking for a solution that would detect something that plugged in that wasn't approved, and isolate those. This product worked really well for that," Shields says.

If someone plugged their own laptop into the network at a branch location, the previous solution may not have been able to detect it. In addition to its IPS capabilities, the new version profiles all of the devices on the network, so that there are no "unknown unknowns".

Patelco has two main datacentres that monitor its 56 locations. The primary datacentre is located outside of Sacramento, California, with the back-up facility in San Francisco. The datacentres have two primary connections. One is an MPLS connection that also connects to the various branch locations, and there is also a point to point 100mbps link. Inside the datacentres, the firm uses mostly Windows Server 2003, although it is gradually moving to 2008. It also runs HP-UX on a mainframe system.

VLan links

Networking-wise, Patelco uses Cisco 6504 switches in the datacentres, which are used to divide the corporate network into VLans, including a dedicated VLan for voice over IP connections to the datacentres.

"We're a hub and spoke network, and all the branches have a connection to both the main datacentre, and to the back-up, via the MPLS network," Shields explains, adding that the CounterACT IPS/NAC device hangs off the Cisco switches to monitor traffic. "So if a branch were to access the internet, it would come back through the MPLS network and out again, and the device monitors whatever is going on."

Although the CounterACT appliance monitors the various devices on Patelco's VoIP network, the company controls access to the VoIP VLan via the firewall, and it rarely takes action on that network because the VoIP system is heavily locked down, Shields explains.

As for the rest of the network, the company largely operates a thin-client environment, but each branch has some "fat" Windows PCs. Shields estimates that each branch has about 50-60 devices including routers and printers. Overall, the devices watch about 2,500 devices on the network.

"The thin clients themselves we're not too worried about, and we've locked down most of the PCs," Shields says. "But the real danger to us is people bringing in their own laptops from home and plugging them into the network."

Asset control

The devices are used for asset control. The CounterACT switch notices when a new device connects to the network. It looks to see if it's a domain computer. If not, it'll check to see whether it's a workgroup, machine and whether it can be logged into using workgroup passwords. If not, it is classified as unknown, and can then put it into a quarantined VLan and granted limited access to the network.

In addition to identifying and isolating unauthorised computers, the system's IPS facility also monitors for suspicious traffic and is able to alert administrators to potential problems. Patelco also drills down to examine network traffic at the individual packet level to look for suspicious activity.

"The things we look for are devices that are trying to talk to multiple endpoints," Shields says. "It also detects SMTP requests, so that we can categorise approved SMTP servers, and issue alerts for devices not in that list that are trying to send SMTP traffic."

Although the company upgraded to the latest version of the CounterACT appliance in a single day, Shields says that the deployment was not entirely without its issues. The company still faces some problems with false positives, he says, adding that his team has to create some whitelists to allow certain devices into the right VLan. "We have some of those issues even with printers of ours that don't fit the relevant category," he says.

The false positives have made Shields wary of giving total control to the CounterACT system. Although ForeScout's literature boasts that 100% of customers block their traffic automatically, he says that he hasn't turned on the feature that can block a device based entirely on the switch port that it connects to.

"It can block machine at the port level. It talks to the ports that the machine is hooked into, so it can send a change to the switch and block that port completely," he says. "However, we haven't turned that on because we worry that there will be some legitimate traffic that it might block," he admits.

Similarly, he has to refine the system so that it doesn't cause unpatched devices to launch Windows Update automatically and update their patch status or anti-virus definitions. "We don't want it to happen automatically on all devices," he says.

Patch management

As with any mature IT operation, change management has to play a significant part in Patelco's procedures. Some mission-critical devices cannot simply be updated with new security patches automatically without being tested, in case it brings down the rest of the system.

ForeScout also lags behind Microsoft when it comes to maintaining a list of patches that should be issued for Windows systems, according to Shields. "ForeScout will update their appliance two weeks after Microsoft comes out with the latest patches," he warns. "We would like to see that window shrink a little bit."

Since upgrading the CounterACT system, the biggest benefit for Patelco has been better asset management, Shields surmises. The latest version of the appliance, which has now been installed at the back-up datacentre in addition to the primary one, detects the current operating system in addition to the patches that have been applied, enabling it to build up a sophisticated picture of the active endpoints on its network.

So, what of the future? The Credit Union still needs to reach the point where the appliance can be relied on to automatically block types of traffic, such as an unauthorised computer, without some form of manual interaction. "We are still being tentative about that, because we don't want to block things that should be allowed," he said.

Still, by gradually updating its appliances to newer versions with enhanced device recognition functions, and by iteratively refining its lists and trusting the CounterACT IPS capability to block more traffic without human interaction, Patelco is hardening its infrastructure.

As we have seen, that has a substantial effect on its customer service - because the last thing a Credit Union wants to have to tell its members is that its cash machines units are infected with a virus. In a world increasingly concerned about cybercrime and online financial fraud, that doesn't give customers a warm, fuzzy feeling.

Best practices

  Don't hand over total control until you're ready

  • Even the most helpful devices can occasionally overstep their bounds, and intrusion prevention is a fine art.
  • Include an element of manual intervention so that you can avoid false positives.
  Refine your rules
  • New threat emerge eveery day, using new exploit characteristics.
  • Ensure that the rules in your intrusion protection system (IPS) reflect the most up-to-date conditions online. 
  Complement IPS with Network Access Control
  • For maximum intelligence, you need to understand what is connecting, along with what traffic you're sending. NAC capabilities can help you to avoid rogue machines connecting to the most sensitive parts of your network in the first place.
  Route all your traffic centrally
  • An IPS is only as good as your routing policy. If you let endpoints punch out to the internet through undocumented, unmanaged routes, then the IPS may not see their traffic.
  • A hub and spoke network infrastructure ensures that your digital sentries see everything. 

  Use VLans effectively

  • Carve your network up into logical segments that can be used to compartmentalise your traffic.
  • Keep low-latency, mission-critical services such as VoIP on their own VLans, so that they will be untouched by traffic from other categories of device.
  • Maintain quarantined VLans that can be used to hold untrusted or suspicious traffic from devices that you don't know.



Read more on Network hardware