Managing a corporate network to plan the Olympics

The race to be ready for the next Olympics, networks dependent on car batteries, and protecting security conference networks against hackers out to prove their skills. Just three of the challenges facing network managers with a taste for the ultimate in challenging environments.

Take Sheng Jiang, for example. He has a lot to keep him up at night. Jiang works for outsourcing firm Atos Origin, which is responsible for building the computing infrastructure for the Olympic Games in Beijing next year.

"It is a very complex project," says Jiang, who is technical services manager for the event. "My challenge is that we have a lot of different stakeholders." Network equipment suppliers, owners of the different venues and network auditors all have to be dealt with.

Jiang has been in Beijing planning the infrastructure for the Olympic Games for the past two and a half years. The games will have three datacentres - a primary and secondary, and a technical operations centre that will be used to manage the technology operations during the event. Jiang has to manage 130 Unix servers, 1,000 Intel servers and 16,000 PCs.

When a competitor passes the finish line, even in competition venues 800km outside Beijing, the team must show this data on screens in under one second.

The timing and scoring system for the Olympic Games is used to deliver data to the information diffusion system in charge of processing the information and deliver it to the internet data feed, which gets it to the official website.

For the broadcasters who play such an important part in the games, a common data information system sends information in real time to an international broadcasting centre, which takes feeds from 40 competition venues, seven of which are outside Beijing.

In the Athens and Turin games, where Jiang learned the techniques he is applying now, systems delivered results in under 200 milliseconds.

"You only get to do this kind of project once in your whole life," says Jiang. "It is very stressful and you get a lot of grey hairs." He points out that there is no second chance, and you realise again the magnitude of what he is doing.

At least Jiang has time to plan, which is a luxury that Emerson Tan does not enjoy. Tan does not dress like your average network manager, but then again he does not work in a normal environment, either.

Tan wears an old military assault vest filled with everything from pocket knives to cable crimpers. You will often find him or a colleague piling together rubbish on a roof to try and create a line-of-sight platform for a small satellite dish, before trying to thread a network cable from it down through a top-floor window.

Tan is a technical expert who flies into disaster-stricken places with MapAction, a charity providing geographical data to disaster relief groups working in the area.

One of the biggest problems facing aid groups in areas like tsunami-stricken Sri-Lanka or earthquake-ravaged Pakistan is that it is hard to work out who needs how much aid, or where they are.

The 2005 tsunami saw one village receiving copious assistance while another village just a few miles down the coast received nothing. The aid follows the TV cameras, Tan says, and like them it does not reach all the right places.

Tan, who works for one of the big four management consultancies and works with MapAction on a voluntary basis, flies into disaster zones laden with IT equipment.

He sets up operations centres, both at central UN disaster coordination sites in major cities, and in the field in the midst of the disaster zone.

The centres support geographic data specialists that venture out to gather information about the injured, as well as local conditions. Information is brought back and fed into a geographical information system donated by software supplier ESRI, where it can be turned into maps to give aid workers better intelligence.

Building networks brings unique challenges, says Tan. "All of your problems come from the environment," he says. Everything that can usually be taken for granted - such as reliable power - disappears. "You have to create an environment where things work. It is all about improvisation and repurposing," he says.

The improvisation starts before he boards the plane to the disaster area. A combination of strict weight requirements and limited budgets forces Tan to modify equipment before it is transported.

"People do not design 802.11g routers with batteries in them, for example. So we have to build our own systems," he says.

On the other hand, heavy batteries have to be removed from UPS equipment and replaced with terminals to make them transportable. Then, when Tan is on the ground, he has to find a car battery to hook it up to. All while negotiating with local bureaucrats for scarce resources such as DSL lines.

Another difficulty is that geographic information systems software is very processor-intensive, meaning that only small, lightweight, high-performance computing equipment will do.

"You could not take a 1u server on a mission because it is about 19 kilos," Tan says. Currently, Tan is working on squeezing a Windows Server 2003 onto a dual core Intel-based Mac Mini using the Parallels virtualisation software.

Such innovations are what building technology in a hostile environment is all about. Because MapAction flies people out on as little as six hours' notice, Tan and his colleagues will sometimes be among the first people on the ground.

If he is there before anyone else, he will set up a printer in the airport arrivals lounge and start printing out maps for the aid workers to use as they arrive, while others go on ahead. There is no supplier certification for this stuff - much of it is based on using your initiative, and working with what you have.

Will Whittaker also gets to travel across the world. As organiser of several security conferences, he gets to travel to London for EUSecWest, to Asia for PacSec, and hangs out in his Vancouver home town for CanSecWest.

Setting up networks for security conferences is stressful at the best of times, because security professionals like to rattle the doors on a network more than most. It is even more stressful when dealing with hotels, whose network configurations may lack sophistication.

"We have to ensure that whatever happens in the conference network does not spill over," Whittaker says. "In Japan, the abuse in our network spilled out to the point where the upstream fell over. They were just flooding it with network traffic."

Typical attacks include spoofed wireless access points. The network is also awash with port scans and probing of clients and hosts, he says.

Whittaker does his best to police the network, to the extent where he will take a spectrum analyser out into the conference to try to track down wireless users who are misbehaving. "You pull out the directional locator and instantly the action stops," he says.

Another common attack is packet sniffing clear text passwords to access services from the network. But then, you might reasonably argue that such people deserve what they get.

"Our policy is that this is a hostile network and you are a security professional and you should be able to defend yourself," says Whittaker. "Consequently, we have some government agencies who now will not bring their laptops to the event."

Imagine such a network where a small percentage of the community has the technical knowledge and motivation to play a lot of time- consuming pranks. Now, scale it up to internet proportions, where you are a single organisation with a lot of technically savvy opponents determined to cause you real trouble.

This is the problem facing Spamhaus, a voluntary organisation that spends its time gathering intelligence on the spammer community.

"If you are committing crimes that no one likes, you do not want to be outed," says Spamhaus volunteer John Reid.

Spammers who control thousands of compromised machines (bots) around the world will direct large numbers of PCs to attack the organisation's website. The spammers often use a straightforward SYN flood attack, a type of denial of service attack where the servers get hammered with network traffic from machines distributed across the world.

The Spamhaus website received a spate of attacks in 2003, and was attacked again in September last year, bringing it down for several hours. Luckily, says Reid, the crown jewels of the Spamhaus operation - the realtime blacklist that ISPs use to block spam - is well protected.

"The blacklist is in so many mirrors around the world in different areas that it does not affect that. The billions of queries that we get each day would look like a distributed denial of service attack in itself if you did not know what it was," he says.

Spamhaus calls on organisations such as NeuStar Ultra Services when it comes under attack. Formerly called UltraDNS, the subsidiary of US clearing house service provider NeuStar is authoritative for about 25% of all internet domains in the world, says CTO Rodney Joffe.

It also provides pro bono services to Spamhaus, letting the non-profit organisation's users conduct DNS lookups for the mail that they receive, which helps it to gather intelligence on spammers.

Joffe is the ultimate escalation point for a "tiger team" of network professionals at NeuStar who move quickly to try and mitigate distributed denial of service attacks against their critical infrastructure.

Small, localised distributed denial of service attacks are occurring almost all the time, says Joffe, but the large attacks often tend to happen late at night on weekends. "A number of hackers are in school in the middle of the day, and if they are in school they do not get to see much.

"They tend to launch attacks when it is convenient for them and they can stay up late, and that tends to be at the weekends," he says. There is also an assumption that organisations may have their guard down at the weekend.

Just as Spamhaus helps protect its blacklist files by mirroring, NeuStar helps to protect its DNS records by replicating them to several large network partners, so that if it comes under attack others can continue to resolve names.

NeuStar also works to mitigate the problem by dedicating a team to gathering intelligence. The team harvests information about hacker activities, for example, to help predict when and how such attacks may occur.

But much of the time, attacks will come from nowhere and the network has to be ready to react. "We never ever stand the tiger team down," says Joffe. "They are on call 24x7 for a reason."

Doubts persist about open source VoIP

www.mapaction.org