A faster, stronger, redundant Olympic network

A steady stream of taxis grinds up the hill to the headquarters of the Athens Olympic Committee on the northern edge of the city....

A steady stream of taxis grinds up the hill to the headquarters of the Athens Olympic Committee on the northern edge of the city.

In the lobby it's all bustle as visitors mill around the accreditation desk and pass through security controls.

But on the second floor the glass-walled technology operations centre sits idle - most of the 135 seats in the control room are empty, and all but one of the screens on the video wall are dark.

There is plenty of work going on behind the scenes, though, with integration still to be completed at some of the smaller venues. And then there is testing: lots and lots of testing.

Claude Philipps, program director of major events at Atos Origin, the lead IT contractor for the Olympic Games, likes to be prepared. "We are ready, but we are still testing, because we want to be sure that every stupid thing that can happen is planned for," he says. "In a normal IT project we could have delivered the application to the customer almost eight months ago."

But the Olympic Games is far from a normal IT project. The deadline is non negotiable, and there are no second chances. Everything must work, from the opening ceremony on 13 August right through to the end.

With all that pressure, Philipps' team is doing its utmost to ensure that the network will not fail. Engineers and programmers are building multiple layers of security and redundancy, using reliable technology, and then testing it rigorously.

Creating a team

In the build-up to the games the team went through two technical rehearsals in which 30 Atos Origin staffers put the network through its paces. They spent a full week simulating the busiest days of the games, Philipps says, dealing with "crazy scenarios of what might happen in every area: a network problem, staff stopped in a traffic jam, a security attack - everything that might happen".

The rehearsal tests people and procedures as much as products. That is important because the IT operating organisation Philipps is building will have grown from nothing to a staff of 3,400 in less than three years. Many staff members are volunteers who train at evenings and weekends to deliver first-line support.

Philipps is getting used to this boom-and-bust cycle of team building, having worked on the event since the 2000 Olympic Games in Sydney. Some of his colleagues can trace their involvement further back, because Atos Origin now owns Sema, which has been developing software for the Olympics since the 1992 competition in Barcelona, Spain.

The two major components of the software that will run on the Olympic network are Atos Origin's GMS (Games Management System), a customised suite of applications that acts as kind of ERP for the Olympics, and the IDS (Information Diffusion System).

GMS will run on Windows 2000 servers in Athens, an upgrade from the Windows NT 4 used at the Salt Lake City games in 2002. "We're not using sexy technology," Philipps says. "The main goal for us is to reduce the amount of risk."

The IDS collects and distributes event results and rankings. Press agencies such as The Associated Press and Reuters get a dedicated feed from IDS, as do certain websites.

Philipps expects the system to deliver 50 million pages of reports to Olympic partner Xerox for printing during the games, largely for those print journalists at the press centre in Athens who still prefer information on paper.

The IDS also serves broadcasters. "We provide a live feed for TV and radio broadcasters [who] are commenting live on the event. This is a real-time system that provides them everything so that they can look smart," Philipps says.

A fail-safe plan

Together, GMS and IDS impose exacting requirements on the network. GMS is used to manage accreditations for the games, so security is vital. Speed, too, is important: Philipps' goal is to have the result on commentators' screens 0.3 seconds after the athlete has crossed the line, complete with rankings, statistics, and biographies.

Yan Noblot, information security manager at Atos Origin, says the key to that is to build in redundancy - and lots of it. "We have doubled everything, because we need 100% availability at games time," he says.

And when he says everything, that goes for the routers and switches at each site, the datacentres that process the results, even the PCs on the desks in the control room.

An SDH (Synchronous Digital Hierarchy) network composed of two STM-1 rings supplied by Greek carrier Hellenic Telecommunications Organisation (also known as OTE, an acronym based on its name in Greek) links the Olympic venues at 155Mbps. "We only use one ring, the other is for redundancy," Noblot says.

The 36 competition venues and 20 or so non competition venues are linked by fibre to the SDH ring. All the venues are connected by two different routes, with a ring for each venue, and traffic from each venue is served by two different telecommunication centres, according to Elpida Trizi, a spokeswoman for OTE. "We have constructed the network in such a way that we are able to provide a service even if one of the routes is damaged," she says.

Event results and data from the games management system are stored in two datacentres hosted by OTE, which also supplies the SDH network. The primary datacentre is located near OTE's headquarters in Marousi, just across the main highway from the Olympic stadium; the other is another several hundred miles away, still in Greece but in a different earthquake zone.

The datacentres sit directly on the SDH ring. "One reason we did that is because we do real-time replication between primary and secondary datacentres," Noblot says.

That direct connection is made through a pair of Cisco Systems 7200 series routers. "We have two of each, at least, for redundancy, configured in such a way that the traffic would be automatically rerouted," Noblot says. "Behind this we have two Catalyst 6513 layer-3 switches running services. We use it to do VLan routing. We also have a firewall and intrusion detection system in the same chassis."

To keep things orderly, Atos designed three different Lan configurations: one for the largest venues, including the Olympic stadium and the water sports centre; another for midsize venues such as the equestrian centre; and one for the many smaller venues.

Each venue has a pair of access routers, with models depending on the venue size, and behind that a pair of switches to send all the traffic to the intrusion detection system. After that comes the distribution layer, a pair of switches for VLans and routing between the VLans. "The very last layer is the access layer where we have Cisco switches that plug directly to the servers and the PCs," Noblot says.

Segmenting traffic

Atos is using VLans both to simplify troubleshooting and to limit damage if anyone manages to break into the network. There are separate VLans for the commentator information system, information diffusion applications, and the game management system. Technical services, directories, management and monitoring, and the on-venue results system each have their own VLans too, sometimes several per venue for the same function.

"The purpose is to segment the traffic so we can monitor it and contain potential issues," Noblot says. "If someone brings in a virus, that would be contained on systems on the same VLan."

Software distribution is another service secured over a VLan. Atos is using Symantec's Ghost and software from LanDesk Software to remotely load software on to PCs.

Anything that can avoid the need for a time-consuming journey to make hands-on repairs is welcome, because the service level agreement specifies a 10-minute resolution time for severe problems. "That's not to take ownership of the problem, but to fix it," Philipps emphasises.

What makes the Olympic Games a unique IT project is that the athletes are not going to stop running just because the server has. As Philipps says, "When we speak about fixing something, it might be a work-around, a decrease of functionality, but the key thing is that the show must go on."

Agam Shah writes for IDG News Service

Read more on Business applications