Accounting, communications, staffing and the website were among the areas that faced massive strains as Oxfam's IT operations responded to the tsunami crisis. Julia Vowler reports on how the charity's technology and IT team overcame the obstacles
On 24 December 2004 Simon Jennings, Oxfam's IT director, was doing a final pre-Christmas check of his department when he bumped into the charity's humanitarian officer, who was to be on duty over the holiday weekend. He gave the officer the IT department's call-out number so it could respond quickly in the event of an emergency.
Within 48 hours Oxfam was playing a major role in responding to the Indian Ocean tsunami - the biggest natural disaster in modern history. As an international aid agency, Oxfam is familiar with humanitarian disasters on a massive scale. It is only a year since the earthquake at Bam in Iran killed almost 30,000 people, and the conflict in Darfur and the famine in Chad are continuing.
Oxfam is operating 37 emergency response actions in 71 countries; work that is critically supported by IT. The charity is divided into eight regions, plus a headquarters in Oxford. Each region has one or two IT support managers who are responsible for at least six countries. Each country has a full-time local IT support officer on the ground. Altogether there are 27 international IT staff, all supported from Oxford by James Glisson, the international network and support manager. There are 70 IT staff in Oxford who support 1,500 overseas staff and 1,300 staff in Oxford.
When news of the tsunami reached Oxfam on Boxing Day, the IT department's first priority was to ensure that the charity's tsunami emergency programme, which was set up immediately, would be able to do its work.
"What made it so difficult was the scale and geographical reach of the disaster. It presented a huge challenge to IT," says Glisson.
Jennings says, "Oxfam had to assess what was needed and how to respond to the disaster, and an IT component was built into the subsequent plan." The biggest immediate problem was communications. Oxfam's response to a disaster is a huge logistics effort: without communications, it would be crippled.
"Our philosophy is to join with the humanitarian staff to determine what the communication needs are," says Jennings. In the case of the tsunami, these varied in different parts of the affected area depending on how badly they had been hit and how good the existing communications infrastructure had been.
"I contacted our IT office in Indonesia on 27 December," says Glisson. "Before we knew the extent of the damage in Aceh, we set up a new site in Nedan for logistics and IT."
In Aceh communications had been all but wiped out. "All that was working was an extremely patchy phone connection. A local ISP offered us wireless connectivity and we put up a mast with a 128Mbyte wireless switch which gave us one e-mail account for 16 people - and no attachments allowed," says Jennings.
To improve the situation, says Glisson, "Oxfam helicoptered in a VSAT (a satellite communications base-station) from Jakarta to Nadan. But Nadan airport was closed to freight - a plane had hit a herd of buffalo - so we ended up convoying the dish to [its destination] Nulabo."
"The critical factor was to provide 24x7 phone coverage and to remain flexible. For example, when Nadan airport closed it was essential that we were able to link the logistics manager in the UK, who was co-ordinating the whole relief programme, to the logistics people on the ground who had the local information about what was needed. We also had to be able to speak to our IT people on the ground who needed support."
Psychological support was just as important for the ground IT staff in the affected areas. "The IT people were working l6-hour days. I asked one what he needed most and he replied, 'sleep'," says Glisson.
Jennings says, "There is a human risk with a sustained effort over a long time. Our guy in Indonesia was shattered and we had to get him out."
Glisson says, "It was more than he could handle, so we sent him back to Jakarta. We also sent our East Asia regional manager, who had come to Indonesia and was also fatigued, back to his base in the Philippines."
IT officers from other Oxfam regions had been brought in to support overworked staff in Indonesia. Glisson sent his deputy because he is a highly experienced engineer.
"We designed the IT so that the officers who support it can be interchangeable," says Jennings. But although that meant the Pakistan officer, who had offered to be on standby for Indonesia could be called on, it also meant Oxfam had to field someone to take his place in Pakistan; and the regional manager also could not do his normal Philippines work.
There was no shortage of IT staff wanting to help. Glisson says, "Our West African manager offered his entire team, but we wanted as little disruption as possible to other areas." So, the IT support officer for Mexico came to Indonesia and Glisson's on-site deputy also recruited an IT assistant. "The IT assistant is a new post. We had no time to train him in the UK where the officers are trained, but it meant that more senior staff could be freed up," says Glisson.
Although Oxfam lost no staff in the tsunami, Jennings says, "We had two offices washed out in Sri Lanka and many staff lost their homes and possessions - their personal commitment to us at this time has been phenomenal."
As well as being under such great personal stress, working conditions were grim. "As any technician who does site installations knows, inevitably they will find they are missing a cable or some part, which they will get from a local shop," says Jennings. "But the guys we sent out sent us back a picture of their local computer shop - the downstairs had been completely washed out. There was no question of 'nipping out for a spare part'. They were trying to set up a new office with printers and PCs, but it all had to be shipped in."
As well as dealing with the overseas IT requirements created by the disaster, there was also an impact on the home front IT. For Jennings, this was where good preparation paid off. "We had our processes and procedures in place," he says. Even so, the UK IT staff were pulling out all the stops.
"On 27 December our HR system briefly went down and we got it back up in 14 minutes - that was our only datacentre activity between Christmas and New Year. We had no maintenance work to get in the way of our tsunami response," says Jennings.
Nor could he afford any hitches in the systems supporting Oxfam's warehouses. "Our warehouse in Bicester had £1.7m worth of emergency stock - from buckets to hygiene kits - which had to be put on charter aircraft and sent to Sri Lanka and Indonesia. The inventory and replenishment systems were crucial," he says.
However, the UK IT system facing the biggest challenge was the fundraising program on Oxfam's website. "We have been taking donations online since 1997," says Oxfam IT project manager Andrew Hatton. "Since the Iran earthquake 12 months earlier we have had a set of procedures that mean we can respond very quickly. We ran through them just before Christmas as a routine dry-run." It was just as well.
"We went live with our tsunami disaster appeal at 2pm on Boxing Day - just hours after the UK heard about the event," says Hatton. "As soon as Oxfam had decided to set up the appeal, any of our IT staff with access to the web could log in and set up an online appeal within minutes."
Donations were also coming in over the phone. "We usually have 10 seats at our donations call centre, and suddenly we needed 50," says Jennings. "We needed technicians working on the switchboard and setting up desktops so staff and volunteers could man the phones from 10am to 10pm. An awful lot of effort went into making it work."
As for the website, transaction levels soared as money came in online. Oxfam hosts its website externally on open source Red Hat Apache servers which can be load-balanced to provide extra capacity. The problem was, given the overwhelming public response to the appeal, just how much extra capacity would the site need?
"The system was designed to be scalable, but it was pretty nail-biting as we did not know when or where transaction levels would peak," says Hatton. The numbers, when they came, were immense.
"Oxfam was the most hit [tsunami appeal] charity website after the Disaster Emergency Committee website," says Hatton. "On our busiest day, 29 December, we had 100,000 visits - that's 10 times our normal level. And every visit was transactional at a 1:1 ratio. Everyone who hit the site gave money. We received as much money on that day as we get for a normal year's donations excluding disaster appeals."
Many donations Oxfam received for the tsunami appeal came in over the web, but the money also had to be collected. "We worked closely with our online payments partner, WorldPay, which we contacted on Boxing Day," says Hatton. "It had been scaling down its overall capacity because the Christmas peak was over, but immediately brought in new capacity for the online donations."
Incoming monies then had to be recorded, a process made more essential because Oxfam must clarify whether raised funds are restricted or unrestricted. Unrestricted funds - general donations - can be spent on any of Oxfam's work, but funds raised specifically in respect to the tsunami could only be used for that purpose. Oxfam's financial control systems had to record them accordingly, says Jennings.
Before the tsunami, "we used to re-key online donations into the legacy finance system, but now we suddenly had a mountain of transactions to process", says Jennings. "An event like this sorts out priorities. Writing an interface to the legacy system had always been promised, but someone just sat down and wrote it in two days flat - which shows what can be done."
Although Oxfam has now closed its tsunami appeal and its work in the affected areas is becoming "business as usual", it remains, because of the sheer scale of the disaster, more than just normal business, says Jennings, Online donation levels, for example, did not return to their pre-tsunami level but have stayed a few points up. "We have a new normal. It has shown that online funding really works and is here to stay," says Hatton.
For Jennings, the experience has demonstrated that the £4.5m a year he spends on IT has proven to be critical in enabling Oxfam to respond to tragic events.
"The biggest challenge is not so much the technology as the people. It is the personal effort that goes into all of this - the commitment is phenomenal."
Oxfam's IT infrastructure
- Oxfam spends about £4.5m a year on IT, employing 70 IT staff in the UK and 27 across the world, supporting 1,300 staff in the UK and 1,500 staff in the 71 countries where Oxfam works
- IT governance is at board level, with IT director Simon Jennings reporting to Oxfam's head of finance
- Oxfam has a global virtual private network, 80% of the UK offices are on ADSL, and an 8Mbyte ISP pipe connects the internet to the VPN carrier
- General business management packages include PeopleSoft 8 for finance, HR and customised project management of its 2,000 projects around the world, plus warehousing software for Oxfam's two warehouses: one for emergency supplies and the other for its trading business and retail shops
- Oxfam has been moving from client-server applications to web-enabled applications, which allows more centralised rather than local applications, increases standardisation and reduces support and licencing costs
- Systems unique to the charity sector include fundraising and donor management systems
- Oxfam's website is hosted on Apache Linux servers
- Oxfam also uses its IT to generate revenue directly - its Festival System is used at Glastonbury to schedule the festival's stewards, for which Glastonbury pays Oxfam a usage fee.