When an IT environment spans the vast, complex landscape of state
government, it's nearly impossible to stop every threat from
cyberspace and the physical world.
Take the state of Delware's IT environment, for example, where a
network comprised mostly of Windows machines serves the needs of
49,000 employees, in addition to thousands of citizens and a
variety of different agencies. It would be bad enough if an online
attack against one agency rippled across the state network or if a
disaster in one municipality disrupted the flow of digital
resources throughout Delaware, but what if the state network were
engulfed by a confluence of incidents all at the same time?
As far as state officials at the Delaware Department of
Technology and Information (DTI) are concerned, the best way to
prepare for chaos is to spring the worst-case scenario on employees
on a regular basis by way of disaster drills.
The likely threat
DTI held the first such "tabletop exercise" last October, along
with the Delaware State Police, the Federal Bureau of Investigation
(FBI) and the Delaware Emergency Management Agency (DEMA). Other
drills are happening "all the time" within the central IT
department, according to Elayne Starkey, the State of Delaware's
CTO.
The ultimate goal is to get everyone thinking about what they'd
do in the face of a massive security incident, she said, so they
can spring into action if ever faced with a real one.
"While the next big exercise is in October," Starkey said, "I
want to do smaller drills with the state police, FBI and DEMA more
often and expand it to include drills that are coordinated with
municipal governments, regional governments and neighboring state
governments,".
Lisa Wragg, the state's disaster recovery coordinator, said the
last exercise involved 80 participants from approximately 10
agencies, two school districts, two universities and a
private-sector financial institution. The state hired Wayne,
Pa.-based SunGard Availability Services -- a disaster continuity
procedure specialist -- as a consultant during the planning stages
and the actual simulation.
The planners thought of doing a simulation involving a major
terrorist attack. In the end, Wragg said, they opted for an
exercise based on what the state considers 70-80% of its risk:
the insider threat.
"We looked at the kinds of problems that could be caused by
malicious insiders," Wragg said, "but we also decided to focus on
what you do if a bunch of things happen at once -- a power failure,
a massive virus infection and a denial-of-service attack."
Who's in charge?
During the exercise, participants were placed into groups based on
their roles and responsibilities, sitting together at large
tables.
Each group worked through the exercise's three stages: pre-event
preparation, event detection and finally response and recovery.
Starkey and Wragg said that the room was constantly buzzing with
debate and activity, and that interaction among groups increased as
the exercise progressed. The importance of communication between
agencies became evident when one group unilaterally decided to shut
down the network to deal with the threat at hand, a move that led
to confusion among the other groups.
"Coming from a technical agency, I was very surprised by that
decision," Wragg said. "But it was quickly flagged as a problem and
resolved. It was definitely an 'a-ha' moment that helped illustrate
why cybersecurity is so complex."
Starkey said it also raised a key question: When it comes to a
cybersecurity incident, who's in charge?
"Generally when there's an emergency in the state, the
law-enforcement agencies or fire departments lead the response,"
she said. "But with a cybersecurity incident and all of the
associated technical issues, DTI must lead the response."
The empty table
Another key moment in the exercise came when participants decided
to establish a command center to address the evolving situation. A
representative from each group was tapped to participate in the
command center.
Wragg said she and her team had earmarked an empty table for a
command center and was pleased when participants saw the need to
create one. But in hindsight, she said, it became clear that the
command center should have been set up a lot more quickly than it
was.
In addition, Wragg said the following points were identified
during the exercise as areas requiring an improved response:
- People in different groups weren't clear on which doors to exit
the building from, or when to exit. Starkey said there have since
been training drills to ensure people know what to do.
- The command center is crucial to maintaining clear
communication lines among agencies during a crisis, and must be
established quickly on the cusp of a major incident.
The best way to work out these kinks, Starkey and Wragg said, is
tokeep holding drills large and small and tweaking security
policies to take the lessons into account.