From time to time my company is contacted by customers reporting a computer crime. They've seen strange things on one or more machines, so we have to come at once and begin our forensic research. A few years ago, one strange case involved a large HP Unix computer that had died on Monday and had had its hard disk wiped completely clean. The system administrator recovered the data on the disk from the backup tape but the machine was dead and had been wiped the next day. The system administrator replaced the hard disk and restored the system again. Once again, the machine was dead the next day.
A hacker was suspected and we were asked to collect the evidence needed to nail them. First, we installed several sniffers to see all network traffic as we expected the hacker to revisit the hacked machine the following night. We also changed the log-on to the hacked machine. All log events were directly sent to another computer, so the log wouldn't be lost if the hacked machine's hard disk was wiped again. We then divided the night in two shifts and waited for things to happen.
At exactly 4.06am the machine died suddenly again. There was no abnormal network traffic at that moment and nobody in the building except us.
The log showed us that the Unix machine had started its automatic backup but that the (new) local tape unit had been improperly installed and used the same interrupt as the SCSI disk device, which caused errors on the disk during the backup. We just had to change one configuration item and the whole problem was solved. Although the administrator remembered the tape unit being changed, he'd omitted to tell us, considering it an unimportant detail.
I thought that was it but then received another call. The machine, it appeared, was acting strangely again, dying every night for no apparent reason. My first thought was that it had to be the tape device, but the administrator assured me that absolutely nothing had changed within the machine configuration or hardware.
So we took our sleeping bags back to the building's computer room and placed sniffers on the network. That night, nothing happened. We assumed the hacker knew we were now on the lookout for him or her, so we installed special devices to hide all our sniffing machines making it impossible for the hacker to see our activity from the network view.
Again, nothing happened. We halted our investigation and the same night disaster struck again. After a while it became apparent that if one of us was in the computer room, the hacker wouldn't appear. But if we left the building for just one night, the machine was dead in the morning.
We decided to place a monitor in front of the computer room window trained on the machine to see if it worked when we weren't there. We were watching the monitor through binoculars from the roof of a building on the other site of the road, when the security car arrived, as it did every night at that time. The two guards left the car and entered the building. After 10 minutes the first man left the building. Just before the second guard left, the computer screen died!
We ran downstairs and caught up with the guards. We asked what they'd done just before leaving. Nothing special, said the second one, just switched off the light in the computer room.
The light switch, it transpired, also controlled the power supply for the computer. The next morning we talked to the administrator. "Oh yes," he said, "The office people had to switch off a photocopier every evening before leaving the building. To make their lives easy, the Xerox socket was connected to the light switch. But when the copier left the office a few months ago, the computer was placed there."
So the administrator was right: the machine's configuration, both hardware and software, had remained unchanged. He was also living proof that when administrators say "Nothing's changed", they shouldn't be trusted.
Sten "s10" Kalenda is the security manager of security specialist PinkRoccade