It's human nature to trust your fellow employees--the people at the coffee pot, on your company softball team, down the hall. That's why it's so natural for IT managers to focus their network defenses on outside rather than inside threats.

What hit UBS PaineWebber on March 4, 2002, shows just how dangerous that one-sided thinking can be. Nothing more than 50 to 70 lines of malicious code--a "logic bomb" that U.S. prosecutors claim was planted by a disgruntled employee--took down about 2,000 servers, leaving 8,000 brokers across the country unable to work. IT teams spent sleepless nights on conference calls with IBM and scrambled to reset servers, trying to undo damage that still, four years later, hasn't been completely repaired.

The details of what happened are pouring out of a trial that began last week in U.S. District Court in New Jersey, where a former systems administrator, Roger Duronio, is charged with computer sabotage and securities fraud. The case paints a nightmare scenario for any IT team: a system failure that forced at least 400 employees to drop what they were doing and troubleshoot. Assessing and repairing the damage cost $3.1 million. In some cases, brokers were down for days, even weeks, depending on how badly their machines were hit, how remote the offices were, and if the branch's backup tapes could be found. The company, now called UBS Wealth Management USA, hasn't put a price on its lost business.

"It was the magnitude of it. How on earth were we going to bring them all back up? How was this going to affect the company?" testified UBS IT manager Elvira Maria Rodriguez, the first witness for the prosecution. "If I had a scale of 1 to 10, this would be a 10-plus."

Trading resumed in the days after the attack, but some servers hit by the malicious code were never fully restored, largely because about 20% didn't have backup tapes. "We were always having issues with these large-scale servers" after the attack, Rodriguez said. It would have taken about a year, she estimated, to make all the servers right again, even if that was all she did. "We just had to learn to live with it," she said.

Money And Revenge
Prosecutors claim that Duronio, 63, of Bogota, N.J., sought revenge against his employer by building, planting, and disseminating a logic bomb (see story, Software Bombs: Simply Tricky) to delete all the files in the central data center's host server and in every server in every U.S. branch office. His motivation allegedly was money and revenge. Assistant U.S. Attorney V. Grady O'Malley said in his opening statements that Duronio wanted to take home $175,000 a year from a base salary of $125,000 and a maximum annual bonus of $50,000. In February 2002, that bonus came in about $15,000 shy of his expectations.

Here's how the prosecution, led by Assistant U.S. Attorney Mauro Wolfe, alleges Duronio committed the crimes: Logging in to the central host server from his home VPN connection, Duronio planted the malicious code months ahead. When he found out that his bonus wasn't all he'd hoped it would be, he demanded that the company give him a contract for a full $175,000 or he'd walk out that day. UBS didn't give him a contract, and Duronio was escorted out the door. But the logic bomb was already planted and the trigger set to go off on March 4 at 9:30 a.m.--just as the stock market opened and trading began. Prosecutors said in court that investigators executing a search warrant at Duronio's home found pieces of the malicious code on his personal computers and in hard copy on his dresser.

According to prosecutors, Duronio intended to profit by buying put options on UBS stock--using $20,000 cashed out of an IRA--that would pay off only if the company's stock took a dive within 11 days. "If he wasn't going to receive that [bonus], he was going to level a catastrophe against UBS that would rock their financial stability--and that would get him the biggest payday of his life," O'Malley told jurors. Despite the attack, UBS's stock didn't drop, and Duronio's investments didn't pay off.

Duronio's defense will point to UBS's inadequate security. Duronio isn't to blame for this "unsophisticated and sophomoric" code that was most likely planted as a prank, said Chris Adams, Duronio's attorney and a partner at Walder, Hayden & Brogan. The real problem, he laid out in opening statements, was that UBS's network was riddled with security holes that left the company open to attack.

Adams hasn't conceded that the code was an inside job, but he's trying to convince the jury that other employees were responsible. Weaknesses in UBS's IT system let someone else using Duronio's ID and password move around undetected in the network, Adams said.

A January 2002 internal audit of the UBS PaineWebber IT department found there were issues with the company's Unix and Sybase security, specifically involving passwords, Adams said. Forty administrators could gain root access using the same password, affecting the system's ability to tell which root user was giving commands, he told jurors.

Rodriguez testified that immediately after the attack began, she stepped out of her office and used an open root access on another systems administrator's computer to monitor what was happening on the network. Asked if it was company policy for an administrator to walk away and leave root access up on a computer, Rodriguez said it wasn't policy, but she wasn't surprised it happened.

Adams asserted that a March 2000 review of the financial firm's VPN showed that another session could open under a user name and password that already was in use. Rodriguez said she wasn't sure if that could be done at the time of the attack, but it can't be done now.

Chaos After the Attack
What's beyond dispute are the problems caused by the attack, and the trial offers a rare glimpse into an IT team in full crisis mode.

Rodriguez, who was in charge of maintaining the stability of the branch servers, got on a conference call that night with some of the 200 IBM tech workers who immediately were sent to the company's branch offices. Rodriguez didn't go to bed that night; she stayed on the conference call the rest of the night. She had plenty of company.

Rajeev Khanna, manager for UBS's Unix systems group at the time of the attack, also didn't go home the night of March 4, 2002. Khanna, who oversaw the recovery process, didn't go home for three days, as his team redirected 400 to 500 UBS workers--application developers, project managers, systems administrators, and database administrators--from their normal jobs to work on the restoration.

"The most important thing was for users to be able to log in to their desktops," he testified. "They couldn't log in. They couldn't do the work they do on a daily basis, in terms of pulling data on their clients, making trades, and checking market data."

The problem wasn't just downed servers. There was mounting chaos in the data center and the Escalation Center, as system administrators and other IT workers flooded in, yelling out questions and suggestions. A room where six or seven people usually work teemed with 20 or 30 by midmorning. By noon, 50 people were working on the downed network, and just an hour later, hundreds were involved across the country.

The problem led to a grim annual ritual for the IT team. To avoid a repeat of the incident, for the next two or three years Rodriguez prepared to fend off a similar attack before every March 4--taking critical servers offline so that if any malicious code still lurked on the network, at least those servers wouldn't be affected. "We had to make sure there was no more business impact," she said.

Beware the Inside Job
Computer attacks by insiders, even by IT professionals, aren't uncommon. With only slight variation from year to year, inside jobs occur as frequently as highly publicized external attacks. Insiders can be more dangerous because of their access privileges and because they're not suspected. "Your system administrators have a lot of power because it's part of the job," says Burton Group analyst Eric Maiwald. "You have some general expectation that they're not trying to cause you harm. If you put too many controls on them, they can't do their jobs.''

Put too few, however, and many sleepless nights may lie ahead.

Continue to the sidebar:
Software Bombs: Simply Tricky