A Day With The Patch Patrol 2

On Tuesday this week Microsoft jolted its customers with 10 new security bulletins and a rerelease of a previous bulletin. Seven of the new bulletins were ranked as "critical" by the software vendor, while three were ranked "important."

The critical flaws place customers' systems at risk for serious security compromises, or even an Internet worm the likes of Code Red, Blaster, and SQL Slammer. This month's batch of patches aim to fix more than 20 vulnerabilities that affect a swath of Microsoft applications, including nearly every supported version of Windows, Exchange 2000, Exchange 2003, and Microsoft Excel.

This creates a race during the next few weeks between hackers and virus writers and the security administrators rushing to patch their business-technology systems before an attack. If history is any indicator, security professionals have from one week to 21 days before hackers' tools used to attack this week's set of vulnerabilities become widely available on the Internet. There's also the possibility that a worm could surface within weeks or even days.

Companies that use patch-management software from companies such as BigFix, PatchLink, Saint Bernard Software, and Shavlik Technologies have a distinct advantage in this race to beat whatever attacks may surface as a result of these Microsoft vulnerabilities. By automating the patching process, they're able to trim days, if not weeks, from the laborious process of updating each vulnerable workstation and server.

On Tuesday, InformationWeek camped at the offices of patch-management vendor Shavlik Technologies to get an inside view of how the company readies and tests its patch-management applications during "Patch Tuesday" so their customers can more efficiently fix their systems before any potential attacks.

At approximately 12:30 CDT Tuesday afternoon Microsoft's 10 new security bulletins hit the company's Web site. Eric Schultze, chief security architect at Shavlik, quickly gets to work poring over the security bulletins and analyzing how consequential each software update will be to Shavlik's customers. "This is big," says Schultze. "There are seven critical vulnerabilities. This is Microsoft's biggest patch announcement so far," he says, reviewing the bulletins for the first time.

Patch Preparation
The moment the Microsoft bulletins hit, Schultze sends an E-mail alert to the Patchmanagement.org mailing list, which is hosted by Shavlik and moderated by Schultze. A similar E-mail is immediately sent to Shavlik's XML Announcements mailing list. This list informs Shavlik customers that the company has begun prepping their patch-management applications to deploy this new round of Microsoft patches.

While Schultze studies the bulletins and inputs the critical information for successful patch deployment into Shavlik's application, a large pepperoni pizza and two cans of diet Pepsi wait for him by his desk. "I always have pizza on patch day," he says.

Minutes after the news of Microsoft's October security bulletins goes public, Shavlik's customer-support phones start to ring. Customers want to know when Shavlik's flagship patch-management software, HFNetChkPro, will be updated so that they can deploy the new patches. "We just got the final patches at the same time as everyone else. Our goal is to have these ready today," Schultze says.

With such a large number of security bulletins hitting Tuesday, it appears to be a stretch goal to have their applications ready the same day. Shavlik needs to prepare its software to be able to spot and deploy more than 30 patches in eight languages, including English, French, German, Italian, Japanese, and Portuguese, both the Brazilian and Portugal dialects.

The heart of the day's work resides within two XML files that Shavlik's software uses to manage the patch process for its customers. The first XML file will be used by HFNetChkPro to assess customers' systems and ferret out which of their systems are vulnerable to attack. The second XML file that needs to be created will instruct HFNetChkPro how to deploy the patch files on their customers' systems. "This is what drives our software," Schultze says.

Shavlik's HFNetChk "engine" is licensed and used by other security vendors, including BindView, BMC Software, NetIQ, and Symantec, within their patch-management applications to scan Windows workstations and servers for un-patched and at-risk systems.

First Schultze preps the XML file that will be used by HFNetChk to scan and assess customer systems for the new vulnerabilities. For about four hours Schultze prepares the patch-assessment files, painstakingly ensuring each bit of information ranging from arcane security bulletin tracking numbers from various security organizations to other minute details about each patch file is correct, so their software will correctly spot vulnerable systems.

Virtual Testing
While Schultze works on Shavlik's patch data files, Karen Helker, quality assurance manager at the company, begins preparing 41 virtual systems, which run various Microsoft operating system and application configurations, to test the XML data files Schultze crafts. "We thoroughly test the detection and the deployment of the patches before making our XML files available to our customers," Helker says.

Helker and her team will use the virtual machines to test how well Shavlik's software spots vulnerable systems and deploys Microsoft's patches in each language. For the October round of patches, that includes more than 240 possible patching configurations--and each needs to be tested. Only one of today's security bulletins contains a vulnerability in a system for which Shavlik doesn't have a pre-built system ready for the test, a vulnerability within Microsoft Exchange 2003 running on a Windows 2000 system. "We had that built on Windows 2003, but not Windows 2000, which is the configuration for this particular flaw," Helker explains.

By 4 p.m. CDT, Schultze passes his XML assessment files to Helker and three other technicians who divvy up the more than 240 configurations for testing. While the Q&A team tests each configuration, Schultze goes to work developing the XML file that will be used by their customers to deploy the actual Microsoft patches.

The building and testing process continues until late Tuesday evening. At 10:30 p.m. CDT, Schultze and Helker hold a meeting to determine how well the process had gone for the day, and to make sure they're ready to post the patch files to their customers by midnight.

"We got it 95% ready," Schultze says. The final hours are spent fixing any minor glitches that the Q&A process uncovers. "This is the detailed stuff," says Schultze.

It's that kind of diligence that Craig Perkins, bank officer and technology manager at Red River Bank, an independent community bank based in central Louisiana, says is "extremely important" to him as the bank's business-technology team navigates through the "labyrinth of corporate host patching."

Red River Bank has been running Shavlik's software for about a year, and Perkins says it has greatly improved their process for patching more than 175 workstations and servers spread throughout seven bank branches. "It's more manageable now," he says.

Perkins says prior to running Shavlik's patch-management software he had occasional "sleepless nights" hoping the hacker tools or worms didn't surface before the bank had time to safely patch its systems.

There's not much sleep for the likes of Shavlik's Schultze and Helker on patch Tuesday. It's just minutes after midnight CDT by the time the process of preparing and testing Shavlik's patch-management software is complete and the files are ready to be published to their customers.

"You missed your same-business-day goal by a few minutes," this reporter prods Schultze. "No we didn't," he says. "We're operating on Redmond time."