Rolling Review Wrap-Up: Application Performance Managers

There's a lot of marketing hype and money swirling around next-generation application performance management. While many IT organizations strive for deep visibility into critical application metrics, they're fearful of products that depend on labor-intensive approaches to deploy, manage, and monitor apps--and for good reason. To find out which APM suites provide insight without too much outlay of effort, we've spent much of the past year testing nine leading products in our Windward Consulting Real-World partner labs.

Results Summary

>> BROAD APPLICATION SUPPORT
was our first assessment criterion. Three products earned perfect scores.

>> PROBLEM DETECTION/REPORTING
is tied to agent deployment. No one hit a home run here.

>> DISTRIBUTED APP MONITORING
was a strength for Nimsoft and Quest.

>> ROOT-CAUSE ANALYSIS
is a problem area for many entries, though BMC showed strong here.

>> STRONG NETWORK INTEGRATION
was another lackluster category.

We designed a thesis around holistic APM products that not only identify problems but take corrective action to resolve performance issues before they impact users and customers. Actions may include allocating additional bandwidth or server processing capacity, even rolling back configuration changes. Where corrective-action capabilities are lacking, organizations will struggle with fixing problems manually, though fast identification and notification are better than nothing. We also suggested that for truly holistic APM, products should locate and predict performance problems across all application components and support all facets of the app infrastructure. Service modeling was another key area; we sought to model critical application services and report on compliance with service-level agreements.

A tall order, but there are dozens of players in this space. We tested suites from BMC, Compuware, Indicative, NetIQ, NetQoS, Network General, Nimsoft, Quest Software, and Symantec. During the review, we had an additional 23 vendors that wanted to participate, while a handful, including CA, IBM, and Hewlett-Packard, declined our invitation or couldn't get software to us in time. We'll try to get more APM products in the lab this year, but we're confident the nine suites tested represent the full spectrum of features in terms of data collection, reporting, correlation, and integration. Note that right before press time, Symantec sold its APM business; see our analysis at theApplication Performance Optimization Immersion Center.

Rolling Reviews present a comprehensive look at a hot technology category, including market analysis, product reviews, and wrapping up with a synopsis of our findings. See more of this Holistic Application Performance Management series and our other reviews at Rolling Reviews.

BEST OF THE BEST

In terms of SLA management for APM, Nimsoft's Nimbus was a real leader. Its flexible and robust SLA reporting engine let us manually build application service-level agreements by coordinating groups of monitored components into a comprehensive service view. We could report SLA performance granularly, for example, during defined business hours and excluding particular time slots, such as maintenance windows. Nimbus also let us exclude a particular component, during a specified time range, within a group of elements that operate under an SLA. This level of detail is helpful when IT must exclude an application component that failed because of a customer-generated outage that falls outside its SLA. This happens all too frequently, yet many APM tools are unable to manage this scenario.

Products focused primarily on collecting network packet information, including Network General (now NetScout) and NetQoS, are best suited for troubleshooting performance issues within the network. These appliances let us drill down and get very discrete packet-level performance data and work to determine the nature of problems. However, dashboards and executive views weren't the best--we typically needed to integrate into other appliances for this capability, not a seamless process.

NetIQ AppManager handily made our short list. Although it relies on agents and, as with all rivals, service models needed to be constructed manually, NetIQ did a great job depicting end-to-end application transactions. Its à la carte pricing also will be attractive for organizations. In the same class, we also liked Quest Foglight's balance of data collection and features. Foglight dashboards can be created easily and set to model high-level business and service views that clearly identify issues as they occur. All that data does come with a price in terms of complexity, however.

If you need a synthetic transaction monitor for Web services, go with Indicative. Its top-down business service lets organizations monitor and manage performance as a true service. Indicative doesn't require IT to manually correlate data to build service views; just drag and drop field-component templates into a logical service grouping. As a bonus, you can be up and running in an afternoon, and maintenance was a breeze.

For large enterprises that want scalability and flexibility and don't mind some complexity, Compuware deserves a look. It's the most complete system we reviewed in terms of data collection. One ServerManager can support as many as 1,000 agents, and you may add multiple ServerManagers, then access and report on all of them in the VantageView console. We didn't test the scalability of this three-tier agent, controller, and console architecture, but we did validate that it can be distributed across machines. Only HP/Mercury and CA/Wily--neither of which agreed to be reviewed--have comparable diversity. Given the cost and complexity of these applications, however, they're not for the faint of heart.

If you already have BMC tools or other supported data collection utilities and are looking to correlate information and make some intelligence out of your data, BMC ProactiveNet is a slam dunk. ProactiveNet is strongest in intelligent correlation of events, invaluable for organizations with SLAs. Its agents are effective at pulling extensive metrics from applications and application infrastructure components. This was the only tool tested that focused on correlation of data in contrast to data collection.

THE REPORT: Who Made Our Short List?

Of the nine products tested, five made the cut. Find our complete APM Rolling Review at: informationweek.com/1173/report_apm.htm

See all our reports at informationweekreports.com