To achieve business flexibility, companies can't depend on hiring a staff of contortionists. For one thing, there aren't enough of them. More importantly, better business intelligence and analysis tools have executives and managers straining to apply newfound knowledge to improve their organization's chances in global markets. To make fundamental changes in business execution, these execs shouldn't be forced to work around systems and processes locked in old ways.

How can IT establish a software infrastructure that is both stable and amenable to business change and innovation? The answer is service orientation. From a business perspective, services make sense. Two or more parties perform services according to a contract, with performance measured by how well the arrangement achieves some desired result. A services perspective also enables decision makers to see overall business performance through the customer's eyes, so resources can be reallocated to keep the best customers loyal. Flexibility is critical: What satisfies customers today may not be good enough tomorrow.

Bend, don't break. Be agile, not fragile. And don't bust the budget. These are the sometimes-conflicting challenges confronting IT managers as they select tools and technologies to succeed with services. Our cover package this month focuses on what's rapidly becoming the centerpiece of service-oriented architecture (SOA) implementations: the enterprise service bus (ESB). Here, we provide a strategic overview of how ESB has developed into SOA's business integration hub. Lori MacVittie, senior technology editor at sister publication Network Computing, presents our joint, comparative review of major ESB products (see "Four ESBs That Won't Cramp Your Style").

From an IT perspective, services must enable end-to-end processes, transactions and information flow. At the same time, they must mask complexity by keeping the guts of how organizations meet contractual and business performance agreements well behind the scenes. Facing a terrain littered with heterogeneous systems inside and outside of their organizations, IT needs SOA: a system of open, shared standards that allow for more affordable and diverse systems integration than has been possible with point-to-point or enterprise application integration (EAI) solutions.

With business flexibility paramount, IT developers are showing interest in SOA's increasingly process-oriented flavor. Later in this article (and in our Field Report on page 28), we'll consider Business Process Execution Language (BPEL), which supports the orchestration of Web services. BPEL lets services development aim higher, toward true business "agility" — so that freaky contortions become the exception, not the rule, in a well-aligned enterprise.

SOA: The Empire Strikes Late

Distributed computing involves multiple systems working together over a network. Parts of an application system usually reside in different locations and thus require a software infrastructure that allows procedures, tasks, events and processes important to the overall objective to happen remotely and reliably. Central control must live in balance with local autonomy.

Never easy or affordable with a clean slate, distributed computing becomes even more complex when necessity forces mediation and data transformation between existing systems. Most of the time, systems remain in their functional or departmental silos, requiring extensive data-, application- and process-integration, including EAI hub-and-spoke approaches, to hold it all together. Making the situation even tougher is that customer transactions and other information — involving, for example, self-service banking, mortgage or travel-planning relationships — now enter the enterprise through multiple channels around the globe, at all times. Mobility and business change are putting formerly fixed points of entry in constant motion.

As was the case in previous decades when midrange and client/server computing exploded on the scene, IT is busy playing catch-up to computing that's already distributed. SOA comprises many goals, but a big one is to ease the long struggle to manage runaway distributed computing among heterogeneous systems. Individually, many of the ideas and techniques coming together in SOA have been tried. SOA goals sound familiar because they are familiar.

Web Services Get the Message

Riding the growth of the Internet, Web services development seemed to toss aside carefully wrought (read: too slow) object-oriented software engineering, modeling, development methods and middleware approaches such as the Common Object Request Broker (CORBA) and Unified Modeling Language (UML) standards. More favored approaches have included Agile Programming and other faster techniques.

In fact, Web services development built on the CORBA foundation to employ SOAP, HTTP and UDDI over TCP/IP networks. The big difference was XML: Through WSDL and other XML implementations, Web services brought XML into play as the new lingua franca for data- and process-rich distributed computing. Today, Asynchronous JavaScript technology (AJAX) is using XML to rebuild some of the methodology structure missing from early Web services development (see "Will Rich Internet Apps Catch the Bus?")

The first generation of Web services generally hewed to the tried-and-true model: point-to-point, remote procedure call (RPC) systems that demand synchronous, tightly coupled connections to ensure complete transactions. But as Web services mushroomed to serve thousands of clients and required interaction, better security and orchestration with multiple, distributed back-end systems, the SOAP-and-HTTP-over-TCP/IP approach hit scalability problems. From an IT management perspective, the distributed future began to look too much like reheated spaghetti code.

As they have before, IT managers turned to message-oriented middleware as a model for what should come next. Messaging employs a middleware bus, over which participating systems "subscribe" to information "published" by applications or other sources. Along with Microsoft, IBM and TIBCO produce the best-known message brokering systems. With the queue as the central hub, messaging software is designed to replace manual programming and maintenance needed to handle prioritization, routing and other issues involving the implementation of business logic among multiple systems.

Real-time business goals make it imperative to separate the Web service messaging system from underlying network connectivity changes. ESB messaging also can reduce the need for programmers to fix the message queue when new systems hop on the bus or cause other changes. Led primarily by Sonic Software, but also PolarLake and Fiorano Software, ESB packages have put messaging at the center of the Web services and SOA picture. Striving to assume their network-independent hub role in the new world of SOA and Web services, EAI and application server providers resell Sonic and other ESB brands (including those from open-source providers, such as JBoss) in their packages.

Java Messaging Service (JMS), part of the J2EE specification, is the essential standard applied by EAI, messaging and application server providers. However, vendors such as Cape Clear Software and Microsoft counter that the standard does not protect organizations from proprietary software shackles that some believe will be an impediment to reaching the fully distributed, loosely coupled SOA vision. Cape Clear and iWay Software are examples of vendors that do not have their own messaging component, viewing that layer as either commodity or part of its customers' existing infrastructure. Cape Clear, iWay, Software AG and even the ESB providers that have their own messaging systems are sensing that their customers' attention is more focused on the potential business value of mediation and XML data transformation above the messaging layer.

Microsoft shops, of course, are most concerned about the world outside of Java. Microsoft does not as yet market an ESB product; instead, it offers BizTalk Server and other elements of the Windows platform for messaging and related functions. Microsoft is a major backer of OASIS Web Services (WS) standards, which could supplant (or at least counterbalance) JMS for critical messaging duties. The biggest task is ensuring reliability so systems can detect, recover from and ultimately avoid failures, without compromising the SOA's loosely coupled, plug-and-play objectives. WS-Reliable Messaging and WS-Reliability build on the SOAP processing model to give asynchronous SOA the same level of transaction quality as synchronous, RPC-style computing.

Reliability is also a key issue in how ESBs manage transaction state in SOAs. ESB vendors use routing and concepts such as itineraries or circuits to pass state from one node to the next. XML lets messages carry a great deal of information as metadata, which describes the endpoints that must receive the messages. Through smart routing and state passing, the system can begin to manage itself through reusable integration patterns. These techniques also let humans or process management systems make changes in midstream if necessary. Metadata-rich messages hold potential as an important integration point for business activity monitoring and BI systems.

Is the growing ESB workload going to turn SOA back into the hub-and-spoke architecture that typified EAI? This might please conventional EAI vendors, who are "having a hard time with SOA," says Ronald Schmelzer, senior analyst with ZapThink, an IT advisory and analysis firm. "They are afraid customers will say, 'Wait a minute, if I take my services and interfaces and just do SOA composition, I might not need this expensive integration middleware.'" Schmelzer doesn't see EAI as a good starting point for ESB. Because EAI is focused around a hub, he feels it's contrary to SOA's goal of decentralized services. "EAI systems pour concrete on business processes, since they tend to solidify existing processes rather than enable an IT environment that allows companies to deal easily with change," he says.

Not all analysts share Schmelzer's view. "The weaknesses of hub and spoke are no longer a big deal," says Ken Vollmer, principal analyst with Forrester Research. "Even the 'single point of failure' criticism is not true anymore, because integrated middleware platforms are clustered and the products take advantage of all kinds of backup and recovery." Forrester's November 2005 ESB report gave high ratings to EAI players that have ESB as part of their comprehensive suites.

Processes and Business Integration

BPEL, which is defined in WSDL (and thus, XML), has become the primary connection between business process management (BPM) and SOA. However, BPEL is likely only the beginning of a growing synergy. As they grow in number and sequence complexity, Web services need the orchestration that process management can provide. Messages and services, which must organize and mediate their interaction, have a lot in common with business processes. And as organizations use BPM to increase efficiency and reusability, they will execute processes and monitor performance through SOA.

Business processes are "acts of communication," according to Robin Milner's pi-calculus theory, which is foundational to BPM. Process engines are good at taking business rules and determining who or what does which activities and tasks to accomplish the process objective. SOA easily becomes the standards-based integration platform that BPM needs.

BPEL's popularity is spurring the Java community to step up its move from code and objects to process development. The Java standards community now has Java Specification Request (JSR) 208 for Java Business Integration (JBI), which is slated to become for services what J2EE has been for applications. JBI also will provide an easier way for existing J2EE systems to expose components as services. Finally, JBI helps BPM by opening up the SOA "pluggable platform" to more than just BPEL.

Gartner defines ESB as a platform that "combines features from several previous types of middleware into one package," including "message validation, transformation, content-based routing, security, services discovery for SOA, load balancing, failover and logging." Functioning as the linchpin of SOA, organizations hope that a heavyweight ESB can lift integration and mediation up to a higher, more strategic level, where the language of communication is business models, rules and processes, not arcane IT concepts like objects.

With SOA, ESB and BPM in sync, when the organization needs to move and change, IT can move and change. Custom development will never entirely go away, but it can focus more on uniquely competitive enhancements and less on routine integration. And contortion? Hopefully, it will go back to being a circus act.