InformationWeek Analytics Research: Federated Search

We all know the pain of a lost piece of vital information, whether a customer proposal, document, or important e-mail. The hurt became even more acute this year, as many were forced to comb through data left behind by laid-off colleagues. That's too bad, because the core technologies to enable federated enterprise search have existed for more than 15 years. Yet according to our InformationWeek Analytics Enterprise Search Survey of 552 business technology professionals, not even one in four organizations uses any type of enterprise search system today.

That's not the worst of it. We dug further and asked how respondents who've adopted enterprise search are using their systems, and whether they provide a unified search capability across network shares, databases, applications, intranets, SharePoint, and desktops, plus consolidation of Web browsing. Of the 24% who've deployed enterprise search, less than 8% provide hooks into multiple silos. That's not quite 2% of the total. What's tripping us up? Technology to wrap all of our enterprise data into a cohesive, searchable whole is available from major players including Google, IBM, Microsoft, and Oracle, as well as specialists such as Autonomy, Endeca, and Vivisimo. All told, there are upward of 50 search systems on the market, and most vendors continue to invest in new feature releases or in acquisitions, as with Microsoft/Fast and Autonomy/K2.

The problem isn't technology. It's the three Ps that plague many an IT initiative: politics, privacy, and perception.

"Skipping over mail search was a policy decision, not a technology decision," says a poll respondent from one large enterprise that searches almost everything but mail. "We just aren't able to handle that type of assault on IT." Translation: This company made a conscious decision to hamstring a system with the potential to make employees more productive to avoid the fallout of a policy change.

We understand the fear: E-mail search is one of the most politically charged areas CIOs will encounter. Almost every organization's official policy is that e-mail is owned by the company and employees have no expectation of privacy, yet almost every survey respondent limited e-mail search to the individual level, with only 3% allowing search within departments or teams.

Unless you have e-mail fully integrated into your CRM and ERP systems--something we rarely see--employees' e-mail discussions with customers, partners, and vendors stay hidden within their in-boxes. If you've ever had to piece together a project gone bad or the backstory to a customer complaint that involves e-mail, you know the grief this entails.

Privacy, or users' perception of it, is an issue you'll need to address head-on. Try saying "IT owns search" at your next company meeting and watch the phone lines to HR light up. We've evolved to an odd dichotomy: Most of us accept that Google and Facebook track our movements and use this data to sell everything from behavioral analysis studies to pay-per-click ads. Users happily hand over identity cards at the grocery store to save a few bucks on cereal. But they'll raise holy hell at the concept of IT indexing their e-mail or Web activity to make everyone more productive. Don't we pay them?

There are also issues involving ingrained perceptions about what enterprise search is and what it can do for the organization. First off, enterprise search is not a "set it and forget" technology, no matter what that vendor rep told you. IT needs to develop policies, train users, analyze searches, and respond to zero hits--you know, that "not found" message. Internally, any empty return should be seen by the help desk as a cry for help by someone who's actually using search but can't find the information she needs. It's a trainable moment.

"It's not about the technology, it's about findability," says the director of search at a global consulting firm that has been expanding its search system for almost 10 years. He calls management of search results the No. 1 key to the success of such initiatives, and he should know. The top concern of our survey respondents was the opposite: that users will be flooded with irrelevant data.

That's where IT needs to step up by digging into different vendors' search systems to understand their algorithms and ranking methodologies and ability to integrate with the organization's infrastructure, including identity stores and e-mail system. There are a lot of suites out there, with a wide array of capabilities. Don't forget, many have their origins in Web search, so they may lack basics like Active Directory or Exchange integration and connectors to certain databases.

Pricey And Risky?

As for cost, the full picture can be hard to pin down, especially if you're including desktop and e-discovery needs into this mix. Initial investments will vary based on the number of documents and data silos to be indexed. To give a sense of the potential investment, we built a scenario involving about 2 million documents, Windows and Unix file shares, Active Directory and SharePoint server integration, additional intranet sites, several secure external Web sites, and 1,500 users accessing the system. We then priced it out against two popular systems, Google Search Appliance and Microsoft Fast ESP. Results are below.

There's also a sense among IT pros we speak with that enterprise search systems are a security risk. However, dismissing search because it's a security risk is a page right out of the Barney Fife manual. If we don't know where all sensitive data resides, keeping people in the dark isn't the answer.

Most organizations run an internal security audit a few times a year. Suddenly, you've rolled in a system that does secure crawling on your network on a daily basis. Think Susie in purchasing won't eventually search on "payroll" and get the 2009 budget worksheet that was misplaced on an unprotected share? Mishaps like this happen all the time without enterprise search, but with an enterprise search system you can bet they'll happen with less effort by the end user.

Better to take the proactive view that federated search can be a godsend for the security team, helping them quickly identify configuration mistakes before Susie accidentally (or intentionally) finds confidential data. Besides, even if you don't officially sanction search, most large organizations have a few rogue systems already in place.

Not on your network, you say?

More than 60% of IT organizations we surveyed either let users load their own desktop search utilities or lack policies restricting or controlling their use, which amounts to tacit permission. Microsoft and Google dominate the desktop search market with free apps that are both easy to install and user friendly.

Problem is, these desktop search utilities can unleash a torrent of traffic on your network to satisfy one user's search needs. That's bad news for IT groups that have let this particular underground flourish--if you don't have a slew of rogue search queries hitting your network every day, you will, and it will hurt. Every index is unique to the desktop--no scale, no aggregating results. In fact, our testing of desktop search utilities showed a whopping 15% per-desktop increase in traffic during initial indexing or full reindexing. Multiply that by 10,000 desktops.

Even the most basic enterprise search product provides a unified index database of your information stores, and all use indexing methodologies that have almost no impact on traffic. They also provide the benefit of collecting results, giving you valuable information to help folks find what they need.

After all, the term "jobless recovery" is now being bandied about as the recession officially ends. As business picks up, workers will be stretched thin. Getting search technology implemented--and used--is a great way to improve productivity. If your organization has endured layoffs, remaining staffers are as eager as they'll ever be to adopt search. Chances are, they've already felt the pain of looking for information that was in the head--or the e-mail store--of a colleague who was let go. Time to throw them a lifeline.

The first step is to ensure you have a good map of all your data silos: file shares, core database, e-mail, even software as a service and cloud apps. Document the search systems available in each. Only then will you begin to see the potential gains that may have been right in front of you for years. Before selecting a search product, reach out to security and applications groups, and especially upper management. With the power of search will come complaints about Big Brother; you need a unified front.

"Before we made the leap and rolled out search, we only thought we were productive," says one IT director we spoke with. "Once we got the organization in tune to the overall search needs, we were able to dramatically increase how rapidly we deliver information to our staff."

Remember: Even if you enforce policy and allow regular mail search, without a solid archiving system that organizes e-mail strings, removes duplicates, and performs automated cleanup, you'll end up with a disorganized jumble. On the bright side, administration of groups isn't as big a deal as you might think; you can piggyback search rights onto access to a co-worker's calendar, for example.

Generation Gap

We described the full integrated search concept, including tracking all searches at a corporate level, to a few sales staffers at a midmarket distributor in New England. All were between 25 and 28 years old. Their response: "Why aren't we doing that?" Then we described the "big picture" of search to the company CEO, including how a federated system not only searches all of the organization's information stores (that got his eye twitching), but that it also enables IT to review the searches and usage patterns of the staff, tweaking results based on their behavior.

To paraphrase his response: "There's no frigging way I'm letting IT know what I'm searching for."

Without spiraling into an Oprah episode, this underscores a broader change happening within the workplace and hitting IT at many levels. The "next-gen" employee is computer savvy, expects to IM at work, doesn't understand why you won't support the iPhone, and yes, wants to search the Web and the network from one window. Can you leverage this influx of workers to jump-start a federated search initiative? Just maybe.