AIIM, the Association for Information and Image Management now also known as the Enterprise Content Management Association, says its research demonstrates the increasing degree to which important business documents (let's call them "needles") can get lost in the e-mail "haystack." They might be in there. They're just impossible to find. According to AIIM's press release on the matter, one of the culprits is an insufficient e-mail retention policy as the number of business-critical documents stored in e-mail systems rises. For $65-$75, AIIM will teach you how to manage the problem. But is this one more reason to........get off those insourced e-mail servers and move to the cloud.

I'm 100% certain that there are companies out there whose e-mail retention policies don't exist, are insufficient, or are not well articulated. But let's be honest about one of the root causes of the problem: insufficient storage.

As long as I've been a user of corporate e-mail, the problem with finding that all-important document that I might have kept in my in-box or some other folder had nothing to do with the articulation of a retention policy. It has to do with the fact that no company wants to make petabytes of e-mail available to their users from real-time storage. It's just too expensive to reliably keep so much storage online, all the time.

Instead, you're asked to keep your e-mail file below a certain storage limit (usually one that's measured in megabytes during a day and age when many attachments exceed a megabyte in size). To do this, we're told we can archive e-mail that's beyond a certain age to our local hard drives. But, there are two problems with that. First, since we're so sensitized to the issue, we try to avoid the archiving step altogether by aggressively deleting mail (including sent mail) that we don't think we'll ever need anymore. Sometimes overaggressively.

Second, when we do save and archive our e-mail, it makes it harder to find an e-mail we might want to dig up later.

These limitations are almost entirely a function of how companies have convinced themselves that they must insource their e-mail and group calendaring systems. Sure, insourced solutions have all sorts of bells and whistles that haven't turned up yet in some of the cloud-based solutions like Gmail. But so many people I talk to say that storage and search are two of their biggest pain points with their corporate in-boxes.

Meanwhile, for free, Google let's you have 6.7 GB of storage per user and, for $50 per user per year, you get "the works" from Google, which includes 25 GB of storage per user. That's so much storage that most users would probably go the entire time they worked for your company without having to really delete a single mail (received or sent). With all their mail essentially in one place, it's easily searched and Google's Gmail search is (a) lightning fast and (b) very configurable when you're looking for that needle in the haystack.

Although I'm a fan of Gmail (the Google Apps version in particular) since I've been using it for so long, I want to be clear. You don't have to turn to Google for these benefits (although Google happens to be a great provider). There are other cloud-based offerings today (Zoho is in closed beta but HyperOffice offers the first 100 MB per user free and every 100 MB thereafter for $3).

Over time, there will be plenty of other cloud-based offerings. All of them should allow you to do away with some of the hardest parts of insourcing e-mail and group calendaring: running the necessary servers and storage and having to tell your users that they must keep their e-mail file sizes down to, what to them, will always be an unreasonable size.

In its press release, AIIM wants you to know that, in an effort to solve the problem (the one where business documents are becoming increasingly difficult to unearth from the e-mail infrastructure), it will teach you a best practices certificate course that, amongst other things, will help you design a great e-mail retention policy for your company. There's a quote from Atle Skjekkeland, the VP of AIIM, that says:

The quote is a great reality check of the old stodgy AIIM. It demonstrates how systemic the problem around e-mail and group calendaring really is. Storing business documents in e-mail is the perfect self-fulfilling policy if you're a seller of an insourced e-mail solution. Of course e-mail is becoming an increasingly important repository for business-critical information. And, on their present course, enterprises will have no choice but to spend more money, hand over fist, to make sure their existing insourced solutions don't disintegrate under the pressure. And, now that AIIM is going to show you how to retain even more e-mail, of course you will have to spend even more on e-mail infrastructure than you anticipated.

Isn't it nice of AIIM to help us stay on that software train for the forseeable future? If I were the VP of AIIM, I wouldn't let the bit about spending "significant or slightly more on e-mail management technologies" go so easily. Why not teach companies how to spend less while offering more to their users (more storage, more better search, etc.)?