The GATE Way to Open Source Text Analytics

A decade ago, at the start of the naughties, the majority of text-analysis systems came from research labs. At the start of the tennies, we can look back on an explosive growth of startups in the area, followed by acquisitions and consolidation, and latterly the arrival of a healthy market supporting a variety of commercial offerings. The drivers of this expansion included:

• The Web. (Yawn! I routinely skip the first paragraph of the papers in my field these days, as they all start by making this point.) More recently and more interestingly, social networking.

• Cost cutting. Replacing costly market research departments with cheap(er) text-mining departments has become the basis of a whole family of text analysis products that mine the "voice of the customer."

So, what prospects for the tennies? More growth, partly because the drivers that arose in the naughties are still with us, but also because of two new factors:

• Maturity of open source text mining. Replacing costly proprietary software licenses with open source is a trend which we've seen in many other areas. A big sticking point for text analytics hitherto has been the lack of a Red Hat or a Canonical to provide enterprise-level support and training, but that's changing now as more companies sign up to support open source, better training programs become available and so on.

• Fear and trembling at the data suppliers. The pressure from Google in this area is relentless ("We'll give away the data that you sell to drive use of our tools"). The data suppliers can see that they have to offer a higher level of service in order to hold onto their customer base, and text mining is pretty much the only game in town. So, for example, Thompson buys ClearForest, sinks large amounts of resource into "Open" Calais, etc. etc.

In GATE's case we're also now seeing faster growth in demand that we attribute to our repositioning in 2009. We have new commercial partners who have funded a raft of new features, we have new products to complement the long-standing developer-oriented offering, and a new training and certification program. The tennies look like being a busy decade.

It seems to be something like 3 new commercial-walkins per week instead of the previous rate of 1 a month (though today [January 26] is only Tuesday and we've had several already this week...) A week ago I did this summary (confidential) of the week's new contacts:

• (top-3 insurance corporation), looking for GATE support with SLA

• (major US IT contractor), an existing ClearForest customer, looking for GATE training

• (SME), a startup doing sentiment stuff for marketing, looking for GATE Teamware

• (big IT corporation), leading CAD supplier, looking for terminology extraction for translation

The week after was similarly productive so the new message seems to be working.

The interest rate shows no signs of slowing BTW; and various other positive indicators have come my way. It really seems like text analysis and semantics are taking off! Strange. I feel like asking people what's wrong with them... but then I've been waiting for this for 15 years.