Do sentiment-analysis tools pass the accuracy test? Here are five tests along with results using freely available products.

Seth Grimes, Contributor

December 16, 2010

2 Min Read

Text five explores (a widespread) inability to distinguish opinion holder from sentiment object. I can't offer you an example from any of the free sentiment Twitter analyzers because none handle this important distinction so far as I know. They analyze at message level, which is obviously insufficient. Instead, consider Appinions (a rebranding of Jodange's opinion-extraction tools).

Check out the results for this search on Barack Obama: The Appinions WCubed tool distinguishes -- and lets you explore results according to -- opinion holder, associated topics, and keywords. As a plus, you can filter results by sentiment polarity (positive, negative, neutral) and type of information source.

Going Beyond

Clearly, many tools that automate sentiment analysis have a long way to go, judging from the free Twitter-analysis tools, to reach accuracy levels that will make them reliable and useful for business decision making. Still, I'm encouraged by progress and also by the market availability of stronger tools, which typically carry out deeper linguisitic analyses of social messages (and enterprise feedback).

Unfortunately, most of the more sophisticated tools/vendors -- Attensity360, Clarabridge, IBM, Lithium Technologies, OpenAmplify, Open Text (disclosure: a consulting client), Radian6, SAS Social Media Analytics, Sysomos -- don't have open, online demo interfaces like Appininions', which is a shame. (Lexalytics does have an online demo, but it was offline for maintenance when I was writing this article.)

Count on continued progress, not only in accuracy but also in other areas that affect the usability and usefulness of automated sentiment analysis. These areas include: Ability to go beyond polarity, to classify sentiment not only as positive/negative/neutral but also according to task-relevant emotion categories such as happy/sad/angry and satisfied/dissatistfied.
Ability to discern intent signals such as purchase plans and service dissatisfaction that's a service-cancellation precursor.
Contextual advertising, the ability, for instance, to ensure that a car ad isn't displayed next to a news report on a vehicle crash.

With a bit of progress, soon even freebie Twitter sentiment tools will pass basic tests. Attention will then shift from accuracy debates to the really hard problem of creating business intelligence from online, social sources.

Seth Grimes is an analytics strategist with Washington DC based Alta Plana Corporation and chair of the Sentiment Analysis Symposium.

About the Author(s)

Seth Grimes

Contributor

Seth Grimes is an analytics strategy consultant with Alta Plana and organizes the Sentiment Analysis Symposium. Follow him on Twitter at @sethgrimes

Never Miss a Beat: Get a snapshot of the issues affecting the IT industry straight to your inbox.

You May Also Like


More Insights