I have yet to see satisfying criteria for assessing social-media analysis tools. Definitive specifications? As Ovum analyst Tony Baer puts it, "that stuff is all around but amorphously defined."
Businesses are spending good money on SMA tools, too often on tools with weak analytics and weak interfaces. I was shocked by one tool I recently saw, both by the tool's shallow sentiment analysis and '90s-vintage analysis capabilities and by the demoing manager's unawareness of same. (Nonetheless, the company recorded $62 million in 2010 revenue, total for all products.)
I suppose the excuse is that social platforms are pretty new, so expectations shouldn't be high. Lame.
Let's raise the bar. I know a worthy challenge when I see one, so here's what I look for in a social analysis tool.
No to Silos, Yes to BI
Start with the thought that most of us are not really interested in analyzing social media, are we? Our real interest is the business value of social content, which lies in messages, people, and networks, not in the platforms themselves.
We're best served by a conception of social networking that transcends any given platform or medium, that bridges into the enterprise systems that mediate and record value-generating ($$$) business transactions.
Yet most SMA tools treat social media in a siloed fashion. They don't recognize the linked mutuality of social and conventional touchpoints -- people post about restaurants they've eaten in, and conversely our purchases are influenced by what we read on social networks. Tools also tend to silo data from each social platform (Facebook, Twitter, blogs, forums, etc.).
As a business analyst, customer support rep, or marketer, I'd want to know not only what was said where online, I'd want also to know who heard the message, whether a message crossed platforms, for instance from a blog or forum posting to Twitter, and what hearers did in response.
Silos mean walls. Down with the walls. Enough with the silos. Understand and account for the networks!
Continue with the admission that while Facebook and Twitter and the like are still young, business-adapted data analysis isn't. Modern BI dates to the late '80s. Every leading BI tool offers not only dashboards and reporting but also dimensional models and pivot tables for interactive, exploratory, visual data analysis. You wouldn't know it by looking at many notable SMA tools (or, by the way, at leading mid-market survey tools). Their best is static dashboards and reports offering limited ability for data filtering and report parameterization. They offer nothing approaching the drag-and-drop, multi-dimensional cross-tabulations that every leading BI tool supports.
The most disappointing example I've seen is heavily-promoted SMA software launched last year by an analytics giant, whose version 3.0 appears still not to benefit from the vendor's deep BI and analytics competence.
What I Look For
SMA silos and weak interfaces are my two big points. Now let's jump down a level. I'll offer a slew of criteria, broken out in six categories. A caveat: Neither the categories nor the criteria are meant to be comprehensive nor deeply explained. I'm looking for gaps, so the criteria I list here relate to elements that are missing from a large proportion of social-media analysis tools now on the market. Any given tool doesn't have to cover every function I'll list -- I'm not a fan of requirements checklists -- but every tool should cover a good selection of the criteria, whichever suit the business goals the tool will support.
On to my categories: Metadata, Resolution, Integration, Alignment, Interface, and Walk the Talk.
Metadata: Even the most basic SMA tools can pull in Twitter, blog, news, Facebook, and other feeds, if not directly, then via services such as Spinn3r, Gnip, Factiva, and Moreover. Raw feeds are fine, but I'm interested in what's in and surrounds the feeds, namely metadata. Metadata records who posted, where, and when; it may also capture location.
Strong SMA will look further at the message "envelope" to discern interrelationships. Is a given message a retweet, a reply, a comment, or some other form of link to an external resource or message? Is it part of an exchange? Let's not look at messages in isolation, as so many tools do. SMA tool makers: Help us understand message diffusion and discourse (threaded conversations) with an analytic that incorporates demographics.
Resolution is the ability to extract data from the content of social postings and other source materials, both non-explicit metadata and information locked in the content. By "non-explicit metadata," I mean primarily identity information. Take my own Twitter account as an example. My profile includes my real name and my account location (which is different from a tweet's location), and I also include, as many Twitter users do, a link to a page with lots more information about me. A strong SMA tool will tap this information.
Content analysis is the real challenge, getting at the entities (names of people, companies, places, products, etc.), facts, opinions, and signals (for example, "I'm shopping for a new car"; "Can anyone recommend a restaurant in Duluth?"). For this, you need sophisticated natural language processing (NLP) and sentiment analysis with the ability to resolve parts of speech and, especially for source materials longer than tweets, to spot co-references including anaphora -- instances where, for example, "Barack Obama," "the president," and "he" refer to the same person.
In my opinion, SMA done right can resolve sentiment at the feature level, related to entities and topics, and can distinguish opinion holder from object. A tweet that says:
@consumerist Gilbert Gottfried Loses Aflac Duck Gig Because He Thinks The Japan Tsunami Is Hilarious
illustrates the latter need: @consumerist ≠ Gilbert Gottfried. I found that tweet, by the way, via a search on "japan tsunami" using the TweetFeel Twitter sentiment tool. TweetFeel correctly saw the "japan tsunami" sentiment expressed as positive; it's sentiment about Gilbert Gottfried that is negative, but it's not "Gilbert Gottfried" that I searched on.
Resolution needs to extend to complex, compound messages. The tweet
@OutsellInc RT @jillfgibson: Love this take of #visualization through the ages RT @infobeautiful Vintage InfoPorn No.1 http://bit.ly/ibl9Uy
involves three tweeters and three submessages (let's call them), which strong SMA would recognize and decompose. Again, message-level analysis ignores information.
The Integration imperative is captured in two snippets from my Down-with-Silos tirade, above. I look for social analysis that "bridges into the enterprise systems that mediate and record value-generating ($$$) business transactions" and that can tell me "what hearers did in response" to social messages.
To integrate, or link records across sources, you need to capture or discern identity. I think the information is more available than most people would suppose, with significant digital sleuthing involved in discerning it. Of course, sleuthing, or even use of openly available or permitted identity information -- few users read terms of service nowadays -- can create a creepiness factor at best and a violation of privacy rules, with reputation implications at worst.
Alignment, to me, means two things: 1) Ability to measure quantities that affect business outcomes and compute indicators that predict outcomes. 2) Usability of results in business that need them. Social engagement that allows a support rep to find and reply to Twitter comments and requests is good -- results can be quantified -- but note that my "two things" didn't use the word "social." I'm looking for analysis tools that measure and predict social's ability to drive business transactions -- money-making outcomes -- as well as how business news will play out on social platforms.
Interface: I claimed earlier that every leading BI tool offers not only dashboards and reporting but also dimensional models and pivot tables for interactive, exploratory, visual data analysis. Pivot tables let you place variables (such as location, age, sex, platform, product) in the row and column (and sometimes the page) axes of tables to create a cross-tabulation.
BI tools will typically let you nest variables in an axis to create a pivot table with several dimensions. You often have a choice of measures -- sums, counts, percentages, calculated values -- and the ability to navigate up and down dimensional hierarchies (such as year-quarter-month-week-day) with automatic value aggregation. I rarely see these capabilities in SMA tools.
Slick-looking tools too-often provide little more than filters, the ability to select values of variables for inclusion or exclusion in a table or chart. Tool developers should look at the BI world to see example of the visual controls, (time-based) animations, ability to create reusable macro expressions, and the sharing and collaborative capabilities they should consider for inclusion in their own tools.
Walk the Talk: I look for clue-ful SMA suppliers. If a company doesn't know how to use social media effectively, or if it won't make an effort, do you really want to trust it with your business? The question isn't moot; anyone who spends time on social platforms can tell strong from weak social engagement and has seen instances of both.
These are elements I look for in a social analysis system. I've written about them here because I too-infrequently find them. Too often, I encounter poorly conceived solutions, based on weak technology, from vendors, established and start-up alike, that should know better.
I apologize for not naming names, but my purpose here is to help readers in their own evaluations and to try to guide the vendors' development efforts. I hope the situation changes, and that I start to find what I look for in social analysis tools.
InformationWeek contributing editor Seth Grimes is an analytics strategist with Washington DC based Alta Plana Corporation. He chairs the Sentiment Analysis Symposium, April 12 in New York, and the May 18-19 Text Analytics Summit.