What To Do About False-Positive Alerts, Non-Collaborative BI and Inaccurate Metrics

I've been hinting at some points for quite a long time, without really spelling them out in written form. So let's fix that. I believe:

I shall explain.False positives on alarm systems are a huge problem. Alarms were ignored (indeed, turned off) on the Deepwater Horizon due to too many being false. When I had a fire in my house, the smoke alarm wasn't on because of its penchant for earsplitting false positives. And false positives held back the software category of intrusion detection systems until it was eventually subsumed by related kinds of security appliance. Similarly, false positives are an under-appreciated problem that restrains the advance of business intelligence software. It's no big inconvenience when, out of your 3 biggest problems this week, your dashboard calls attention to 17 of them. But suppose you want to make alerts more granular, take them up in quantity a couple of orders of magnitude, and have your smartphone pinged each time the alert condition is met. Whoa. Now those bogus warnings start to be more of a consideration. Why would you want some kind of "push" alerting? Use case circumstances include but are not limited to anything that might herald:

Why would these alerts ideally be granular and high-volume? Well, you might have lots of machines, lots of trucks, lots of customers, lots of potential investments, and so on. I could list many more examples, but these probably suffice to at least start the discussion.

I'm agnostic about what the UI would be. Sounds going off on your mobile phone? Email on your mobile device? An app more like a chat window? Something more like a mobile dashboard? Whatever it is, false positives would really screw up your work day.

Classical enterprise dashboards suffer from a similarly fundamental problem -- dashboard technology is optimized for a screwed-up enterprise analytic methodology. That is:

Yes, you can do drilldown or even data exploration, QlikView-style or otherwise. You can communicate around the BI tools, whether via general portal technology or something built-in. Vendors keep trying to make it easier for you to build your own reports, charts, and dashboard elements. But you can't very easily define and track your own metrics, and I don't see a lot of effort toward making that better.

Perhaps vendors don't try to provide strong metrics management because doing so might lead to -- gasp! -- more than "one version of the truth." If so, such thinking is regrettably misguided. Examples of formulas -- i.e., metrics -- that should and can not be cast in stone* include:

*At least as a general rule -- we surely can think of some simple cases or exceptions where extreme cookie-cutterness is just fine.

It's not that there shouldn't be preferred or default metrics for these quantities. Rather, my point is that individuals should be allowed to:

Why? Because the person closest to the situation may be the one best able to assess it, and also because recognizing that fact is a really good idea from the standpoints of employee morale, engagement, and even development. (Or you might say -- shiny buzzword! -- employee "empowerment".) In particular, when a boss and her subordinate disagree about the ideal way to calculate a metric, they should be able to track both versions side-by-side. When there's a significant difference between the two, that could be a fine time for an out-of-the-norm decision.* If that ability were available, portal-style BI collaboration would almost automatically come into its own.

*What kind of decision that is of course depends on the specifics of the case. If only one of two competing metrics says you should bother giving extra care to a specific customer, the answer is easy - you give it. In other cases, an actual meeting or other conversation may be in order to decide what to do.

A metric is just a function. To offer BI users the flexibility I want, it should be very easy to use these functions as inputs into other, custom functions. I.e., I'm talking about something with the flexibility of a stripped down Excel, although the UI would be more like that for a rules engine. And if you think about it, that's exactly the same functionality needed to personalize your alerts and weed out false positives.

One last point -- what I'm calling for could greatly increase the BI performance burden, perhaps even by three orders of magnitude. Well, that's what we have all this great new super price-performance analytic database technology for. And anyhow the throughput hit won't come overnight. It will be interesting to see where the boundaries end up between DBMS and inside-the-BI-tool analytic processing. But overall, the analytic DBMS industry -- and the hardware vendors backing them up -- should be able to handle anything the BI vendors throw at them.

Related links

"Push" alerting technology could be much more granular and useful, but is being held back by the problem of false positives. Metrics passed down from on high didn't work too well in Stalin's USSR, and haven't improved sufficiently since. A piece of the solution to both problems is an engine for setting and modifying metrics definitions.