TDWI Selection Bias: It Depends Whom You Ask

The saying "There are three kinds of lies: lies, damned lies, and statistics" is attributed to Benjamin Disraeli, and it's nicely illustrated in a couple of Intelligent Enterprise reality-check photos from this week's TDWI conference.

Check out the TDWI Image Gallery photos posted by Intelligent Enterprise Editor-in-Chief Doug Henschen. Executive Summit attendees used the "dots method" to identify Important BI Technologies and Biggest BI Challenges. You get simple histograms showing what's hot and what's not. And you get clear illustrations of "selection bias": not TDWI's fault, but an effect to keep in mind when you assess formal and informal research findings.The Important BI Technologies results: 1 Dashboards/Scorecards; 2 Predictive Analytics; 3 Operational BI; and 4 BI Portals. My favorite topic, Text Analytics, is way down the list in a third tier. That's the reality check. (I hope this won't jeopardize the Text Analytics session planned for the August TDWI Executive Summit.) Open Source BI is even lower on the list.

These results aren't lies; no, they're statistics. Selection bias comes into play because a BI-themed session will attract folks grappling with technologies and solutions that are, well, in the solid BI mainstream. If you're out in front looking at text analytics, a BI-themed summit is probably not the place for you. Similarly, as I have reported, it's Java developers and not IT execs who are most interested in Open Source BI. It's a good guess that text-analytics types and OSBI types tend to self-select away from TDWI.

TDWI recognizes this effect. Philip Russom's 2007 report, BI Search and Text Analytics: New Additions to the BI Technology Stack, includes the explanation:

You'll find similar, similarly understandable examples of selection bias in the Biggest BI Challenges dots results, where #1 is "Gaining consensus on data definitions." Yup, that sounds like a BI/DW manager or exec speaking; the rest of us are stuck in our cubes, waiting for the execs to get back from TDWI in Vegas, keen to consensusize us.The saying "There are three kinds of lies: lies, damned lies, and statistics" is attributed to Benjamin Disraeli, and it's nicely illustrated in a couple of Intelligent Enterprise reality-check photos from this week's TDWI conference. Check out the TDWI Image Gallery photos for clear illustrations of "selection bias": not TDWI's fault, but an effect to keep in mind when you assess formal and informal research findings.