Ian Ayres: Success Starts With Random Tests - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Software // Information Management
Commentary
4/2/2008
01:40 PM
Doug Henschen
Doug Henschen
Commentary
Connect Directly
LinkedIn
Twitter
RSS
E-Mail
50%
50%

Ian Ayres: Success Starts With Random Tests

"If you're not relying on the twin pillars of regression analysis and random tests, you're making a big mistake." This was the key message delivered this morning at the Gartner BI Summit by keynoter Ian Ayres in his talk on "Why Thinking-By-Numbers is the New Way to Be Smart." Drawing from his book "Super Crunchers," Ayres said random testing will give you "at least a 5 percent to 10 percent improvement in any measure you care about."

"If you're not relying on the twin pillars of regression analysis and random tests, you're making a big mistake." This was a key point delivered this morning at the Gartner BI Summit by keynoter Ian Ayres in his talk on "Why Thinking-By-Numbers is the New Way to Be Smart." Drawing from his book "Super Crunchers," Ayres said random testing, in particular, is the best way to get into statistical algorithms, and he guaranteed the measure will give you "at least a 5 percent to 10 percent improvement in any measure you care about."Ayres' larger point was that it's time for companies to move away from gut feel, tribal knowledge and intuition in decision making and move instead toward crunching the numbers. He cited dozens of interesting mainstream "Super Cruncher" examples in which statistical analyses are being applied to decisions:

Epagogix is helping the movie industry predict film revenue before they go into production by coding up scripts, stars, directors and other known variables and then doing regression analysis against historical results. Pre production, Epagogix predicted that the Nicole Kidman film "The Interpreter" would gross $69 million, and, in fact, it ended up grossing $73 million - a more accurate forecast than most movie studios are used to.

Professor Orley Ashenfelter of Princeton caused a stir in the wine industry by predicting the quality of wines based on statistical analysis of growing conditions (average temperatures, rainfall, etc.) compared with historical growing conditions for top wines.

eHarmony does regression analysis of personality characteristics to suggest possible matches that other dating sites with simpler methods might miss.

Farecast predicts whether airline ticket prices will go up or down based on regression analysis of historical pricing patterns, and it even displays a confidence level and sells fare insurance on its predictions.

Delaycast predicts the likelihood of flight delays based on historical patterns tied to up-to-the-minute variables such as weather conditions.

A big believer in random tests, Ayres used the technique in picking the name of his book, finding that Web ads for "Super Crunchers" outperformed "The End of Intuition" (another name he had in mind) by 63 percent.

"One reason I like random tests is that the results are so easy to communicate, and easier it's easier for people to understand and believe than regression analysis," said Ayres.

Ayres recommended Web sites as the natural place to start random testing (and he touted the free Google Web Optimizer as a helpful tool). Want to find out if online marketing drives or cannibalizes offline sales? Start by randomly turning on and off the online marketing campaigns at random times in randomly chosen cities. Keep track of the impact, and once you have the results, segment the results to see if the impact varied among different groups. Ayres cautioned practitioners not to throw off the results by picking test cities or times in a non-random fashion. rather than determining everything will a random selection tool such as a coin toss.

Even if the humans involved in a decision are "subject matter experts," Ayres says countless studies show that they're more accurate than statistical analyses only about 6 percent of the time, usually when there are few variables. As the number of variables and complexity increases, empirical analysis wins every time."If you're not relying on the twin pillars of regression analysis and random tests, you're making a big mistake." This was the key message delivered this morning at the Gartner BI Summit by keynoter Ian Ayres in his talk on "Why Thinking-By-Numbers is the New Way to Be Smart." Drawing from his book "Super Crunchers," Ayres said random testing will give you "at least a 5 percent to 10 percent improvement in any measure you care about."

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Slideshows
10 Ways to Transition Traditional IT Talent to Cloud Talent
Lisa Morgan, Freelance Writer,  11/23/2020
News
What Comes Next for the COVID-19 Computing Consortium
Joao-Pierre S. Ruth, Senior Writer,  11/24/2020
News
Top 10 Data and Analytics Trends for 2021
Jessica Davis, Senior Editor, Enterprise Apps,  11/13/2020
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
Why Chatbots Are So Popular Right Now
In this IT Trend Report, you will learn more about why chatbots are gaining traction within businesses, particularly while a pandemic is impacting the world.
Slideshows
Flash Poll