The best way to find a winning idea is to test far more of them than feels sensible, and to test them before launch, not after.
Companies that do this find the outliers that instinct would have filtered out.
In 2012, a Microsoft employee proposed a small change to how Bing displayed ad headlines. Managers ranked it low priority, and it sat untouched for more than six months. When an engineer finally ran a quick test, the change lifted revenue by 12%. That was worth more than $100 million a year in the US alone. It became the best revenue-generating idea in Bing’s history.
Nobody predicted it. The people whose job was to judge ideas had judged it unimportant. The only thing that revealed its value was a test.
At EPIC’s recent Growth Exchange event, Ogilvy’s Rory Sutherland argued that companies that test more tend to be disproportionately more successful. His reasoning was simple. Companies that test a lot end up testing things that seem silly, random or counterintuitive, and those are often the ideas that pay off most. If you only test what is obvious, you may miss the idea that changes everything.
If that is true, the winners are not the companies with the best instincts. They are the ones that test widely enough to find the outliers.
Does testing improve performance?
Jeff Bezos has said that Amazon’s success is “a function of how many experiments we do.” The evidence backs him. In a 2022 study, Koning, Hasan and Chatterji tracked high-tech startups before and after they adopted A/B testing. Among adopters, performance, measured by website visits, improved by 30% to 100% after a year. The gains came from using testing as a strategy. Firms launched more new products, scaled promising ideas and dropped weak ones faster.
Testing also corrects for how poorly experts judge ideas. At Microsoft, around two-thirds of all changes were found to have a negative or neutral effect on the metric they were designed to improve. When even carefully selected ideas so often fail to deliver their intended effect, opinion alone is an unreliable filter. Without testing, ideas that make things worse go live and reach customers.
Why the best idea is the one you don’t expect
The returns to new ideas are not evenly spread. A 2020 study examined about 1,500 experiments run on Microsoft Bing and found extremely fat tails: most tests made little difference, while a small number produced outsized gains, with the top 2% of ideas accounting for roughly three-quarters of the total benefit. These winners were hard to predict in advance, and Bing’s own pre-screening showed little ability to spot them. In that situation, the authors show, the better strategy is to test more ideas, even with smaller samples, rather than concentrating resources on a few large, high-powered tests.
Winning ideas often look wrong before they are tested. In his 2019 book, Alchemy, Rory Sutherland describes a charity fundraising test where the most rational-looking envelope cut donations by more than 30% against a plain control. Three less obvious designs raised them by more than 10%. In a world in which these “illogical” ideas are not tested, the charity would have tried only the rational one, and the three designs that actually worked would never have been found.
The lesson is simple. To find the winner, you have to test a wide variety of ideas. Testing only what seems sensible protects you from embarrassing failures, but it also caps your upside.
What does it cost to skip the test?
Skip the test, and the market runs it for you, at a much higher cost.
J.C. Penney learned this at scale. In 2012, new CEO Ron Johnson scrapped coupons and sales in favor of everyday low prices, across every store at once. When a colleague suggested piloting the change, he reportedly replied that they didn’t test at Apple. Sales fell 25% that year, and the company lost $985 million.
Netflix made a similar mistake with pricing. In July 2011, it split its DVD and streaming plans, effectively a 60% price rise for customers who wanted both, and applied it to its entire US customer base. It lost 800,000 US subscribers the following quarter, and its market value fell by 70% within four months.
A 2021 study tracked 83,719 new products across 31 US consumer packaged goods categories. One in four was no longer bought a year after launch. After two years, the figure was about 40%. For each of those products, the verdict came only after the money had been spent.
Every idea gets tested eventually. The only question is whether you test it before launch, when it is cheap, or after, when the market does it for you.
The testing gap
The obvious alternative is a controlled trial: try the idea on part of the market before rolling it out. For a web page, that is easy. It can be changed for a slice of customers and read within days, so a team can run hundreds of tests a year. For a new price, product or proposition, it is rarely that simple. A beer brand can reprice a pint in half the pubs it supplies, but it risks confusing drinkers, straining relationships with licensees and showing its hand to competitors. A packaged goods company can put a new pack format into a few hundred stores of one retailer, but that means a dedicated production run, winning the shelf space and waiting three months or more before repeat purchase shows whether it is more than a novelty.
Each of these tests is possible. The problem is how many you can afford. If a few ideas deliver most of the returns, and nobody can reliably say in advance which ones, the only way to find them is to test many.
Controlled trials make that impractical. At the cost and pace of a regional pilot, a business might run a handful a year, so teams ration their tests and spend them on the ideas they already believe in. The ideas most likely to become outliers, the unusual ones nobody is championing, never get tried. The search stops at the obvious.
What these decisions need is a way to test dozens of ideas, not a handful, before committing money in market.
How can you test dozens of ideas before launch?
Choice-based conjoint is that kind of experiment. Respondents make a series of realistic choices between competing offers whose features, price and brand vary systematically. Because each element varies independently, the analysis can isolate how much each one drives choice, much as a classic A/B test isolates a single change. But where an A/B test checks one change at a time in market, a conjoint study tests many elements at once, before launch. Companies can test offers that do not exist yet, simulate responses to thousands of combinations and estimate how much customers will pay for each feature.
Marriott used this approach to design Courtyard in the early 1980s. A conjoint study of 50 hotel attributes found that business travelers placed little value on features full-service hotels treated as essential, such as room service and an upscale restaurant. Marriott designed around the data, test marketed the concept and rolled it out nationally. Courtyard now has more than 1,300 hotels worldwide.
Courtyard shows what a broad search can find. In conjoint studies, unusual features or unexpected claims can sit alongside the safe options, at a fraction of the cost of an in-market test. The ideas nobody believes in get a fair hearing, and companies can decide on evidence which ones deserve to be brought to market.
Send us the ideas that didn’t make the shortlist. Whether it’s a new price, product or proposition, or a list of options too long to pilot in market, EPIC can help you test them side by side before you commit budget to any of them.
The companies that gain most from experimentation are not the ones with the best instincts. They are the ones willing to test the idea nobody believes in.
References
- Azevedo, E. M., Deng, A., Montiel Olea, J. L., Rao, J., & Weyl, E. G. (2020). A/B testing with fat tails. Journal of Political Economy, 128(12), 4319-4377.
- J.C. Penney Company. (2013). Fourth quarter and full year 2012 results. Company release, 27 February.
- Kohavi, R., & Thomke, S. (2017). The surprising power of online experiments. Harvard Business Review, 95(5), 74-82.
- Koning, R., Hasan, S., & Chatterji, A. (2022). Experimentation and start-up performance: Evidence from A/B testing. Management Science, 68(9), 6434-6453.
- Mattioli, D. (2013). For Penney’s heralded boss, the shine is off the Apple. The Wall Street Journal, 25 February.
- strategy+business. (2012). Netflix wasn’t all wrong. strategy+business, 2 April.
- Sutherland, R. (2019). Alchemy: The surprising power of ideas that don’t make sense. WH Allen.
- TIME. (2011). Netflix loses 800,000 subscribers after price hike, Qwikster debacle. TIME, 24 October.
- Victory, K., Nenycz-Thiel, M., Dawes, J., Tanusondjaja, A., & Corsi, A. M. (2021). How common is new product failure and when does it vary? Marketing Letters, 32, 17-32.
- Wind, J., Green, P. E., Shifflet, D., & Scarbrough, M. (1989). Courtyard by Marriott: Designing a hotel facility with consumer-based marketing models. Interfaces, 19(1), 25-47.
EPIC Insights provides expert-led product and pricing conjoint software that develops actionable retail, wholesale and food service insights in just 7 business days. By combining a cutting-edge research platform with deep domain expertise, EPIC empowers global leaders like Coca-Cola, L’Oréal, Kraft Heinz, PepsiCo, Decathlon, Philips and Harley Davidson to make precise, data-driven commercial decisions at scale.
