I'm a big fan of testing and optimisation. When user feedback is so valuable, yet so readily available, and real-time, can you really justify not incorporating it into your working practices and developing a culture around test and learn? Smart, digitally focused businesses like ASOS, Netflix and Zynga have been enormously successful through doing just that. But there are some things that optimisation can't do. Like come up with a radical, game-changing solution to a problem. An idea that is entirely different from any which is currently in play. A disruptive innovation.
For that, you have to have a vision. Testing won't give you disruptive ideas. Focus groups won't either. Then is the time when you rely on the art, not the science, in innovation. Then is the time to embrace a bit of chaos.
Agile organisational cultures needn't preclude discordant ideas. In fact they should thrive on them. The companies that really get it, are those that encourage divergent, not convergent thinking around a powerful vision, and then test and learn as (and not before) they build and execute it.
(HT to this post on Apple design hacks for the title inspiration for the post)