Pre-testing reduces effectiveness

In the current issue of the Marketing Society's journal 'Market Leader', Les Binet & Peter Field make the somewhat contentious claim that, far from improving the chances of getting an effective ad out, quantitative pre-testing actually reduces your chances of success.

Their data shows that ads that have not been quantitatively pre-tested have a 71% chance of being effective whereas those that have been pre-tested have only a 44% chance of success:

Their analysis is based on the IPA dataBANK of 880 case studies. The 'Market Leader' piece is based on their recent book Marketing in the Era of Accountability which analyses the dataBANK in detail.

In the article Binet & Field provide very little to explain this poor performance by the pre-testing industry apart from deriding the overreliance by pre-testers on 'standout' as a metric.

Whilst pre-testing has it's detractors (and I must admit to being less than comfortable with the claims made about most if not all of the pre-testing methodologies that I have encountered), I'm sure that there are factors at play here apart from the use of the wrong metrics. For me the biggest factor influencing the result we see above is the difference in the cultures of the clients that pre-test and those that don't.

Comments

[Malcolm Murdoch]

Ideas that are excellent often convince people not to pretest them. This might also be a factor - if the client's not convinced enough to approve it without research then maybe the idea isn't good enough?

[Scamp]

Hilarious!

Though I definitely agree with you that "the biggest factor influencing the result we see above is the difference in the cultures of the clients that pre-test and those that don't."

Nike and Honda never test; dfs and Halifax do. But if dfs and Halifax stopped pre-testing overnight they wouldn't suddenly start producing Nike-quality ads.

[Jason Lonsdale]

Excellent stuff!

Yep, the certainly culture is part of it, but there is strong evidence that pre-testing will all-too-often lead to work which adheres to the advertising norms of the category.

Respondents will generally prefer the familiar over the original, because it follows their conception of what an ad for that category is like.

Slavinshly believe in pre-testing and you end up making a beer ad wherein three guys (always three -one is a loser, two are gay) walk into a pub, something funny happens and they all have a drink.

Ignore the pre-testing and you get to create 'Surfer' or 'Jacques de Florette'.