GETTING THE MOST FROM YOUR CATALOG TESTS

THE CATALOG DOCTOR
GETTING THE MOST FROM YOUR CATALOG TESTS
First published on MyTotalRetail.com blog November 2015
© 2015 Susan J. McIntyre                                
            
PATIENT: “Doc, we do tests, but I don't feel that we learn much from them. What are we doing wrong? What should we change?”
 
CATALOG DOCTOR: "Try building your tests differently to get clearer, more actionable results. Here are some tips for what to do, and what not to do.”
 
OPTIMIZING AN OFFER TEST
 The goal: determine which type of offer delivers the best sales and Return On Investment in both the short and long run. Here are 3 keys to getting useful results.

1. Assign a unique promotion code for the test offer, and have that code appear only in the test catalog.
2. Look at test results both early and late — they often change over time.  Early results may make “A” appear to be the winner, but 6-month or 12-month results may show that “B” is the actual winner.
3. Track multiple result indicators including Response Rate, Average Order, Sales-per-catalog, and Return on Investment. You may decide that Response is more important than ROI (for example, if your offer is to prospects and your goal is adding the most new names to the file), or you may decide that ROI is most important (for example, if your goal is maximizing profitability). 

DON'T DO THIS: splash the test-offer code all over the website, where folks not in the test group can see and use it. That may be easier operationally, and it may garner more sales overall, but it's not a test at all, so you'll learn nothing.
 
 OPTIMIZING A HOLDBACK TEST
 Also known as “holdout”, this test answers the question “Do we really need to mail a catalog since 90% of our sales come over the web?” There are 2 keys to getting clear results.

1. Identify the names to “hold back” (that is, names that won't get mailed the catalog). Flag those names and keep those exact same names held back for the entire period of the test. And keep them held back even if they order during the test period.  DON'T DO THIS:  keep adding and subtracting which names get held back and which get mailed while you're still in the test period.
2. Let the hold-back test run for long enough to really tell if sending a catalog is making a difference or not. Example: if you normally mail 6 catalogs during a 4-month period, then hold back all 6 catalogs from your holdback names during that period. Whatever your catalog timing, holding back a sequence of 4 to 8 catalogs should be enough to give a clear reading.  DON'T DO THIS: just skip one catalog mailing instead of a longer-term mailing sequence. Skipping only one catalog won't tell you if you do or don't need to mail catalogs at all, it will just show that some names may do as well with fewer (but not zero) catalogs.

 OPTIMIZING A CREATIVE TEST
 Protect your control. That is, don't redesign your control at the same time as you test a new design. This applies to copy, design, photography, rebranding or a combo. When testing, you need to (A) protect the sales that your control has already been delivering, and (B) see if new, different creative can deliver even more sales.
 DON'T DO THIS:  True story. An in-house design team was building the control catalog. An outside design firm had been hired to build a new test design. The outside firm had to submit all test pages to the creative director of the in-house team. Whenever that in-house creative director saw something in the test catalog that he liked, he incorporated that new idea into his control catalog. When mailed, the test and control delivered approximately the same results (because they ended up looking about the same as each other).  Result: nothing learned.