Testing Subject Lines: English or Chinese?

This week I was finally able to take a little time out to do some A/B testing on my E-newsletter.

The list I tested on has recipients from all around the world, but the main bulk of folks are in the USA and China. I wanted to know which language of subject line would get better opens: English or Chinese

To do this testing, I used Mail Chimp, an online email mass mailing system. I really like it, because the user interface is super intuitive. I knew I could count on them for the testing functionally to have everything I wanted. Honestly, the reporting didn’t turn out to be incredibly robust: it only tells you which subject line got more clicks, but no metadata about those clicks. But for my purposes today, it did tell me everything I needed to know.

For today’s subject line, I used a piece of copy that I have previously used as a subject line which had gotten a solid open rate. Because I know that the subject content is effective, I would be able to attribute any major fluctuations in open rates to the language, which was what I am testing for.

All in all, this test went smooth. This test showed that English language subject lines had a slightly higher open rate.

Unfortunately, there is another aspect of this email which I didn’t take into account. I suspect that this was important influencing factor on open rates—that pesky caption! The caption shows up underneath the subject line, and can also influence open rates. That copy was completely new, hadn’t been tested yet, and was in English in both versions. So as of right now, it’s hard to say what is causing the change in open rates.

Even though I was unable to completely pinpoint any major insights, its a good jumping off-point for future tests. I am looking forward to running a new version of this test next week.