Three Testing Triumphs

Three Testing Triumphs

I have run thousands of split and multivariate tests in my career. Here are three highlights involving a full website, a web application and emails.

Continually tested the corporate website post-redesign for a massive 159% increase in form completions in 10 months.

Company: Hair Club

In February 2012 was completely redesigned to make SEO improvements related to site speed, page tagging, content and organization, as well as overhauling the brand. Despite pre-usability testing, best practices and sleek design, the new site had a slightly lower conversation rate at launch.

Key pages on the site were A/B tested multiple times. Elements throughout the site were multivariate tested as well. Items as small as the text of the header and color of a button to the entire sidebar form and navigation, which are displayed on every page. Some tests yielded higher improvements than others, but the site was continually testing one or more mutually exclusive items/pages. A management tool was even created to list ideas, schedule and log results.

Here are a few specific examples:

  • A form was added to the homepage, improving the website visit to form-fill conversion rate by 37%.
  • Several versions were created for the appointment scheduling page. The winning page increased the lead to appointment rate 21% by keeping the calendar, timeslots and submission button above the fold for X x 800 screen resolutions, but keeping the center address and directions below the fold.

(Adding together all the conversion rate improvements for this corporate website, as well as 100+ microsite/landing pages and emails, totals a 7000% increase over the last 3 years).

New Website:
Older Version (2011):


Redesigned a 10 year old application to improve user satisfaction by 20% and save around $75,000 a year despite legacy back-end infrastructure limitations.

Company: FPL (Fortune 200 Company)
The application is for reporting a power outage. To identify pain-points, qualitative feedback was gathered from users of the application and several expert review session were held. During the design phase, Photoshop files were reviewed by both a random non-employee group and the working and management team. And, of course, at both the prototype and draft phase the actual new design was usability tested. The strategy was adjusted during all testing phases, which yielded the positive results.

The following were key issues:

  • Users did not have or know their account number.
    The Solution: Options were added to find the account using the website login, account phone number and SSN.
  • Users wanted to follow-up on already submitted forms.
    The Solution: A new feature was added and highlighted to allow users to check the status using select information from existing internal systems.
  • FPL needed to reduce unnecessary calls, saving an average of $500 per ‘rolled truck’.
    The Solution: The most common cause was identified to be breaker resetting, so detailed information as well as an instructional video was added, visible on all pages. All power outage was separated from some power outage and flicker/dimming lights to allow for more specific self-troubleshooting instructions.
  • Branding and design was poor (the application did not look legitimate) and did not follow best practices.
    The Solution: Especially since drop-off is not actually in issue with this form, the application was kept within the normal site header and footer. Fonts and colors were updated to match brand standards. Links to other related applications were added and highlighted, such as a Power Outage Map Tracker or requesting a tree trimming. A slider was also used to show the user where they were in the process.

New Application:


Tested and redesigned database emails to yield 7 times more sales.

Company: Hair Club
To achieve this improvement, the new emails followed best practices, including responsive mobile display and text to image ratios. The emails were also segmented into four categories by gender and language (English vs. Spanish) and personalization, such as product interest, name and city, was utilized as appropriate. CTAs were made clearer and more were added, including a phone number. Four different subject lines were A/B tested for each email, as well as four hero images. The landing pages also matched the email creative for consistency.

The improvement lead to just under half a million dollars in annual revenue.


Leave a comment