Hamburger vs Menu: The Final AB Test

In case you missed this post about an AB test on mobile menu icons, make sure you check out the comments. There are some very interesting insights about A/B testing and its shortcomings.

The post went a tiny bit viral, and suddenly it wasn’t just my mother reading this blog.

Three things I learned:

1. This iconhas lots of names: hamburger, sandwich, and even hotdog ?! What it actually is, is a list icon. We’ve just co-opted it to mean a navigation menu.

2. When something gets noticed, some people get a little mean (source)

3. One commenter said I was the Dunning–Kruger effect in action. This phenomenon is when you try to sound clever but are actually a dumbass.

Thanks for the vote of no-confidence.

In this hyper-connected world full of rockstar developers and super-smart designers, I’m humbled on a minute-by-minute basis. I might need to start attaching positive affirmation stickers on my laptop.

The Final Hamburger A/B Test

I do enjoy A/B testing, but conclude what you want. I’m not an expert, nor am I advising anything, but sharing what happened on a single website.

Using a commercial A/B testing service can get very expensive very quickly, and well beyond the budgets of small-time web designers and developers. So, hopefully, these posts are helpful for some of you.

If you are using social sharing buttons, you might find these tests interesting.

Variation 1

Bordered list icon (hamburger).

Variation 2

Bordered word menu.

Results

240,000 unique mobile visitors were served the A/B test.

VariationUnique VisitorsUnique ClicksHamburger1205431211Menu1211521455

The test was large enough to achieve statistical significance.

The MENU button was clicked by 20% more unique visitors than the HAMBURGER button.

Where things get interesting is when we break down the data a little:

Unique VisitorsHamburger ClicksMenu ClicksiOS1480979060.61%11430.77%Android872452160.25%2370.27%

There is very little difference in the Android user preference, but their lack of engagement is disturbing.

Interpretations

Hamburger icons may appear to be ubiquitous, but they are not the only option.

There is an issue that is much more important:

Android users are almost 3x less likely to click a navigation button than iOS users.

Why?

Caveats

• These are the results from one website (see more about demographics here).
• The test was done using some in-house code, so I cannot guarantee the perfect execution of code across all devices. I do not have time or capacity to rigorously test code like the big commercial AB testing services like Optimizely. Bear in mind that to run this test with Optimizely would have cost $859 (I kid you not).
• I can’t measure intent with this test. I’m measuring clicks on a webpage. Maybe the user thought menu as a list of food to order. Maybe they wondered what the hamburger icon was and tapped it. Who knows. AB testing cannot tell you this.

Source