Strategy & Best Practices

Marketplace A/B test results

By Mary Becica / Jan 09, 2012

I recently ran an A/B test for the marketplace home on appcelerator, comparing engagement and lands to the profile pages.

Control:

AB-testing-control.png

Variation:

AB-testing-variation.png

The result: No winner! The changes were not significant enough to have a clear winner. The details:

The variation had a -4% change in engagement (that's any clicks on the page at all), and +3% change in getting users to the app profiles. These percentages are too small to be significant. Looking at the heatmaps reveals more:

AB-testing-control-heatmap.png

AB-testing-variation-heatmap.png

what does it all mean?

We all thought that the varation would win, since it shows more on the page. More == better, right? More things to click == more clicks? But the first test is one of engagement. This means that users are clicking around and not necessarily leaving the page, but doing multiple clicks. The sliders have something going for them - they are interactive. User thinks there might be more there that they want, so user keeps clicking. That's engagement. You can see it in the heatmap - one click on the control means you've moved the slider, and you'll probably click again. One click on the variation design means you've left the page.

It is nice to note that the variation got more users to the profile pages. This could be because we're showing more stuff to click to get there, but +3% is a bit measley. For that metric, more == more. But only 3% more. We can certainly do better than that.

The Google

Remember how we debated the pros and cons of google's marketplace? It is very relevant here. They probably did a test of engagement of the before and after, and saw something similar to what we're seeing. Before: lots of things to click, all clicks take you away to a new page. After: lots of things to click, clicking leads to more interaction on the page you are on. Google went for the interaction winner. (It is also good to note here that Google ramped up its marketing engine when they launched the new store, which is rumored to be why the numbers jumped so extremely).

Do we want interaction? What does having interaction mean for an ecommerce customer?

You walk by a vaguely interesting store. It's got all sorts of attractive things in the window but you're not sure what their function is yet, and it seems like the store has lots of different types of stuff in it. What do you do?

I'd say, browse. I'll make the leap here to say browsing can be measured best by engagement. My first step as an interaction designer is to connect user expectation to experience; if the user is looking for a browse experience, we should improve ours.

What we've got now.

We've built the site with a much more page-to-page interaction model. Engagement for us can be measured across pages: user goes to marketplace, listing, filters on listing, goes to profile, clicks to another profile from the "also bought" slider. However, page-to-page engagement requires a user who is already excited about the types of products, and is hunting. It is more difficult to get the lost and confused user (think bell and sasktel, here) to engage in this way, and easier to get them to engage on one, non-threatening page.

This is why we are seeing so much engagement on the listing page, and so much bounce on the marketplace. The user is finding their browse experience on the listing; they get feedback and control on what they see, don't have to make any commitments, and aren't interrupted by page loads.

Intention is best

You may ask, why have a marketplace page at all? What is the difference between the listing and the marketplace, if the marketplace becomes an interactive experience?

The marketplace is a curator. The channel can design its layout, choose their categories, and decide what apps to feature over others (that's half built). It is the first communication to the user: "This is what you should be looking for in this store". The listing page is a search results page, which the user should only land on after they have tried searching (this is not the current behavior, unfortunately), and has no bias. They are both distinct pages, useful for different types of browsing.

What next?

I'm going to begin building a new marketplace homepage with our shiny new app api (thanks Christophe). Using backbone.js, we'll be able to easily call our api with attributes as they are required. I am hoping to get the entire catalog to quickly load as the user requests it. I am going to do page mockups of the interactions first before we do visual design. This is a bit of an inverse of design process, but I'd like to get us thinking in interactions for this page before visuals, especially since our visuals are fluid (channel-specific).

In a dream world, lets say we get a high engagement rate on the marketplace page. People are filtering and finding more relevant things. They understand the scope and size of the marketplace better because the page expresses it clearly. Does this lead to more eventual conversion? We'll just have to run some more tests, get more feedback, and keep iterating.