The ABCs of Product Experimentation

Product experiments are one of the most powerful tools product managers can leverage to build better products faster.

Teams that recognize the power of product experiments are well-positioned for the future. By encouraging your team to set simple goals, share new ideas, and test them at a quick pace, you’ll be able to use experimentation as a way to improve all aspects of your business.

Incorporating experimentation into the development process helps teams:

• Stop guessing (no more building useless features);
• Innovate through customer insight;
• Iterate quickly.

Product development vs. product experiments

Product development is about making your product solve your customers’ problems. And the process needs to be frictionless to be successful.

But how do you solve a problem you don’t know it exists? How can you find out what would improve your customers’ experience?

Product experimentation is the key. By improving the value that is already developed, you can make customers’ experience more delightful.

And the best part about product experiments is that they can be conducted at any stage in the product development cycle. You can test an idea before you launch it, or you can improve a feature after it has already been rolled out.

We’re pleased to see more and more teams looking for ways to iterate gradually and validate their ideas with data. More experimentation means faster development and better products.

But like any trend, product experiments can easily go wrong. We want to save you the pain of costly mistakes, so here’s how you could play with product experiments.

3 simple but surprisingly practical product experiments

1. Fake door testing

Fake door tests, also known as painted door tests, can be used to validate customer demand. When you’re not sure whether a feature is worth developing or not, you can create a minimal user experience (for example, a button with the name of the feature on it), and measure how many users engage with it.

If enough users click on your button, you can confidently add that feature to your product roadmap.

2. Solving debates with data

A product team’s goal is to make sure the language in the product is descriptive. Using product experiments, you can debate ideas that can be validated with data. Having data to inform how you can update your product via an experiment is extremely valuable.

Run tests until your results are statistically accurate, then go for the winning option. There’s no need for heated debates within your team.

3. Making your key features more visible

First, find out which product features lead to higher retention rates. Once you identify them, use heat maps to find out how discoverable these features are for customers.

Chances are, your best features are not always the easiest to find. And you could really benefit from changing that. By making your key features more visible, you get more usage and, in turn, a higher retention rate.

“One competitor everyone has that they don’t realize is change. Change is the number one competitor we all have when we’re selling any product in any market. People don’t like change.” – Sean Sheppard

Keeping this in mind, it becomes even more important to conduct experiments before tweaking your product. You’ll have the opportunity to discuss more product experiments and growth hacking with Sean at How to Web 2019.

Meanwhile, let’s take a look at how Fever, the mobile event discovery startup, uses experimentation to improve their product.

Case study: How Fever used experimentation to rapidly improve key business metrics without hurting the user experience

Fever is an event and experience technology company focused on inspiring users and helping event organizers generate demand. Using a recommendation algorithm, the app features personalized listings and transactional capabilities for everything from music festivals and gigs to theatre, fashion, restaurants, and pop-ups.

Launched in 2014, the Fever mobile app and website, together with its Secret Media Network, reaches and inspires over 12 million unique users per week across its key markets (London, New York, Paris, and Madrid), allowing them to discover the best events in their city.

As the startup reached break-even and switched to rapid growth mode, new challenges emerged: how do they rapidly improve key business metrics without hurting the user experience?

After an initial analysis, the team set out to implement the following changes:

• Data-driven development – KPIs, metrics, dashboard, data deep-dives;
• A/B experiments;
• Qualitative research.

Everyone wants to be as data-driven as Google, Netflix, or Booking.com, but many companies fall into the common pitfalls of focusing on the wrong metrics. Here’s what Fever did:

The grid view experiment

The Fever app shows one event at a time in a feed. The team hypothesized that showing six events per screen in a grid view, although with less detail, will grow impressions, click-throughs, and, eventually, purchases.

 

 

The experiment version raised the number of events seen by users by only 11% on average but decreased CTR by 23% (possible reasons included less info on each event, more distraction) and reduced conversions to purchase by 24%. Overall, revenue per user went down by 23%.

The Fever team learned it’s essential to give users enough information in the feed to make a decision before entering the event.

Keyword search experiment

The Fever app offers a curated list of events + the ability to filter by category and area. The team thought that adding keyword search will help users discover events of interest quicker and will lead to more conversions.

 

 

This experiment was a success. Both CTR and conversion to pay went up by around 7% each, and total revenue per user shot up by 12%!

The best products and the most successful companies may seem like they were predestined, but anything worth talking about requires iteration. Having said that, data and experiments are only half of the growth story. It’s just as important to step back and try to identify the key driving forces of growth in a company.

Remember, product experimentation paves the way to innovation. And innovation is not about big budgets and large teams. “Having a lot of money is not a measure for being innovative” (Dan Toma).

Want to learn more about product experimentation and how your startup can make a difference from Dan Toma? Join us at How to Web Conference 2019 on the 30th & 31st of October.

Get your early bird ticket here: https://www.howtoweb.co/buy-tickets/

Illustration credits: vecteezy.com

0