Concept Testing: Research Methods and Examples
The Idea in Brief
THE CHALLENGE
Approximately 95% of new product launches fail, according to Clayton Christensen, a professor at Harvard Business School. Some well-known failures in history are New Coke in 1985, Crystal Pepsi in 1992, and Google's Glass project.1
WHY IT HAPPENS
Product launches can fail if companies don't understand customer needs. They can also fail when there is a poor product-market fit.
THE SOLUTION
Thankfully, there are ways to mitigate product launch failures. Conducting market research and testing concepts before launching can improve your chances of success.
Choose among different market research types based on what stage of product or service development you are in and your business need. For instance, you can check if there is demand for your product. You can determine your market size, potential share, focus on key features, create pricing models, and more.
“Concept testing is an essential step in the product development process as it helps reduce the risk of failure and optimize the final product… concept testing provides valuable insights into consumer preferences, identifies potential issues, and allows for necessary adjustments before significant resources are invested. This process helps ensure that the product aligns with market demand, ultimately increasing the likelihood of its success.”
— REID & CRAWFORD, 2022, SPRINGER
Why Test New Business Ideas?
A product that seems great on paper could be a miss in real life. When you include concept testing in your product development process, it can reduce the risk of your product failing. It also helps you save resources as you work toward achieving success and profit. Ask the right questions and do the right tests early on to validate your idea and improve your chances of success.
In Testing Business Ideas, Bland and Osterwalder suggest using concept testing and experimentation to assess two main risks:
Desirability risk:
The risk that customers aren’t interested in your ideaViability risk:
The risk that you can’t earn enough money with your idea
Test concepts carefully when developing a product or service. This helps find dangers and can even give insights on branding and messaging strategies at the beginning of the process.
For example, Qwerry helped Kolabree test their use-case hypotheses, validate product-market fit, and prioritize product features. Insights like these can directly influence production design and your future user experience.
We applied this method for concept testing with Kolabree and provided additional insights beyond their original goals. We used cost-effective research methods allowing us to run shorter experiments with better data quality. Our data informed brand strategy, marketing strategy, messaging themes, buyer personas, market segmentation, and the marketing campaign, going beyond what some typical concept test surveys usually provide.
MARKET RESEARCH RULES OF THUMB
Follow these guiding principles in your concept testing process:
At first, go as cheaply and quickly as possible.
If you learn early that your idea won’t work, you can stop or pivot before spending valuable resources.Don’t put all your eggs in one basket.
When you can, use complementary experiments to layer pieces of evidence together. This reduces uncertainty even more.Go for the tightest case.
Use experiments that give you as close to real-life evidence as you can get, always balancing research and business needs. This usually means balancing costs with the level of research rigor and sample sizes, for example.
Concept Testing Examples
Concept tests can be divided into two types: Discovery Tests (desirability risk) and Validation Tests (viability risk). And this framework lines up nicely with common Business Model and Value Proposition Canvas frameworks. We appreciate Bland and Osterwalder (2020) for their helpful ideas on testing business concepts. We have included these ideas in our processes as a research firm.
Too many concept test approaches exist to outline them all in one article. We’ll start with a focus on two discovery tests and quickly introduce some validation experiment types. We’ll dive deeper into more concept testing types of both kinds in future articles.
Start by using this method to evaluate new ideas with your product team, product manager, and marketing team members.
DISCOVERY
Customer Interviews
Discovery Surveys
VALIDATION
Validation Surveys
Mock sales
Customer Interviews
OBJECTIVE
Get feedback on your new idea with your target market to understand their pain points, needs, and wants. These qualitative interviews allow you to go deeper into your audience’s thinking and get customer feedback to understand:
why your new product is (or is not) needed
how it may (or may not) be used
how it will (or will not) enhance the customer experience
whether there is willingness to pay for it
These interviews also provide inputs into discovery and validation surveys further in the process.
APPROACH
Customer interviews are qualitative interviews. Your approach to these can be simple. Remember, cheap and fast at first.
1. Identify your ideal audience
Who will benefit from this new product or service offering? You’ve probably already answered this in your business model canvas. These are the people you want to talk to in this early-stage research.
For B2C concepts, consider demographics like age, geography, and income level as well as behavioral characteristics. For B2B concepts, consider job title, level, industry, business needs, etc.
2. Develop your guide
Your guide should explain the idea to the customer with a detailed description of your product or service, or a mock-up. Articulate the specific gains and pains it addresses.
During interviews ask about those pains and gains. Did you get them right? Probe on how they think your concept will solve for them. You can ask what they are willing to pay, and other things.
Keep the interview short but not rushed (30-45 minutes is good). Your guide should use open-ended questions. For example:
"Can you describe challenges or problems you face related to this product?"
"What features or qualities are most important to you in a product like this?"
"Have you used any other products to address this problem? How do they compare?"
3. Recruit
Talking to customers can be helpful. They might share their thoughts with you for free or you could offer them a discount on a current product perhaps.
Or you can offer incentives for their time. Qwerry uses Tremendous to easily compensate research participants. Research from Tremendous indicates incentives play a crucial role in attracting participants and collecting high-quality data. They increase response rates by up to 19% while improving participant retention and response quality.”3
At this stage, about 15-20 interviews is a good target.
If you are interviewing different subgroups, aim to interview 10-15 people in each group. This could include men, women, different generations, and so on.
4. Conduct interviews and analyze
Use recordings and transcriptions to pull out direct quotes or even short video clips to include in your final report. These can help in telling a compelling story to stakeholders.
5. Consider non-customers
We called these “customer interviews” because they are a cost-effective option for this early-stage research. Another option is to recruit outside participants through a research firm. Investing wisely in people who are not your customers can give you a broader perspective and improve your evidence.
We mentioned layering pieces of evidence earlier in our research as a rule of thumb, this is an example. This can also generate new early adopters if your interviewees are excited about your product idea.
Discovery Surveys
OBJECTIVE
Surveys are usually considered quantitative, but they can be qualitative. Use short, open-ended survey questions to uncover pains and gains, unmet needs, and more – with more scale than with Customer Interviews.
Qwerry uses a Discovery Survey to identify attributes needed for our brand tracking and category driver research. Based on the results, we create a list of attributes. We often uncover things client teams didn’t think about. The approach helps us understand market needs – strengthening our research evidence.
APPROACH
1. Develop your survey
Since it is open-ended, it should be short. Our Discovery Survey has about 5-7 questions following a 5W’s approach. In our recent coffee category research, we asked:
Why do you usually go to a coffee shop or café?
When do you do you usually go to a coffee shop or café?
What activities or occasions prompt you to go to a coffee shop or café?
Who else influences whether you visit a coffee shop or café?
These are just select examples. Use information from customer interviews to create questions. These questions should link different research findings. This will help you build a clear story based on evidence.
2. Program, Test, Field
Use a survey platform to host a Discovery Survey or engage with a research partner who will do this for you. A research partner will also be able to recruit and field the survey with your target customer profile.
At this stage, about 50 survey completes is sufficient and cost- and time-effective.
3. Analyze
Based on your results, update your business model and value proposition canvas. You can also update your brand strategy, marketing message, feature pipeline, and so on.
Then, move into validation experiments.
Validation Surveys
OBJECTIVE
Quantitative surveys gather data to analyze ideas, spot patterns, gauge opinions, and comprehend behavior. You can use this information to make conclusions about a larger population. That last point is key. Results you can model to the broader market will help you validate the viability of your idea.
VALIDATION SURVEY EXAMPLES
The methodology you use will vary based on your product or service and what you are trying to validate most.
A MaxDiff method helps you assess many messages or product features simultaneously, beyond what one person can accurately prioritize. Then you can stack rank them. This helps project managers focus on feature development and marketing managers create messages for their campaigns, among other tasks.
Conjoint methodology helps you understand what customers value in a product or service. It also helps predict price sensitivity and can show how brand affects decision making. A conjoint study creates a tool that lets you compare different product setups. It also helps evaluate against competitors, analyze price sensitivity and feature changes, and predict market share.
APPROACH
You should work with a specialist in these methods, like an internal research team or an external research partner. These tests need special skills and software to design research, run surveys, gather data, and analyze results.
Use your Discovery Test results in Validation Tests to build on research and strengthen your evidence. Follow best practices in quantitative research to avoid bias and ensure the most reliable results.
Mock Sales
OBJECTIVE
“Mock sales” experiments simulate steps in the marketing and sales process to gather real-world feedback. They measure what consumers do when presented with your product, not just what they say they will do.
This kind of data provides the best evidence you can get from concept testing. However, the advanced validation surveys and analyses mentioned above are strong indicators and can be easier to execute.
MOCK SALES EXAMPLES
Examples include online ads, landing page tests, and presales.
APPROACH
1. Online Ads: create an online ad that communicates your product’s value proposition. Target your ideal audience, and measure performance such as click through rate. You can do this with social media ads, paid search, etc.
2. Landing Page Tests: create a landing page for your product with a call to action such as “Sign up for more information.” Plan an approach to get landing page visits from your target audience and measure sign-ups vs. visits. This is an effective real-world measure of product viability.
3. Presales: sell your product before it is ready for purchase. You can test demand on a small scale and test different pricing models. Create a landing page with different price options, allowing buyers to pre-purchase. Monitor your page analytics like sales conversion rates to gauge interest.
Case Study
Tesla's Model 3 Presale Test
In 2016, Tesla accepted $1,000 deposits for the Model 3, even though the car was still in development. This pre-order strategy served multiple purposes:
The significant number of pre-orders validated demand for the Model 3, and Tesla’s investment in development.
They secured funding, as the deposits provided Tesla with immediate capital to fund further development and production.
Tesla validated real interest in the product, backed by financial commitment. This reduced risk of a product failure upon launch. The pre-orders also created a sense of exclusivity and anticipation. This fostered a loyal customer base before the product was available.
