Product Testing
Step six in the 9II9 product lifecycle is Test. The testing phase of the product lifecycle is one of the biggest sources of delays for a product team. Without the ability to identify and resolve issues early, and implement processes to keep old issues from resurfacing, a product team can consume valuable development time that could be better spent working on new features.
Functional Testing
As the development team begins working and marking certain tasks as done from their side as work packages are being delivered at the end of each sprint, the complete set of features will go through a round of functional testing. There are many other types of testing that can be conducted, such as security, performance, and compatibility testing. However, the most critical testing that should be done is to verify that the design and functionality works as expected on the target devices.
Similar to usability testing, functional testing can be formal or informal. The formal version involves writing test cases with scenarios and expected results. The result of each is captured as a pass or fail status. The informal version involves exploratory testing, where the product owner or other stakeholder will simply use the product on whatever test devices are available on hand and try to discover defects.
Usability Testing
Great Product Managers have a deep appreciation for design and can quickly identify a good design vs a bad design. A deep sense of empathy towards the user and the pain that the product seeks to alleviate is the reason why Product Managers perform user testing on a product to validate design assumptions and discover points of friction. Designing a product often involves making guesses and assumptions. Navigating a sea of opinions from various stakeholders can be difficult for a Product Manager. Introducing data from real-world user testing is the most effective way to win an opinion war.
Below is a sample video recording of one of multiple user testing sessions that were executed through UserTesting.com that informed major usability improvements in the Bloom smart video app. Send an email if you would like a copy of the script that was used for this test.
Below is a sample user research guide and script that was created for mobile and tablet apps. Drop a note if you want a cope of this script to use in your next app project.
A typical user testing session involves recruiting up to 8 participants from outside the organization that are representative of the target users of the product. Beyond 8 participants, a point of diminishing returns is reached, whereas the participants fail to reveal any new meaningful insights.
The test starts with devising a test plan containing various scenarios and tasks that the user should complete while speaking out loud. The session requires that a facilitator guide the one-on-one session and a note taker captures insights. The session usually lasts 45-90 minutes and the results are presented as a final report for the Product Manager to review and prioritize the issues that are worth resolving.
Exploratory Testing
When there are time constraints and quick validation of a product’s design and functionality is needed, exploratory testing is a useful tool. As the name implies, this involves “exploring” the core features and functions of the product looking for functional defects. This same activity should be done on as many target devices as possible so that there is a reasonable level of confidence that the product will perform as expected on the top devices that the target audience is likely to use to access the product or service.
Performance Testing
Speed is a major benchmark for a product, since the product’s core purpose gives users the ability to access content and information in a quick and frictionless way. Load times and page speed have direct impacts on key metrics such as retention and bounce rate.
When writing user stories for a product that needs to load content on a screen in a browser or app, acceptance criteria for how fast the complete page or screen needs to fully populate with content should be included. For example, a typical web page should load in the browser completely in less than 5 seconds.
Copy Testing
Copy testing is often covered as part of usability testing and involves testing the words, phrases, and wayfinding elements that provide affordances that help propel users forward and backwards through the desired scenarios in your product. Understanding which copy to use to give users a proper understanding of what they are being asked to do, what they should expect to happen next, and how to get back to where they were, are key aspects of delivering a good user experience that properly manages user expectations.
A/B Testing
In some cases, testing with users to determine which design works best may result in a tie or a situation where there is no clear winner. At other times, there may be two very strong design directions for a component or screen. In such cases, A/B testing helps to pick the winning design. There are several tools available in the market for running A/B tests. Two great tools are Optimizely, which can support your entire product portfolio and Google Optimize. These tools allow you to setup experiments to test your hypotheses about how a user may interact with your product.
Cafe Testing
A more casual approach to testing and validating a product with real users prior to releasing a product into the market is through café testing. This involves approaching a stranger in a coffeeshop or café and asking them to use your product in exchange for purchasing them a cup of coffee. With this approach, there is no excuse for not being able to validate your product through user testing. The same could be done at dinner parties or social events and allows users to test your product in a natural and unscripted context.