How to beta test, a robust example from award-winning learning app Pocket Prep

Surely all software companies are familiar with beta testing, but how well do you actually do it?

If you’re not all that confident with answering this question, hey, that’s cool too, because you’ve found the right article to get some practical ideas!

We talked to Colin Ulin, senior software engineer at Pocket Prep, about their low-effort high-outcome beta testing strategy and design-thinking development process. 

Pocket Prep is an e-learning SaaS company providing all sorts of exam preparation courses via mobile and online at reasonable prices. Every day, 50,000 students, trainees and educators benefit from the platform. The award-winning team consists simply of 5 developers and 13 other talented staff members.

Now that you know the basic background of the product and team, let’s jump right into their beta test setup.

A feedback-driven example of beta testing

When a new product is ready to beta test, the first task is to invite testers. Instead of employing random software testers to go through different test scenarios, Pocket Prep gets actual customers to try out the new application. They would have a dozen schools, think of them as accounts, and around 50 users in the testing environment. 

To collect feedback, the first method is regular meetings between business development managers and customers.

The second, and the more self-service way, is that customers can send screen captures within the application with their real-time experiences via Usersnap’s widget.

It’s just as important to ensure the UX of the feedback process as it is to ensure the quality of your testing product. The on-screen annotation tools and simple feedback form by Usersnap help users effortlessly highlight issues and present suggestions. Like this:

It takes one minute to send a feedback with a customer feedback software. Compared to writing an extensive email or filling out a technical bug report that takes more than 10 minutes every time.

Now, let’s talk about the timeframe.

Many software companies nowadays tend to go for the “do now, apologize later” approach for shipping new features and products. But at Pocket Prep, the beta test and improvement process take 2 to 3 quarters. 

“We are keen on doing it right the first time. If we ship fast but not great, it will only upset our users. And this is also how we’ve built strong customer relationships.”

– Colin Ulin, senior software engineer

Listening to feedback and building smart solutions to those pain points takes time. But in return, your product can harvest memorable UX and customer loyalty.

What kind of feedback should you be looking for?

During the beta test stage, the most common feedback you can expect are bug reports and feature requests. This helps the development team and management team to understand if the product is bug-proof and feature complete.

For Pocket Prep, to uphold the trust their users have in them, it’s essential the product launch is on par with users’ expectations.

Beta testing feedback goes beyond standard bug reports and feature requests. They are looking for UX defects and identifying missing elements to close the expectation gap. 

“We get feedback from beta users about what they think is broken, when in fact it’s just the feature not working as they expected. This helps us understand that the feature is not ready yet.” 

– Colin Ulin, senior software engineer

And because Usersnap’s tools allow users to easily share feedback instantly anytime, users won’t hesitate to click on the feedback button.

Once launching the feedback widget, a toolbar appears on the screen for making annotations on the screen capture. You can draw, highlight and/or pin comments directly on the screenshot.

Metadata, such as URL, browser data, device info and javascript errors, are automatically recorded and attached to the ticket.

Getting the right context empowers Colin and the team to find the fix and solve issues faster. In addition, when the users don’t need to worry about entering a long list of “boring” details every time they send feedback, the happier they are to share.

And with more feedback, the QA process becomes more robust.

When is the beta test phase done?

Although Usersnap is great for collecting beta testing feedback, the issues won’t fix themselves. 

The development and design teams at Pocket Prep work together in refining the new flows and adding new requirements to features. They tend to ship a couple of versions to test and gather more user feedback.

If after some iterations, they get positive feedback or stop receiving complaints, then it’s champagne time. But if similar feedback continues, then they would know the solution(s) did not work.

There is no exact cutoff point in beta testing. However, the amount and type of user feedback can be taken as references along the way.

How you decide when beta testing is complete also depends on your objective. Pocket Prep adheres to a meticulous approach because they believe making studying and exams fun means their app and UX also need to be enjoyable.

Customer feedback as the source of product solutions 

Some customer feedback is very tangible and actionable. For example, a screen recording feedback of a non-loading CTA button. But others require more digging and analysis, such as the product onboarding process.

To explore the root problem and evaluate a holistic perspective, Pocket Prep holds “expert panels” where different departments are invited to join forces in the design thinking process. 

• Empathy is at the start of the design thinking process, and you can use customer feedback as evidence to establish common grounds for customer empathy. At Pocket Prep, they like to distill the feedback to topical threads and begin the conversations from there.

• Next, they define what to build or do. Using the feedback threads again, they would prioritize and vote to create their roadmap. 

• For ideation, the team highly appreciates customer feedback as key references in making design decisions. Especially if it’s a new feature’s kick-off meeting.

• In the prototyping and testing stages, customer feedback should be the driver. The need for feedback, as well as the convenience of a customer feedback tool to centralize the tickets and events, have been praised wildly by Pocket Prep’s team. 

Even for internal communication, in comparison to just using Slack messages, Usersnap shortens and brings visual clarity to the discussions. With Slack, messages get lost easily, while Usersnap’s dashboard is designed to track feedback items and assignees in an organized manner.  

“As a developer, I’m used to leading long training sessions on new tools. With Usersnap, we simply just added the tool and told users to click the Usersnap button if they wanted to provide feedback. Everyone has found the tool to be incredibly intuitive yet inclusive of all the features we need.”

– Colin Ulin, senior software engineer

Now you’re ready to run your beta test program

Pocket Prep is realizing its mission to make studying engaging and fun by listening to user feedback every step of the way. 

We have learned so much from their story – the dedication in building outstanding products and UX. And we hope you too have gained some applicable insights. If you have, share this case study with your team or network to inspire them!

To give you a head start in yuor upcoming beta test, try the bug tracking widget and the beta test widget on Usersnap for free in the first 15 days!

And stay tuned for another insider’s look on Pocket Prep’s website rebranding story coming up next month!