Gathering user feedback is a key element of a DevOps strategy, but you have to be proactive in collecting that feedback.
For DevOps teams to make a successful product (and maintain that product to improve user retention), they need to understand how people are using the app, and what they think about it. In DevOps, everything revolves around communication between product managers, software developers, and operations professionals, but open channels to customers and active users are relatively scarce. Product managers may have a finger on the pulse of collected user feedback, but that can only get you so far.
The problem with passive measurement
Unfortunately, relying on the soft trickle of user ratings and reviews isn’t the most reliable way to gauge user reactions. There are several reasons for this:
Selection bias. First, you need to understand that the types of users you get feedback from are going to skew your results. Generally speaking, only the users most pleased and least pleased with your app are going to go out of their way to leave feedback (unless you step up your efforts to get a read on the “average” user’s opinion; more on that in the next section). Accordingly, only the most egregious issues with the app are going to be acknowledged. Early on, that may be helpful, but if you want to build the best app possible, you’ll need a more even spread.
Sample size. In the early stages of development, you probably won’t attract many user reviews. Again, unless you’re intentionally pushing for more inbound reviews, you’re only going to get results from the most invested users (positively or negatively). Scientifically speaking, the bigger your selection sample is, the more reliable your results will be.
Qualitative interpretation. Reading user reviews may be able to point you in the general direction of certain problems, but it won’t tell you anything about how those problems arose, or what the user’s experience actually looked like. Accordingly, you may base your conclusions on a set of misleading or incomplete information.
Timing. You’ll want to establish a feedback loop with your customers as early as possible, but relying on passive user reviews is going to slow down the process significantly. This is especially true because it can be hard to attract reviews early on in your product development cycle.
The better strategies
These are the strategies you should use to improve your understanding of user experiences:
Remote software. You can start by implementing remote control software on devices both in and outside your network running the app, which will allow you to view and/or take control of users’ screens while they interact with the app. This approach is useful in guiding users, so you can see exactly what’s unfolding in real time, take screenshots, manage configuration settings, chat with users during the session and more.
Eye tracking software. You can also adopt eye-tracking software, which gathers less data about user behavior but is more scalable in most situations. Here, your software will track mouse movements and gestures made by your users, and may aggregate this data so you can understand the “average” paths users take to reach their in-app destinations.
In-depth qualitative interviews. If you’re interested in hearing how users describe your app (and their experiences with it), opt to conduct more in-depth qualitative interviews. Stage one-on-one interviews after a set period of app interaction, and ask deep questions about their thoughts while using it. This is far superior to collecting comments from user reviews, since you’ll have the opportunity to probe for elaboration.
Big data analysis. You can also crunch big data to find more general trends about the “average” customer experience. You can look at things like time spent on page, total conversions, and abandonment/exit rates to pick out key trends and evaluate how you can boost them. Incorporating data visualization and working with analysts can improve your efficiency here, allowing you to scope out broad trends and turn them into actionable insights.
It won’t take you much time or cost you much money to implement these tactics and collect this data to learn more about app performance. Once you have them in-hand, every wing of your collective team will have a better read on user impressions, and will be able to work together to turn those impressions into actionable takeaways.
Larry Alton is an independent business consultant specializing in tech, social media trends, business, and entrepreneurship. Follow him on Twitter and LinkedIn.
The InformationWeek community brings together IT practitioners and industry experts with IT advice, education, and opinions. We strive to highlight technology executives and subject matter experts and use their knowledge and experiences to help our audience of IT ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
2017 State of IT ReportIn today's technology-driven world, "innovation" has become a basic expectation. IT leaders are tasked with making technical magic, improving customer experience, and boosting the bottom line -- yet often without any increase to the IT budget. How are organizations striking the balance between new initiatives and cost control? Download our report to learn about the biggest challenges and how savvy IT executives are overcoming them.
Infographic: The State of DevOps in 2017Is DevOps helping organizations reduce costs and time-to-market for software releases? What's getting in the way of DevOps adoption? Find out in this InformationWeek and Interop ITX infographic on the state of DevOps in 2017.
Digital Transformation Myths & TruthsTransformation is on every IT organization's to-do list, but effectively transforming IT means a major shift in technology as well as business models and culture. In this IT Trend Report, we examine some of the misconceptions of digital transformation and look at steps you can take to succeed technically and culturally.