What Is The Difference Between A User Interview And A Usability Test

User Interviews; Analyzing Your Data For Insights

18 minutes read

Conducting user research has many facets. The decisions and processes you make, right from the word go, affect the nature of the research results you get. It also affects the value of your research findings to the product development process. It is important to take into consideration the entire project from the beginning so that you can have the best insights to work with. 

One of the most critical components to consider during the initial stages is data analysis. This is because no matter the type of research you choose to conduct, you are likely to come up with a whole lot of data that is going to need some figuring out. Without a plan from the start, it will look daunting and frustrating. 

But with a plan for subsequent or even ongoing data analysis, you can extract what you need out of the volumes of transcripts you have pooled to review without breaking much sweat. Even though user interview data analysis typically comes after all interviews have been conducted and data has been gathered, you can greatly improve your research and its findings if you start much earlier. 

Having the end in mind will help you carve the best stories out of your research, get the most critical insights, and achieve a better ROI from your investment in research.

How Do You Analyze A Customer Interview? (User Research Analysis)

What is a User Interview

Data comes in the raw form of numbers, and in most cases, numbers do not come with any meaning except that which we give them. The goal of user interviews is not to get a bunch of zeros and ones, but rather actionable insights that help you move the needle toward your product development. This means the crucial part of analyzing interviews and the data they bring depends largely on how you choose to communicate them. To do a great job at this, you have to intimately understand your data, and decide what you are going to communicate and the story you both want to tell. The next step after all research is data analysis, regardless of the type of study research.

User interviews analysis simply refers to all the tasks, processes, and measures put in place to make sure that the data gained from the research can be transformed into actionable intelligence in the product development process. Every user research ends in a set of deliverables, but only after careful analysis can deliverables be developed, or created. Remember, data can tell several stories, and it is up to you, the researcher, to come up with the best stories that your data and insights tell. All of this depends on your user interviews analysis. What story is your project telling? – that is the summit of your research.

Start With The End in Mind; Analysis Right From The Beginning

The last thing you want for your research is to start your analysis after you are buried in so much data. You need to start thinking about analysis right from the beginning. Your interview process should be in a way that helps you analyze small findings. Your target users should be screened to be sure you have the right people to talk to in your UX research. This is especially crucial if you are sourcing interview users from social media and other public sources that are not your own app users. Finally, your research methods should be in a way that you can readily analyze your data. That is the way to get the best results from your project.

How you intend to analyze your results should start from the research design and continue till data has been collected. And at any point in time of the process, you need to keep your research objectives and goals in mind.

If your goal is to build a detailed and intimate understanding of who your audience and ideal users are, what motivates them, and how they meet those motivations on a daily basis, then your research is better off being designed to identify patterns in the daily behavior of the interviewees. In addition, you will be on the lookout for demographic details, lifestyle patterns, and various attitudes of users. On the other hand, if your objective is to test a new feature or upcoming app, then your focus could be on pain points, satisfaction points, and trying to rank them based on priority.

It is useful to have the hypothesis you are testing at the back of your mind and try to get as much information to ensure that you can come to an objective conclusion. But you don’t always have to wait till the end. You can start running small analyses once data starts to come in. It is typical to have 5 interview sessions a day. At the close of the day, transcripts could be examined to start looking for patterns and answers that could help you conclude your hypothesis. It may reveal some deficiencies in your research design and questions and help you iterate your research process for the best results.

Having an idea of the end from the beginning is not just about planning, but also regards results and the necessary adjustments needed to achieve them by the end.

Discovery On The Go

Product development is heavily dependent on quality research through user research, interviews, and testing. UX design is also based on thorough research. Building a product entirely on your idea or fantasies about what it should be, do and look like is the fastest road to costly mistakes. 

The research will help you discover the nuggets you need to create the product users will love. Whether you are thinking about features, functions, and other aspects of your application, it is important to find out the opinions of real users. It isn’t only expedient to go through such a discovery process, but also helps save time and resources. It is more expensive to fix a development problem than to build a solution from scratch. 

In some cases, you may have a series of really good use-cases for your product. but upon discovery through research and analysis, you realize that there are more pertinent use cases that could even be developed much faster than competitors haven’t caught up to. Finding out such insights can quickly set you up for success in an overly competitive environment. 

A discovery like any other research, starts with the stakeholders, what they want for the product, and the objectives created around those visions. Next is to create a strategy to get accurate user personas, experiences, and opinions to reach the organizational goals.

It Never Stops

The analysis never stops and starts from the beginning. It starts with the advent of the research and continues till after interviews are done and the final analysis and deliverables are concluded. One of the best ways to keep tabs on progress, and key insights is by taking notes, which tl;dv can be of great help. 

By recording sessions as well, you can share snippets or entire sessions of videos with stakeholders for appropriate action. Taking notes does not only help you analyze stuff on the go; it helps you keep an account of things as they happen so that you do not forget important things during the final stages of your research project. One way to do this, whether with qualitative or quantitative research, is to have short sessions with your fellow researchers after each interview or session and review the responses. 

Ask yourselves if the responses lead to achieving your objectives. Delve into what you need to do to improve the process, questionnaire, and the entire research. This will create a synergy between team members and help each member get into the others’ minds about the proceedings.

Not All Qualitative and Quantitative Data is The Same

Pooling data at the end of your research is one of the easiest ways to get discouraged by the sheer amount of data that you have and make analysis feel daunting. Don’t fret, not all data is the same. And at this stage, knowing how to find gold in your data is the skill you need. To achieve the best results, start by categorizing your data into various analysis areas that will allow you to weigh different data in terms of importance and priority.

Having such a list is important in order to determine what is important now, and what will be nice to have later on. This kind of analysis is particularly important when you have few resources to execute your findings; you want to focus on only the things that will bring maximum impact to your users and help you achieve your designated objectives.

That means putting away all the nice ideas in a basket for later and allowing the development team to build what is absolutely necessary at each point in time. Imagine you come across an idea to improve an aspect of your product, but it isn’t as important as other ideas. It won’t be in your best interest to put that at the top of your list. Knowing the objectives of the user research will help you perform the best analysis, knowing what exactly to put in your recommendations and in what order of priority.

Prioritization starts with the UX research design, based on the organizational objectives stakeholders have set for the project. To ensure that you get the necessary information to meet those requirements, you need to keep the objectives at the back of your mind and have a way of sorting your data.

Prioritizing Your Analyses Can be Done in Two Main Ways

1. Research Thematic Areas

Thematic analysis is basically putting your data into “buckets” for proper consideration since certain buckets of insights may be more important than others. By breaking down your data into well-organized buckets, you are able to consider every little detail based on its theme as well as the general importance of that thematic bucket. One way to start doing this is to decide on code colors and match them to a theme. When reviewing transcripts, use each color whenever you meet a certain theme and color code it. Using multiple codes is ideal to cover a number of themes. This can be done at the end of the day or shortly after each session to ensure that you do not lose perspective on each interview. Together with your notes which tl;dv can help you with, you can easily digitally color code your auto-generated transcripts from tl;dv.

When creating themes, you need to have a specific meaning, definition, or scope surrounding each theme. It is ideal to have a theme when you come across it a couple of times in interview data or transcripts.

According to the Nielsen Norman Group, there are six (6) steps for data analyses for thematic areas.

Step 1; Gather Your Data

Step 2; Read all of your data from start to finish.

Step 3; Code your text with thematic areas

Step 4; Create Codes necessary for new themes

Step 5; Take a break. (A day will do!)

Step 6; Evaluate Your theme for fit

One of the important things to help other team members (if you are not a lone wolf) understand your process is making a legend or a coding key when working with multiple codes. An example of a text coding key is;

  • Red – pain points
  • Green – positives
  • Gray – user suggestions
  • Yellow – apps used daily
  • Example of Thematic Diagram

Source; NVIVO BLOG

2. Affinity Diagrams

The affinity diagram is a way to visually organize all the facts by putting them in various categories (or topic clusters). The affinity diagram is often done with a pen and cardboard, but it can also be perfectly executed in a kanban style with tools like Trello. The affinity diagram is called different names such as collaborative sorting, snowballing, and sometimes affinity mapping.

Example of affinity mapping

Source; Leow Hou Teng

What is the Difference Between Affinity Diagrams and Thematic Areas in User Research?

Hierarchical thematic analysis is a process of breaking down your data into themes and then sub-themes. This is done by reading through all the transcripts, interview notes, and other relevant data. And then coding your data for various themes. Thematic areas are generally broader in scope than affinity diagrams. An affinity diagram will present smaller topics or clusters that can be further explored. 

The thematic analysis involves sorting your data into themes in order to better consider each detail, while affinity diagramming involves grouping facts together by topic. Both methods can be used to develop a better understanding of your users and improve your product accordingly. They are not the same and can be used side by side in your research analysis depending on the perspectives you want to pursue. 

How to Analyze Different Types of Data

How to Prepare for a User Research Interview

Analyzing Qualitative UX Data from User Interviews

One thing you may quickly notice with qualitative data is that it gets to the extremes and seems chaotic. This is because it is highly subjective. Also, you will end up with a lot of data, most of which may be repetitions and stuff you cannot use. But that is what it takes as user interviews typically are open-ended, allowing the user to express their opinions without filters from the questions asked. This also means analyzing the data depends largely on you, the UX researcher, and how you want to handle the data.

When performing qualitative user research analyses, pay attention to;

  • Patterns that come up in various thematic areas.
  • Findings that surprised your team.
  • Moments (topics) of great emotion for users.
  • Users liked and disliked the product.
  • Features that are popular among users
  • Use cases that your current UI does not support quite well.

But if you are performing exploratory research for your product discovery stage, you will find that that data may contain both qualitative attitudinal data and qualitative behavioral data. By using affinity diagrams and thematic analysis, you can uncover the hidden gems in the data. 

For you to perform thematic analysis, you will need 3 things; data, research preferences (based on team and goals of research), and the context and constraints of analysis. Armed with these, you can perform your data analysis with software, through journaling or through affinity diagrams. 

Using Software for User Interviews Data Analysis

Qualitative research would typically give you a lot of data to deal with. That is usually very difficult to analyze without using any software. In such a research, UX researchers rely on Computer-Aided Qualitative-Data–Analysis software or CAQDAS such as Provalis Research Text Analytics Software, Quirkos, Qiqqa, Dedoose, Raven’s Eye, webQDA, Transana, HyperRESEARCH, and MAXQDA. The benefit of going the software route is that you can perform very thorough research and work with a wider team. But at the same time, it takes a lot of time to code and use software and could be quite restrictive based on the software and the knowledge the team has about using the tool. 

Journaling For User Interview Analysis

The grounded theory method works well with journaling. It involves writing the various ideas and insights you gain from reviewing the transcripts and recorded videos of user interviews. It works with thematic analysis where there are various themes and their sub-themes where ideas can be categorized as the analysis goes on. By using annotation methods, either digital or manual, you can highlight data, facts, and ideas that are relevant to your research and record the notes created from this kind of reflection. It allows the researcher to think deeply but that is what makes it difficult; only one researcher can engage in such a process, making it difficult to collaborate with others. But it is also cheap and allows flexibility and there is always a well-documented process to how the results were achieved by looking back at the notes from journaling. 

Using Affinity Diagrams for Qualitative Research Analysis

As already detailed above, affinity diagrams are a good way to get ideas from your sessions and out to a board where you can see connections between the ideas. You can create an affinity board using Trello virtually or go the manual way; coding text with colors and cutting them out and finally putting them on a physical board. Whether you are going with a physical or virtual board, you can benefit from recording your sessions with tl;dv, writing your notes and having auto-generated transcripts which you can use long after your sessions have ended. 

Coding Your Data

When you are doing any type of thematic analysis of your data, you will need to do some kind of coding. Coding is basically labeling segments of text or data that you consider valuable based on certain themes while reviewing your transcripts. Each code indicates in shorthand, what the highlighted text is about. A code typically has a name, a short descriptive text on what it means in the project, and lastly, a color to be used when highlighting text. 

There are two types of codes; descriptive and interpretive.

Types of Code 

Descriptive 

A descriptive code tells or describes what the text or data is all about. It is typically for labeling data or what the text means. An example is “how language is acquired ” used when dealing with interviews for a language app where users are asked about how they learn languages that they currently speak. 

Interpretive

Interpretive coding goes the extra mile, compared to descriptive. It adds some analytical meaning to the text or data and often expresses how the research team assesses the information. Using our language app user interview example, an interpretive code would be “self-reflection” added to the text where the participant is trying to recollect their daily language experiences or their beliefs about the languages that they speak.

Analyzing Quantitative UX Data from User Interviews

Analyzing quantitative UX user research data or qualitative data is primarily aimed at answering a set of questions with the aid of the responses you get through the patterns you observe in the data collected. 

This could be how people use your app, why they prefer certain things and their general behavior about your product. You could also ascertain the general level of satisfaction that users get from your product using qualitative research. Depending on the scale of the research, this may be a ton of data or a simple spreadsheet to work with. 

Your task would be sifting through the data to gain values for certain variables to make your conclusions on your hypothesis. Also, be on the lookout for demographics of users as that could give you perspective when it comes to certain patterns you notice in your data. 

Don’t be quick to judge the entire data by the majority. And when you get your values, you might want to find out why those results are prevalent. Data leads to information, and information leads you to user stories that will move your product to the next level of development.

When performing quantitative user research analyses, pay attention to;

  1. What features are used most?
  2. Needs that are not currently met and priority.
  3. Differences in experiences in using your product.
  4. The time it takes to do something using your product vs the ideal.
  5. Features that need to be improved asap.

Where It All Ends; Recommendations

All user UX user research ends in recommendations based on insights. Your analysis process determines the quality of your findings and recommendations. It doesn’t matter whether you performed qualitative or quantitative research; they should both end with a list of recommendations, based on the original user research goals and objectives delivered to the appropriate stakeholders for discussion and implementation. 

You want to look through the data for trends, behavior patterns, usage insights, and common stories prevalent among your users. It isn’t enough to know that users find it difficult to find the “Shop Now” button, but instead, your research should come with a recommendation such as “put Shop Now button at the top banner”. 

Your research findings need to be actionable and help craft a good recommendation after analysis.

Creating A Compelling User Research Presentation

Giving a presentation on your user research findings is a great way to share your work with others and get feedback on your findings. But it can also be a daunting task, especially if you’re not used to presenting or do not have much experience with design.

Here are a few tips to help you create an engaging user research presentation:

  1. Make sure your presentation has a clear structure and flow.
  2. Use visuals to help tell your story.
  3. Use data and quotes from your users to support your findings. Focus on answering the “why” of data.
  4. Be prepared to answer questions from your audience.
  5. Practice your presentation beforehand so you feel confident when delivering it.

Conclusion

User interviews are an ongoing way to keep your product development team focused on what absolutely matters at the moment, the now for your users. Through well-executed research and well-analyzed results, you can set up your product for success now and in the future through ongoing discovery. How users use your product may evolve, and you need to be there to notice these trends. Remember to use tl;dv to record, generate free transcripts, and take notes of important insights. And when you need to share raw footage of an interview video, easily cut your video with the in-app video editor on tl;dv. Happy user research!