- Usability Testing tends to be very expensive, but it is experimental compared to analytics. 
- Analytics is very inexpensive, nice when seeing what users do in the wild. 

Difference between Usability testing and Analytics

- Used to detect or prioritize problems and to monitor performance
- Describe what user are doing with the product, where they go, what they click, what features they use, where they come from, and which pages they decide to leave the site or app and etc. 

University of Michigan - Ann Arbor

- Quantitative Usability Testing (Benchmarking)
- Web Analytics (or App Analytics)
- A/B Testing or Multivariate Testing
- Card Sorting
- Tree Sorting
- Surveys or Questionnaires
- Clustering Qualitative Comments
- Desirability Studies
- Eyetracking Testing

Popular Types of Quantitative Research

- Used to track usability over time and also comparing with competitors.
- It is not used often, but similar to qualitative usability testing. Difference would be quantitative testing is focused more on collecting metrics like time on task or success. 
- Can be done in person, remote moderated, or remote unmoderated.

Quantitative Usability Testing (Benchmarking)

Web Analytics (App Analytics)

- Used to compare two design options
- Comparing two different live versions of the same UI which are shown to different users to see which version performs best. 
- Great for deciding among different variations of the same design. (Ex - Tracking how many times a certain button was clicked)
- Downside: Some teams fail to run the tests as long as they should, and make risky decisions based on small numbers.

A/B Testing or Multivariate Testing

- Used to determine information-architecture labels and structures
- Studies are done by participants given content items and asked to group and label items to where it makes sense to them.
- Could be done in person, or remotely
- Quantitative analysis of the percentage of participants who created similar groupings can help establish which categorization approach would be understandable to most users.


Card Sorting

- Evaluates a hierarchical category structure, or tree, by having users find the locations in the tree where specific tasks can be completed.
- Useful in identifying if an IA structure, labels, and placements agree with people’s expectations.

Tree Testing

- Used to gather information about your users, their attitudes, and behaviors 
- Can produce combination of quantitative and qualitative data — ratings, proportions of answers for each choice in a multiple-choice question, as well as open-ended responses. 

Surveys and Questionnaires

- Used in identifying important themes in qualitative data
- Counting the frequency of instances when a particular theme is mentioned (Ex. Frequency of a user complaint or of a UI problem)
- This technique is less of data-collection methodology, and more of an analysis approach for qualitative data.
- Great way to mine numerical data from large amounts of qualitative information.
-  Downside: Time consuming.



Clustering Qualitative Data

- Used in identifying attributes associated to your product or brand
- Quantify and measure some quality of a product — such as aesthetic appeal, brand strength, tone of voice. (% of respondents describes the design as “Sleek”)


Desirability Studies 

- Used in determining which UI elements are distracting, findable, or discoverable
- Require special equipment that tracks users’ eyes as they move across an interface. 
- Help identify which interface and content elements need to be emphasized or deemphasized, to enable users to reach their goals.
- Very expensive in terms of cost


Learn Before

Related

Learn After