Making comparisons with Tree Testing

- able to evaluate hierarchy based on a real-world scenario
- Because sketching or prototyping  wireframes  of a website or application is not needed, testing can occur very early in design process

Tree Testing: Pros

(1) prepare tree/ hierarchy on excel spreadsheet, listing only one category per row 
(2) create tasks for tree (ex: find a specific product on company's website or test potential problem that a new category may pose)
(3) create instructions that do not give hints to users about the correct categorization and are concise

Tree Testing: Preparation and Formatting

Researchers can use paper prototype if in person or for remote, online softwares such as userzoom.com & Treejack. 

Tree Testing: Tools

- Because it is usually remote and unmoderated, researcher can not ask follow-up questions
- User experience of design can't be tested because no visual styling or mockups are implemented

Tree Testing: Cons

Tree testing focuses on evaluating category labels. This type of study is called "tree" testing because it is assessing hierarchical menus. It is often implemented remotely and unmoderated, and primarily used to follow up card sorting. It also identifies if Information Architecture (IA) structure, labels, and placements align with users’ expectations.

University of Michigan - Ann Arbor

- The methods are very different based on costs, methods, and needs 
- Each method requires a different minimal sample size for statistical significance difference 


Different Types of Quantitative UX Research

Research methods used by researchers to collect qualitative or quantitative data in studies.

Data collection methods in UX research

Whitenton, Kathryn. “Tree Testing:  Fast, Iterative Evaluation of Menu Labels and Categories.” Nielsen Norman Group, 7 May 2017, https://www.nngroup.com/articles/tree-testing/.

Tree Testing:  Fast, Iterative Evaluation of Menu Labels and Categories

World Leaders in Research-Based User Experience. (n.d.). Quantitative user-research methodologies: An overview. Nielsen Norman Group. Retrieved March 5, 2022, from https://www.nngroup.com/articles/quantitative-user-research-methods/ 

Quantitative user-research methodologies: An overview

- Quantitative Usability Testing focuses more on using metrics to measure time to complete a task or success, etc. 
- Sample size: 35 participants or more 

Quantitative Usability Testing

- Analytics-based experiment to test differences in products, which help decide which variation of design is the best 

A/B Testing or Multivariate Testing

- A flexible tool to test people’s attitude and behavior in many contexts: live websites, emails, and usability testing 

Surveys and Questionnaires

- It quantitatively analyzes qualitative data, not a qualitative data collection method

Clustering Qualitative Data

- Quantitatively measures the desirability of products 

Desirability Studies 

- It is used to track users’ eye movements and understand where they look at in the interface, which helps designers and researchers to understand which content to emphasize or deemphasize 

Eye-tracking Testing

Tree Testing 

A Card Sort is used to understand how people think information within a website should be organized and named. It is often used to create or design an information architecture. 

- In a typical card sorting session, participants group the given topics into categories that make sense to them. 
- Users may also make changes to the cards you provided them by: removing a card they find unnecessary, creating a new card, renaming a card, changing the definition of a given card, and/or placing cards they feel belong that to multiple categories. 
- This can be carried out using sticky notes, paper cards, or a card sorting software.


Card Sorting in UX Research

User surveys

- Contextual Inquiry

The study will be conducted in the natural environment of the participants. For example, if the participants are students, the environment would be their classroom instead of an office.

Ethnographic field studies in UX research 

Contextual inquiry is a type of ethnographic field study that involves in-depth observation and interviews of small samples of users to gain a deep understanding of work practices and behaviors. Typically, it is conducted during the early discovery stages for a new feature or product. 
Key definitions:
- Context: The research takes place in the users’ natural environment as they conduct their activities the way they normally would. The context could be in their home, office, or somewhere else entirely
- Inquiry: The researcher watches the user as she performs her task and asks for information to understand how and why users do what they do

Contextual inquiry is useful for many domains, but especially well suited for considering users’ interactions with complex systems and in-depth processes as well as the point of view of expert users.

Contextual inquiry

Users are accompanied one-on-one with a researcher in a lab setting, and are told to complete tasks on a product or service.

Usability-Lab Studies

Participants contribute to the study by presenting their ideal experiences with reasons. The necessary design elements and creative materials will be provided by the researchers.

Participatory Design

"Tightly scripted usability studies are performed with several participants, using precise and predetermined measures of performance."

Usability Benchmarking

Researchers conducting usability testings remotely, with tools "such as screen-sharing software and remote control capabilities."

Moderated Remote Usability Studies

Having users install research moderator tool software on their devices and observe their process/thoughts while they complete a task.

Unmoderated Remote Panel Studies

Open and/or close ended information issued by a chosen set of users via a feedback link, button, form, or email.

Customer Feedback

Researchers invite random site visitors to answer questions about their goals of using the website and their feedback. Researchers then observe their subsequent behavior.

True-Intent Studies

Researchers observe user behavior through the recording of screens or pages that users click or see. "It requires the site to be instrumented properly or the application to have telemetry data collection enabled."

Clickstream Analysis

It could be either a quantitative method or a qualitative method. "Participants are offered different visual-design alternatives and are expected to associate each alternative with a set of attributes selected from a closed list."



Desirability Studies

User's Experience Measurements

Qualitative Data

A research method where a researcher in a structured setting asks an individual questions about any given topic for the purpose of collecting data about that topic.  From a UX perspective, interviews are valued insofar as they solicit the real opinions and experiences that users have when interacting with any given site or system, and the information collected can help improve that system. 

User Interviews

A qualitative research method where participants log interactions and information related to a specific context over a period of time. Researchers give participants a device to record their lives related to the studied products. 




Diary/Camera Studies in UX Research

Conducted after figuring out your target users. It uses a prototype to explore how a user will use a product. You can use a real user or someone on the design team can act as the user. There will be another colleague that will record any difficulties or concerns raised by the team. A guideline for the walk-through can also be created. This is different from usability testing as it is not used to test hypotheses or part of the rigorous iterative design process.

Walk-throughs in UX Research

A questionnaire is a form designed to collect information from respondents through a series of questions. A variety of question types can be included such as open-ended, multiple choice, or a scale (e.g. On a scale of 1-5, rate your satisfaction with this product).

Questionnaires in UX Research

A focus group is a qualitative data collection method that involves small groups of people participating together in an interview focused on a specific issue or topic. The dynamic interaction among group members can often elicit richer, more detailed information than what might be uncovered in a traditional one-on-one interview. For example, this approach has become a standard technique in business and industry for understanding consumer tastes and preferences. Because these group discussions yield extensive qualitative data, the sessions are typically recorded and transcribed to facilitate thorough later analysis.

Learn Before

Related

Learn After