Overview

This usability study aimed to improve the initial device setup process for multiple devices by testing user satisfaction, efficiency, and effectiveness.

I compared two refined device grouping user interfaces against the existing process using monadic unmoderated testing (240 users on Maze.com).

Results surprisingly showed that the current method remains the most efficient for setting up the devices. This finding led us to refocus efforts on understanding user device grouping preferences. A follow-up study will inform the development of a framework for device grouping for device setup.

My Role & Project Details

‍Timeline: 4 weeks
Tools:
• UserTesting.com for recruitment
• maze.com
• Confluence
• Jira
• Quip/Word Doc + Excel sheet
‍My Role: As a Solo UXR on the team, my responsibilities were
• Propose and execute efficient and effective research plan encompassing research questions
• Analyze the data and present actionable next steps to the team.

Product Space & Context

A leading provider in home security solutions offers a variety of products including video doorbells, outdoor cameras,indoor cameras, and alarm systems. Their distribution network includes their own platform, major online retailers, and other stores, making them a well-recognized brand. Consumers considering a home security system often compare this company's offerings with competitors to find the best value.

User Problem

For device onboarding and set up process, each device including 3rd party devices are listed in a category representing different product types. Majority of categories are not needed from a technical perspective because the category is identified from scanning QR code. This approach adds extra steps and cognitive load for users to decipher which category does the product belong.

Moreover, some devices are presented outside the category for marketing purposes. The approach of having a new product category for every new product tier isn't sustainable.

On average a user spends 14 sec. on categories list and 89% try two or more categories before setup.

Business Problem

To streamline the user experience, the focus is to simplify 'out of box' experience for users.

Goal of the Study

Assess the usability of 2 new setup initiation flow
• Validify if users prefer categorization before scanning the device during the set up.
• What are users expectation with the search functionality when users are finding the device
• Evaluating the user satisfaction with the new approaches towards initiating the setup flow.

Method

Unmoderated usability testing
• Monadic Test (each participant will initiate the setup of just one device in one of the prototypes)
One tester per task will reduce the recency and learning bias to give us a first impression of each tester to set up the device.
• Sample = 240 Participants (recruited through usertesting.com)
• 16 Devices (Core, Non-Core and 3rd Party Devices);
• 5 users per device per prototypes

Prototypes Tested

We tested 3 prototypes & used the same task and questions for each device.
• Prototype 1 featured a dedicated scanner page and a separate list of devices ineligible for scanning.
• Prototype 2 presented a new two-page device categorization system with subsequent steps to initiate the scanning process or device setup
• Prototype 3 replicated the existing workflow, including an extensive device categorization list followed by subsequent device setup steps.

Task Details

Context: Your objective today is to initiate the setup process accurately among the available options. You have just bought a brand new <device name> and are excited to initiate the setup process.

Task: Please initiate the setup process to connect your <device name> to the application.

Description: You may start by clicking on the ‘Set Up a Device’ button in the menu or on the main screen. Please think out loud during the entire process.

Condition: If users used/clicked on the search bar:
Q: Please write down keyword(s) for your search request Q: Initiate the setup process to connect your <device name> without using the search bar.

Post task Questions: Q: On a scale of 1 – 5, how easy was it to start the setup of <Device name>? (1 = Not Easy at all & 5 = Extremely Easy).
Please provide reasoning to your score.
Q: What did you dislike about this setup flow? Why?
Q: What did you like about this setup flow? Why?

Research Findings

• Users did not find a pre-setup confirmation necessary to verify correct device selection. Instead, a post-setup confirmation screen detailing successful registration was preferred to provide user reassurance of device enrollment.

• Prototype 1 (direct scanner page) proved twice as efficient as Prototype 2 (new grouped device categories) for setting up Core devices. Users found Prototype 1's direct scanner approach quicker, simpler, and more effective, with fewer steps.
However, users encountered difficulties with non-Core and 3rd party devices on prototype 1 that lacked QR codes. The "set up a device" page did not provide clear setup process guidance for these devices. 

• Prototype 3, replicating the current flow, was twice as efficient as Prototype 1 (direct scanner) and Prototype 2 (grouped categories) for setting up 3rd-party devices.

In prototype 2's grouped device categories did not align with users' mental models, leading to confusion. The two-page categorization in Prototype 2 also increased setup time.

• 21% of users (across Prototype 1 and 2) utilized the search bar to expedite device discovery. Searches were evenly split between those using only the product category and those combining brand and product names for more precise results.
Examples:
‍• Product category: smart plug, intercom, garage door, mailbox sensor, camera
• Brand and product: ABC* smart plug, XYZ* smart water valve, LMN* mailbox sensor etc.

A small percentage of users searched solely by brand name (e.g., abc*, xyz*)

• Users anticipated that the search bar would accurately capture their search terms. For instance, searches like "PQR* garage door opener," "Garage door opener by PQR*," "pqr* door," or "PQR*" should yield consistent results for the same product.

‍^{*Brand names are white labeled for confidentiality reason}

Analysis

Through affinity maps, and task analysis, I grouped common insights and highlighted the main themes answering each research question

Due to confidentiality restrictions, specific project details cannot be shared. However, I am presenting my process and general observations from my research work.

Recommendations

• Simplify categorization: A follow-up card sorting test will be conducted to determine optimal device groupings based on user preferences

• To enhance user experience and address challenges with non-standard device setups, clear instructions and a prominent call-to-action should be provided on the initial setup page to guide users towards the correct setup path.

• Enhance category clarity by adding descriptive labels. Replace generic terms like '...and more' with concrete descriptive examples. For example, add specific device names that can be found under 'Accessories.  

• Prioritize proper visual hierarchy: Users failed to distinguish between primary and third-party devices, often seeking third-party devices within primary device sections. Implementing clear content hierarchy and visual emphasis on headings will guide user focus and improve navigation

• Users appreciated the visual distinction of device icons through color and imagery. However, the gray icons for 3rd party devices gave an impression that the categories are inactive.  

• Enable search filtering for quickly finding the device for setup initiation.

Next Steps

The unexpected finding that the current device setup method remains the most efficient for 16 devices shifted the project focus. To optimize the experience further, another research was prioritized to understand user preferences for device grouping.
A follow-up study will gather insights to inform the development of a tailored device grouping framework within the app, aiming to enhance user satisfaction and streamline the setup process.

Reflections & Learnings

Foster transparency within the team by clearly communicating goals, focus, timelines, and next steps. Regular meetings and shared notes ensure everyone is aligned.

Early project discussions can streamline production by getting validated user feedback and avoiding re-work.