Overview

This study focuses to understand the cause of user churn rate in Jagline mobile app and enhance user experience to increase customer retention & acquisition.

I designed & executed a mixed-method research plan to uncover, quantify, prioritize usability problems, & provided recommendations to refine the design.

I noticed increased user satisfaction, a jump of 38.7% in the SUS score from the pre- and post-design implementation from the usability study, with the average SUS score landing at 95.3% post-design iteration.

My Role

Duration: 8 Weeks
‍My Role: UX researcher + Usability tester
‍Team: 2 User researcher
‍Method & Tools Used:
Contextual inquiry
Semi-structured user interview
Task scenarios For usability testing
Observation
Descriptive statistical analysis

Product Space & Context

Jagline is a IUPUI shuttle service that runs around the Indianapolis campus. Its mobile application updates users of different routes, live shuttle position and provides the shuttles schedule.

User Problem

The application, built without the perception of its user, had a declining user base. There were ~93% reported complaints at the backend database over the time of 2 months.

Business Problem

The high rate of complaints is directly impacting the operational efficiency and diminishing the value proposition on a necessary element in the university experience.

Goal of the Study

In this evaluative research, the critical high-level goals were to understand the following:Research questions

• Easy Accessibility of the information
• Efficient usage of the application
• Frustration points in the application(if any)

Method

To develop better picture of what was going on, I used a triangulated approach with three research methods

• Contextual Inquiry: To get to know the participant’s needs, and use cases
‍• Task Based Analysis: Observe user perform two predefined tasks with success & failure criteria
‍• Heuristic Analysis and Cognitive Walkthrough: To access the severity of the usability of the application with the user research

I planned, recruited and conducted 8 interviews at a participant’s convenient Location. The participants were randomly selected to reduce the voluntary response & selection biases.

Blend of first time users, daily users, and intermediate users helped to get an impartial assessment of the usability of the JagLine application.

I designed the evaluation tasks with the pre-set criteria to measure the task success and failure. It helped me understand the efficiency of the application

• Navigating from a known to an unknown location: This task measured users' ability to find the nearest shuttle stop from a familiar location to reach an unfamiliar destination.

• Navigating to and from unknown locations: This task evaluated users' ability to locate the nearest shuttle stops for both starting and ending points when both locations are unfamiliar.

These scenarios were chosen to understand users' mental models for finding shuttle stops in various situations and to identify potential usability issues

Research Findings

• Research revealed a disconnect between the user’s expectation with the application to application services:  ‍6 out of 8 participants expected the application to assist in the navigation to reach the bus stop, building recognition, along with providing shuttle information.n.

• Inaccessible to carry out new tasks using the app: All users expressed difficulty with tackling the unfamiliar location tasks. The onboarding experience was not designed keeping in mind participants’ unfamiliarity with the map area. Because of this the app has a greater learnability curve.

• Lack of proper information display: 5 Users wanted a display of the building names on the map as opposed to the current way i.e., street names. Many students were unfamiliar with the location, and couldn’t map the nearest / destination bus stops.

The list of user issues surfaced from observation and task based analysis:

• The Average SUS Score Of 56.7 From 8 Users Highlights The Usability And Inconsistencies In The Application. Because of the smaller sample set, I calculated the 80% confidence interval SUS score for the actual population. The calculation Indicates that we can be 80% confident that the population's true SUS score is between 39.73 and 60.97. The lower boundary of the 80% confidence interval does not dip below 40%.

Analysis

• The Hierarchical Task Analysis Of Each Task Gave The Structured Breakdown Of The Steps. It Made Us Analyze The Sequential Difference In Our And Users’ Perspectives Of Completing The Task.

• Merging The Difference Laid Out The Gap And Scope Of Improvement.

• Research painted a clear picture of our new personas. The task analysis and environment analysis allowed us to access the user's context and bring the critical content for our personas.

Recommendations

Based on these personas and user struggles, I recommended these design changes.

Impact

I conducted the usability testing of the new design with a new set of users and gave them the same task to perform. Testing showed that the implemented changes were intuitive and provided a more enhanced experience. The users raved about the revised flow during the second usability test.

8/8
Participants successfully completed the tasks

38.7%
Jump in average SUS score of 8 participant

93.9 - 96%
80% confident that the SUS score for actual population will be in this range

Reflection & Learnings

• Insights + Analytics worked in parallel to guide the necessary design actions. Each step strengthened the shortcomings of other methods.

• Understand the phenomenon and user behavior through your research. It has built confidence in me to direct the design directions.