Grocery Shopping AR app for Hololens

A case study on conceptualizing, designing and prototyping a mixed-reality grocery shopping application for Hololens-II

Last year in 2020, in my final semester of graduating from University of Michigan, I pursued my capstone project in Mixed Reality space. I, with my team members Hannah Wang(Designer) and XYZ (designer) identified problems with cross-cultural grocery shopping that people face. Especially in a multi-cultural country like US that comprises of special grocery stores from various countries.

Sneak peak into experience

Final video experience

1. Grocery list curation

   

2. Aisle Interaction

   

3. Navigation & List viewing

   

Motivation & defining scope of problem

Our team proposed a Hololens-based application to help people with grocery shopping, especially for people who are open to exploring international food and recipes, and people who hope to optimize their shopping experience.

Research

Users

• People who like to explore new international items in the grocery store

• People who cook recipes in different cuisines and do recipe based grocery shopping

• People who like to save time while shopping in large stores

Survey and results

We conducted a survey with these users to understand their behavior towards shopping for international food items in the store and recipe based navigation in the stores. The survey was also to understand the barriers people face while shopping in stores not from their native country. We had 82 participants for the survey!

Initial sketching and storyboarding

Credits: Hannah Wang, for sketching the concepts

User Journey

Prototyping in 3D - translation & rotation

We used Adobe Aero to prototype the flows and use the figma UI components we made as the first iteration of our UI.

1. Hand interaction to see the list - design explorations

We used resources provided by facebook(Link) to prototype how the hand interactions will look like. We also took motivation from Hololens interaction for Windows Start button.

Hololens windows startup gesture

Oculus virtual hands tools and resources (https://design.facebook.com/toolsandresources/virtual-hands/)

Prototyping hand gestures using virtual hands on a grocery cart

2. Distance estimation for items in grocery store

Adobe Aero was an accessible tool to create AR prototypes, so we imported our UI elements to Adobe Aero to create some video clips that mimic the AR user experiences.

We understood the various levels at which we need to design our UI based on the distances of the UI elements from the user. So we designed and prototyped based on those distances and came up with three guides: ~.5 cm  away, 1 m away and 2 m away. These distances covered the functionalities of our prototype.

User is 0.65 cm away and item is held up close

User is 2 m away. User is looking at items on aisles from a distance

User is in front of the screen and interacting with it (<1m or 1 arm distance away)

User is holding the cart & interacting with the screen (<1m /1 arm distance away)

Design System

We referred to the Microsoft Mixed Reality documentation for designing components in Figma. We cam up with our own set of colors, typography hierarchy and design of components used across the application.

Link to Figma for design system

Testing prototypes in Hololens-2

I am working on testing out the designs in hololens 2 using Microsoft Mesh.

I will be updating this space with my analysis and iterations on the design very soon. Stay tuned!