Kakao Voice AI
A project brief received from Kakao ai to conceptualize an AI service using speech recognition and object recognition technology in the mobile environment.

Duration: 1 week

Role

Solo Project

UX Design

UI Design

Research

kakao-vui-cover2.jpg
Artboard 2.png

Concept designs for an ai cooking assistant that finds recipes, gives step by step instructions, saves recipes to the user's cookbook, and calculates unit conversions along with general tasks such as playing music or sending text messages.

Inspiration

I love inviting over my friends, and cooking them a meal. And I'll often find a new recipe to try on my phone. But it's always such a hassle to access the recipe while both my hands are wet and full of ingredients. I need to look at the recipe on my phone, search online for unit conversion, and read texts from my friends, but the screen often turns off.

After getting a voice ai assistant in my house, I've found that it's much easier to look for unit conversions and set timers. But there still isn't a way for me to easily follow recipes on my phone while cooking.

This pain point was the inspiration for an ai cooking assistant.

User Testing and Role Play

In order to better understand the accessibility of our smart mobile devices while cooking, I conducted a diary study where I recorded myself cooking from a recipe. I counted the # of time when I used the mobile device while cooking and noted the reasons why I needed my  device. 

During the 20 minutes I was cooking using a recipe, I needed to look at the phone 10 times for these reasons:

  • View the ingredients list multiple times

  • Read the next steps, but the phone kept turning off

  • Convert unit measurements

  • Look at the photos in the recipe for reference

Artboard Copy.jpg
Artboard.jpg
Artboard Copy 2.jpg
kakao VUI_6.jpg
kakao VUI_7.jpg
kakao VUI_8.jpg
kakao VUI_9.jpg

Current Experience Diagram

Studying Voice AI on Mobile Devices

Since I didn't own a kakao voice assistant, I took a trip to the kakao store to understand it's personality and how it works. I wanted to know its loading time to answer commands, its ability to answer general and specific questions as well as it's tone of voice and language.

 

In my interaction, I also learned that Kakao could not answer my command "헤이 카카오, 파스타 레시피 찾아줘" (Hey Kakao, find me a recipe for cooking pasta).

I also reviewed various voice assistants that I could get my hands on from Google, Siri, Clover by Naver, KakaoMap on my mobile device, to carefully observe their interactions.

IMG_1402.PNG
IMG_1394.PNG
IMG_1403.PNG
IMG_1398.PNG
IMG_1397.PNG
IMG_1396.PNG
Screen Shot 2018-11-16 at 1.05.40 AM.png
Screen Shot 2018-11-16 at 1.06.01 AM.png
IMG_1375.png
IMG_1425.png
IMG_1426.png
IMG_33AB18E6B8AE-1.jpeg

Sketch Ideation

High level call flow

kakao VUI_2.jpg
kakao VUI_3.jpg

Dialogue Flow of new voice AI Experience

Dialogue diagram.png

Dialogue diagram (click to zoom)

Explorations

explore7.png
explore8.png
explore9.png
explore10.png
explore11.png
explore12.png
explore1.png
explore2.png
explore3.png
explore4.png
explore5.png
explore6.png
 

Final Designs

Voice interaction states

Motion prototype

Finding recipes using kakao voice ai

1.png
2.png
3.png
4.png
5.png
6.png

Retrieving recipes and getting recipe directions through kakao voice ai

8.png
9.png
11a.png
11.png
12.png
13.png
14.png
15.png

Next Steps

  • Prototype the above voice interaction with another person or through a voice recording

  • Refine the language to achieve most natural human voice interaction

  • Go through all the possible ways a user might state the command

  • Cover multiple use cases for voice ai interaction while cooking

  • Design for those multiple use cases

  • Further refinement for accessibility

  • Potential consideration for localization to english