PROJECT 3 | mysite

Project 3: Human-Robot Interaction

Goal of this project

Goal of this project is to develop an innovative yet feasible solution toward education using the Pepper and Nao Robot developed by Softbank Robotics. We wanted to build something that is more than just a yoga instructor which is already done by so many companies. We also wanted to build something that is useful, something the current education lacks and utilize Pepper or Nao’s automation and sensing forte to address it.

Brainstorming Process

Throughout the brainstorming phase, we identified several solutions in tackling education to make teaching more efficient for students of different levels. We came up with the following solutions:

Yoga Instructor that shows moves properly
Q&A Tutor that can give personalized education and individualized feedback based on the student’s level and progress of learning
Teacher Training that can mimic the behaviors of different types of students (especially disturbing students) and give feedback to the teachers on how to effectively deal with them
Sex-ed Teacher that teach about the awkward topics regarding consent, harassment, and safe sex
Video call extension that captures gestures of callers and connect users efficiently
Campus Directory Assistant that helps direct visitors to the right locations on campus
Specific Language Tutor that adopts native pronunciation and gives more personalized learning path for students of different experience in the language
Cultural/Behavior coach that teaches the different manners and etiquette across different cultures

After voting within our team, we decided to focus on the language instructor, teacher trainer, and the sex-education teacher. We think that these three are more unique than the other ideas and have not really been fully implemented yet in the current market.

Storyboarding

After deciding the three ideas, we planned out each of them further by making storyboards.

Here are the storyboards for each of the ideas.

Speed Dating

We conducted speed dating interviews for two of our potential users. Upon describing our features and purpose of the robot ideas, we also showed them the storyboard flow.

In speed dating, we learned:

Use robots to improve not to replace humans
Humans are mostly better than robots because they can adapt to different situations and responses
However, a human may judge others and create discomfort, while robots might be more comfortable to approach and talk to as they lack emotions
Human-human interaction can be positive or negative
Difficult to use robots to replace humans if human-human interaction is essential

Point-of-View

Final User Testing and Feedback

Prototype

For our prototype, we decide to use Pepper instead of Nao as its bigger size allows for a more similar replacement of a teacher and more convenient for students to interact with.

We broke down our prototype into three scenarios:

Pepper initiates healthy consent conversation
Pepper shows relevant biology video according to user request
Pepper answers awkward sex questions from user

In the three scenarios, we used Pepper’s facial recognition to detect whether user is male or female for deciding which the right contraception video to play. We also implemented speech recognition to detect certain words/sentences, and head/arm sensing to detect gestures.

Scenario 1

Scenario 2

Scenario 3

Demo Video

As we were making the prototype, we were also taking and editing video clips alongside. Here is the final video demo.

My Contributions and Reflection

In this project, I actively involved in coding the Pepper robot to perform the functions we need. I worked collaboratively with my teammate, Jim, to code out our three scenarios in the lab.

In the coding process, we were met with some issues with the Choreographer IDE. The Choreographer sometimes has a bad connection with the Pepper robot, so a lot of the times, we have to either restart the IDE or Pepper. We also spent a lot of time experimenting with the code, since the dragging sensing and logic blocks take a while for us to familiarize ourselves. We also found out that some sensing works better than others with Pepper. For example, in some cases, we swapped head sensing with speech recognition as Pepper responds more sensitive to touch than to speech.

In addition, I helped out with gathering a few of the video clips and editing parts of them for the demo video. Since the Pepper interaction and the video it displays on its screen go hand-in-hand with our video demo, I collected a few high-quality animations for relevant sex-education.

Moreover, I helped out with brainstorming our ideas, conducting our user interviews, and developing user point-of-view.

Overall, this is a great experience for me, especially I always dreamed of working in the robotics industry. Therefore, being able to code one of the most popular consumer/industrial robot Pepper, is a wonderful opportunity for me to learn how to build something useful with its functionalities (e.g. movements, sensing, logic, etc.). In this project, I learned these things:

It is really fun and exciting yet sometimes challenging to get the robot to work as intended. Software bugs and the hardware-to-software connection get in the way
Needs smarter test cases and solutions to minimize time spent on testing and debugging
Efficient division of labor is key to success within a team
It is important to get feedback before and after each implementation
Continuous improvement to make the robot better and better, according to the target audience’s needs

Last but not least, thanks to my great team! Without their help and skillsets, we could not have made such a rewarding and fun project!