R&D Project

Discussion Capture

Helping to augment the teaching experience in active learning classrooms by providing facilitators with insights into multiple ongoing live conversations.

Company

Sony

Role

Product Designer

Date

May 2018 - December 2019

Summary

Project Background

Through various University visits conducting formative user research, as well as a broader consultancy study, our team determined that active learning was a growing pedagogy within higher education. Our team was tasked with creating a product that could leverage Sony's existing expertise with microphones and cameras, to create a better teaching experience for active learning facilitators. Our team decided to partner with Marcelo Worsley, a multi-modal learning analytics professor from Northwestern University, to create a system that could provide professors with real-time data about group discussion in their classroom.

My Role

As the lead user experience designer, I was responsible for formulating and conducting user research studies, analyzing the results to help guide our project roadmap, and designing the product in its entirety. In addition to these responsibilities, I facilitated a project kick-off in Atsugi, Japan, helping to bring together multiple disciplines (project planning, business development, marketing, engineering, and design) to determine our team's problem statement, business outcomes, user outcomes, and potential features.

Result

This research project was an invaluable learning experience for our team and helped guide future projects Sony would pursue in the educational space. Being our team's first exposure to creating a product that leveraged speech-to-text and natural language processing, it helped us to gain a better understanding of limitations and future potential. Furthermore, through our many conversations with faculty, our leadership team recognized how important it would be to involve multiple institutions in the creation of new technologies for the classroom. With the support of 11 U.S. colleges and universities, our division would help to establish the "Future Learning Collaborative".

This project would lead to the submission of a publication to the Association for Computing Machinery, titled: Chemistry Pods: A Multimodal Real-Time and Retrospective Tool for the Classroom. The paper describes our discussion capture system (codenamed "Chemistry Pods"), and how it can be leveraged to aid instructors in supporting group collaboration in a classroom or laboratory. 

The Process

Strategize

Project Kickoff

I ran a project kickoff with my team following the "lean UX" framework.  

Initial Hypotheses

We generated a few initial hypotheses that we could test based on our preliminary user and market research:

"We believe that professors have difficulty knowing whether or not student groups are discussing relevant topics in an active learning class.  So, if we display a list of spoken "keywords" then professors will be able to tell if the table is discussing relevant topics."

"We believe that professors don't know how much individual students are contributing in their group discussions. So if we provide an angular distribution of sound then professors will be able to identify who is contributing the most in a group conversation. "

Proto-Persona

I generated multiple proto-personas we could use to guide our initial brainstorming. 

Brainstorm

I then ran multiple brainstorming sessions with my team to get our creativity flowing and help to generate an MVP we could test. 

Discover

Research

To better understand how professors plan, facilitate, and evaluate their active learning classes, I conducted a contextual inquiry study and numerous user interviews. These studies provided me and my team with a greater understanding of the context of use for our application, and greater insight in regards to problems faced by both students and faculty in active learning.

Design

Version 1 - Mobile App

As an MVP, we created a simple prototype application that allows users to record audio using multiple “ReSpeaker” microphone arrays. This application is then able to display real-time information about the recorded discussions using linguistic analysis and audio directional information. For this prototype, we utilized Google's Text to Speech for audio transcription and question detection. We also utilized an algorithm created by Northwestern University, based on the Linguistic and Inquiry Word Count text analysis algorithm, to generate specific "discussion features".

Test

University Workshops

My team and I facilitated multiple workshops at Universities across the US and Canada. The goal of these workshops was to test our assumptions regarding active learning and expand our understanding of our users by facilitating discussions with faculty. We wanted to hear more about their experiences with active learning, better understand their mental models, explore how they currently solve common active learning challenges and gauge their reactions to our prototype. 

User Research Database

Our team was beginning to collect large amounts of disparate data from various sources and it was becoming exceedingly difficult to analyze the data through our traditional method of sharing using our inter-company wiki. I decided to create a user research database, allowing our team to quickly and easily identify trends from various schools. 

After analyzing the data, we discovered multiple recurring themes amongst various schools. The top recurring themes we discovered: it's hard to keep students engaged, there is too much overhead running an active learning class, it's difficult to gauge student comprehension, and it's challenging to provide equal attention to each group.

We planned a second demonstration at Duke University, so I decided to design a second iteration of the application to address some of the recurring themes.

Iterate

Version 2 - iPad App

This version of discussion capture, named Discussion Plus, was designed for use on an iPad to still allow for professor mobility while allowing for the display of more information.

ReSpeaker 3D Printed Case

Our team decided to switch to use the ReSpeaker Core V2, a six-microphone array, allowing for better audio capture. We needed an enclosure for the circuit board, so our team purchased a 3D printer, I learned to 3D print, and created this enclosure for use in our demonstration. I expanded the enclosure to also accommodate a USB battery back as in our initial in-class tests the wires and required power sources became an inconvenience for my team and the student participants. 

Closing Notes

Our research and prototyping would help guide multiple subsequent undisclosed Sony initiatives undertaken in the post-secondary education space. Furthermore, it sparked a new project which would leverage this capture device in simulation-based training for nursing students. The final system allowed instructors to check multiple qualities of the discussion such as the usage of instructor-specified keywords, relative participation of each individual, the speech acts students were using and different emotional characteristics of group language.