People who are d/Deaf or hard-of-hearing (DHH) often rely on visual signals to perform everyday tasks, and while many are happy with this system, others want to explore ways to interact with sounds.
Researchers from the UW Makeability lab have launched a smartwatch app, SoundWatch, that alerts DHH people to nearby sounds. The app, now available on Google Play Store, can detect a range of sounds from a bird to a car, achieving an accuracy rate of about 86% in lab tests.
It all began in a parking lot, when Dhruv “DJ” Jain , computer science Ph.D. student and lead researcher, realized he was walking in front of a car for a little too long. As a person who is hard of hearing, he couldn’t hear the engine and eventually relied on visual cues — the moving car and a very angry driver — to know that he was blocking the way.
Jain was embarrassed but was also inspired to create a sound alert system that notifies people of sounds in their environment.
His first system, HomeSound, uses machine learning to recognize sounds in the homes of DHH participants. From a dog’s bark to a doorknock, the system can recognize up to 19 unique sounds based on a dataset of over 31 hours of recording.
DHH participants who had HomeSound installed in their home were surprised by the experience. Four participants reported that HomeSound made them realize that there are more sounds in their homes than expected.
“Putting plates in the cupboard, the water running — there are so many different kinds of sounds going on around, and they didn’t know,” Jain said.
Jain wanted to experience sounds outside of the home, so he transitioned to using a smartwatch. Now, people only have to lift up their wrists to engage with the sounds around them. Similar to HomeSound, the SoundWatch app prompts users to take action based on a sound alert, like moving out of the way after hearing an ambulance's siren or just being present in the moment.
“When I’m hiking, I can hear the sound of water running down the stream,” Jain said. “It makes me feel more connected in nature.”
Since the app’s launch was fairly recent, the research team is still collecting data on SoundWatch’s user experience.
Jain’s upcoming project will create a feature within SoundWatch that allows users to customize sounds based on their needs. For instance, if a person wants SoundWatch to differentiate between a front door knock and a back door knock, they can train the app by recording a few knocks from both doors.
“Training a machine learning model with lots of data is hard enough,” Jain said in an email. “People don't have a lot of time, so it'll be interesting to see how we can train only using a few samples.”
People who are d/Deaf or hard of hearing are not the only users of SoundWatch. Those who can hear, but are wearing headphones or watching TV, can use the app to get alerts from sounds that are too quiet to hear.
By catering to all hearing abilities, Jain compares this to being in a noisy bar — people appreciate having subtitles on the TV regardless of whether they are hard of hearing or not.
“My overarching vision is to transform the way people experience, engage with, and deal with the sounds around them, and SoundWatch is a step in the direction,” Jain said in an email.
Reach reporter Anh Nguyen at firstname.lastname@example.org. Twitter: @thedailyanh
Like what you’re reading? Support high-quality student journalism by donating here.