Sound, touch, and technology: When you don't have to hear to experience music

Can a Philly music and technology expert find new ways for people who are deaf or hard of hearing to experience music?

On November 14, LiveConnections honored Drexel University’s Youngmoo Kim with the nonprofit’s annual Connector Award. Kim, the director of Drexel's ExCITe Center, is a natural collaborator for LiveConnections, an organization whose mission is to create programs that inspire learning and build community through collaborative music-making.

Down-to-earth music through tech

Kim’s background, experience, and expertise are a fascinating blend of engineering, technology, and music. He is a professor of electrical and computer engineering at Drexel and has extensive experience in music performance, including eight years as a member of the Tanglewood Festival Chorus, the chorus of the Boston Symphony Orchestra. As director of the ExCITe Center (which stands for Expressive and Creative Interaction Technologies), he works with both graduate and undergrad students in a multidisciplinary approach to technology and how it affects community.

“People think of university research as ‘ivory tower’ — but we love doing stuff out in the real world with real people,” says Kim.

The collaboration with LiveConnections involved Kim and his students designing interactive live experiences for LiveConnections using computing and large-screen projections to help kids visualize and learn about how sound works and how technology enables us to make music.

In video clips from World Café Live (where LiveConnections is a resident) and the Philadelphia Science Festival, you can see this technology in action. “We use color to help participants to visualize elements of the music,” Kim explains. “Color shows what the different instruments are doing, how the sound changes through the piece of music.” Participants follow on the large screen and also through an app on their smartphones.

Can we feel the difference?

Based on this technology’s success, Kim and LiveConnections are partnering again on an upcoming LiveConnections project for people with hearing disabilities, in which he and the ExCITe Center will create a way for those who are hard of hearing to visualize music and participate in a large community singing event.

“People who are deaf or hard of hearing can often feel music… They feel the bass,” Kim says. “Electronics can give touch sensation feedback, and our vision is to use technology to translate sound into different sensations of touch that people who are deaf or hard of hearing could access.” Kim acknowledges that feeling the rhythm will be the easy part — but that translating timbre, what makes one musical sound different from another, is a more significant challenge.

“Can we translate the more subtle parts of a piece of music?” Kim wonders. “Could a participant experience the difference between acoustic and heavy metal?”

In exploring this question, Kim will test out his new technology with people who are deaf or hard of hearing. He hopes to create a new way to translate music to them.