Music for Your Skin
Haptic devices could soon become more than a novelty, revolutionizing how we experience everything from music to motion pictures.
I settle into an orange camp chair that has been stripped and fitted with 16 small, round voice coils, or speakers without loudspeaker cones. Carmen Branje hands me a pair of noise canceling headphones connected to his smartphone and launches a white noise app with instructions to jack up the volume as high as possible. Soon I feel as if I’m underneath a rushing waterfall. Branje hits play on a nearby computer and the chair begins to vibrate, though I can no longer hear the whir. The beat plays fast across my skin, the buzzes fluctuating from soft to intense as they flit up and down my back.
After a few minutes, the buzzing stops and I remove the headphones. Happy or sad? Branje asks. Happy, I respond. Definitely happy.
Audio-visual technologies have long dominated the media landscape, but touch-based, or haptic, technologies, could soon become more than a novelty. As far back as the 1950s, renowned psychologist and tactile researcher Frank Geldard noted that our narrow focus on the ears and eyes ignored other perfectly good channels of communication. The skin, he wrote in 1960, “is a good break-in sense: cutaneous sensations, especially if aroused in unusual patterns, are highly attention-demanding.”
Someday we may have a new genre of “music” based not on sound but touch.
Nowadays, with so many gadgets clamoring for the attention of our ears and eyes, using the skin as an alert system has started to gain traction. Cell phones buzz. The handheld controller in Wii vibrates when a user knocks out his opponent in a mock boxing match. General Motors recently installed vibrators inside the seats of its luxury cars to alert drivers when somebody enters their blind spot or when they’re drifting too far to one side. Researchers are looking at placing touch sensors along the body to improve bowing technique on the violin, facilitate rehabilitation after an injury or stroke, allow coaches to direct players on the field without yelling, and even help astronauts stay oriented in space.
But here, at the glass-walled Inclusive Media and Design Centre at Ryerson University in Toronto, Canada, the goals are loftier. Branje, who was a week shy of defending his doctoral thesis when I visited, believes our sense of touch can do a lot more than receive alerts. Cell phones vibrate at a single frequency, and so does the buzzer under one’s butt in a Cadillac. But Branje, himself a musician, thinks touch can be used to mimic an octave or the feel of a melancholic song. Someday, he says, maybe we’ll have a new genre of “music” based not on sound but touch. “Instead of an mp3,” he says, “I would send you a vib [vibrational] file.”
Hearing Speech Through Skin
Nobel laureate Georg von Békésy, a biophysicist at Harvard University, frequently lamented that those studying hearing, like himself, rarely researched the skin. Békésy’s research, inspired by his desire to understand difficult to access parts of the ear via the skin, showed that acoustic elements like pitch, loudness, and rhythm all had touch-based equivalents.
Anatomy backs up his discoveries. The skin is underlain with four mechanoreceptors that each respond to different forms of touch, such as a light tap, pressure, or pain. But in the 1980s, researchers found that those same receptors could respond to—or “hear”—different frequencies. Compared with the tens of thousands of mechanoreceptors in our ears, known as hair cells, the resolution of the tactile system is terrible. But their existence illustrates striking similarities between the two systems.
Much of our early understanding of the skin’s ability to receive sound comes from a line of research that blossomed in the 1970s and ’80s: enabling deaf people to “hear” the vibrational patterns that make up speech through their skin. That idea first emerged at a deaf school in the 1920s, says Janet Weisenberger, a psychologist at Ohio State University in Columbus, and an early researcher in the field. At that time, researchers essentially affixed miniature loudspeakers to user’s hands and fingers, but because the skin can’t feel anything above 1,000 hertz—which is where we distinguish among different vowels and consonants—users were able to detect that someone was speaking, but not comprehend what was being said.
Weisenberger and her team have studied a 16-vibrator device that could circumvent the skin’s frequency limitations by utilizing its abundant surface area. For instance, Weisenberger arrayed the device across users’ arms. A high-frequency note like the sound of the letter “s” would buzz near the elbow while a lower frequency note like “oo” would buzz near the wrist. The name Sue would thus vibrate in quick succession at the elbow followed by the wrist. In that way, people could be trained to recognize the patterns that went along with different vowels and consonants, Weisenberger says. The device “transformed frequency into location.”
That finding, coupled with others such as helping users distinguish between words that appear identical on the lips (look in a mirror and say pat and bat) significantly improved the accuracy of lipreading, which alone enables a user to understand a frustratingly low 30–60% of a conversation.
Research into haptic speech could give the deaf greater access to music.
But before the device could reach prime time, another technology soon upended that work. Cochlear implants, which rose to prominence in the early 1990s, reroute sound from the damaged hair cells in the inner ear directly to the auditory nerve. Today, deaf individuals with cochlear implants have greater access to speech and sound than ever before. Funding and interest in haptic speech soon dried up. Yet the information gleaned from that work could serve another purpose—giving the deaf greater access to music.
Even those with cochlear implants stand to benefit. Most cochlear implants have been made to maximize an individual’s ability to hear speech, which is much less dynamic and occupies a narrower frequency range than music. A soprano may sing four octaves higher than her speaking range, for instance. In an essay titled “ My Bionic Quest for Boléro ,” that appeared in Wired magazine in 2005, writer Michael Chorost writes about his experience with his first implants. “I could hear the drums of Bolero just fine. But the other instruments were flat and dull. The flutes and soprano saxophones sounded as though someone had clapped pillows over them. The oboes and violins had become groans. It was like walking color-blind through a Paul Klee exhibit.”
Feeling the Horror
The “scary scene” from the movie The Sixth Sense runs for almost two minutes without dialogue. In it, a young boy comes face-to-face with the ghost of a girl. She is puking violently. The boy screams and runs. He is breathing heavily and sobbing. Eventually, though, the boy returns to the girl and offers his help. The music starts off low and slow, then crescendos when the boy first encounters the girl, before fading again. As a viewer, I’m terrified right alongside the boy. But then I watch the scene a second time—on mute. Almost immediately, I’m bored. What, after all, is a scary movie without scary music?
Deb Fels, Branje’s thesis advisor and head of the Inclusive Media Center at Ryerson, has dedicated her career to making media more accessible to people with disabilities. Several years ago, Fels, began addressing the shortcomings in relaying sound in film and television to deaf and hard of hearing audiences. Closed captioning, she says, is usually handled not by the creative team, but outsourced to people who have nothing to do with the show. “You get crap,” she says. “You get spelling mistakes, grammar mistakes, missing stuff, and that’s how deaf people consume that particular piece of media. They don’t have any alternatives.”
“We basically put a speaker on your body and turn your body into an ear.”
Fels set out to change that. In the late 2000s, she and her team designed and began testing the Emoti-Chair, the same chair I sat in. Why shouldn’t people feel the noises on screen, too, they thought? Like Weisenberger, Fels’ team realized that to convey as much information as possible via the chair, they had to utilize the skin’s generous surface area. In a hearing person, sound travels as vibrations through the ear and into a spiral-shaped cochlea covered in tens of thousands of tiny hair cells. Hair cells tuned to hear higher frequencies reside toward the front portion of the cochlea, while those tuned to lower frequencies are embedded deep inside. So Maria Karam, then a post-doctoral fellow in Fels’ lab, essentially unraveled the cochlea and arrayed higher and lower frequency “hair cells” across the body by distributing the speakers throughout the chair. “We basically put a speaker on your body and turn your body into an ear,” Karam says.
When I watch the scary scene from The Sixth Sense on mute from the chair, the effect is electrifying. Soft buzzes flit across my lower back during those initial creepy moments and then the whole chair vibrates intensely—the crescendo—before returning to the low frequency buzz. To see how scared people could get by just feeling the vibrations, the Ryerson team had 59 volunteers sit in the chair and watch clips from both The Sixth Sense and The Shining while they measured their galvanic skin response, or GSR. GSR is the same physiological response measured during lie detector tests; it spikes whenever a person is agitated. They found that the participants’ GSR blipped during suspenseful scenes, or when the chair’s vibrations were more muted, but soared when participants were startled—like when the boy in The Sixth Sense encounters the puking girl. The chair, says Branje, can “give you those tinglies without sound.”
The Deaf Concert
Until the birth of his daughter a year and a half ago, Branje played percussion in a punk band. On a recording he sent me, the music is metallic and gritty.He’s always loved music, Branje says, and he grew up playing the piano, but his obsession was computers. As an undergraduate computer science major at Ryerson several years ago, Branje stumbled across an ad Fels had placed for a research assistant. She wanted someone to create software to add an additional layer of narration to film and television for blind viewers. Branje liked the work, so instead of going directly into industry after graduation as was his plan, he became a graduate student in Fels’ lab.
Branje soon started working on the Emoti-Chair, the timing of which coincided with his punk band days. Snagging gigs and playing in front of live, throbbing audiences became part of his new reality. His work in Fels’ lab, though, left him wondering what it would be like to lack access to that world. So in 2009, Branje lined up four bands and a DJ and hooked up five chairs to the sound system. Rather than playing high and low frequencies along different parts of the back, the various instruments and vocals were funneled to different speakers inside the chairs.The event, billed as the first deaf-accessible concert, became a media sensation.
Yet Branje wanted more. Even though many deaf listeners felt inspired, Branje knew the sounds they were feeling were a watered-down version of the real thing. He wanted to devise a system designed specifically for touch. Branje set to work creating a keyboard, one that didn’t play sound but instead funneled vibrations to the Emoti-Chair, a project that became the cornerstone of his doctoral work. When Branje thinks big, he sees his work as not a way of just presenting music to the deaf, but an entirely new art form accessible to all. Paintings appeal to the eye; music appeals to the ear; vibrotactile music appeals to the skin.
Composing Vibrotactile Music
Yet transferring music to skin is far from straightforward. Part of the problem is speed. The human ear, which has evolved to respond immediately to an external threat, is capable of hearing auditory changes that happen over just a few microseconds, yet the skin takes milliseconds. It’s glacially slow by comparison, says Roger Cholewiak, a haptics researcher who retired from Princeton University in New Jersey in 2004. “ The Flight of the Bumblebee may not be well appreciated on a tactile display.”
But music that is too slow will cause the skin to adapt or tune out the vibration. A person will notice the feel of a shirt when she first puts it on but will get used to it in less than a second,” Cholewiak says. Moreover, not only does the skin “hear” at a much lower frequency than the ear, it also struggles to discern between notes that are very close in pitch, like adjoining keys on a piano. That discernment gets worse the higher the frequency.
“Vibration is always there when we’re close to the music-makers.”
Altogether, that suggests that vibrotactile music must be low in frequency, slow, but not too slow, and dynamic—but only to a degree.
While the challenges are indeed daunting, Frank Russo says the payoff could be profound. Russo, a cognitive neuroscientist at Ryerson and co-inventor of the Emoti-Chair, is interested in how we perceive sound outside the ear. He’s found, for instance, that the skin can discern timbre, which is to say it can feel the difference between instruments like a violin and a piano even when they are playing the same note at the same loudness. For Russo, touch and sound are fundamentally related.
“Vibration is always there when we’re close to the music-makers,” he says. “But for at least a hundred years we’ve had recorded music and we’ve had the opportunity to be removed from the music makers.” Reuniting what we feel and what we hear, then, could, at the very least, make the experience of listening to one’s radio in the living room considerably more exciting.
Reawakening Our Tactile Sense
Back at the lab, Branje plays me another tune. This one is slow and starts very low along my back. It clearly feels sad. The tunes were composed on Branje’s keyboard, which doesn’t look much like a normal keyboard. It is large and shaped like a semicircle so that composers can always feel the pulse against their back. The keys themselves are divided up into five sections, each of which map to different sets of voice coils in the Emoti-Chair, and each section has eight keys ranging from 40 to 421 hertz. So if Branje taps the 40-hertz key at the left-most section of the keyboard, a person sitting in the chair would feel a low-pitched buzz along their lower back. If Branje hits the same 40-hertz key on the far right of the keyboard, that same buzz would play along the shoulder blades.
Once he had created a prototype for the keyboard, Branje set out to see how he could convey the emotion typically found in music through touch. Some aspects of music, Branje says, transfer perfectly from ear to skin, such as rhythm and tempo, though he suspected other elements would prove more challenging. For instance, in Western music, major keys connote happiness, while minor keys connote sadness. But what would the equivalent of songs written in major and minor keys feel like? It’s a black box, Branje says. “There is no such thing as a vibrotactile song.”
To begin sorting it out, Branje asked professional music composers to write happy or sad vibrotactile songs. Someone who had spent years conveying emotion via sound might intuitively understand how to convey those same feelings through touch, he reasoned.
His early results are promising. When Branje played those compositions to volunteers and had them report what emotion they thought the songs conveyed, they often reached the same conclusion. Branje then broke down each composition by musical elements, such as tempo, amplitude, frequency, length of individual notes, and how often a piece jumped along tracks (or speakers along the back). He found that tempo, or the number of notes per second, and frequency had the greatest significance. “If you use a higher frequency and a higher tempo people are going to think your songs are happier,” he says.
With his doctorate now complete, Branje now has a full-time job in industry. But his keyboard remains in use. Recently, it, along with several chairs, were hauled two hours southwest, to London, Ontario. There, at the new VibraFusion Lab, visiting artists can learn how to incorporate touch into their work. Rather than teach the finer details about how the skin works, the specifics of which are still being sorted out anyway, David Bobier, the artist heading up the lab, suspects people will discover its limits through experimentation. But he also hopes that the process will reawaken people to the beauty of touch. “We are,” he says, “tactile people who have been numbed by our other senses.”
Photo credits: Hannah/Flickr (CC-BY-NC-SA) and Carmen Branje.