On Nov 10th, the diamond wedding anniversary of Jacqueline Bouvier Kennedy and John Fitzgerald Kennedy, National Geographic Channel premiered Killing Kennedy, a well made movie about the story behind the JFK assassination. Apart from the beautifully made movie itself, NatGeo also launched an equally touching Web experience: http://kennedyandoswald.com
The simple yet stunningly rich experience of the website is one of the best examples of web storytelling through the use of not only great photography, videos, but also an important element: sound.
The beautifully mastered voice-over in the background, the piano, the birds, the whirlwind, the snapping of cameras…all of these non-visual elements instantly add another dimension to the experience, drawing the audience into the emotional space-time created by the artists. Try do a simple test, browse the site with sound on, and then sound off; you will find yourself shifting in between two worlds, an immersive world of the story, and a world that stood outside. That’s the power of sound. It renders your mind and brings the storytelling to life.
According to the research conducted by Dr. Vinoo Alluri from the University of Jyväskylä, Finland, sound is the only medium that lights up the entire brain under fMRI scan, compared with partial light-ups of visual stimulations. The researchers found that music listening recruits not only the auditory areas of the brain, but also employs large-scale neural networks. For instance, they discovered that the processing of musical pulse recruits motor areas in the brain, supporting the idea that music and movement are closely intertwined. Limbic areas of the brain, known to be associated with emotions, were found to be involved in rhythm and tonality processing. Processing of timbre was associated with activations in the so-called default mode network, which is assumed to be associated with mind-wandering and creativity.
“Our results show for the first time how different musical features activate emotional, motor and creative areas of the brain,” says Prof. Petri Toiviainen from the University of Jyväskylä. “We believe that our method provides more reliable knowledge about music processing in the brain than the more conventional methods.”
Film makers are the masters of creating compelling and convincing storytelling experiences, and no one else understand how powerful sounds are for storytelling than film makers.
“The power of sound to put an audience in a certain psychological state is vastly undervalued. And the more you know about music and harmony, the more you can do with that.” - Mike Figgis
Similar to film making, interactive Web experiences engage our audio-visual senses, only with interactivity and no constraint of time. The challenge with using sound on Web is time; its very difficult to synchronize sound when you can not control people’s visual flows and sequences. However, the degree of freedom in people’s interactions with the Web can also serve as a great opportunity to use sound in very creative ways, bringing more immersion and realism to the experience itself. The key is ‘context’. The use of sound in interactive experiences has to be contextual and responsive, for example, the sound of birds and waves are triggered as ambience when the audience is viewing photos of the ocean.
When the right sounds are used with the right contexts and responsiveness, the experience can not only be much more engaging and memorable, but also influencing people’s behaviours. According to studies, with the right use of sound effects and background music on storytelling based user experience, there’s a significant improvement on key metrics such as click-through rates and time spent, as well as social sharing and potentially conversions. In other words, when used right, sound brings better business results. According to research, with the help of music and sounds, audience could understand better the story being created and have a more enjoyable experience while experiencing it.
Although HTML Web has existed for over two decades, the use of interactive sound on a mass scale on the Web is still something relative new, or even undervalued. If you do a Google search on the subject, many ‘best practices’ recommend against using sound on the Web, for a couple of reasons: Firstly, it’s hard to synchronize sound with the right contexts of the Web. Secondly, the technologies and internet bandwidth just weren’t there yet, so latency and performance have always been top concerns. Last but not least, there’s a lack of talent pool and experts in designing interactive sound UX, so instead of doing it wrongly, it’s better to avoid it altogether! But it doesn’t mean we should ignore the power of sound and keep silence.
With that, before concrete Web specific audio UX principles and methods are established through industry practices and research, many methods and frameworks of the traditional film making can be learned and borrowed. For example, the D3S (Dynamic Story Shadows and Sounds) framework, was built with the main objective of increasing the understanding and enjoyment of a viewer of an interactive story generated in a virtual environment with autonomous virtual agents. It follows two parallel layers when considering music execution: event sounds and background music. Event sounds are used to underscore actions of the virtual characters that occur in the scene. Differently, background music, offers some of the score functions, with a special focus on enhancing the understanding of the story. In D3S, this type of music is classified in four different categories: character themes, background music that emphasizes emotions, background music for key moments and background music as filler.
Certain musical features can dynamically change according to the evolution of the environment. In D3S we considered: volume, instrumentation, and tempo.Volume is associated with emotions intensity. Different instruments are associated with different characters so that the audience has a better perception of what is happening in the story and who is doing what. The third parameter manipulated was music tempo which is associated with environment’s arousal.
More specifically, the association between instruments and characters is a good way of hinting which actions a certain character is doing, helping the audience to identify them. Changes in volume of sounds associated to actions between characters have an influence on the perception of the strength of the relation between them. Themes with features associated to happiness (such as major mode and faster tempo) might suggest that the character is happy, while themes with features associated to sadness (such as minor mode and slower tempo) might suggest that he is sad. Background music can also have a big impact about what is happening in scene – If we have two characters acting with a type of music, the audience might think they are doing something. If we change the type of music radically, they might think they are doing something completely different. From the results obtained, we can draw some conclusions about the importance of music associated with virtual characters, emphasizing the importance that sound and music has in these characters perception, and eventually in their believability.
The above is just a brief intro of how important music and sound can play in creating immersive and emotional digital experiences. I see a big trend coming with rich sound enabled digital multi-dimensional experiences along with the emerging technologies of wearable computing and multi-screen experiences. Humans have long been using sound as a way to learn and interact with the physical world, and there’s no reason why we should not use sound as a key interface in digital world. A big paradigm shift is coming.
For more information, please feel free to leave you comment below or contact BOZ UX.