On Nov 10th, the diamond wedding anniversary of Jacqueline Bouvier Kennedy and John Fitzgerald Kennedy, National Geographic Channel premiered Killing Kennedy, a well made movie about the story behind the JFK assassination. Apart from the beautifully made movie itself, NatGeo also launched an equally touching Web experience: http://kennedyandoswald.com
The simple yet stunningly rich experience of the website is one of the best examples of web storytelling through the use of not only great photography, videos, but also an important element: sound.
The beautifully mastered voice-over in the background, the piano, the birds, the whirlwind, the snapping of cameras…all of these non-visual elements instantly add another dimension to the experience, drawing the audience into the emotional space-time created by the artists. Try do a simple test, browse the site with sound on, and then sound off; you will find yourself shifting in between two worlds, an immersive world of the story, and a world that stood outside. That’s the power of sound. It renders your mind and brings the storytelling to life.
According to the research conducted by Dr. Vinoo Alluri from the University of Jyväskylä, Finland, sound is the only medium that lights up the entire brain under fMRI scan, compared with partial light-ups of visual stimulations. The researchers found that music listening recruits not only the auditory areas of the brain, but also employs large-scale neural networks. For instance, they discovered that the processing of musical pulse recruits motor areas in the brain, supporting the idea that music and movement are closely intertwined. Limbic areas of the brain, known to be associated with emotions, were found to be involved in rhythm and tonality processing. Processing of timbre was associated with activations in the so-called default mode network, which is assumed to be associated with mind-wandering and creativity.
“Our results show for the first time how different musical features activate emotional, motor and creative areas of the brain,” says Prof. Petri Toiviainen from the University of Jyväskylä. “We believe that our method provides more reliable knowledge about music processing in the brain than the more conventional methods.”
Film makers are the masters of creating compelling and convincing storytelling experiences, and no one else understand how powerful sounds are for storytelling than film makers.
“The power of sound to put an audience in a certain psychological state is vastly undervalued. And the more you know about music and harmony, the more you can do with that.” - Mike Figgis
Similar to film making, interactive Web experiences engage our audio-visual senses, only with interactivity and no constraint of time. The challenge with using sound on Web is time; its very difficult to synchronize sound when you can not control people’s visual flows and sequences. However, the degree of freedom in people’s interactions with the Web can also serve as a great opportunity to use sound in very creative ways, bringing more immersion and realism to the experience itself. The key is ‘context’. The use of sound in interactive experiences has to be contextual and responsive, for example, the sound of birds and waves are triggered as ambience when the audience is viewing photos of the ocean.
When the right sounds are used with the right contexts and responsiveness, the experience can not only be much more engaging and memorable, but also influencing people’s behaviours. According to studies, with the right use of sound effects and background music on storytelling based user experience, there’s a significant improvement on key metrics such as click-through rates and time spent, as well as social sharing and potentially conversions. In other words, when used right, sound brings better business results. According to research, with the help of music and sounds, audience could understand better the story being created and have a more enjoyable experience while experiencing it.
Although HTML Web has existed for over two decades, the use of interactive sound on a mass scale on the Web is still something relative new, or even undervalued. If you do a Google search on the subject, many ‘best practices’ recommend against using sound on the Web, for a couple of reasons: Firstly, it’s hard to synchronize sound with the right contexts of the Web. Secondly, the technologies and internet bandwidth just weren’t there yet, so latency and performance have always been top concerns. Last but not least, there’s a lack of talent pool and experts in designing interactive sound UX, so instead of doing it wrongly, it’s better to avoid it altogether! But it doesn’t mean we should ignore the power of sound and keep silence.
With that, before concrete Web specific audio UX principles and methods are established through industry practices and research, many methods and frameworks of the traditional film making can be learned and borrowed. For example, the D3S (Dynamic Story Shadows and Sounds) framework, was built with the main objective of increasing the understanding and enjoyment of a viewer of an interactive story generated in a virtual environment with autonomous virtual agents. It follows two parallel layers when considering music execution: event sounds and background music. Event sounds are used to underscore actions of the virtual characters that occur in the scene. Differently, background music, offers some of the score functions, with a special focus on enhancing the understanding of the story. In D3S, this type of music is classified in four different categories: character themes, background music that emphasizes emotions, background music for key moments and background music as filler.
Certain musical features can dynamically change according to the evolution of the environment. In D3S we considered: volume, instrumentation, and tempo.Volume is associated with emotions intensity. Different instruments are associated with different characters so that the audience has a better perception of what is happening in the story and who is doing what. The third parameter manipulated was music tempo which is associated with environment’s arousal.
More specifically, the association between instruments and characters is a good way of hinting which actions a certain character is doing, helping the audience to identify them. Changes in volume of sounds associated to actions between characters have an influence on the perception of the strength of the relation between them. Themes with features associated to happiness (such as major mode and faster tempo) might suggest that the character is happy, while themes with features associated to sadness (such as minor mode and slower tempo) might suggest that he is sad. Background music can also have a big impact about what is happening in scene – If we have two characters acting with a type of music, the audience might think they are doing something. If we change the type of music radically, they might think they are doing something completely different. From the results obtained, we can draw some conclusions about the importance of music associated with virtual characters, emphasizing the importance that sound and music has in these characters perception, and eventually in their believability.
The above is just a brief intro of how important music and sound can play in creating immersive and emotional digital experiences. I see a big trend coming with rich sound enabled digital multi-dimensional experiences along with the emerging technologies of wearable computing and multi-screen experiences. Humans have long been using sound as a way to learn and interact with the physical world, and there’s no reason why we should not use sound as a key interface in digital world. A big paradigm shift is coming.
For more information, please feel free to leave you comment below or contact BOZ UX.
This has recently been published on Fact Company. Some very interesting and insightful facts that may help shape your next social strategy.
1. THE FASTEST GROWING DEMOGRAPHIC ON TWITTER IS THE 55–64 YEAR AGE BRACKET.
This demographic has grown 79% since 2012.
The 45–54 year age bracket is the fastest growing demographic on both Facebook and Google+.
For Facebook, this group has jumped 46%.
For Google+, 56%.
2. 189 MILLION OF FACEBOOK’S USERS ARE “MOBILE ONLY”
Not only does Facebook have millions of users who don’t access it from a desktop or laptop, but mobile use generates 30% of Facebook’s ad revenue as well. This is a 7% increase from the end of 2012 already.
3. YOUTUBE REACHES MORE U.S. ADULTS AGED 18–34 THAN ANY CABLE NETWORK
Did you think TV was the best way to reach the masses? Well if you’re after 18–34 year olds in the U.S., you’ll have more luck reaching them through YouTube. Of course, one video won’t necessarily reach more viewers than a cable network could, but utilizing a platform with such a wide user base makes a lot of sense.
4. EVERY SECOND TWO NEW MEMBERS JOIN LINKEDIN
LinkedIn, the social network for professionals, continues to grow every second. From groups to blogs to job listings, this platform is a rich source of information and conversation for professionals who want to connect to others in their industry.
5. SOCIAL MEDIA HAS OVERTAKEN PORN AS THE NO. 1 ACTIVITY ON THE WEB
We all knew social media was popular, but this popular? Apparently it’s the most common thing we do online. So next time you find yourself watching Kitten vs. Watermelon videos on Facebook, you can at least console yourself with the fact that the majority of people online right now are doing something similar. Social media carries more weight than ever. It’s clearly not a fad, or a phase. It continues to grow as a habit, and new platforms continue to appear and develop.
6. LINKEDIN HAS A LOWER PERCENTAGE OF ACTIVE USERS THAN PINTEREST, GOOGLE+, TWITTER AND FACEBOOK
Although LinkedIn is gathering new users at a fast rate, the number of active users is lower than most of the biggest social networks around. So more people are signing up, but they’re not participating. This means you’re probably not going to have as good a response with participatory content on LinkedIn, like contests or polls, as you might on Facebook or Twitter.
7. 93% OF MARKETERS USE SOCIAL MEDIA FOR BUSINESS
Only 7% of marketers say they don’t use social media for their business. That means there are lots of people out there getting involved and managing a social media strategy. It’s becoming more common to include social media as part of an overall marketing budget or strategy, as opposed to when it was the outlier that no one wanted to spend time or money on.
8. 25% OF SMARTPHONE OWNERS AGES 18–44 SAY THEY CAN’T RECALL THE LAST TIME THEIR SMARTPHONE WASN’T NEXT TO THEM
It’s pretty clear that mobile is a growing space that we need to pay attention to. And we’ve all heard the cliché of smartphone owners who don’t want to let go of their phones, even for five minutes. Well, apparently that’s not too far from the truth. If 25% of people aged 18–44 can’t remember not having their phone with them, there are probably very few times when they’re not connected to the web in some way.
9. EVEN THOUGH 62% OF MARKETERS BLOG OR PLAN TO BLOG IN 2013, ONLY 9% OF US MARKETING COMPANIES EMPLOY A FULL-TIME BLOGGER
Blogging is clearly a big focus for marketers who want to take advantage of social media and content marketing. This is great, because blogging for your business has lots of advantages: you can control your company blog, you can set the tone and use it to market your product, share company news or provide interesting information for your customers. With only 9% of marketing companies hiring bloggers full-time, however, the pressure to produce high-quality content consistently will be a lot higher.
10. 25% OF FACEBOOK USERS DON’T BOTHER WITH PRIVACY SETTINGS
We’ve seen a lot of news about social media companies and privacy. Facebook itself has been in the news several times over privacy issues, Instagram users recently got in a kerfuffle over changing their terms of service, and the recent NSA news has seen people become more conscious of their privacy online. But despite these high-profile cases of security-conscious users pushing back against social networks and web services, Velocity Digital reports that 25% of Facebook users don’t even look at their privacy settings.