Changing soundscapes: how mixed immersion is shaking up the world of immersive audio experiences
“The truth is, for me, it’s obvious that 70, 80 per cent of a movie is sound. You don’t realise it because you can’t see it.”
— Danny Boyle, film director
The team at Mixed Immersion describe themselves as pioneers in immersive audio production and delivery, a 3D audio production company with a focus on creating amazing immersive, sound-based experiences within experiential, virtual reality, augmented reality and live performances.
With visual effects (VFX) agencies using lighting, textures and reflections to build virtual 3D worlds, Mixed Immersion refers to itself as the world’s first ‘AFX’ company, capturing, designing, mixing and blending cinematic quality soundtracks, atmospheres and ambiences to work in parallel to their visual counterparts. Their content is platform agnostic and sits on any type of immersive or interactive experience including theatre, cinema, theme parks, experiential and headphones.
“When it comes to 3D spatial audio, we are finding that it is not just a part of the VR scene but it’s solid enough as its own medium that it’s creating its own immersive technology market,” claims Dave Black, Mixed Immersion’s co-founder and commercial director. “We are moving towards a place where it is not AR or VR per se, but everything immersive.”
Putting the team together
And they certainly have a very strong pedigree in the immersive technology world they are attempting to shake up. Dave Black was an early pioneer in the AR and VR world and one of the key figures behind the AR and visual discovery platform Blippar, which is now being labelled as the next billion-pound British start-up. En route to founding Mixed Immersion, he also headed up marketing at one of the UK’s leading digital content studios, REWIND.
His co-founder and technical director, Gareth Llewellyn, is a re-recording mixer and audio specialist who has worked on the 3D audio versions of several major Hollywood films and has been involved in the development and application of new 3D audio technologies since 2010. While the third co-founder, Jon Olive, who is also the company’s creative director, has worked on sound and music production on blockbusters including The Mummy and the Harry Potter films, and consulted on experiential projects such as the Harry Potter exhibit at Universal Studios Theme Park in Orlando.
And unlike other companies in the immersive technology sector who have often struggled to find the mix of skills, Dave says they have been fortunate that their extensive list of industry contacts has allowed them to hire the people they need – highly skilled freelancers who have worked on top grossing global productions. “We’ve developed quite a specific and niche area, offering a premium audio service and have found some amazing people who want to work with us. We know the groups of people who can create amazing things, people who are the best in the business at what they do.”
Why is spatial audio important?
3D audio is a relatively recent development in cinema and works on both tiered speaker systems and VR headphones due to the ability to instantly track the head movements of the user. Dave Black describes it as the successor to surround sound, taking a flat, horizontal plane of audio and adding another above to bring in height. He explains that height is one of the most important factors for how we hear in the real world and allows us to process layers of spatial information to “feel” the environment and the atmosphere that resides within it.
“We constantly use our hearing to understand what is around us, where things are and how close they are to us. It is now possible to replicate this in VR and to do so means a massively increased sense of presence,” Dave explains.
The framing that has always been present in 2D film, is now nonexistent in VR and the director can no longer direct the viewers’ gaze as they can look where they like. Filmmakers can no longer zoom, tilt, cut or pan, so the ways with which users can follow a narrative are being massively challenged and redefined with the grammar of how we describe the processes and workflows being placed in the hands of the VR pioneers.
Spatial audio is now the essential tool to highlight where the action is and direct the viewers eyes using the same natural audio cues and spatial information that we use in the real world to understand our environments.
The technology used
Mixed Immersion combines spatial audio content with interactive and immersive technologies, delivering solutions over speakers, headphones or a bespoke blend of the two. They offer experiential experiences through immersive audio, fully immersive, spatial and atmospheric VR and AR audio production and 3D audio capture, design, mixing and full production for traditional film and music.
Spotting a gap in the market for creating true 3D audio for the new wave of film production, they developed processes to build layered environments with spatial sounds, capturing a wide range of sound layers and rebuilding them using a knowledge of 3D platforms such as Dolby Atmos, Auro 3D, ISONOS and DTS:X.
Providing a convincing spatial environment requires a high level of technical expertise, equipment and systems – “there are no short-term fixes to capturing realistic and believable audio, regardless of what new-to-the-market VR companies might claim,” explains Dave. He continues:
“We have developed our own capture techniques which allow us to monitor the reverb and reflection of sound within real world environments and recreate them in any environment. This could be a theatre, concert hall, church, kitchen or even in the woods or on a beach. This process is very similar to Photogrammetry techniques used in construction which use high end laser scanners to take a ‘point cloud’ of 3D information and can then be reproduced in any 3D development platform.
We call this ‘acoustic fingerprinting’ as no two places ever sound the same and those used to the sound of a specific venue will be able to hear those very familiar acoustics.”
Mixed Immersion has also developed workflows and techniques for both 360° and 3D VR experiences, where the audio is required to be either directional for standard narrative or interactive for a user-led narrative. They have extensive experience working with game engines and 360° video editing tools, having already worked on a number of well-known VR experiences and recently have been working closely with an Oscar-winning director to create a world-first in 360° full sensory, VR production.
With their agnostic approach to content creation, they can use the same material across all available delivery platforms, apps and media and advise on the best platforms for displaying the created content. Their audio dialogue services include sound-to-picture mapping, voice-over recording, artist acquisition and disguise and manipulation techniques.
“We are also experts in large scale, immersive speaker installations and have a strong history in 3D audio systems,” says Dave. “Using the same technologies, we can scale these experiences down to single-person or crowd-sized headphone installations where each user will have slightly different experiences, depending on their locations.”
Coping with the challenges
“We’re not just building a company or a brand, we’re building an entire industry,” Dave explains. And the issue with any new development is persuading people that it needs to exist. “Not only are we educating audiences to the possibilities of this new world in audio, we’re having to educate potential clients also,” Dave continues.
“In film, the whole experience is about sitting forward and with audiences focused on the action, they don’t notice the sound. Film directors say that 3D sound just distracts people from what’s happening on the screen but in VR, it’s important to have 3D audio. However, audiences – and practitioners – don’t realise the importance of that audio as they can’t see it.”
Explaining that it’s currently a challenge to persuade clients to allocate budgets for audio when making 3D interactive experiences, such as games, Dave says there is not much appreciation for audio as part of the overall experience – and this is something they are working to change. But – “we have to get people to try it to believe it”. Once they do, the whole conversation becomes a whole lot easier!
“The difference between 2D audio and 3D audio is unreal but there are not many platforms to try it out on – there’s no YouTube for 3D audio for example. We need to allow more consumers to try it out to increase the demand.”
Projects to date
The company’s work to date includes the world’s first “interactive boot” for Adidas, fitted with sensors and real-time visuals that responded to users’ gestures and movements, alongside a 3D sound cloud that moves audio around the user using a technique called “granular synthesis”.
They have provided spatial audio environments and sound design for the pilot of the VR series, Rose Coloured, a future-based experience where AR and AI tools are used by the public to help understand their new world.
Working with two-time Oscar winner, the Indian composer and producer AR Rahman on his first VR film, they used the latest techniques in spatial audio production to help create an immersive viewing experience combing stereoscopic visuals, audio, scent and viewer-movement.
They worked for the Guardian on its Underworld experience, adding effects to enhance ambience and provide deep immersion, including the noise of manhole covers opening and the splashing of rats through the water.
Next on the horizon is the Somnai immersive theatre production in London which launches in March 2018. Described as “a layered reality show” created by merging real world theatrics and the latest immersive technology, the team will provide sound and speaker design, as well as managing on-set audio-based narration and direction.
What does the future look like?
“We feel there’s been a first movement of VR and AR, now there’s going to be a secondary movement where spatial audio assumes a key importance. The future of this audio will come out of headphones and will have the same impact on the world as that of the movement of audio from mono to stereo. Augmented reality audio is something that we should all watch out for in the next couple of years.”
— Mixed Immersion