Fully Immersive Spatial Audio and Why It Matters In VR

Fully Immersive Spatial Audio has the potential to change the face of the VR industry as we know it.

It’s not a new technology by any means, and it’s been around for at least 100 years prior to this moment – but it’s never quite been applied to the development of Virtual Reality quite like now, and we’re seeing great leaps and bounds in technology and the way people are immersed in their virtual worlds.

Virtual Reality becomes pretty useless to the user when it isn’t an immersive experience, and the burden falls on the developers of VR hardware and software to ensure that the experience is as close to real-life as possible. It has to make you feel like you’re right in the middle of what you’re looking at – and it’s not enough to be looking at a scene through VR glasses when you want the full experience.

True immersion for Virtual Reality involves being able to engage all of the senses at once – sight, smell, taste, touch and sound; though we know that taste and smell aren’t nearly as necessary for VR to work as the rest. Could we go as far as to say that the senses required for VR immersion is sight, touch and sound?

How do you judge whether or not something is real? In the first Matrix movie, Morpheus asks Neo if what he can smell, taste or touch is real – and then goes on to show him that the construct of “real” he had known was just about fooling these human senses with computerization.

While we haven’t blacked out the sun yet like in the heavily Dystopian tone of the Matrix movies, the movie got the idea of Virtual Reality pretty spot on. If you’re going to fool anyone into thinking it’s real, fool the senses.

To believe that you’re truly immersed in your virtual reality experience, sound and how you experience it is one of the most important parts of the whole VR thrill.

Reacting to Sound

Sound activates a primal reaction within the brain, and sound is often the very first way we react to something; for example, while you’re playing a first person shooter, you’ll be paying the most attention to the sound to find the position of the opposing player – on the other hand, while immersed in a VR movie, your mind will pay the most attention to the sound to associate both the sound and the visual aspects with each other.

To put it in its most simple terms, the brain asks itself three questions:

What am I seeing?

What am I hearing?

Do these two things match up?

The first two questions should be answered by the VR experience already – and if one of the first two questions aren’t immersive enough to the user then the brain’s answer to the third (more important) question that tells your brain how immersed it is (or not) will be a simple no.

We already have the visual aspects of virtual reality developed to a much more advanced degree than some years ago: Virtual Reality goggles are becoming cheaper and easier to access; projects like Google Cardboard have even made experimentation with VR easier.

But when it comes to developing the sound aspects that fit with the visuals, fully immersive spatial audio could be the answer to finally providing the perfect VR experience.

Why Regular Audio Isn’t Enough

If you want a practical example of why regular audio isn’t enough for real VR immersion, then you’re going to need to do a little experiment. You can use even the cheapest headphones for this.

There are many songs that illustrate the concept of stereo practically so that you can hear why it matters: Many classics by the Grateful Dead or Led Zeppelin use effects that seem to “bounce” between one headphone to another. This effect is the stereo effect – and it sounds pretty damn cool, especially through the right quality headphones.

But that’s not all it is: It’s also something very important to your brain.

Audio tells you where things are. If you were to close your eyes and listen, your brain relies on the position of sound to form a clearer picture – and the same applies to VR, when your visual input is being replaced with a picture seen on a VR screen.

In order for your brain to be truly immersed in the VR experience, your brain needs to be able to relate to 3-dimensional sound – sound that tells you more about the environment you’re in, which is (in this case) a virtual one.

If the audio is off, your brain will notice that something is “wrong” and it just won’t sound right.

When It “Doesn’t Sound Right”

When things around you don’t sound quite “right” to your brain like we’ve just explained above, your brain automatically assumes that there’s something wrong with the entire environment – and the whole VR setup starts to disappear for your brain, much like the glitch in the Matrix when the cat appears twice.

“It means they’ve changed something.” Your brain knows something is wrong when the audio doesn’t match, when it doesn’t truly immerse you in what you’re seeing – but when your brain trusts the two inputs fully, then it tends to trust the entire concept of immersion more.

You can fool your brain into experiencing a fuller virtual reality world – but only if you use sound immersion right.

What the Hell is Fully Immersive Spatial Audio?

Fully Immersive Spatial Audio isn’t a “new” technology by any means; it’s something sound engineers have known about for decades – but we’re seeing more and more reason to use this technology within VR because it offers a far more “full-bodied” sound experience that gets us a little closer to experience real virtual reality.

Fully immersive spatial audio makes use of several different techniques in order to round off the sound experience, turning it into a truly 3D sound experience that “positions” different sounds in the room – or virtual environment – so that your brain forms a much fuller image.

There are many techniques used in the development of fully immersive spatial audio, including the use of heightened speakers that add an element never-before-seen in speaker systems: Height.

Think about your average “surround sound” system – it surrounded the viewer as much as it possibly could, but one of the elements that was very rarely accurately added was height: Sure, you could tell that a sound was coming from one side or another, but you couldn’t tell the finer details like how close to the floor this sound might have been.

Are you starting to see the difference between simple surround sound and fully immersive spatial audio?

Fully immersive spatial audio offers three dimensions, and space to sound where you previously wouldn’t have had such a deeply immersive experience: It offers height, depth, reality: The real elements that you’re looking for when we’re talking about VR.

Immersive Sound and Training Developers

The applications of immersive sound for achieving real reality within VR are immense – and it also opens up a whole new world of opportunity for both sound engineers and developers, who might have to explore the field of immersive sound more in the form of workshops.

There are plenty of workshops (and them some) that explain the basics of immersive sound, how it works and how they can apply practical techniques to their sound recording (and playback) to ensure for a better sound experience.

Scaling Through Sound

Sound tells you everything you need to know about your immediate environment, and spatial immersive sound attempts to fill in all of the gaps that your brain needs to know about the essential elements.

• Where am I in relation to the sound?
• How far is the sound from my immediate position?
• How high or low is the sound?
• What’s the distance of the sound?

If your brain doesn’t receive a 3D answer about these questions – one that positions the brain, the sound and the environment all at once – then the experience just can’t be called immersive enough to be called real Virtual Reality.

The importance of fully immersive spatial audio can’t be underestimated for anyone who needs to have the fully immersive experience. If you want to fool the brain completely, it turns out that you have to fool the right senses at the same time.