Or how your brain needs your eyes to help trick your ears into hearing what’s not there.
Or how your brain needs your eyes to help trick your ears into hearing what’s not there.
In the past few months, I’ve experienced two different audio technology demos that have given me more insight into audio processing than I have had in my entire career. And I’ve come to the conclusion that you really have to “see” audio in order to believe in the experience that you’re hearing.
Of course, this concept isn’t really new. Think of blind taste testing, or the popular mystery box game where people stick their hands into a box and try to guess what they’re touching. As humans, we need visual confirmation of the input from our other senses in order to process it.
I knew this of course, but I misjudged the extent of the difference that sight brings to the table. This is why it’s so hard to appreciate audio technologies such as surround, binaural, 3D or physically-based sound. You can read about it and understand what is happening, but you may not always be able to put a finger on what you’re hearing.
Many surround sound systems try to mimic multiple audio sources to trick you, commonly called HRTF processing, or Head Related Transfer Function. Object-based surround sound like DTS:X and Dolby Atmos take the concept further by treating sound like an object that can be moved around in a 3D space, so you don’t just hear sounds front, or left, or back, but all around around you.
NVIDIA on the other hand is pushing path-traced audio through their VRWorks Audio SDK, which is technically ray tracing, but for sound instead. This is sound that can be processed by graphics cards such as NVIDIA’s new Turing-based GPUs capable of real-time ray tracing. By ray tracing the path of sound, not only can you project audio, it’s also calculating the changes in quality of the sound from all the surfaces it passes through before it reaches your ears.
Between all the fancy graphics demos during NVIDIA’s Quadro RTX launch, what struck me the most was a VR demo of NVIDIA Holodeck, a collaborative VR design environment that was retrofitted to showcase path-traced audio. The dynamism of real-time, path-traced audio was immediately apparent in a VR environment. For example, I could tell the difference between music that came through a glass door and a wooden door, or the qualities of speech from an empty room to one filled with furniture. It was clearly a very different experience than the usual surround sound processing you hear from games.
The second demo that blew my mind, was at a personalized fitting for Creative’s Super X-Fi Amp. Super X-Fi is Creative’s attempt at surpassing headphone limitations. Billed as “Headphone Holography”, the idea is to trick you into believing that you’re hearing sound naturally from a larger room, rather than tiny speakers right next to your ears. It does this through complex calculations of how sound travels and ultimately hits your ears, and as such, is very dependent on both the type of headphones used and the shape of your head and ears. This is why Super X-Fi works best when it is personally calibrated with professional equipment in a studio environment, which was what I experienced.
During the fitting, I could see the speaker setup in the room, so my eyes helped my ears to trick my brain into believing that audio from the headphones were actually originating from the speakers in the room. When the time came to test Super X-Fi calibration by turning it on an off, the audio projection was so convincing that some of the people in my session took off their headphones just to check if Creative was pranking us. The aural illusion was uncanny.
However, the general masses who will buy this product will only be privy to an app-based configuration and not the full calibration. Now, I don’t doubt the technology actually works, but for someone who has never experienced the same combined audio-visual stimuli as I have, the result of turning Super X-Fi on will be greatly diminished.
You’ll definitely still be able to tell a difference, but is it different good? Different bad? Is it any different other types of equalizer settings that many audio apps already have? How do you know what a studio sounds like? What if you actually like how your headphones sound without all the processing?
Tl;dr
Digital audio processing is advancing at a tremendous pace with exciting breakthrough developments. Ironically, all this audio magic still doesn’t really work unless you are first able to see what you’re hearing. Even the two technologies mentioned here—ray tracing audio and headphone holography—were developed from visual concepts.
What struck me the most was a VR demo of NVIDIA Holodeck, a collaborative VR design environment that was retrofitted to showcase path-traced audio.
PICTURE 123RF
