Gestalt Audio Storytelling
Audio has tremendous power to influence our perception of reality.
In media, music is the most common expression of this, and the effects of music on the mind are well known. Non-musical audio, such as foley, is also incredibly important in making artificial worlds real. However, sound extends well beyond the sense of hearing. It is generally accepted that human beings cannot perceive sounds above 20 kHz, but the physiological effects of high-frequency components (HFCs) in audio are well-documented. A common example of this fact is the sound design of horror movies; leveraging theater sound systems, high-pitched sounds that just barely meet the range of hearing set audiences on edge, increasing their heart rates and making them sweat without fully realizing why. Many people become ill near faulty machinery or on rolling train cars, not simply because of the motion, but because of the inaudible frequencies that the rubbing metal parts broadcast. Pieces by artists such as Janet Cardiff also show that these harmonics can be utilized as a positive element in auditory experiences.
In this experiment, I intended to explore how these psychoacoustic effects can be combined with traditional music theory, spatial mixing and entrainment techniques to create a sonic experience encompassing the full possibilities of human auditory senses. To structure this, the experience can be framed as a narrative, allowing the listener’s imagination to realize and personally contextualize the audio data. My final iteration of this idea is a standalone audio track meant to be experienced as an individual journey; however, these ideas have potential for experiences between multiple participants and senses.
Stage 1: Concepting
The first thing I did to explore this topic was research the different effects of audio I could play with. In doing so, I talked with Robert Harari, the resident aficionado of psychoacoustic research in Stevens Institute of Technology's Music and Technology program. The culmination of this research was a short audio experience for two, exploring the ability of sound to re-contextualize a social experience.
Psychoacoustics_Demo_v2.mp3Duration: 2:17. The intended experience for this audio is listening with another individual.
Both participants should be listening through headphones and starting playback at the same time.
Eye contact should be held through the duration of the piece.
Stage 2: Prototyping
Iteration 1
Next, I experimented with communicating a narrative entirely through audio by introducing a more structured context and environmental noise. The above storyboard outlines a potential audio sequence incorporating music, foley and dialogue to deliver a narrative for individual imaginative experience; the audio below is a mockup of some of those structured elements, approximating a basic form of how that audio could sound.
Binaural_Test.mp3Duration: 1:19. The intended experience for this audio is listening as an individual through a sound system capable of detailed binaural reproduction, such as a pair of headphones.
Iteration 2
After this, I made a jump to incorporating visual and interactive elements to further draw the listener into a full experience. To do so, I created a progressive audio track incorporating HFCs, and a small playable game that provides some context and control to the player. As the player progresses through the game, the audio also progresses, tying both elements together.
EvolvingGameplayAudioPrototype.mp4A demonstration of the game with evolving audio tied to gameplay. Caution: The beginning of the track broadcasts ultrasounds that may make certain people feel ill or motion sick.
On deploying this demo with my test audience, I discovered that the visual and gameplay components draw too much attention away from the audio, causing the focus of the experience to be on less impactful elements of the experience. In addition, the intended deployment method of a web-based game had compatibility issues with devices, which is not a problem with audio files. Because of this, I decided to focus on a pure audio experience for the final iteration of this experiment.
PsychTrainAudioFull.mp3Duration: 2:14. The full musical progression featured in the game. I encourage the listener to experience the track on its own to contrast with the visually accompanied version. Caution: Contains some ultrasounds.
Stage 3: Final Product
My final creation for this experiment is a freeform audio narrative, following as a similar but more refined expansion on the concept of my first prototype iteration with a fresh theme and strategy. As all projects have infinite iterations, this work functions as a touchstone for my exploration of the concept and a culmination of this phase of research.
Audio Experience: A Walk
FinalIteration_Version2.mp3Duration: 2:20. The intended experience for this audio is listening as an individual through a sound system capable of detailed binaural reproduction, such as a pair of headphones.
I recommend relaxing, closing your eyes, and letting yourself be transported to the world the sound inspires in your imagination.
Caution: Contains some ultrasounds.
Bibliography
Oohashi et al., Inaudible High-Frequency Sounds Affect Brain Activity: Hypersonic Effect, 2000, https://journals.physiology.org/doi/full/10.1152/jn.2000.83.6.3548
G Gerra, et al., Neuroendocrine responses of healthy volunteers to `techno-music': relationships with personality traits and emotional state, International Journal of Psychophysiology, Volume 28, Issue 1, 1998, Pages 99-111, ISSN 0167-8760, https://doi.org/10.1016/S0167-8760(97)00071-8
Madden, The Resonant Human: The Science of How Tempo Affects Us, 2014, https://sonicscoop.com/the-resonant-human-the-science-of-how-tempo-affects-us/
***
An interesting phenomenon of selective auditory disturbances worldwide: "Have you heard 'the Hum'?". BBC News. 19 May 2009.
Duloc, a 2016 album by artist Worthikids integrating strong musical/skeumorphic audio narrative elements: https://worthikids.bandcamp.com/album/duloc