Grimshaw has been interested in game sound for some time and how sound helps create an immersive experience. He is also interested in how games sonify others in a multi-player game (how you hear others). He is also interested in virtual reality and how sound can be used to give verisimilitude.

Why rethink sound? He started by discussing problems with definitions of sound and trying to redefine sound to understand sonic virtuality. The standard definition is that sound is a sound wave. The problem is that there are really two definitions:

• sound is an oscillation of pressure or sound wave, or
• sound is an auditory sensation produced by such waves (both from the ANSI documentation)

He mentioned another definition that I rather liked, that sound is “a mechanical disturbance in the medium.” This is from an acoustics textbook: Howard, D. M., & Angus, J. (1996). Acoustics and psychoacoustics. Oxford: Focal Press.

Not all sounds produce an auditory sensation (like ultrasound) and not all sensations are created by sound waves (eg. tinnitus). For that matter, sound also gets defined as that which happens in the brain. The paradox is:

• Not all sounds evoke a sound, and
• Not all sounds are evoked by sound.

He then talked about the McGurk effect when what we see overrides what we hear. Mouth movements cause us to hear differently so as to maintain a coherent version of the world. Perhaps sound waves are not all there is to sound. See https://www.youtube.com/watch?v=G-lN8vWm3m0

He provided some interesting examples of sounds that we interpreted differently.

What is interesting is that we defined sound as the sound source, as in that sound is a bird. This shows how our everyday sense of sound has nothing to do with waves.

Then there is the question of “where is the sound?” Does the sound come from the ventriloquist or from their dummy. The effect is known as synchresis (see here). We have the ability to keep more than one mapping system of sound sources. We locate sound very well, but in some cases, like cinema, we locate it in the world of the film. We can separate the location of the heard sound from its actual location.

Some other definitions include:

• Democritus said sound is a stream of particles emitted by a thing (a phonon)
• Sound is an event (Aristotle)
• Sound is the property of an object
• Sounds are secondary objects and pure events
• Sound is cochlear (involving sound waves) and non-cochlear sound (synaesthesisa)

Needless to say, the language of science around sound is very different from our everyday language.

His definition is for “sonic virtuality”:

Sound is an emergent perception arising primarily in the auditory cortex and that is formed through spatio-temporal processes in an embodied system.

A sonic aggregate is all the things that go into forming the perception of sound (like what you see.) Some is exosonus (what is outside) and some is endosonus (non sensuous components).

He talked about how for some animals there might be a sense of “smound” which is some combination of sound and smell.

The emergence of sound can determine epistemic perspective, though in some cases the perspective forms the sound. Imagined sound is just as much sound as exosonus sound.

In sonic virtuality, sound localization is a cognitive offloading of the location of sound onto the world. We tend to put sound where it makes sense for it to be.

Cognitively what seems to happen is that we form hypotheses about sound aggregate and eventually select emergent version of sound. This is embodied cognition which is time pressured – ie. pressured to decide quickly. We don’t know for sure.

Immersion and Presence

He then shifted to talking about games and the difference between immersion and presence. Immersion is supposedly objective – how close to reality is the simulation of sensory stimuli. Presence seems more subjective.

The way we locate sound out in the world is what leads to differentiation of self and not-self and that leads to sense of presence. Sound tells us about space.

If we want a better sense of presence in virtual reality – is increasing the simulation the way to go? VR systems try to deliver discrete sensory stimuli of greater and greater verisimilitude.

RV or real virtuality suggests a different approach – that of an appropriate level of stimulation that lets the brain make sense of the virtual space. You want the brain to actively engage.

If we model sound as perception then can we extract it? Can we extract sound? This is an area called neural decoding. (Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011). “Reconstructing visual experiences from brain activity evoked by natural movies.” Current Biology, 21, 1641–1646.) It seems they can now reconstruct what someone saw from the brain imaging.

Sonification

At the end he talked about sonification which connects to the 3DH project. Sonification is the audio equivalent to visualization. What is the value of representing data with sound? He gave some examples of sonification:

• A geiger counter is a sonification of radiation
• In radio astronomy sonification is used to help finding interesting or anomalous moments in radio waves. We can’t stop listening the way we can look away.
• PEEP (PDF) is a tool that sonifies network activity.

If we can transform large amounts of data into sound, what would we do? Each sensory modality has some things it is good at and some it is not so good at.

• Sound is good at time.
• Ambiguity is hard to visualize and often left off. Sound might be a way to keep ambiguity.
• Sounds can have meaning that could be used (but sound waves do not.)

Are there some sound primitives? Yes! there are some sound primitives that seem to be evolutionarily encoded in us like the sound of something rapidly approaching. Our brains seem to be attuned to certain sound wave attacks. What are the sound primitives that we can manipulate?

• Attack
• Loudness
• Tone(s)
• Texture (timbre)

Discussion

Some of the points that came up during discussion include:

Are there ways that sound can contradict vision as in a Jacques Tati movie like Playtime? It turns out that in most situations vision dominates hearing, but in others hearing can override vision. It seems that hearing is very sensitive to temporal changes, as in changes in rythym.

Are there ways of understanding the cultural and social in interpreting of sound?

Note

This was updated with corrections from Grimshaw.