Listening to Geoscientific Data
Interactive spatial audio sonification for geoscience and science-art collaboration by Karen Mair and Natasha Barrett.
Photo from Crush 2 (2011), an interactive sound-art installation exploring the microscopic forces released during the process of crushing rocks.
Sonification is the use of non-speech audio to convey information or display data. Since our ears are sensitive to unusual sonic events or sound patterns, sonification provides a potentially powerful way to perceptualise (or hear) our data.
The project ‘Listening to Geoscientific Data’ investigates sonification in two areas:
- As a science-art collaboration to engage and encourage interest in the fields of geoscience and cross-disciplinary collaboration.
- As a technique to enrich visual displays or provide an alternative method with which the geoscienist can explore multi-dimensional data.
By applying methods of audification, parameter mapping sonification and model-based sonification, we perceptualise data from computer models of earthquake processes and from mechanically induced rupture experiments. The models are inherently 3D and to best appreciate this spatial aspect of the crushing and sliding events that occur as faults slip, we use ambisonics (a sound field recreation technology). This allows the position of individual events to be preserved, generating a virtual 3D soundscape so we can explore faults. The addition of 3D audio to the sonification tool palate allows us to more accurately connect to spatial data in a novel, engaging and accessible approach.
During sonification, events such as grain scale fracturing, grain motions and interactions are mapped to specific sounds whose pitch, timbre and volume reflect properties such as the depth, character and size of the individual events.
In the science-art project active listeners can explore the 3D soundscape and listen to evolving processes in the data by physically moving through the space or navigating via a 3D mouse controller. The soundscape can be played over a loudspeaker array or over headphones.
To explore the potential of interactive sonification specifically for geoscience, the user should ideally be able to, through the use of haptic controllers, scale data ranges and modify mapping in real-time as the data is ‘sounding’. Although there already exist a handful of custom made interactive sonification programs, they are generally task specific and constrained to pitch mapping, noise modification and basic stereo panning. Our solution tackles the challenges by considering the complete timbral and spatial sound palette within a framework of sonic interaction design (SID).
Abe, S., and Mair, K. 2005. Grain fracture in 3D numerical simulations of granular shear.
Geophysical Research Letters, vol 32, L05305, doi: 1029/2004GL022123.
Abe, S., and Mair, K. 2009. Effects of gouge fragment shape on fault friction: New 3D modelling results. Geophysical Research Letters, Vol 36, L23302, doi:10.1029/2009GL040684, 2009.
Barrett, N. 2012. 18’30 real-time interaction performance of Crush rendered to stereo. https://vimeo.com/65065359.
Barrett, N. 2012. Aftershock video documentation. https://vimeo.com/66334772.
Barrett, N. and Mair, K. 2013. Aftershock: a science-art collaboration through sonification. Organised Sound, Vol 19:1. In Press.
Hollerweger, F., 2008. An Intoduction to Higher Order Ambisonic.
Mair, K., and Abe, S. 2011. Breaking up: Comminution mechanisms in sheared simulated fault gouge. Pure and Applied Geophysics, doi:10.1007/s00024-011-0266-6, volume 168, number 12, p2277-2288, December 2011.
Mair, K., and Abe, S. 2008. 3D numerical simulations of fault gouge evolution during shear: Grain size reduction and strain localization. Earth and Planetary Science Letters, vol 274, pp. 72-81, doi:10.1016/j.epsl.2008.07.010.
Schubnel, A., Thompson, B. D., Fortin, J., Gueguen, Y., Young, R. P. (2007) Fluid-induced rupture experiment on Fontainebleau sandstone: Premonitory activity, rupture propagation, and aftershocks. Geophysical Research Letters, Vol 34, L19307, doi:10.1029/2007GL031076.