︎

Subcortical Matters

Data Sonification for Augmented Reality Experience  
2023


https://www.maximelecalve.com/on-going-fieldwork/2023/12/18/rubbing-as-fieldwork-workshop-on-site-writing-with-technologies-of-captures-with-melt-and-shauna-janssen-and-speclab-at-kunstgewerbemuseum

https://speculativerealitieslab.pubpub.org/pub/yp1wtprt/release/5

https://www.matters-of-activity.de/en/activities/12820/site-writing-with-technologies-of-capture


The subcortical matters projects aim to create an immersive experience based on new developments on neurological data obtained by fMRI. Through data processing techniques, it is possible to perform a tractography of neural paths within the white matter of the brain, thus identifying specific connection that relate to certain functions of the brain.

Within neuro-chirurgical practices, this is important in the removal of gliomas (tumors), so that such paths are not damaged in the process.

We are interested in exploring fMRI data to visualize subcortical tracks. Beyond the standard Diffusion tensor imaging (DTI) = One direction we aimed to include the Constrained spherical deconvolution (CSD) = multiple directions technique to acquire a more comprehensive visualization of said data. We take advantage of data processing of voxel (3D pixels) that signal not only the position of data but also its direction though spherical harmonics.

Sonically the project consists of three layers. First it should include a basic and pragmatic UX sound design that is coupled with the VR experience. This includes reinforcing certain movements and gestures that the user might encounter. Second it should include a semantically representative layer, where data configurations glioma location in relation to neve tracks. This includes the use of distortion to signify the sound of the glioma or certain modulations to signify its position. Finally, the overall configuration of the voxel data should create a soundscape that represents the configuration and nuances of different patients.