Synesthetic Interfaces

Athina Papadopoulou & Harpreet Singh Sareen

firstpage2

Idea and Motivation: Synesthetic interfaces

We traditionally think of sensory augmentation as rendering one or more sense more acute, enhancing our perception beyond the human level. Or, similarly by adding an extra-sense not possessed by humans, such as magnetoreception. Contrary to this approach, we suggest that we can achieve sensory augmentation only by engaging the full spectrum of all of our existing human senses. The idea of increasing our sensory interaction with objects is not a new idea; there is a substantial amount of research in the field of multimodal interaction on how to integrate in digital interfaces not only vision but also sound and touch. However, even if multimodality in this approach allows us to interact with virtual environments similarly to a physical one, it doesn’t really enrich or augment our perception. Synesthetic interfaces can go beyond mere simulation of the physical experience through enhanced modes of human-machine interactions that promote creativity.

Case Study: Pottery Machine with Sonic Feedback

maxresdefault
Pottery machine with sonic feedback. Papadopoulou, A. & Singh Sareen, H. 2016

To explore the idea of synesthetic interfaces we designed experiments and prototypes for the development of a pottery machine with sonic feedback. Through this case study we aimed at gaining a better understanding of the shape-sound correlations as well as experiment on possible implementations of a multimodal creative system founded upon synesthetic associations. The basic idea for the implementation of the synesthetic pottery machine is that the sonic feedback system reads the profile of the ceramic pots in the making and generates sounds that match the specific shape. One basic reason that we chose to study pottery as a case study is that the symmetric nature of the pots facilitates the translation of shape to sound. In fact, rule-based profile reading of pots is an already established method in archeology as it is used for pottery classification and 3d simulation based on found fragments [1]. Thus, we believe that the method we propose could easily become intuitive to the pottery makers and the scientific community surrounding this craft.

Drawing1Edit
Pottery machine with sonic feedback. Papadopoulou, A. & Singh Sareen, H. 2016

Background & Related work: Synesthesia and Creativity

Synesthesia is a condition that allows one to have sensory impressions on an additional sensory modality that the one stimulated. Although traditionally perceived as a malfunction of the brain, synesthesia has recently been discussed as a creative state of the mind that offers augmented cognitive and sensory abilities [2]. According to studies, people that naturally acquire synesthesia after a sudden incident, demonstrate increased creativity, and/or augmented cognitive abilities in a certain activity, or even excellence in certain artistic endeavor with no prior knowledge or experience. This all happens because the brain becomes rewired elevating some skills beyond usual levels and providing cognitive access to “hidden” parts of the brain [2]. Recent evidence from cognitive experiments demonstrates that synesthesia, is not only an innate or naturally acquired condition but it can be learned upon training [3]. In other words, through training we can increase our sensory interactions with the world and acquire augmented creative skills.

51Xbv1cLk7L._SY344_BO1,204,203,200_
Synesthetes as “Superhumans.” Brogaard B, and Marlow, B. The Superhuman Mind (Cover)

Synesthesia as a form of creative human-machine symbiosis was explored by the cybernetician Gordon Pask in the 1950’s. In 1953, Gordon Pask developed the Musicolour System, a machine that used sound as inputs and projected colors as outputs. The Musicolour System was used in musical performances to promote a creative exchange between the conductor and the machine [4].

musicolour
Gordon Pask, Musicolour machine1953

Artists have also been exploring synesthetic cross-modal correlations. In the Bauhaus school, Wassily Kandinsky experimented with the integration of different sensory modalities through “synesthetic” multimedia experiments [5]. Today, contemporary artists like Timothy Laiden explore visual-auditory correlations focusing on the visual manifestations of sounds [6]. Research in cognitive science has shown that certain sound-shape associations may have a common embodied and cognitive basis. For example, as the “Bouba/Kiki effect” demonstrates, people tend to associate rounded shapes with rounded vowels and unrounded vowels with angular shapes [7]. The physical responsiveness of materials and shapes to sound can also indicate ways to define sound-shape correlations. Research in human echolocation demonstrates methods for perceiving shapes and materials through sounds [8].


Design and Implementation: Method, Experiments and Developed Prototypes

Before implementing the synesthetic pottery machine system we conducted a pilot experiment to test whether the chosen sound-shape associations would be perceptible by the users. Because in human echolocation methods an increase in amplitude or frequency corresponds to different properties in spatial features, we decided that a focus on either invariant frequency and variant amplitude or variant frequency and invariant amplitude would be more intuitive for the users.

pots-11
Group 1. Basic Shapes

We finally chose to vary the frequency as it generated more pleasant sound mappings. For the experiment we designed four groups of shapes. The first group consisted of very basic 2D shapes, the second group of basic 2D shape combinations, the third group of complex 2D shape combinations and the fourth group of 2D pottery profiles. We used the first group – the group of basic shapes – to train the participants in the shape-sound associations by asking them to listen to the sounds while observing the corresponding shape on a computer screen. After this quick training session, we played the sounds of the second group – the basic shape combinations – and asked the participants to draw the shapes. This time the corresponding shapes were not shown to the partici

pots-12
Group 2. Shape Combinations
pots-13
Group 3. Advanced Shape Combinations

pants. The same procedure was repeated for the third group of shapes – complex shape combinations – and the fourth group – pottery profiles. The system was reading the profile from top to bottom mapping each pixel to sound sequentially. The sound output was set to produce higher frequency when the pixel was further away from a set boundary and lower frequency when the pixel was closer to the set boundary.

pots-10
Pottery Profiles

Seven students took part in this pilot experiment. On average, most of the participants were able after a short training to identify the first and second group of shapes although the sharp angles and smooth curvatures were often difficult to be distinguished. The third group of complex shapes was difficult to be perceived through sonic information. Judging from the participants comments, a longer duration of the produced sound would facilitate shape perception. The fourth group – pottery profiles – was not identifiable in detail but, on average, the basic curvatures – positive/negative – and sequence of shapes were reproduced by the participants in the drawings. Again, longer duration of the produced sound would facilitate shape perception. We also noticed that participants with music knowledge performed better on the experiment. Another observation was that although distinguishing a separate tone for each tone sounded more musical, smoothing out the difference between the different frequencies into a continuous sound facilitated shape perception.
Two prototypes were developed to test sound mappings of pots. For the first prototype we used a Kinect sensor and programmed the system to read the profile of the pot from top to bottom, reading each detected point sequentially. We tested bost smoother and more segmented mappings. Capturing the profile was a real-time process where the user could also observe on the screen a red line being drawn on the pot, matching the speed of the sound mapping. As in the previously described shape-sound experiments, the profile shape was mapped into a sound output based on distance-frequency correlation where the distanced points would correspond higher pitch than the points read closer.

The second prototype was made using an array of 16 pairs of infrared LED emitters and receivers which were controlled through a microcontroller. Similarly to the first prototype, we programmed the system to map the distance of the object to the sensor to a range of frequencies (the closer the object, the lower the frequency). The second prototype although not as precise as the first, allows for a more intuitive seeing-hearing-doing interaction which we believe would prove to be useful in the actual process of making.


Usage Scenario: Synesthetic Pottery-making

Although limited at the moment on shape sound correlations based on distance and easiness of perception, the synesthetic pottery machine could potentially allow for a creative feedback loop to take place between the user and the system. The craftsman could train the sonic system on their aesthetic preferences and the sonic system could in turn help the craftsman to be trained on synesthetic associations. If a sound-to-shape correlation could be established for a variety of combinations of amplitude and pitch, then any sequence of sounds, whether an instrument, or song could be translated into shape. Eventually different pottery styles could emerge through different music genres and styles.

Drawing2Drawing2
Conclusion and Future Work

Research in cognitive science has demonstrated the creative effects of cross-modal interaction and sensory integration. Moreover, synesthesia has lately been regarded as condition that can foster creativity. Previous work in cybernetics and the arts has demonstrated ways to combine seemingly unrelated modalities promoting synesthetic associations. In this project we explored the idea of synesthetic interfaces as a means to induce cross-modal correlations in the pottery making process. Through the study case of pottery profile sound mappings we demonstrated that simple samples can be easily perceived through sound. We believe that after the necessary training, more complex shapes, and combinations of these would be also easily perceived by the user/maker. Another aspect that we aim to further explore is how the shape-sound associations affect the makers’ aesthetic judgement and choices of form in the making process.


Citations

1. Kample, M. & Sablatnig, R. (2007) Rule based system for archaeological pottery classification. Pattern Recognition Letters, 28 (6), pp. 740-747
2. Brogaard, B., & Marlow, K. (2015). The superhuman mind: Free the genius in your brain. New York: Penguin
3. Bor, D. et al (2014) Adults can be trained to acquire synesthetic experiences. Scientific Reports 4
4. Reichardt, J. (ed) (1971). Cybernetics, Art and Ideas. New York Graphic Society
5. Ione, A. & Tyler, C. (2003) Neurohistory and the Arts: Was Kandinsky a Synesthete?. Journal of the History of Neurosciences, 12 (2) pp. 223–226
6. http://theshapeofsounds.com/
7. Maurer D, Pathman T & Mondloch CJ (2006). “The shape of boubas: Sound-shape correspondences in toddlers and adults” (PDF). Developmental Science 9 (3): 316–322.
8. Kolarik AJ., Cirstea S., Pardhan., S., Moore BC., (2014), “A summary of research investigating echolocation abilities of blind and sighted humans,” Hearing Research, Volume 310, April 2014, Pages 60–68

 

Contact :
Athina Papadopoulou (athpap@mit.edu)
Harpreet Singh Sareen (sareen@mit.edu)