7.9 Sensory Congruity and Multisensory Integration

From an ergonomic perspective, if we can understand how to use the multisensory features of a product, then our understanding is consistent with how it was designed to be used (Schifferstein & Spence, 2009). This consistent relationship is called sensory congruence and happens when the sensory message is aligned with the overall product interaction experience. A congruent message fits into our perceptual and cognitive understanding, whereas an incongruent message does not. Looking at a hammer, you would likely expect it to be heavy and if it is, upon picking it up, you will likely use it without thinking about it. However, if it feels light, you may wonder what is wrong with it and if it will work. That heavy feeling is providing a congruent message – it is what it seems to be and what you think it should be based on your familiarity with the product. In our example below, a thoughtfully designed product package can present a clear message to the user about the product, whereas a product package that conceals what is inside does not provide any message about its contents.

Congruent visual messaging between product and package

Integrated multisensory experiences also provide a congruent perception through sensory synthesis (combining of senses). For example, taste and smell are experienced together as flavour; audio and visual effects are often combined to provide robust, immersive movie experiences; and the integration of our visual vestibular system in our inner ear contributes to our sense of balance. In these latter contexts, one sense helps the other focus and prepare for the next stimulus (Park & Alderman, 2018; Stein, 2012).

Sensory Incongruity and Novelties

In contrast to a congruence approach, some designers use the element of sensory surprise in their products to offer a more stimulating design. For example, you may expect a stool that looks like a sawed-off log to be hard like a log, yet it turns out to be a soft pillow that merely looks like a log (this was initially discussed in the section about colour as material deception, Chapter 3). When our expectations are different from the reality of the product or experience, this is called sensory incongruity (Ludden et al. 2007, 2008).

Visual-tactile sensory incongruence: it’s a pillow!

Studies by Ludden, Schifferstein & Hekkert (2007, 2008) suggest that evoking the element of surprise or intrigue in a product can be seen as a uniquely pleasurable product experience. Sensory incongruities can be deliberately incorporated into a design to capture consumer attention or make a product unique and recognizable through associations with memories. However, designers should consider avoiding garish or kitschy incongruities to add meaning to a product – to avoid detracting from its long-term value (Ludden, Schifferstein & Hekkert, 2007). Designing novelties into products can be beneficial and fun or harmful and aggravating, depending on the context. Designers need to recognize and apply incongruities suitably to enhance an experience. Consider, for example, which bar of soap pictured below would you prefer to use.

How much sensory incongruity surprise is appropriate?

We can experience different types of sensory incongruities or sensory novelties. Hidden novelty is another type of sensory incongruity; it refers to products that look familiar at first sight yet have unexpected properties that may be shocking. The surprise arises when interacting closely with the product, offering an unexpected sensation, material, or output (Ludden et al. 2007, 2008). While the log pillow is a form of sensory incongruity and hidden novelty, there are some designers who take the concept of hidden novelty to the extreme. For example, Chef Ben Churchill (2020) and Tuba Geçkil (2021) are chefs who independently create desserts that challenge sensory perceptions. Churchill’s sweet offerings (a dirty dish sponge, a decomposing orange, or a soiled ashtray, amongst others) initially look unappealing but taste delicious. Whereas Geçkil’s hidden novelty cakes look like everyday objects (an iron, a toaster, a roll of toilet paper, a raw piece of chicken, for example), only to surprise viewers when they are sliced into, revealing pieces of deliciously layered cake.

Tasty hidden novelty!

Another sensory novelty is a visible novelty, which occurs when an element looks unfamiliar and we have no former references or expectations to relate to (Ludden et al. 2007, 2008). For example, the image below captures the unusual novelty that Luigi Colani introduced to camera design in the 1980s (Colman, 2009). At the time, his camera was such a novel concept that it was unfamiliar to camera users who were used to holding small rectangular boxes to take photos. Today, Colani’s once-novel tactile affordances have been interpreted into common curvilinear and ergonomic features of the camera’s shell and no longer seem novel. In the case of visible novelty, we may experience a surprising reaction to its true nature. Some of us may try to make associations with other familiar elements or products but remain uncertain about its reality until exploring its surface and weight.

Visible Novelty

Ludden et al. (2007, 2008) found that both hidden and visible novelties evoke surprise. Their participants tended to be more reactive in their facial and vocal expressions to hidden novelties and more exploratory with visual novelties by investigating and interacting with the products (Ludden et al., 2007, 2008).

Synaesthetic Experiences

Different stimuli are perceived as crossmodal correspondences

As we have noted, most of us engage with products, environments and services as multimodal experiences, however, some individuals process sensory stimulation in unique ways, which we refer to as synaesthesia. Synaesthesia is a neurological condition where incoming information that is normally perceived by a particular sense stimulates other senses too; a person may experience stimuli from one sensory channel through a different sensory channel as a crossmodal correspondence. Wassily Kandinski was well known for perceiving tones of sounds associated with particular colours – where a trumpet or bugle is yellow, and a flute, cello, or organ is blue (Kandinsky, 1910). However, synaesthesia is not limited to auditory and visual combinations; some people relate colours to letters of the alphabet, and others relate sound to pain levels in their bodies; the combinations of senses are diverse. These are involuntary experiences, typically involving two sensory modalities that inform one another, which vary among individuals but are consistent for any given person (Schifferstein & Spence, 2009).

Synaesthetic sound associations

True synaesthetic experiences are natural and cannot be induced, although designers can use synaesthesia as inspiration in designing multisensorial products (Hummels & Overbeeke, 2006). As an example, synaesthetic-like features can be experienced in the use of electric toothbrushes. In experiments using sound to enhance the toothbrushing experience, louder and higher sounds were related to roughness, whereas softer and lower sounds were related to gentleness. People also perceived fabrics as softer when imbued with pleasant scents (Schifferstein & Spence, 2009). Interestingly, most of us are usually unaware of these crossmodal effects (Schifferstein & Spence, 2009).

Crossmodal effects inspired by synaesthesia

The scope of the discussion about multisensory principles and applications in design would not be complete without considering the multisensory aesthetics of movement, which we discuss in the following section.