7.1 Introduction

Earlier chapters explore the application of sensory features to facilitate positive design outcomes. They provide you with foundational knowledge about the significance of designing for each of the senses and set the stage for the more complex multimodal (multiple sensory modes) interactions presented in this chapter. This chapter examines the sensory modalities we experience when using a product and how they can be integrated into a greater whole that contributes to our individual interactions with it. It also explores how our holistic and multimodal experience depends upon the synergy of our many senses interacting with products, environments, and services at the same time.

Multimodal/Multisensory synergies across senses

We perceive our surroundings and interactions as a combination of sensory interactions, not as separate sensory layers or modalities (Ludden, Schifferstein & Hekkert 2007; Fulkerson, 2013; Park & Alderman, 2018). For example, in previous chapters, we learned the importance of our sensory modality of vision for most activities, especially for orienting and scanning our surroundings. When our visual perception is combined with tactile coordination, we experience the familiar multisensory experience of hand-eye coordination, e.g. taking notes during a lecture, playing a musical instrument, hammering a nail, or sending a message on any electronic device. We perceive this as a fully integrated sensory experience rather than separate sensory visual and tactile layered experiences. In addition to visual and tactile sensory experiences, we may simultaneously depend on auditory and proprioceptive feedback (discussed later in this chapter) for locating our body position and navigating through environments (Park & Alderman, 2018). These multisensory abilities depend on integrating complex layers of sensory modalities that systematically support and enhance user experiences. In this chapter, we discuss approaches designers take to accommodate these complex layers and add new terms that describe them.

The designer’s job is to determine the appropriate multisensory design features that best support a wide range of interaction contexts. Since each of us processes sensory feedback differently, we need to learn more about accommodating a range of perceptual variations through design. In many cases, it may not be possible to dissect some multisensory interactions into their individual sensory modalities. In these cases, our senses are so intertwined that we perceive them as truly blended multisensory interactions that we call crossmodal. For example, vision and touch often provide overlapping crossmodal information; people can see and feel the shape, size, or surface roughness of an object (Schifferstein & Spence, 2009).

In collaboration with other experts, designers determine the problem space and the design brief that defines the design variables. The combinations of multimodal, aesthetic, and manufacturing variables can be infinite, and how they work together determines the experiential value of the design solution. Given this daunting task, having an understanding of multimodal considerations could contribute to innovation, flexibility, and acceptance, and avoid a fixation on the visual modality — still dominant in design today (Schifferstein & Spence, 2006). In this chapter you will learn:

• How we process multisensory experiences from a perspective that is relevant to design.
• How multisensory layers can provide opportunities for design features that add value to product interactions.
• How sensory dominance affects the interactions people have with products at different phases of use.
• Why it is important to design features that contribute to a feeling of delight in product interactions.
• How multisensory design principles can support a variety of use scenarios that lead to positive outcomes.
• Unique applications for multisensory experiences that enable us to achieve tasks, given different sensory abilities and perceptions.
• How interpreting a synaesthetic approach can lead to richer design interactions.
• Awareness of the multisensory aesthetic properties of kinetic and collapsible design that add depth to our product experiences.