104 – Mulsemedia
Yevgeniya Sulema
Annotation

This course explores the integration of multiple sensory modalities into digital media systems to create immersive and interactive experiences. The course begins with an introduction to mulsemedia, outlining the concept of engaging multiple senses – sight, sound, touch, smell, and taste – in multimedia environments. Students will gain an in-depth understanding of mulsemedia hardware and techniques, including 3D visualization through 3D displays and 3D printing, and 3D image capturing techniques. The auditory aspects of mulsemedia are covered through surround sound recording and reproduction systems. The course presents ways of employing haptics that introduce the sense of touch into human-computer interaction. The course also delves into olfaction-based hardware for smell detection and smell reproduction, and gustatory technologies like taste detection, analysis, and taste simulation as well as food 3D printing. The course briefly covers mulsemedia standard MPEG-V and software that enable the integration of multisensory content in interactive systems. Finally, students will explore mulsemedia applications, ranging from entertainment, virtual reality, and gaming, to healthcare, education, and marketing. This course provides a comprehensive foundation in the design, development, and application of multisensory media technologies, preparing students to create immersive experiences that engage all human senses.

Objectives

Upon completing this course, students will acquire the following knowledge and competences.

Knowledge:

The core principles of multisensory media, including how visual, auditory, tactile, olfactory, and gustatory stimuli can be integrated into digital environments

3D visualization technologies, including 3D displaying methods and 3D printing for real-world object creation

3D image capturing, covering techniques such as depth sensing, stereoscopic cameras, and volumetric capture systems

The concepts of surround sound technologies for both recording and reproducing spatial audio experiences

Haptic technology for simulating touch and interaction in digital environments

Olfaction-based hardware for smell detection and smell reproduction learning how scent can be used in virtual and augmented environments

Gustatory-based systems for taste detection, analysis, and taste simulation, along with their applications in 3D food printing

MPEG-V standard and mulsemedia software tools

Applications of mulsemedia, including its use in entertainment, virtual reality, gaming, education, healthcare, marketing, and more.

Competences:

Multisensory experience design

Technical skills in mulsemedia hardware

Software development for mulsemedia

Critical analysis and application of mulsemedia

Keywords
Mulsemedia, 3D Visualization, Surround Sound, Haptics, Olfaction, Gustatory
Date of Creation
30.09.2024
Duration
30 hours
Language
English
License
ISBN
Literature
  1. Veltman Kim H. Notes for A New History of Media, Scales and Disciplines. Strategies for a Creative Future with Computer Science, Quality Design and Communicability, San Marino, 2014.
  1. Ghinea G., Ademoye O. A User Perspective of Olfaction-Enhanced Mulsemedia. Proceedings of the international conference on management of emergent digital ecosystems MEDES’10, 2010. 277–280.
  1. Ullah, A.; Liu, Y.; Wang, Y.; Gao, H.; Wang, H.; Zhang, J.; Li, G. E-Taste: Taste Sensations and Flavors Based on Tongue’s Electrical and Thermal Stimulation. Sensors 2022, 22, 4976.
  1. Joseph Nathaniel Kaye. Symbolic Olfactory Display. Massachusetts Institute of Technology 2001.
  1. Geng J. Three-dimensional display technologies. Advances in Optics and Photonics, 2013, 5(4):456–535. https://doi.org/10.1364/aop.5.000456
  1. Tal et al., "Mulsemedia in Telecommunication and Networking Education: A Novel Teaching Approach that Improves the Learning Process," in IEEE Communications Magazine, vol. 57, no. 11, pp. 60-66, November 2019, doi: 10.1109/MCOM.001.1900241
  1. A. Simiscuka, G. Fadda, V. Popescu, M. Murroni, and G. Muntean. " Immersive Live Concert: A Multi-Sensory Experience Based on Real-Time Lyrics Detection from Spatial Audio Data", in Proceedings of the ACM International Conference on Interactive Media Experiences Workshops, Niterói/RJ, 2025, pp. 7-11, doi: https://doi.org/10.5753/imxw.2025.1139
  1. GIMP – GNU Image Manipulation Program. https://www.gimp.org/
  1. What is ASCII (American Standard Code for Information Interchange)? https://www.techtarget.com/whatis/definition/ASCII-American-Standard-Code-for-Information-Interchange