Surface as Interface: Converting Everyday Objects into Percussive Sound Systems
On going Research
This ongoing research explores how everyday objects—such as chairs, desks, and furniture—can be transformed into expressive musical interfaces through embedded sensing and embodied interaction. By attaching vibration-based sensors to non-instrumental surfaces, physical gestures like tapping, knocking, and rubbing are captured and mapped to percussive sound output in real time.
The project investigates how users intuitively interact with familiar objects when they are recontextualized as sound-producing interfaces. Rather than relying on traditional controllers or visible pads, the surface itself becomes the input, allowing sound to emerge directly from physical material and gesture.
This research sits at the intersection of Human–Computer Interaction, tangible interfaces, and sound-based media systems, with a focus on calibration, gesture sensitivity, material response, and perceptual feedback. The system remains intentionally open-ended, serving as a platform for experimentation rather than a finalized instrument.
Tech Stack / Sensors
Hardware
• Piezo vibration sensors (contact-based surface sensing)
• Microcontroller (Arduino / Teensy / Raspberry Pi Pico)
• Audio interface (signal routing and amplification)
• External speakers / amplified sound system
• Piezo vibration sensors (contact-based surface sensing)
• Microcontroller (Arduino / Teensy / Raspberry Pi Pico)
• Audio interface (signal routing and amplification)
• External speakers / amplified sound system
Software
• MIDI / OSC signal mapping
• Max/MSP / TouchDesigner / Ableton Live (sound synthesis & interaction mapping)
• Custom calibration logic for gesture sensitivity and velocity
• MIDI / OSC signal mapping
• Max/MSP / TouchDesigner / Ableton Live (sound synthesis & interaction mapping)
• Custom calibration logic for gesture sensitivity and velocity
Interaction Model
• Surface vibration → sensor input
• Gesture intensity & surface location → mapped percussive parameters
• Real-time audio feedback through speakers
• Surface vibration → sensor input
• Gesture intensity & surface location → mapped percussive parameters
• Real-time audio feedback through speakers
Figure 1: System overview illustrating how everyday hand gestures on a surface are captured using a piezoelectric sensor, processed through a microcontroller, and rendered as responsive percussive sound through a portable speaker.
Instruvis: Play Music Virtually and Visualize the Data
A gesture-based wearable system for music performance and real-time data visualization
Published in Springer’s Human–Computer Interaction International (HCII 2018) Conference Proceedings.
Overview
Instruvis is an interactive system that allows musicians to perform and produce music virtually using wearable technology. By combining micro-inertial and magnetic sensors with machine learning, gestures are trained and recognized as musical notes through MIDI inside a digital audio workstation. The platform enables musicians to create anywhere—without traditional instruments—transforming movement into sound and sound into visual form.
The project explores how human gestures, intelligent systems, and visual representation can create immersive musical experiences, bridging creativity with computation. Published at Springer’s HCII 2018 conference, Instruvis demonstrates how wearable technologies can redefine music performance and interaction design through real-time feedback and sensory augmentation.
Design & Technology
Hardware
• Intel Curie compute module
• Accelerometer, gyroscope, motion sensors
• Bluetooth wireless communication
• Intel Curie compute module
• Accelerometer, gyroscope, motion sensors
• Bluetooth wireless communication
Software
• Machine learning–based gesture recognition
• Arduino-based signal processing
• MIDI mapping to digital instruments
• Real-time color visualization engine
• Machine learning–based gesture recognition
• Arduino-based signal processing
• MIDI mapping to digital instruments
• Real-time color visualization engine
Interaction
• Gestures mapped to musical notes
• Performance feedback through dynamic visuals
• Portable and wireless design for mobile creativity
• Gestures mapped to musical notes
• Performance feedback through dynamic visuals
• Portable and wireless design for mobile creativity
Intel Curie Chip
Visualization used with Instruvis translating MIDI data to colors
Gesture Intelligence
Instruvis uses a neural network–driven pattern recognition system trained on movement data across the x, y, and z axes. Each gesture becomes a “class” assigned to a musical note. Because every performer moves differently, the system adapts by learning user-specific gesture profiles, enabling highly personalized music creation. This creates an interface that feels expressive, not mechanical—closer to a living instrument than a controller.
Visualization as Expression
Sound is translated visually using color, mapping amplitude, frequency, and pitch to visual parameters in real time. Visualization enhances emotional understanding and performance awareness, turning sound into motion and color. The result is a hybrid experience where data becomes artwork and music becomes architecture.
Evaluation & Impact
Testing with musicians and media artists revealed strong engagement with gesture-based creation and visual feedback. Instruvis supports:
• Experimental music performance
• Interactive installations
• Educational applications
• Multisensory storytelling
• Interactive installations
• Educational applications
• Multisensory storytelling
These findings informed refinements in gesture detection and visualization mapping for improved responsiveness and expressive control.
Reflection
This project shaped my ongoing interests in:
• Human–Computer Interaction
• Creative AI systems
• Wearable media
• Multimodal interaction
• Interactive sound environments
• Creative AI systems
• Wearable media
• Multimodal interaction
• Interactive sound environments
Instruvis represents my commitment to designing systems that empower creative freedom, reveal hidden data, and blend engineering with art.
Publication
Ismail, A. (2018). Instruvis: Play Music Virtually and Visualize the Data. HCII, Springer LNCS.