SynesthesiARP: Real-Time Visual Sonification Synthesizer
Embedded Systems | Audio Signal Processing | Computer Vision
SynesthesiARP is a self-contained, real-time synthesizer that transforms live video into expressive soundscapes, running entirely on a custom Linux kernel build for the BeagleBone Black. It captures visual features—brightness, warmth, texture, and object count—and maps them to audio parameters such as waveform, pitch, sequence variation, and reverb depth.
I led the software development for vision feature extraction, audio synthesis, and threading architecture. I also supported kernel optimization and Buildroot configuration to ensure low-latency, reliable deployment on embedded Linux.
Software Architecture & Contributions
Real-Time Dual-Thread Design: Developed vision and audio threads in C++ with lock-free atomic communication and sub-20ms latency.
Feature Extraction (OpenCV):
• Brightness via grayscale mean
• Warmth using channel-weighted RGB ratio
• Texture from Canny edge density
• Object count using adaptive thresholding + morphological filtering
Audio Synthesis (ALSA):
• Custom arpeggiator with pitch, waveform, and rhythm modulation
• Lookup table oscillators, envelope shaping, and reverb feedback loop
• Low-level ALSA output for stable timing under high CPU load
System Integration:
• Preallocated all audio buffers and delay lines to avoid dynamic allocation
• Combined chord sequencing and audio generation into one loop to reduce mutex contention
• Performance profiled using custom metrics to verify responsiveness
Platform Optimization:
Contributed to trimming Linux kernel and Buildroot configuration (USB, ALSA, OpenCV only)
Helped resolve cross-compilation conflicts and OpenCV version mismatches under WSL
Supported bootable SD-card build with minimal driver set (~80MB image)