Echoes

Physical Space Architecture and Equipment
“Echoes” is a physical space with dimensions of approximately 4,5 meters curvilinear base, and 6 meters in height, constructed with eco-friendly materials. It is a parametrically designed structure made of interlocking 15mm wood pieces cut using CNC precision and coded for easy assembly. These interlocked pieces shape 40 frames of various shapes and dimensions that are mounted next to and on top of each other. The structure is designed for rapid and efficient assembly and disassembly, and with transportability in mind for deployment in various contexts. The frames are clad with two 8 mm-thick layers of cork that seamlessly mold to the complex curvatures of the frames. The cork layers also provide acoustical protection and contribute to the desired sound reverberation. The inner surfaces are dressed with cork fabric for tactile considerations and visual continuity.  The top part of the installation creates a protective cap that also serves as the carrier of all equipment, hidden from the visitor’s eye. The bottom surface of this cap is white to reflect the dynamic lights that are projected by a short-throw projector mounted on it. The projector projects dynamic light events on a 1.60 meters milky half-spherical plexiglass with a 160 cm diameter that, from the interior, is perceived as a dome. This surface is positioned 1.20m below the cap and approximately 4.50m above the ground.

Autistic individuals frequently seek smaller, sensory-friendly spaces to help them self-regulate. The installation’s curvilinear, nature-inspired geometry creates a smooth transition entry zone from a larger space to an intimate curvilinear interior. Its geometric composition, material, light, acoustics, and overall detailing follow evidence-based research to provide a personalized, sensory-friendly, and soothing experience. The installation is organized in three interconnected environments: In “outside: a collective experience,” participants engage with “Echoes” while they approach it from an exposed environment initially through its totem-like scale and playful geometry, and also with a series of light events that emerge from within and change slowly and dynamically in light intensity and coloration. This infrastructure acts somewhat as a faro/lighthouse that invites people closer. Participants then enter its intimate, interior space that functions as a “retreat” for an individual, dynamically adjusting the light and its sonic qualities based on their movement and breathing rhythms. Movements like rocking, vocalizing, or fidgeting are often misunderstood—but they’re essential forms of self-expression. Participants are encouraged to interact with a dynamic projection, “the self-reflecting moon,” that echoes their presence, encouraging self-awareness and playfulness, as a soothing process.

In terms of equipment, the installation space is equipped with three RealSense d435i cameras: one overhead for position tracking and two at slightly above eye level for detectors that need a full body view (skeleton tracking, face detection, etc.). A mini-PC is mounted on top of the structure, hidden from visitors, alongside a custom PCB designed as an Arduino shield for controlling other physical devices (like LED strips, lights, fans). The setup also includes the downward-facing short-throw projector, power supplies for all components, and the necessary cabling. Along the curvilinear base of the structure, there are LED strips to illuminate the space. Additional equipment supporting audio generation includes two FMCW mmWave radar sensors, positioned at an angle of approximately 105 degrees to each other, to ensure full coverage of the chamber area. The sensors are mounted at a height that aligns as closely as possible with the torso level across a range of human heights. The setup is completed with two speakers mounted on the top ring of the structure for minimal visual obstruction. All cameras and sensors that exist within the space, mounted on its walls, are encased for minimal visibility in 3d printed components designed with a geometric synergy to the overall structure and with materiality and texture that addresses tactile sensory needs. Together, these form the core of the room’s interactive setup in precise coordination with the physical aspects and geometric characteristics of the structure.
Diagrams, drawings, models, and videos for all the above can be found here: Miro

Application Architecture
The application is built using Python, React, and C (for the Arduino programming). The system is organized into three software units as Docker compositions for easy deployment and reproducibility during development. The separation of software units was chosen to provide the ability to deploy different parts of the application remotely to address the high computational demands of computer vision algorithms. During development, it was determined that the added latency of remote computation was unacceptable to the real-time demands of the experience, so all units were deployed locally on the same machine. The three software units of the system are:

– Management Unit: It includes the backend (FastAPI + SQLAlchemy) and frontend (React UI).
– Processing Unit: It receives input from different sources (camera, mmWave sensors) and distributes it to the appropriate detection modules depending on the source. These, in turn, forward their outputs to other modules that translate the detections to the interactive media or logic of the room (moon animation, audio, lights)
– Room Unit: Composition of processes that interface with the physical installation. They handle capturing inputs from cameras and mmWave sensors to forward to the processing unit using gRPC streaming. They also handle the outputs of the processing unit, which they forward to the projector, LED strips, and other interactive devices.

Sensing & Detection Modules 
Multiple Python-based detectors were implemented using custom or third-party tools. All detectors are working in a maximum throughput state in order to deal with the real-time nature of the application. For detectors that work based on the eye-level camera’s feed, they are always dual detectors, one for each eye-level camera, to increase coverage and accuracy. The detectors have a set of customizable settings that can be tweaked from the application UI.

– Skeleton Pose Tracking: Estimates human body posture by detecting and connecting key skeletal landmarks in real time. Implemented using the pose landmarking module of Google Mediapipe which uses a variant of the BlazePose model for 3D pose estimation. Pose Tracking Video
– Face Tracking: From the previous pose tracking solution, a custom method was implemented where, by using the position of different pose landmarks, it is possible to determine a bounding box for the position of the face on the image without performing face detection, which achieves high accuracy with minimal extra computation cost. The cropped face can then be provided to the modules that perform face detections.
– Face Landmark Tracking: Detects face landmarks (like mouth, eye, brow, nose etc.) and blendshapes that can be used to recognize facial expressions. Implemented using the face landmarking module of Google Mediapipe, which uses BlazeFace, a face mesh model, and a blendshape model for the whole detection pipeline.

–  Sentiment Analysis: A pre-trained open-source Hugging Face model was used for face sentiment analysis, after experimenting with a few to identify the one that produces the best results in the setting of the application.

– Position Tracking: For position tracking, a custom solution was implemented using the OpenCV library with an overhead depth camera. The adopted approach uses the depth frame from the camera above, applies a distance cutoff to remove the floor from the depth data, and then applies a mask to hide parts of the room (like walls) that are static in the frame (since the shape of the room is known). That way, when someone enters the room, they are the only tall enough object to appear in the depth data, and after applying the filtering methods, clustering can be performed to obtain the shape of the object and determine the position of the person in the room.

Position Tracking Picture (Video1, Video2)

– Arm Tracking: The arm tracking module calculates the orientation of the arms. It combines the pose estimation from both detectors for higher accuracy and translates them to a set global coordinate system. Arm Tracking Video
– Vital signs tracking: The Texas Instruments mmWave SDK 3.5 was used to flash the IWR6843ISK sensors for vital signs monitoring. Vital signs data is transmitted via a USB-to-UART bridge. A dedicated listener was developed to parse and process the relevant messages related to the application. Specifically, each mmWave sensor captures the user’s breathing rate along with their position with respect to the mmWave sensor’s antenna. The data received from both sensors are evaluated and post-processed, to assess user activity, distinguishing between stationary and moving states. Based on this assessment, it applies different methods to estimate the breathing rate, ensuring context-aware measurements.

Other Modules

– Presence Tracking: The intended experience is for participants to walk into the room and begin the experience automatically without taking any particular actions. To achieve this, a presence tracker was implemented that uses the detections from other modules to gauge the confidence that a person is in the room. The detections used are from the position tracker and two skeleton trackers (one for each eye-level camera) to increase robustness. A patience factor is also used to mitigate false positive/negative detections and avoid starting or ending sessions prematurely.
– Distributor Module: To deal with the computational demands of the application, every highly computational module was implemented as a separate process (Python multi-processing). Since large amounts of data (multiple image frames per second) are sent between different processes, a custom distributor was implemented that uses pipes for maximum throughput.

Interactive Media & Room Effects
Detector outputs drive interactive experiences in the installation space:
– Moon Animation: This module is in charge of rendering a “moon” animation and modeling its behavior. The rendering of the moon animation written in OpenGL. The logic of the behaviour is modeled using data from detector modules, like position and velocity, and it controls variables like rotation, phase, color, and brightness to achieve an interactive nature to participant actions in the room or the lack thereof. The behavior logic is implemented in Python. 
– Interactive Devices: A PCB was designed and printed as an Arduino shield to expand the limitations of an Arduino in controlling multiple devices, both in numbers and in power demands. The microcontroller is used as an interface between the application and the devices placed around the structure. It is capable of guiding up to 5 analog LED channels and switching on and off up to 56 devices (lights/fans). It is currently controlling a 2-channel warm white, cool white LED strip. The light/fan devices are not being used in the current phase of the project, but the software and the hardware developed are able to support them for a future application.

PCB on top of the installation

Audio production: This module dynamically transforms live data on breathing rate and presence, calculated during vital signs tracking, into an immersive three-layered sound environment. The audio evolves in real-time based on the fluctuations in the data, creating an immersive auditory experience. It generates and manipulates multiple sound layers—ambient ocean sounds, bird calls, and wave samples—where each layer responds to variations in the data. The module processes incoming data, adjusting sound playback parameters like volume, layering, and transitions dynamically. It records the resulting audio output in real time, creating an evolving sonic landscape that reflects participant physiological states and environmental interactions.

– Backend Infrastructure
A FastAPI + SQLAlchemy backend acts as the bridge between the processing unit, UI, and database, handling real-time session data saving, communicating configuration changes, and serving recorded session data.
– Frontend Session Management & Configuration UI
A browser-based user interface built with React provides session management and configuration tools. This enables operators to initiate, monitor sessions, download past session data, and change settings of the application. It also provides tools to calibrate the devices.
– Live Feed Visualization UI
A lightweight HTML interface paired with a FastAPI streaming server delivers real-time camera feeds from the detector modules. Initially designed for development, this component is also suitable for production monitoring.
– Data Handling
Anonymous data for each session is stored, enabling longitudinal study of user interaction while respecting privacy.

Availability: The application is available locally as it is as containerized services (via Docker) and can be accessed on a browser with a computer on site.

Echoes’ interior environment with self-reflecting moon and embodied ambient sound activated by the visitor’s movement and respiration. The space-body sonification stems from research on sounds comforting to autistic individuals.

Draft video “Echoes” overall experience. (Final version expected early October)

Static view of the exterior of Echoes with slightly changing light, echoing the presence of a person inside the installation