Had a blast setting up and presenting The Living Room, an interactive installation specially created for London Design Week.

Utilizing TouchDesigner alongside computer vision models, I implemented pose detection to track participants' movement and position in real-time. The installation dynamically responds, prompting visitors to engage with their surroundings in a way that balances activity and rest.

One of my main interests lately is the great power Computer Vision models have to enable real-time spatial interactions in physical venues. Since the release of Kinect-like cameras, artists and designers have used pose tracking models for interactivity.

But with the recent wave of AI models, hardware requirements to carry out these operations have practically disappeared.

Today, a simple home monitoring camera is enough to carry out complex scene understanding operations such as pose, face and gesture detection, or time-consistent video segmentation.

MediaPipe is an open-source framework created by Google, meant for building pipelines to perform computer vision inference over arbitrary sensory data such as video or audio. By using a GPU Accelerated MediaPipe Plugin for TouchDesigner I was able to perform Body Tracking in a live video stream inside Touchdesigner.

By using two home monitoring cameras, I was able to transmit a video feed via wifi by using the Real-Time Streaming Protocol

To perform a location estimation, I divided the floor in a square grid. In order to determine the location of visitors, I took the right ankle position of the pose tracking coming out of MediaPipe. This comes out as a 2d coordinate in screen space.

Considering camera placement and frustum, I carried out a digital raycast in the direction of that UV coordinate. I then checked where this ray intersected a virtual replica of the floor grid.

I took common game development approach to calculating what I called 'Room Health'. Each grid cell had a health measure, which would get updated on each tick (game loop update). According to the number of people inside the cell and inside neighbouring cells, the health value would go up or down, capped between 0 and 100. This was reflected by a visual representation on the wall.

This installation was a part of the museums Digital Design Weekend programme, called 'A World of Many Worlds': A unique showcase of hybrid digital-analogue artworks, exploring the multiplicity of worlds that make up the one we see every day

You can check the full programme here