Name and Contact Information

  • name: Fanny Cacilie
  • website: GitLab, GitHub, Linkedin
  • gitlab username: @caciliefanny
  • timezone: UTC-03:00

Title

Motion Capture in WebXR: a search for the uniqueness in replaying live performances

Short description of work done

During my participation in the Google Summer of Code program, I executed a proposed project focused on crafting a prototype platform for generating XR experiences to allow users to relive live performances through motion capture technology.

My work revolved around two primary modules. Firstly, I tackled motion capture using MediaPipe Pose, an open-source tool for real-time pose detection. This module was integrated into a live streaming context. The second module was responsible for developing Augmented Reality (AR) sessions on the web using WebXR. To enable seamless integration between systems, I employed Web Sockets, comprising a server-side Web Socket, a client-side Web Socket, and a web application.

The Web Socket server established a socket server, capturing video frames from a camera through MediaPipe’s pose estimation. These frames were then transmitted to clients in a pickled format, with size information appended. On the client side, the Web Socket module received data packets, reconstructed video frames, and presented them using OpenCV’s Python package.

The web application was built using Three.js, forming a 3D scene. This application initialized scene elements, camera, renderer, lighting, and a 3D object. The scene was WebXR-compatible, offering Augmented Reality experiences. The animation loop ensured continuous updates and rendering of the 3D object, creating an interactive experience. Furthermore, the application established a WebSocket connection, logging received messages to the console.

These integrated components facilitated communication between servers and clients, allowing transmission and display of video frames. The aim was to create an augmented reality experience featuring a 3D object on a web page. However, direct manipulation of video frames within a WebXR module for AR sessions remained a work-in-progress, detailed in the “What’s left to do” section.

What code got merged

The project has been crafted within its own dedicated repository: Motion Capture in WebXR, containing the code that drives the application. The following topics represent the main contributions:

Description: Implement the WebXR server to create Augmented Reality session for a diverse array of devices.

Description: Design the proof of concept developed to detect and output landmarks of human bodies.

Description: Establish the Web Socket communication protocol to transmit real-time video frames of motion capture.

What code didn’t get merged

  • …

What’s left to do

Currently, a significant task remains on the agenda regarding the system’s functionality and integration. At this juncture, the real-time communication of video frames, specifically those detecting human poses through MediaPipe Pose, is solely feasible between Python-based clients and servers using the Web Socket approach in a testing module. However, a pivotal objective is to bridge this gap, enabling seamless communication and integration between the motion capture module which is a Python-based system and the JavaScript-based WebXR environment responsible for crafting AR experiences.

To accomplish this, the integration of both programming languages becomes imperative. This amalgamation would pave the way for more comprehensive and versatile interactions within the system. This challenge holds the key to advancing the system’s capabilities. It will serve as the foundation for a crucial future enhancement: the direct manipulation of video frames within the WebXR module during AR sessions.