Name and Contact Information

Title

Face Tracking to Improve Accessibility and Interaction in the Metaverse with Satellite

Short description of work done

During Google Summer of Code, I extended Satellite, SAT’s fork of Mozilla Hubs, to support using Google MediaPipe as as an alternative “controller” so that users can use their face, as well as opening and closing their eyes and mouth, to perform actions within the 3D environment. This required diving into the Hubs codebase, as well as learning and understanding how to effectively use MediaPipe, both of which I had no prior experience with.

In addition, I also developed a simple user input rebinding system within Hubs specific to the MediaPipe controller to facilitate users rebinding facial gestures to any Hubs action they please - something not present in the Hubs codebase. This was the most “difficult” part of the project, and is elaborated further in the “What’s left to do” section.

Work not directly related to integrating Hubs and MediaPipe, but still important to our ultimate goal of accessibility, was also accomplished. Specifically, this includes a dedicated Satellite category within Hubs’ preferences, which allows users to rebind actions associated with facial gestures (as mentioned above) as well as adjust sensitivity values and enable/disable the display of the MediaPipe webcam panel and associated debugging tools (e.g. the use of emojis to draw the location of the users’ eyes, and whether they are open or closed).

What code got merged

The bulk of the work related to my GSoC project can be found in the second merge request listed below:

What code didn’t get merged

In addition to the code I wrote in my Satellite fork which got merged, I also wrote other code. The first of which being exploratory MediaPipe code, to be found here:

I also wrote a Dockerfile and published it on my personal GitHub account to address issue #5471 in the Mozilla Hubs repository, so that other users with the same bug could find relief. The repository is located here:

What’s left to do

Although I consider the project to be a great success that I hope will serve as a foundation for further developers (or GSoC contributors!), there are a couple of key areas which ought to be addressed before this project can become useful to the general public as an accessibility tool/alternative means of input within Hubs:

Improve Face Tracking Accuracy and Consistency

MediaPipe FaceMesh provides the developer with a mesh of face landmarks to use as they please. The approach I settled on to “translate” this mesh into actions within Hubs is straightforward and easy for new developers to understand, but lacks the accuracy and reliability to be useful to end users. In particular, it is quite error prone when the orientation of the user’s webcam is not perfect; for example, when the user turns their face to move the camera within Hubs, there are opportunities for the eyes to be occluded, which can accidentally trigger eye-based actions, leading to user frustration. A more robust implementation of this logic, located in mediapipe-controller.js, is necessary.

Improve and/or Design a System for Dynamic User Input Reassignment in Hubs

There is no API within Hubs for developers of custom controllers to easily “plug into.” Since the focus of this project was on accessibility improvements, I wanted a way for users to rebind facial gestures (such as opening and closing the mouth) to other actions within the 3D environment. Therefore, I had to essentially jerry-rig my own input rebinding subsystem to facilitate this.

Unfortunately, this has led to quite a few odd bugs, such as Mozilla Hubs (or AFRAME?) freezing the browser if two face-based gestures are bound/rebound to the same action. This is obviously a severe limitation and speaks to the necessity of refactoring Mozilla Hubs’ user input system to enable this out of the box, an endeavor that requires a more thorough understanding of the Hubs codebase than I possess.

Miscellaneous Quality of Life Improvements

There are a variety of QoL improvements that I think are important but which I did not have time to complete - and should serve as low-hanging fruit to introduce a new developer to the project. For example:

  • Enable/disable use of the MediaPipe controller via an HTML query parameter, similar to @paperManu’s original work integrating LivePose with Satellite/Hubs
  • Add preferences option to change speed of camera movement in Hubs when using the MediaPipe controller (at present, it moves at a fixed and unchangeable rate)
  • Support additional actions, such as those associated with the Pen in Hubs
  • Allow user to resize the MediaPipe debug panel (at present, its resolution is fixed)