Mobile movement coaching

Summary

An MVP for comparing student and coach movement.

The platform needed to recognise specialist movements on consumer devices and compare them with a coach’s demonstration.

Custom pose models for on-device 2D tracking, with server-side 3D reconstruction, temporal refinement and comparison against digitised coach movement.

Tech Challenge

• Creation of a custom dataset for the model to enable the capturing of special movements, which are unseen by existing models trained on public datasets.
• The on-device 2D pose model was designed for high frame rates on mobile hardware.
• Translation of 2D-body points to 3D by a separate model which is running on a server, also responsible for the temporal refinement.
• Triangulation and 3D reconstruction from multiple, ideally three, video sources.
• Comparison of digitized coach's video and data inference from a device.

Solution

• Coach's video digitization: processes video, translates 2D-body points to 3D-body and saves for future comparison.
• On-server comparison and analytics: compares digitalized teacher video and inferred data from a device and returns analytics.
• On-device inference: 2D-pose estimation for a student on a device.

Outcome

An end-to-end coaching MVP connecting camera input with movement comparison and feedback.