CV-Powered Personal Coach Platform for Amateur Athletes

Summary

• Millions of amateur athletes worldwide aim to improve their performance, while very few have access to world-class professional coaches and star athletes.
• Our client develops a platform that provides access to the best coaches in ballet, dancing, martial arts among other sport types.
• The platform enables aspiring sportsmen to learn different physical activities by using their own devices anytime anywhere. Through any phone or computer camera, the platform compares athlete's movements to the ones of a professional coach while providing live analytics and recommendations on how to improve the performance of each move.
• Our team used advanced Computer Vision technologies to build an end-to-end MVP that enables professional-to-student physical activity learning from any mobile device.

Tech Challenge

• Creation of a custom dataset for the model to enable the capturing of special movements, which are unseen by existing models trained on public datasets.
• Development of optimized architecture which achieves 100 FPS on average for 2D-pose estimation task on mobile devices in inference mode.
• Translation of 2D-body points to 3D by a separate model which is running on a server, also responsible for the temporal refinement.
• Triangulation and 3D reconstruction from multiple, ideally three, video sources.
• Comparison of digitized coach's video and data inference from a device.

Solution

• Coach's video digitization: processes video, translates 2D-body points to 3D-body and saves for future comparison.
• On-server comparison and analytics: compares digitalized teacher video and inferred data from a device and returns analytics.
• On-device inference: 2D-pose estimation for a student on a device.

Impact

• We've built a model that creates a dynamic 3D body of a teacher both from 1 source and 3 source video cameras using server GPUs.
• The model tracks a student athlete using any camera device (computer or mobile device), making all inference on a device and comparing only the results on server GPUs.