Spatial Alignment of Captured Inertial Measurement Unit Trajectory and 2D Video For Golf Swing Analysis

Abstract

A method for spatial alignment of golf club inertial measurement data and two-dimensional video for golf club swing analysis is provided. The method includes capturing inertial measurement data of a golf club swing through an inertial measurement unit (IMU), and sending the inertial measurement data from the inertial measurement unit to a computing device. The computing device is configured to align a first axis of the inertial measurement data to a first axis of further inertial measurement data from a mobile device, estimate translation of the inertial measurement data to two-dimensional video captured by the mobile device, align second and third axes of the inertial measurement data of the golf club swing to second and third axes of the two-dimensional video, and overlay a projected golf club trajectory based on the aligned captured inertial measurement data onto the two-dimensional video.

What is claimed is: 
     
         1 . A method for spatial alignment of golf club inertial measurement data and two-dimensional video for golf club swing analysis, comprising:
 capturing inertial measurement data of a golf club swing through an inertial measurement unit (IMU); and   sending the inertial measurement data of the golf club swing from the inertial measurement unit to a computing device, so that the computing device aligns a first axis of the inertial measurement data of the golf club swing to a first axis of further inertial measurement data from a mobile device, estimates translation of the inertial measurement data of the golf club swing to two-dimensional video of the golf club swing captured by the mobile device, aligns second and third axes of the inertial measurement data of the golf club swing to second and third axes of the two-dimensional video, and overlays a projected golf club trajectory based on the aligned captured inertial measurement data of the golf club swing onto the two-dimensional video.   
     
     
         2 . The method of  claim 1 , further comprising:
 attaching a marker to the IMU, so that the two-dimensional video captures the golf club swing and the marker, wherein alignment of the second and third axes of the inertial measurement data of the golf club swing to second and third axes of the two-dimensional video is based on detecting the marker in video frames.   
     
     
         3 . The method of  claim 1 , further comprising:
 activating a flashing light attached to the IMU, so that the two-dimensional video captures the golf club swing and the flashing light, wherein alignment of second and third axes of the inertial measurement data of the golf club swing to second and third axes of the two-dimensional video is based on detecting the flashing light in video frames.   
     
     
         4 . The method of  claim 1 , wherein:
 the first axis is a z axis, wherein the z axis is oriented according to gravity;   the second axis is an x axis, wherein the x axis is oriented according to a ball running direction; and   the third axis is a y axis, wherein the y axis is orthogonal to the x axis and the z axis.   
     
     
         5 . The method of  claim 1 , wherein:
 an origin of a coordinate system for the inertial measurement data of the golf club swing includes a location of a golf ball prior to or at moment of impact of a golf club head to the golf ball.   
     
     
         6 . The method of  claim 1 , wherein:
 the computing device solves pitch and roll angle by first axis alignment; and   the computing device solves yaw angle by second and third axis alignment.   
     
     
         7 . A method for spatial alignment of golf-club inertial measurement data and two-dimensional video for golf club swing analysis, performed by a computing device, comprising:
 receiving captured inertial measurement data of a golf club swing from an inertial measurement unit (IMU) attached to a golf club;   receiving, from a device having a video camera, a two-dimensional video of the golf club swing;   determining translation of the inertial measurement data of the golf club swing to the two-dimensional video of the golf club swing;   aligning first and second axes of the inertial measurement data of the golf club swing to first and second axes of the two-dimensional video; and   overlaying a projected golf club trajectory onto the two-dimensional video, based on alignment of the inertial measurement data of the golf club swing and the two-dimensional video.   
     
     
         8 . The method of  claim 7 , wherein the device having the video camera is a mobile device, and further comprising:
 receiving, from the mobile device, inertial measurement data of the mobile device in association with the two-dimensional video; and   aligning a third axis of the inertial measurement data of the golf club swing to a third axis of the inertial measurement data of the mobile device.   
     
     
         9 . The method of  claim 7 , wherein the device having the video camera is a mobile device, and further comprising:
 sending a video having the overlaid projected golf club trajectory on the two-dimensional video of the golf club swing, to the mobile device for display on the mobile device.   
     
     
         10 . The method of  claim 7 , further comprising:
 detecting a marker on a floor in the two-dimensional video; and   calculating rotation and translation from marker coordinate space to video camera coordinate space, wherein the determining translation is based on a measured distance between a golf ball and an origin of the marker or detecting the golf ball in a two-dimensional video frame and estimating a size of the golf ball.   
     
     
         11 . The method of  claim 7 , wherein the aligning first and second axes comprises maximizing a similarity between a re-projected captured golf club line based on the captured inertial measurement data of the golf club swing and a detected golf club line from a plurality of two-dimensional video frames. 
     
     
         12 . The method of  claim 7 , wherein the aligning first and second axes comprises minimizing an error between a re-projected captured golf club line based on the captured inertial measurement data of the golf club swing and a detected sensor location of the IMU from a plurality of two-dimensional video frames. 
     
     
         13 . The method of  claim 7 , wherein the receiving the captured inertial measurement data and the receiving the two-dimensional video are by wireless connection. 
     
     
         14 . A tangible, non-transitory, computer-readable media having instructions thereupon which, when executed by a processor, cause the processor to perform a method comprising:
 receiving, from an inertial measurement unit (IMU) attached to a golf club, inertial measurement data of a golf club swing;   receiving, from a mobile device, a two-dimensional video of the golf club swing in association with inertial measurement data relative to the mobile device;   aligning a first axis of the inertial measurement data of the golf club swing to a first axis of the inertial measurement data relative to the mobile device;   determining translation of the inertial measurement data of the golf club swing to the two-dimensional video of the golf club swing;   aligning second and third axes of the inertial measurement data of the golf club swing to second and third axes of the two-dimensional video; and   overlaying a projected golf club swing trajectory onto the two-dimensional video, based on alignment of the inertial measurement data of the golf club swing and the two-dimensional video.   
     
     
         15 . The computer-readable media of  claim 14 , wherein the method further comprises:
 reading gravity sensor readings from the mobile device, wherein the aligning the first axis of the inertial measurement data of the golf club swing to the first axis of the inertial measurement data relative to the mobile device comprises aligning gravity direction captured by an IMU of the mobile device and gravity direction captured by the IMU attached to the golf club.   
     
     
         16 . The computer-readable media of  claim 14 , wherein the method further comprises:
 displaying a video of the overlaid projected golf club swing trajectory on the two-dimensional video;   receiving, from user input, an identification of a location of the IMU attached to the golf club in a video frame;   re-aligning the second and third axes of the inertial measurement data of the golf club swing to the second and third axes of the two-dimensional video, based on the identification of the location in the video frame; and   overlaying a revised projected golf swing trajectory onto the two-dimensional video.   
     
     
         17 . The computer-readable media of  claim 14 , wherein the method further comprises:
 forming a background model, based on averaging video frames;   subtracting the background model from a plurality of video frames;   detecting edges in the plurality of video frames with the background model subtracted; and   selecting a longest line from among the edges, to detect a golf club in the plurality of video frames.   
     
     
         18 . The computer-readable media of  claim 14 , wherein the method further comprises:
 detecting a golf ball in frames in the two-dimensional video, based on searching for a circular pattern; and   eliminating one or more false positives, based on symmetry of intensity and gradient direction of illumination of the golf ball.   
     
     
         19 . The computer-readable media of  claim 14 , wherein the determining translation is based upon detecting a golf ball in one or more frames of the two-dimensional video and estimating a size of the golf ball in the one or more frames of the two-dimensional video. 
     
     
         20 . The computer-readable media of  claim 14 , wherein method further comprises:
 transforming the inertial measurement data of the golf club swing from a coordinate system of the IMU attached to the golf club to a coordinate system of a camera of the mobile device, based on the aligning the first axis, the translation, and the aligning the second and third axes, wherein the overlaying is based on the transforming.   
     
     
         21 . A tangible, non-transitory, computer-readable media having instructions thereupon which, when executed by a processor of a mobile device, cause the processor to perform a method comprising:
 receiving, from an inertial measurement unit (IMU) attached to a golf club, inertial measurement data of a golf club swing;   capturing, on the mobile device, a two-dimensional video of the golf club swing in association with inertial measurement data of the mobile device;   aligning z axis of the inertial measurement data of the golf club swing to z axis of the inertial measurement data of the mobile device;   determining translation of the inertial measurement data of the golf club swing to the two-dimensional video of the golf club swing;   aligning x and y axes of the inertial measurement data of the golf club swing to x and y axes of the two-dimensional video;   overlaying a projected golf club swing trajectory onto the two-dimensional video, based on alignment of the inertial measurement data of the golf club swing and the two-dimensional video; and   displaying, on the mobile device, the overlaid projected golf club swing trajectory on the two-dimensional video.