Comment information generating apparatus and comment information generating method

Abstract

A comment information generating apparatus includes: a video obtaining unit which receives a video; a comment input receiving unit which receives position information of an object in the video and a comment that is displayed with the object and is caused to track the object starting from a predetermined timing; a target time determining unit which determines, to be a target value indicating a time length, a target time during which the comment is displayed, based on the comment; an object motion determining unit which determines, for the object, a motion trajectory having a time length substantially equal to the target time and along which the comment is caused to track the object; a comment information generating unit which generates comment information including the comment and the motion trajectory of the object; and an output unit which outputs the comment information.

1 . A comment information generating apparatus which (i) receives a video, position information of an object in the video, and a comment that is displayed with the object and is caused to track the object starting from a predetermined timing, (ii) generates a motion trajectory of the object along which the comment is displayed with the object and is caused to track the object, and (iii) outputs comment information including the comment and the motion trajectory of the object, the comment information generating apparatus comprising:
 a video obtaining unit configured to receive the video;   a comment input receiving unit configured to receive inputs of the position information of the object in the video received by the video obtaining unit and the comment that is displayed with the object and is caused to track the object starting from the predetermined timing; and   an object motion generating unit configured to generate the motion trajectory of the object based on a length of the comment received by the comment input receiving unit.   
     
     
         2 . The comment information generating apparatus according to  claim 1 , further comprising
 a target time determining unit configured to determine a target time during which the comment is displayed, based on the length of the comment received by the comment input receiving unit, the target time being a target value indicating a time length;   a comment information generating unit configured to generate comment information to be output including the comment and the motion trajectory of the object generated by the object motion generating unit; and   an output unit configured to output the comment information generated by the comment information generating unit.   
     
     
         3 . The comment information generating apparatus according to  claim 2 ,
 wherein the target time determining unit is configured to calculate the target time which lengthens as the comment received by the comment input receiving unit lengthens, and   the output unit is configured to output the motion trajectory having the time length which lengthens as the comment input to the comment input receiving unit lengthens.   
     
     
         4 . The comment information generating apparatus according to  claim 3 ,
 wherein the target time determining unit is configured to calculate, as the target time, a value obtained by multiplying a unit display time which is a predetermined display time per character and the number of characters in the comment received by the comment input receiving unit, and   the output unit is configured to output the motion trajectory having, as the time length, the value obtained by multiplying the unit display time and the number of characters, when the comment is input to the comment input receiving unit.   
     
     
         5 . The comment information generating apparatus according to  claim 3 ,
 wherein the target time determining unit is further configured to calculate, as the target time, a predetermined visual recognition time required to visually recognize the characters, when the calculated target time is shorter than the predetermined visual recognition time, and   the output unit is configured to output, as the motion trajectory, a motion trajectory which has a time length larger than or equal to the visual recognition time even if the comment is shorter than the visual recognition time, by outputting the motion trajectory which corresponds to a longer one of the visual recognition time and a value obtained by multiplying a unit display time which is a predetermined display time per character and the number of characters in the comment when the comment is input to the comment input receiving unit.   
     
     
         6 . The comment information generating apparatus according to  claim 2 ,
 wherein the output unit is configured to output a plurality of the motion trajectories which are different from each other for a plurality of the comments received by the comment input receiving unit, when the plurality of the comments have different numbers of characters and are provided at a same position in a same frame.   
     
     
         7 . The comment information generating apparatus according to  claim 2 ,
 wherein the object motion generating unit is configured to calculate a plurality of the motion trajectories for the object which is in the video and shown by the position information received by the comment input receiving unit, using a plurality of motion estimation methods or motion estimation parameters, and determine, for the object, the motion trajectory which is selected as having a time length closest to the target time from among the plurality of the motion trajectories calculated for the object.   
     
     
         8 . The comment information generating apparatus according to  claim 7 ,
 wherein the object motion generating unit is configured to calculate a plurality of the motion trajectories for the object which is in the video and shown by the position information received by the comment input receiving unit, using, as the plurality of motion estimation parameters, one kind of (i) allowance parameters having different values which affect trackability of the object, (ii) search window regions having different sizes, and (iii) feature values having different values, and determine, for the object, the motion trajectory which is selected as having a time length closest to the target time from among the plurality of the motion trajectories calculated for the object.   
     
     
         9 . The comment information generating apparatus according to  claim 7 ,
 wherein, in a case of occurrence of a “motion estimation impossible state” in which the motion trajectory closest to the target time cannot be determined for the object using the plurality of motion estimation methods or motion estimation parameters, the object motion generating unit is further configured to determine whether the “motion estimation impossible state” is caused by occlusion or a scene change, and switch between object motion determining methods based on a result of the determination.   
     
     
         10 . The comment information generating apparatus according to  claim 9 ,
 wherein when determining that the “motion estimation impossible state” is caused by the occlusion, the object motion generating unit is configured to determine the motion trajectory having the time length closest to the target time for the object by extrapolating a motion trajectory of the object from a frame at which the occlusion occurred to a following frame, based on a motion trajectory of the object to the frame at which the occlusion occurred.   
     
     
         11 . The comment information generating apparatus according to  claim 9 ,
 wherein when determining that the “motion estimation impossible state” is caused by the scene change, the object motion generating unit is configured to determine, for the object, a motion trajectory to a frame at which the scene change occurred as the motion trajectory to be output.   
     
     
         12 . The comment information generating apparatus according to  claim 9 ,
 wherein the object motion generating unit is configured to determine that the “motion estimation impossible state” is caused by the scene change, when a difference between luminance histograms of frames composing the video is larger than or equal to a predetermined threshold value, and that the “motion estimation impossible state” is caused by the occlusion, when a difference between the luminance histograms of the frames composing the video is smaller than the predetermined threshold value.   
     
     
         13 . The comment information generating apparatus according to  claim 7 ,
 wherein when a motion trajectory among the plurality of the motion trajectories of the object calculated using the plurality of motion estimation methods or motion estimation parameters is shorter than the time length corresponding to the target time by a predetermined time or more, the object motion generating unit is configured to determine, as the motion trajectory, a motion trajectory which is closest to the target time by connecting, to a forward end of a motion trajectory of the object, a motion trajectory which is estimated between (i) the frame having the position information and the input comment received by the comment input receiving unit and including a position which is shown by the position information received by the comment input receiving unit and (ii) a frame positioned backward on a time axis.   
     
     
         14 . The comment information generating apparatus according to  claim 7 ,
 wherein when a motion trajectory among the plurality of the motion trajectories of the object calculated using the plurality of motion estimation methods or motion estimation parameters is shorter than the time length corresponding to the target time by a predetermined time or more, the object motion generating unit is configured to generate a motion trajectory having a time length closest to the target time as the motion trajectory of the object shown by the position information received by the comment input receiving unit from among the plurality of the motion trajectories calculated for the object starting at positions within a predetermined distance range from positions shown by the position information of the object received by the comment input receiving unit.   
     
     
         15 . The comment information generating apparatus according to  claim 7 ,
 wherein when a motion trajectory among the plurality of the motion trajectories of the object calculated using the plurality of motion estimation methods or motion estimation parameters is shorter than the time length corresponding to the target time by a predetermined time or more, the object motion generating unit is configured to generate a motion trajectory having a time length closest to the target time as the motion trajectory of the object shown by the position information received by the comment input receiving unit from among the plurality of the motion trajectories calculated for the object based on positions within a user-specified range including positions shown by the position information of the object received by the comment input receiving unit.   
     
     
         16 . The comment information generating apparatus according to  claim 7 ,
 wherein when a motion trajectory among the plurality of the motion trajectories of the object calculated using the plurality of motion estimation methods or motion estimation parameters is shorter than the time length corresponding to the target time by a predetermined time or more, the object motion generating unit is configured to divide the object into regions, and generate a motion trajectory having a time length closest to the target time as the motion trajectory of the object, the generated motion trajectory ending in one of the regions.   
     
     
         17 . The comment information generating apparatus according to  claim 2 ,
 wherein the object motion generating unit is further configured to generate, as the motion trajectory, a motion trajectory having a time length closest to the target time, modify the generated motion trajectory for the object to be a virtual motion trajectory from a position at which a comment received by the comment input receiving unit is provided, based on a relative position relationship between the position at which the comment is provided and a center of gravity of the object shown by the position information received by the comment input receiving unit, and output the modified motion trajectory.   
     
     
         18 . The comment information generating apparatus according to  claim 2 , further comprising
 a target time adjusting unit configured to adjust the target time by lengthening the target time with an increase in a motion velocity of the object, based on the motion trajectory of the object generated by the object motion generating unit,   wherein the object motion generating unit is further configured to generate, for the object, a new motion trajectory having a time length corresponding to the target time adjusted by the target time adjusting unit, the new motion trajectory being for displaying the comment with the object shown by the position information and causing the comment to track the object.   
     
     
         19 . A comment information generating method comprising,
 receiving a video as an input;   receiving inputs of position information of an object in the video received in the receiving and a comment that is displayed with the object and is caused to track the object;   generating a motion trajectory of the object based on a length of the received comment;   generating comment information to be output including the comment and the motion trajectory of the object generated in the generating of the motion trajectory; and   outputting the comment information generated in the generating of the comment information.   
     
     
         20 . A non-transitory computer-readable recording medium storing thereon a program causing a computer to execute the comment information generating method according to  claim 19 .