Audio/visual editing tool

Abstract

Analyzing data sequences, such as video and audio segments, and extracting metadata from the data sequences for providing information related to events of the data sequences. The metadata extracted is used for making editing decisions, such as for an audio-visual production comprised of a collection of video and audio segments. The metadata identifies events of importance in the video and audio segments, which may then be applied to the editing process. A user interface including the extracted metadata in a timeline format facilitates editing the audio-visual production.

1 . A user interface for integrating visual data and audio data together for creating an audio-visual work, said user interface comprising:
 a visual timeline associated with a visual data sequence comprising one or more visual elements, said visual timeline depicting the beginning and end of each of said one or more visual elements, said visual timeline further including visual event markers indicating the corresponding time on the timeline of a particular visual event in the visual data sequence; and   an audio timeline associated with an audio data sequence comprising one or more audio elements, said audio timeline corresponding to the same time scale and position in time as the visual timeline, said audio timeline depicting at least one audio characteristic of the audio data sequence over time, said audio timeline further including audio event markers indicating the corresponding time on the timeline of a particular audio event in the audio data sequence;   said visual event markers and audio event markers providing visual indications regarding events within the respective visual data sequence and audio data sequence.   
     
     
         2 . A user interface as set forth in  claim 1  further comprising an overlay audio timeline associated with an overlay audio data sequence comprising one or more overlay audio elements, said overlay audio timeline corresponding to the same time scale and position in time as the visual timeline and the audio timeline, said overlay audio timeline depicting at least one audio characteristic of the overlay audio data sequence over time, said overlay audio timeline further including overlay audio event markers indicating the corresponding time on the timeline of a particular overlay audio event in the overlay audio data sequence, said overlay audio event markers providing visual indications regarding events within the overlay audio data sequence. 
     
     
         3 . A user interface as set forth in  claim 2  wherein each of said visual event markers, audio event markers, and overlay audio event markers further comprises at least one of an icon indicative of the nature of the corresponding event and a value indicator indicative of the relative value of the corresponding event. 
     
     
         4 . A user interface as set forth in  claim 2  wherein a value indicator of at least one of said visual event markers, audio event markers, and overlay audio event markers indicates increased value when two or more of said event markers correspond to a substantially similar time on their respective timelines. 
     
     
         5 . A user interface as set forth in  claim 2  wherein at least one of said visual event markers, audio event markers, and overlay audio event markers corresponds to a selection element for selection by a user, said selection element adapted for selection of the event by a user and movement of the event to another time location on the respective timeline by the user, thereby also moving at least a portion of the visual data sequence or the audio data sequence, or the overlay audio data sequence associated with the selection element with the respective event marker. 
     
     
         6 . A user interface as set forth in  claim 5  wherein at least one of said visual event markers and audio event markers can be selected and snapped to the location of another of said visual event markers and audio event markers located on another of said timelines 
     
     
         7 . A user interface as set forth in  claim 1  wherein at least one of said visual event markers and audio event markers further comprises at least one of an icon indicative of the nature of the corresponding event and a value indicator indicative of the relative value of the corresponding event. 
     
     
         8 . A user interface as set forth in  claim 1  further comprising at least one of a selection element for selecting the inclusion or exclusion of event markers indicating a particular type of event and a selection element for increasing or decreasing the granularity of the placement of the event markers. 
     
     
         9 . A user interface as set forth in  claim 1  wherein the visual data sequence comprises at least one of video segments and still digital images said user interface further comprising a selection element for automatically adjusting the timeline position of at least one of said digital images including the image of a particular item to appear substantially adjacent other video segments or digital images also including the image of the same item. 
     
     
         10 . A user interface as set forth in  claim 1  further comprising an aggregate timeline depicting the visual data sequence comprising at least one of textual messages, video segments, images, effects, and transitions and the audio data sequence comprising at least one of speech, music, and recorded sounds. 
     
     
         11 . A user interface as set forth in  claim 1  further comprising a selection element for selecting a time along each timeline, such that selection of the same time on each timeline increases the granularity of event markers within at least one of the visual elements and the audio elements occurring at said selected time. 
     
     
         12 . A method for generating a user interface for an audio-visual work, said method comprising:
 extracting metadata associated with a visual data sequence according to at least one characteristic of said visual data sequence;   extracting metadata associated with an audio data sequence according to at least one characteristic of said audio data sequence;   determining intersections of metadata from two or more of said extracted metadata occurring at substantially the same time;   associating metadata and metadata intersections with a visual timeline associated with the visual data sequence;   associating metadata and metadata intersections with an audio timeline associated with the audio data sequence; and   integrating the visual timeline and the audio timelines onto a user interface.   
     
     
         13 . The method of  claim 12 , further comprising:
 displaying an overlay audio timeline associated with an overlay audio data sequence and comprising one or more overlay audio elements, said overlay audio timeline corresponding to the same time scale and position in time as the visual timeline and the audio timeline;   depicting, via the displayed overlay audio timeline, at least one audio characteristic of the overlay audio data sequence over time; and   displaying overlay audio event markers in the overlay audio timeline, said overlay audio markers indicating the corresponding time on the overlay audio timeline of a particular overlay audio event in the overlay audio data sequence, wherein the overlay audio markers provide visual indications regarding events within the overlay audio data sequence,   wherein the overlay audio data sequence further comprises one or more overlay audio elements.   
     
     
         14 . The method of  claim 13 , further comprising providing at least one of an icon indicative of the nature of the corresponding event and a value indicator indicative of the relative value of the corresponding event for each of the following: the overlay audio event markers, one or more visual event markers indicating the corresponding time on the visual timeline of a particular visual event in the visual data sequence, and one or more audio event markers indicating the corresponding time on the audio timeline of a particular audio event in the audio data sequence. 
     
     
         15 . The method of  claim 14 , further comprising:
 increasing the value of a value indicator of each of the visual event markers, the audio event markers, and the overlay audio event markers when two or more of said event markers correspond to a substantially similar time on their respective timelines; and   indicating said increase in value via the user interface.   
     
     
         16 . The method of  claim 14 , further comprising:
 providing a selection element for selection by a user, wherein at least one of the visual event markers, audio event markers, and overlay audio event markers corresponds to said selection element;   receiving, via said selection element, a selection of an event by the user and movement of the event to another time location on the respective timeline by the user; and   moving at least a portion of the visual data sequence or the audio data sequence, or the overlay audio data sequence associated with the selection element with the respective event marker.   
     
     
         17 . The method of  claim 16 , further comprising:
 receiving a selection of at least one of the visual event markers and audio event markers; and   snapping to the location of another of the visual event markers and audio event markers located on another of the timelines   
     
     
         18 . The method of  claim 12  further comprising at least one of a selection element for selecting the inclusion or exclusion of event markers indicating a particular type of event and a selection element for increasing or decreasing the granularity of the placement of the event markers. 
     
     
         19 . The method of  claim 12  wherein the visual data sequence comprises at least one of textual messages, video segments, images, effects, and transitions and the audio data sequence comprises at least one of speech, music, and recorded sounds,
 wherein when the visual data sequence comprises still digital images, the method further comprises providing a selection element for automatically adjusting the timeline position of at least one of the digital images including the image of a particular item to appear substantially adjacent other video segments or digital images also including an image of the same item. 
 
     
     
         20 . A system for generating a user interface for creating an audio-visual work, said system comprising:
 a metadata extraction module for extracting metadata associated with a visual data sequence according to at least one characteristic of said visual data sequence and for extracting metadata associated with an audio data sequence according to at least one characteristic of said audio data sequence, wherein an intersection of metadata from two or more of said extracted metadata occurs at substantially the same time and wherein each metadata and the intersection of metadata is associated with a timeline location corresponding to a timeline relating the data sequences; and   a user interface metadata association module for associating metadata and metadata intersections with a visual timeline associated with the visual data sequence and an audio timeline associated with the audio data sequence, said user interface metadata association module integrating the visual timeline and the audio timelines onto a user interface.