Method for creating view-based representations from multimedia collections

Abstract

A system that is capable of generating a multiplicity of representations from a set of multimedia objects, each with a potentially different form, is disclosed. In addition, a system that is capable of generating representations that include thematic groupings in addition to the traditional purely sequential groupings is disclosed. Analysis of metadata to generate derived metadata creates a more comprehensive repository of information about media assets. Using the abundance of metadata, embodiments of the methods disclosed teach uniquely grouping and prioritizing media assets into a storyboard, which is in turn mapped onto view-based representations based on a selected output modality.

1 . A method of creating one or more view-based representations from a set of multimedia objects comprising:
 using a processor to analyze the multimedia objects and corresponding recorded metadata to generate derived metadata;   applying a selected method to analyze recorded and derived metadata;   ordering and grouping the set of multimedia objects according to said analysis;   prioritizing the set of multimedia objects according to said analysis;   selecting a specific output modality; and   generating a view-based representation of the set of multimedia objects for the selected modality.   
     
     
         2 . The method of  claim 1  wherein the ordering, grouping, and prioritization is persisted independently of the view modality. 
     
     
         3 . The method of  claim 1 , further comprising saving the ordering, grouping, and prioritization data independently of the elected output modality. 
     
     
         4 . The method of  claim 1  wherein the view-based representation further comprises computing an emphasis score for members of the set of multimedia objects. 
     
     
         5 . The method of  claim 1  wherein the view based representation of the set of multimedia objects for the selected modality consists of a subset of the multimedia objects as determined by the priority. 
     
     
         6 . The method of  claim 1  wherein the grouping consists of one or more of the following techniques: chronology, frequent itemset mining, face detection, face recognition, location clustering, object detection, object recognition, event detection, and event recognition. 
     
     
         7 . The method of  claim 1  wherein the ordering consists of a hierarchical arrangement of the multimedia objects from the groups. 
     
     
         8 . The method of  claim 1  wherein all the elements of select groups of the grouped set of multimedia objects are determined to be semantically equivalent. 
     
     
         9 . The method of  claim 7  wherein the semantic equivalence is determined by one or more of the following classes of information: temporal, location, people detection and recognition, and visual similarity. 
     
     
         10 . The method of  claim 8  wherein visual similarity is determined by comparing block-based color histograms. 
     
     
         11 . The method of  claim 6  wherein the set of multimedia objects of the hierarchical arrangement are arranged in sequential order or indeterminate order. 
     
     
         12 . The method of  claim 7  wherein the set of multimedia objects or contains one or more representative members of the set are selected to represent the set. 
     
     
         13 . The method of  claim 1 , wherein the views in the view based representation of the set of multimedia objects for the selected modality are determined by the ordering and grouping. 
     
     
         14 . The method of  claim 2  wherein a graphic user interface is provided that incorporates a visual representation of the ordering and grouping of the set of multimedia objects according to said analysis. 
     
     
         15 . The method of  claim 13  wherein the graphic user interface provides a description or indication of size for the groupings. 
     
     
         16 . The method of  claim 4  wherein the selected modality includes a multimedia presentation, an interactive multimedia presentation, a photobook, a printed collage, a virtual collage, or a digital slide show. 
     
     
         17 . The method of  claim 1  wherein the grouping method consists of identifying a plurality of thematic sets for each group, wherein each of the thematic sets is determined from the derived metadata and where each thematic set contains a subset of the digital images. 
     
     
         18 . The method of  claim 16  wherein for each group, a subset of the corresponding thematic sets is selected by evaluating each of the identified thematic sets responsive to one or more of the following: the probability of occurrence, the size of the thematic set, the user preferences, or other derived metadata. 
     
     
         19 . The method of  claim 1  wherein the derived metadata includes one or more of the following: the frequency with which an asset has shared with others, the number of people with which an asset has been shared, or social media feedback such as the number of “likes,” comments, star ratings an asset has received. 
     
     
         20 . A method of generating one or more view-based representations of an output modality for multimedia assets comprising:
 a processor analyzing recorded metadata and user-provided metadata associated with multimedia assets of a multimedia collection;   the processor generating derived metadata based on the recorded metadata and the user-provided metadata;   the processor organizing the media assets in one or more groupings;   the processor prioritizing the media assets in each grouping;   the processor prioritizing the one or more groupings;   the processor generating a storyboard;   mapping the storyboard as one or more view-based representations according to a selected output modality; and   rendering on a display the one or more view-based representations.   
     
     
         21 . The method of  claim 20 , wherein recorded metadata comprises metadata automatically recorded by an image capture device upon capturing a media asset. 
     
     
         22 . The method of  claim 20 , wherein user-provided metadata comprises metadata provided by a user via an interface located on an image capture device or via an image editing application. 
     
     
         23 . The method of  claim 20 , wherein the step of generating derived metadata comprises applying one or more of the techniques chosen from the group consisting of temporal event clustering, geographic naming, scene classification, materials class extraction, low-level feature extraction, content-based image retrieval, face detection, face recognition, facial clustering, semantic event clustering, image value indexing, and video key frame extraction. 
     
     
         24 . The method of  claim 20 , wherein the storyboard is a hierarchical grouped set of the multimedia assets. 
     
     
         25 . The method of  claim 20 , wherein user-provided metadata comprises indication of preference recorded by users of a social networking application or social networking website. 
     
     
         26 . The method of  claim 20 , wherein the processor groups the media assets strictly according to a chronological paradigm. 
     
     
         27 . The method of  claim 20 , wherein the processor groups the media assets according to a thematic paradigm. 
     
     
         28 . The method of  claim 20 , wherein the processor groups the media assets according to a hybrid chronological-thematic paradigm. 
     
     
         29 . The method of  claim 20 , wherein the groupings comprise sequential segments and parallel segments. 
     
     
         30 . The method of  claim 20  wherein the processor groups events according to a chronological paradigm and groups sub-events according to a thematic paradigm. 
     
     
         31 . The method of  claim 20 , wherein the output modality is selected from the group consisting of hardcopy print media and softcopy digital presentation. 
     
     
         32 . The method of  claim 20 , wherein prioritizing the media assets in each grouping is performed as a function of the recorded metadata, the user-provided metadata, and the derived metadata. 
     
     
         33 . A method of grouping and prioritizing media assets in a multimedia collection comprising:
 apply a frequent itemset mining method to compute one or more sets of thematic groupings of the media assets, wherein the thematic groupings are constrained by one or more feature descriptors belonging to one or more feature categories; and   computing an asset priority score for every asset in each of the one or more sets of thematic groupings, wherein the asset priority score is a function of the one or more feature descriptors.