Visual feature tagging in multi-view interactive digital media representations

Abstract

Provided are mechanisms and processes for visual feature tagging in multi-view interactive digital media representations (MIDMRs). In one example, a process includes receiving a visual feature tagging request that includes an MIDMR of an object to be searched, where the MIDMR includes spatial information, scale information, and different viewpoint images of the object. A visual feature in the MIDMR is identified, and visual feature correspondence information is created that links information identifying the visual feature with locations in the viewpoint images. At least one image associated with the MIDMR is transmitted in response to the feature tagging request.

1 . A method comprising;
 receiving via a communications interface at a server a visual feature identification request for a first multi-view interactive digital media representation (MIDMR) of an object, the first MIDMR of the object including spatial information;   identifying via a processor a visual feature in the first MIDMR of the object based at least in part on the spatial information, wherein identifying the visual feature comprises comparing the first MIDMR with a plurality of reference MIDMRs, wherein comparing the first MIDMR with the plurality of reference MIDMRs comprises comparing the scale information of ate first MIDMR with scale information of the plurality of reference MIDMRs; and   transmitting from the server via the communications interface a feature identification message associated with the first MIDMR in response to the feature identification request.   
     
     
         2 . The method recited in  claim 1 , wherein identifying the visual feature comprises processing user input that identifies a first location on the first MIDMR. 
     
     
         3 . The method recited in  claim 1 , wherein the spatial information is determined at least in part based on inertial data. 
     
     
         4 . The method recited in  claim 1 , wherein identifying the visual feature further comprises:
 selecting a reference MIDMR that is similar to the first MIDMR,   identifying a reference visual feature associated with the reference MIDMR, and   locating the reference visual feature in the first MIDMR.   
     
     
         5 . The method recited in  claim 1 , wherein identifying the visual feature comprises:
 determining an object type associated with the object,   identifying a predefined visual feature associated with the object type, and   locating the predefined visual feature in the first MIDMR.   
     
     
         6 . The method recited in  claim 5 , the method further comprising:
 identifying a visual feature in a second MIDMR of the object, wherein the first MIDMR of the object represents the object at a first point in time and wherein the second MIDMR of the object represents the object at a second point in time.   
     
     
         7 . The method recited in  claim 6 , the method further comprising:
 comparing the visual feature in the first MIDMR of the object to the visual feature in the second MIDMR of the object to identify a change in the object between the first time and the second time.   
     
     
         8 . The method recited in  claim 7 , wherein the object is a vehicle and wherein the change in the object represents damage to the object. 
     
     
         9 . The method recited in  claim 1 , wherein the spatial information comprises depth information. 
     
     
         10 . The method recited in  claim 1 , wherein the first MIDMR includes a plurality of different viewpoint images of the object. 
     
     
         11 . The method recited in  claim 1 , wherein the spatial information comprises three-dimensional location information. 
     
     
         12 . The method recited in  claim 1 , wherein the visual feature represents a physical location on the object. 
     
     
         13 . The method recited in  claim 1 , wherein the spatial information comprises visual flow between the different viewpoint images. 
     
     
         14 . The method recited in  claim 1 , wherein the MIDMR of the object further comprises three-dimensional shape information. 
     
     
         15 . A system comprising:
 a processor; and   memory configured to store instructions, the instructions configured to cause the processor to:   receive via a communications interface at a server a visual feature identification request for a first multi-view interactive digital media representation (MIDMR) of an object, the first MIDMR of the object including spatial information;   identify via a processor a visual feature in the first MIDMR of the object based at least in part on the spatial information, wherein identifying the visual feature comprises comparing the first MIDMR with a plurality of reference MIDMRs, wherein comparing the first MIDMR with the plurality of reference MIDMRs comprises comparing the scale information of the first MIDMR with scale information of the plurality of reference MIDMRs; and   transmit from the server via the communications interface a feature identification message associated with the first MIDMR in response to the feature identification request.   
     
     
         16 . The system recited in  claim 15 , wherein identifying the visual feature comprises:
 selecting a reference MIDMR that is similar to the first MIDMR,   identifying a reference visual feature associated with the reference MIDMR, and   locating the reference visual feature in the first MIDMR.   
     
     
         17 . One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:
 receiving via a communications interface at a server a visual feature identification request for a first multi-view interactive digital media representation (MIDMR) of an object, the first MIDMR of the object including spatial information;   identifying via a processor a visual feature in the first MIDMR of the object based at least in part on the spatial information, wherein the visual feature represents a physical location on the object, wherein identifying the visual feature comprises comparing the first MIDMR with a plurality of reference MIDMRs, wherein comparing the first MIDMR with the plurality of reference MIDMRs comprises comparing the scale information of the first MIDMR with scale information of the plurality oil reference MIDMRs; and   transmitting from the server via the communications interface a feature identification message associated with the first MIDMR in response to the feature identification request.   
     
     
         18 . The one or more non-transitory computer readable media recited in  claim 17 , wherein the spatial information is determined at least in part based on inertial data. 
     
     
         19 . The one or more non-transitory computer readable media recited in  claim 17 , wherein identifying the visual feature comprises:
 determining an object type associated with the object,   identifying a predefined visual feature associated with the object type, and   locating the predefined visual feature in the first MIDMR.   
     
     
         20 . The one or more non-transitory computer readable media recited in  claim 17 , the method further comprising:
 identifying a visual feature in a second MIDMR of the object, wherein the first MIDMR of the object represents the object at a first point in time and wherein the second MIDMR of the object represents the object at a second point in time; and   comparing the visual feature in the first MIDMR of the object to the visual feature in the second MIDMR of the object to identify a change in the object between the first time and the second time, wherein the object is a vehicle and wherein the change in the object represents damage to the object.