Search with joint image-audio queries

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing joint image-audio queries. In one aspect, a method includes receiving, from a client device, a joint image-audio query including query image data and query audio data. Query image feature data is determined from the query image data. Query audio feature data is determined from the audio data. The query image feature data and the query audio feature data are provided to a joint image-audio relevance model trained to generate relevance scores for a plurality of resources, each resource including resource image data defining a resource image for the resource and text data defining resource text for the resource. Each relevance score is a measure of the relevance of corresponding resource to the joint image-audio query. Data defining search results indicating the order of the resources is provided to the client device.

What is claimed is: 
     
         1 . (canceled) 
     
     
         2 . A computer-implemented method performed by a data processing apparatus, the method comprising:
 receiving a joint image-audio query from a client device, the joint image-audio query including query image data defining a query image and query audio data defining query audio;   deriving text from the query audio data;   determining, from the text, positional information that describes a position of a query object in the query image data, the position being a subset of the query image and in which the query object is depicted;   determining query image feature data from the query image data, the query image feature data describing image features of the subset of the query image;   providing the query image feature data and the text data to a relevance model as input;   ordering, by the data processing apparatus, the identified resources according to corresponding relevance scores generated by the relevance model; and   providing, by the data processing apparatus, data defining search results indicating the order of the identified resources to the client device.   
     
     
         3 . The computer-implemented method of  claim 2 , wherein the query image data and the query audio data were paired as the joint-image audio query at the client device. 
     
     
         4 . The computer-implemented method of  claim 2 , wherein the relevance model is a joint image-audio relevance model that is trained to generate the relevance scores for a plurality of resources based on a combined relevance of the query image feature data to image feature data of the resource and the text derived from the query audio to text of the resource. 
     
     
         5 . The computer-implemented method of  claim 4 , wherein the joint image-audio relevance model generates relevance scores based, in part, from only image feature data determined from the subset of the image data. 
     
     
         6 . The computer-implemented method of  claim 2 , wherein the relevance model generates relevance scores based, in part, on the positional information. 
     
     
         7 . The computer-implemented method of  claim 2 , wherein the text further defines one or more restrictions on the search results, and the relevance model generates relevance scores based, in part, on the one or more restrictions. 
     
     
         8 . A system, comprising:
 a data processing apparatus; and   a computer storage medium encoded with a computer program, the program comprising instructions that when executed by the data processing apparatus cause the data processing apparatus to perform operations comprising:   receiving a joint image-audio query from a client device, the joint image-audio query including query image data defining a query image and query audio data defining query audio;   deriving text from the query audio data;   determining, from the text, positional information that describes a position of a query object in the query image data, the position being a subset of the query image and in which the query object is depicted;   determining query image feature data from the query image data, the query image feature data describing image features of the subset of the query image;   providing the query image feature data and the text data to a relevance model as input;   ordering, by the data processing apparatus, the identified resources according to corresponding relevance scores generated by the relevance model; and   providing, by the data processing apparatus, data defining search results indicating the order of the identified resources to the client device.   
     
     
         9 . The system of  claim 8 , wherein the query image data and the query audio data were paired as the joint-image audio query at the client device. 
     
     
         10 . The system of  claim 8 , wherein the relevance model is a joint image-audio relevance model that is trained to generate the relevance scores for a plurality of resources based on a combined relevance of the query image feature data to image feature data of the resource and the text derived from the query audio to text of the resource. 
     
     
         11 . The system of  claim 10 , wherein the joint image-audio relevance model generates relevance scores based, in part, from only image feature data determined from the subset of the image data. 
     
     
         12 . The system of  claim 8 , wherein the relevance model generates relevance scores based, in part, on the positional information. 
     
     
         13 . The system of  claim 8 , wherein the text further defines one or more restrictions on the search results, and the relevance model generates relevance scores based, in part, on the one or more restrictions. 
     
     
         14 . A computer storage device encoded with a computer program, the program comprising instructions that when executed by a client device cause the client device to perform operations comprising:
 receiving a joint image-audio query from a client device, the joint image-audio query including query image data defining a query image and query audio data defining query audio;   deriving text from the query audio data;   determining, from the text, positional information that describes a position of a query object in the query image data, the position being a subset of the query image and in which the query object is depicted;   determining query image feature data from the query image data, the query image feature data describing image features of the subset of the query image;   providing the query image feature data and the text data to a relevance model as input;   ordering, by the data processing apparatus, the identified resources according to corresponding relevance scores generated by the relevance model; and   providing, by the data processing apparatus, data defining search results indicating the order of the identified resources to the client device.   
     
     
         15 . The computer storage device of  claim 14 , wherein the query image data and the query audio data were paired as the joint-image audio query at the client device. 
     
     
         16 . The computer storage device of  claim 14 , wherein the relevance model is a joint image-audio relevance model that is trained to generate the relevance scores for a plurality of resources based on a combined relevance of the query image feature data to image feature data of the resource and the text derived from the query audio to text of the resource. 
     
     
         17 . The computer storage device of  claim 16 , wherein the joint image-audio relevance model generates relevance scores based, in part, from only image feature data determined from the subset of the image data. 
     
     
         18 . The computer storage device of  claim 14 , wherein the relevance model generates relevance scores based, in part, on the positional information. 
     
     
         19 . The computer storage device of  claim 14 , wherein the text further defines one or more restrictions on the search results, and the relevance model generates relevance scores based, in part, on the one or more restrictions.