Answering questions using environmental context

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving audio data encoding an utterance and environmental data, obtaining a transcription of the utterance, identifying an entity using the environmental data, submitting a query to a natural language query processing engine, wherein the query includes at least a portion of the transcription and data that identifies the entity, and obtaining one or more results of the query.

1 . A computer-implemented method comprising:
 receiving audio data encoding (i) an utterance and (ii) background audio data;   obtaining a transcription of the utterance;   identifying an entity using the background audio data;   submitting a query to a natural language query processing engine, wherein the query includes at least a portion of the transcription and data that identifies the entity that is identified using the background audio data; and   obtaining one or more results of the query.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising outputting a representation of at least one of the results. 
     
     
         3 . The computer-implemented method of  claim 1 , wherein the entity is identified further using the utterance. 
     
     
         4 . The computer-implemented method of  claim 1 , further comprising generating the query. 
     
     
         5 . The computer-implemented method of  claim 4 , wherein generating the query comprises associating the transcription with the data that identifies the entity. 
     
     
         6 . The computer-implemented method of  claim 5 , wherein associating further includes tagging the transcription with the data that identifies the entity. 
     
     
         7 . The computer-implemented method of  claim 5 , wherein associating further includes substituting a portion of the transcription with the data that identifies the entity. 
     
     
         8 . The computer-implemented method of  claim 7 , wherein substituting further includes substituting one or more words of the transcription with the data that identifies the entity. 
     
     
         9 - 10 . (canceled) 
     
     
         11 . A system comprising:
 one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
 receiving, from a computing device, audio data encoding an utterance recorded by the computing device, and image data encoding an image captured by the computing device; 
 obtaining a transcription of the utterance; 
 identifying an entity using the image data; 
 submitting a query to a natural language query processing engine, wherein the query includes at least a portion of the transcription and data that identifies the entity that is identified using the image data; and 
 obtaining one or more results of the query. 
   
     
     
         12 . The system of  claim 11 , the operations further include generating the query, wherein generating the query comprises associating the transcription with the data that identifies the entity. 
     
     
         13 . The system of  claim 12 , wherein associating further includes tagging the transcription with the data that identifies the entity. 
     
     
         14 . The system of  claim 12 , wherein associating further includes substituting a portion of the transcription with the data that identifies the entity. 
     
     
         15 . The system of  claim 14 , wherein substituting further includes substituting one or more words of the transcription with the data that identifies the entity. 
     
     
         16 - 17 . (canceled) 
     
     
         18 . A computer-readable storage device storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
 receiving (i) audio data encoding an utterance, and (ii) environmental data;   obtaining a transcription of the utterance;   identifying a title of an item of media content using the environmental data;   submitting a query to a natural language query processing engine, wherein the query includes at least a portion of the transcription and data that identifies the title of the item of media content that is identified using the environmental data; and   obtaining one or more results of the query.   
     
     
         19 . The computer-readable storage device of  claim 18 , the operations further comprise generating the query, wherein generating the query comprises associating the transcription with the data that identifies the entity. 
     
     
         20 . The computer-readable storage device of  claim 19 , wherein associating further includes tagging the transcription with the data that identifies the entity. 
     
     
         21 . The computer-readable storage device of  claim 19 , wherein associating further includes substituting a portion of the transcription with the data that identifies the entity. 
     
     
         22 . The computer-readable storage device of  claim 21 , wherein substituting further includes substituting one or more words of the transcription with the data that identifies the entity. 
     
     
         23 . The computer-implemented method of  claim 1 , further comprising:
 providing, to a query generation engine, the transcription of the utterance and the data that identifies the entity; and   receiving, from the query generation engine, the query based on providing the transcription of the utterance and the data that identifies the entity to the query generation engine.   
     
     
         24 . The computer-implemented method of  claim 1 , further comprising:
 determining, for each result of the one or more results, a score based on a matching between the query and the result;   comparing, for each result of the one or more results, the score to a threshold score; and   based on comparing the score to the threshold score, identifying one or more particular results of the one or more results associated with a score that satisfied the threshold score.