Binaural Sound in Visual Entertainment Media

Abstract

A method provides binaural sound to a listener while the listener watches a movie so sounds from the movie localize to a location of a character in the movie. Sound is convolved with head related transfer functions (HRTFs) of the listener, and the convolved sound is provided to the listener who wears a wearable electronic device.

1 .- 20 . (canceled) 
     
     
         21 . A method that provides binaural sound to a listener watching a movie in virtual reality (VR), the method comprising:
 displaying, with a head mounted display (HMD) worn by the listener, a VR movie theater with a VR movie screen and a VR listener that represents the listener wearing the HMD to watch the movie that includes a character;   tracking, with the HMD worn by the listener, head movements of the listener to determine a head orientation of the listener with respect to a location of the character on the VR movie screen; and   convolving, with a processor in the HMD and based on the head orientation, a voice of the character to originate as the binaural sound at a sound localization point (SLP) that exists in empty space not occupied by a tangible object but at a location on the VR movie screen where the character appears on the VR movie screen such that from a point-of-view of the listener the voice of the character externally localizes where the character is on the VR movie screen as opposed to localizing to speakers located at a perimeter of the VR movie theater; and   providing, with the HMD worn by the listener, the voice of the character to originate as the binaural sound at the SLP that exists in empty space not occupied by the tangible object but at the location on the VR movie screen where the character appears on the VR movie screen such that from the point-of-view of the listener the voice of the character externally localizes where the character is on the VR movie screen.   
     
     
         22 . The method of  claim 21  further comprising:
 calculating a distance between the VR listener and the character on the VR movie screen; and 
 adjusting a loudness of the voice of the character to be commensurate with the distance between the VR listener and the character on the VR movie screen. 
 
     
     
         23 . The method of  claim 21  further comprising:
 calculating azimuth angles between a line-of-sight of the listener and different locations where the character is on the VR movie screen as the character moves across the VR movie screen; and 
 convolving, based on the azimuth angles between the line-of-sight of the listener and the different locations where the character is on the VR movie screen, the voice of the character with head-related transfer functions (HRTFs). 
 
     
     
         24 . The method of  claim 21  further comprising:
 determining a size and shape of the VR movie theater where the listener watches the movie; and 
 convolving the voice of the character with room impulse responses (RIRs) based on the size and the shape of the VR movie theater. 
 
     
     
         25 . The method of  claim 21  further comprising:
 solving a problem of sound being played at a higher volume to compensate for listeners seated at distances from the VR movie screen by playing the voice of the character according to individual preferences of the listeners in the VR movie theater. 
 
     
     
         26 . The method of  claim 21  further comprising:
 providing the movie to multiple listeners seated at different locations in the VR movie theater; and 
 convolving the voice of the character for each of the multiple listeners watching the movie in the VR movie theater based on the different locations where each of the multiple listeners are seated in the VR movie theater. 
 
     
     
         27 . The method of  claim 21 , wherein the SLP where the listener hears the voice of the character follows movements of the character moves such that the listener continues to hear the voice of the character originate from an image of the character as the image of the character moves across the VR movie screen. 
     
     
         28 . A non-transitory computer-readable storage medium that stores instructions that one or more electronic devices execute as a method that provides binaural sound to listeners watching a movie in virtual reality (VR), the method comprising:
 displaying, with a head mounted display (HMD), a VR movie theater that includes a plurality of VR seats, a VR movie screen, and VR listeners seated in the plurality of VR seats watching the movie in the VR movie theater;   displaying, with the HMD, a character in the movie at a location on the VR movie screen;   processing a voice of the character so the voice of the character originates as the binaural sound to the listeners at a sound localization point (SLP) at the location on the VR movie screen where the character is located; and   processing the voice of the character so the SLP of the voice of the character moves and follows the location of the character as the character moves to different locations on the VR movie screen.   
     
     
         29 . The non-transitory computer-readable storage medium of  claim 28  further comprising:
 processing the voice of the character with different sets of head-related transfer functions (HRTFs) for each one of the VR listeners seated in the plurality of the VR seats. 
 
     
     
         30 . The non-transitory computer-readable storage medium of  claim 28  further comprising:
 calculating a distance between each of the VR listeners and the character on the VR movie screen; and 
 adjusting a loudness of the voice of the character to vary inversely with a square of the distance between each of the VR listeners and the character on the VR movie screen. 
 
     
     
         31 . The non-transitory computer-readable storage medium of  claim 28  further comprising:
 calculating an azimuth angle between each of the VR listeners and the location on the VR movie screen where the character is located; and 
 processing the voice of the character for each of the VR listeners based on the azimuth angle between each of the VR listeners and the location on the VR movie screen where the character is located. 
 
     
     
         32 . The non-transitory computer-readable storage medium of  claim 28  further comprising:
 tracking head movements of the VR listeners with respect to the location of the character; and 
 changing, based on the head movements, transfer functions processing the voice of the character so the voice of the character continues to emanate from the SLP at the location on the VR movie screen where the character is located while head movements of the VR listeners change. 
 
     
     
         33 . The non-transitory computer-readable storage medium of  claim 28  further comprising:
 distinguishing a voice of a narrator in the movie between the voice of the character by providing the voice of the narrator in stereo sound that internally localizes and the voice of the character as the binaural sound that externally localizes. 
 
     
     
         34 . The non-transitory computer-readable storage medium of  claim 28 , wherein the voice of the character originates in empty space on a same plane as the VR movie screen. 
     
     
         35 . The non-transitory computer-readable storage medium of  claim 28  further comprising:
 processing a sound in the movie so different listeners simultaneously hear the sound that externally localizes to different locations relative to the VR movie screen, wherein the different locations include on the VR movie screen and off the VR movie screen. 
 
     
     
         36 . A method that provides binaural sound to a listener watching a movie in virtual reality (VR), the method comprising:
 displaying, with a head mounted display (HMD), a VR movie theater that includes VR seats, a VR movie screen, and a VR listener watching the movie in the VR movie theater;   tracking, with the HMD worn by the listener, head movements of the listener to determine a head orientation of the listener with respect to a location of a character on the VR movie screen;   processing, with one or more processors and based on the head movements, a voice of character as the binaural sound that originates at a sound localization point (SLP) that exists at a location on the VR movie screen where the character appears on the VR movie screen such that from a point-of-view of the listener the voice of the character externally localizes to the location where the character is on the VR movie screen; and   processing, with the one or more processors and based on the head movements, the voice of character to move and to follow movements of the character as the character moves on the VR movie screen such that the voice of the character continues to originate from the SLP that exists at the location on the VR movie screen where the character appears as the character moves.   
     
     
         37 . The method of  claim 36  further comprising:
 determining a distance between the VR listener and the VR movie screen; and 
 adjusting a loudness of the voice of the character based on the distance between the VR listener and the VR movie screen. 
 
     
     
         38 . The method of  claim 36  further comprising:
 processing the voice of the character with head-related transfer functions (HRTFs) to provide the voice of the character as the binaural sound; and 
 selecting the HRTFs based on which one of the VR seats the VR listener sits in. 
 
     
     
         39 . The method of  claim 36  further comprising:
 processing the voice of the character with head-related transfer functions (HRTFs) in response to which one of the VR seats the VR listener sits in; and 
 changing the HRTFs processing the voice of the character in response to the VR listener moving to another one of the VR seats. 
 
     
     
         40 . The method of  claim 36  further comprising:
 increasing a level of realism that the listener experiences while watching the movie by providing a sound to originate at a SLP in empty space behind the listener when the sound originates behind the character.