Video-conference device and method

Abstract

A video-conference device for providing an augmented view of in-room participants in a video-conference. The image sensor is configured to capture images comprising the room and the in-room participants. The device comprises a depth sensor configured for measuring the distance to each of the in-room participants having a processing unit configured for providing the augmented view of the in-room. The processing unit is configured to performing an image processing of the captured images from the image sensor to virtually identify each of the in-room participants. The processing unit is configured to determining a desired virtual position for each of the in-room participants in the augmented view. The processing unit is configured to performing a virtual scaling of each of the in-room participants to the desired virtual size.

1 . A video-conference device for providing an augmented view of in-room participants in a video-conference, the device is configured to be arranged in a room, where a first number of participants is the in-room participants present in the room while the video-conference is held, and a second number of participants is far-end participants present at one or more different locations than the room, wherein the device comprises:
 an image sensor configured for capturing images, the image sensor being configured to capture images comprising the room and the in-room participants;   a depth sensor configured for measuring the distance to each of the in-room participants;   a processing unit configured for providing the augmented view of the in-room participants by a processed/augmented video based on the captured images from the image sensor and the measurements by the depth sensor;   
       wherein the processing unit is configured to:
 obtaining the measurements of the distance to each of the in-room participants by the depth sensor; 
 obtaining the captured images by the images sensor; 
 performing an image processing of the captured images from the image sensor to virtually identify each of the in-room participants; 
 determining a desired virtual position for each of the in-room participants in the augmented view; 
 determining a scaling factor for a desired virtual size of each of the in-room participants; 
 performing a virtual scaling of each of the in-room participants to the desired virtual size in the augmented view; and 
 performing a virtual positioning of each of the in-room participants in the augmented view. 
 
     
     
         2 . The device of  claim 1 , wherein the processing unit is configured for performing the virtual positioning of each of the in-room participants in the augmented view such that the in-room participants are virtually positioned closer to each other and closer to the image sensor capturing the images. 
     
     
         3 . The device according to  claim 1 , wherein the processing unit is configured for performing the virtual positioning of each of the in-room participants in the augmented view such that the relative positions of the in-room participants relative to each other are maintained. 
     
     
         4 . The device according to  claim 1 , wherein the device is configured to be located in one end of the room; and wherein the image sensor, the depth sensor and the processing unit of the device are co-located. 
     
     
         5 . The device according to  claim 1 , wherein the processing unit is configured to:
 determining a distance A to each of the in-room participants thereby obtaining an actual position (x1,y1,z1) of each in-room participant;   determining a distance B between the in-room participants;   determining a relative location of the in-room participants based on the determined distance A and distance B;   determining the scaling factor N for the desired virtual size of each of the in-room participants based on the determined distance A and distance B; and   determining the virtual position (x2,y2,z2) of each of the in-room participants in the augmented view based on the determined relative location of the in-room participants.   
     
     
         6 . The device according to  claim 1 , wherein the depth sensor comprises one or more of:
 a time of flight (ToF) sensor configured for determining path lengths;   an infra-red sensor configured for determining distances based on reflected light;   an acoustic sensor configured for determining distances based on reflected audio.   
     
     
         7 . The device according to  claim 1 , wherein the image processing of the captured images to identify the in-room participants is performed by using one or more of:
 virtual image segmentation;   virtual cut-out to virtually separate the in-room participants from the background;   virtual image resizing;   virtual seam carving.   
     
     
         8 . The device according to  claim 1 , wherein the image processing is performed in two-dimensions (2D) and/or in three-dimensions (3D). 
     
     
         9 . The device according to  claim 1 , wherein the depth sensor is configured for measuring the distance to objects/parts of the room, and wherein the image sensor is configured to capture images comprising the room with no in-room participants, and wherein the processing unit is configured for:
 mapping the room with no in-room participants, by determining distances C between the depth sensor and one or more objects/parts of the room.   
     
     
         10 . The device according to  claim 1 , wherein the processing unit is configured for defining a suitable illumination in the augmented view based on the time of day of the video-conference. 
     
     
         11 . The device according to  claim 1 , wherein the processing unit is configured for performing blurring of objects in the augmented view of the room. 
     
     
         12 . The device according to  claim 1 , wherein the processing unit is configured for:
 in accordance with a determination that a criterion for the virtual positioning of an in-room participant is not satisfied:
 forgoing performing the virtual positioning of the in-room participant in the augmented view, and 
 virtually placing the in-room participant in a dedicated picture/box in the augmented view. 
   
     
     
         13 . The device according to  claim 1 , wherein the device further comprises:
 a display configured for displaying the augmented view and the far-end participants; and/or   an audio output transducer configured for transmitting audio from the far-end participants to the room.   
     
     
         14 . A system comprising at least two video-conference devices according to  claim 1 . 
     
     
         15 . A method, performed in a video-conference device, for providing an augmented view of in-room participants in the video-conference, the device is configured to be arranged in a room, where a first number of participants is the in-room participants present in the room while the video-conference is held, and a second number of participants is far-end participants present at one or more different locations than the room, wherein the device comprises an image sensor configured for capturing images, the image sensor being configured to capture images comprising the room and the in-room participants; a depth sensor configured for measuring the distance to each of the in-room participants; and a processing unit configured for providing the augmented view of the in-room participants by a processed/augmented video based on the captured images from the image sensor and the measurements by the depth sensor;
 wherein the method comprises, in the processing unit:
 obtaining the measurements of the distance to each of the in-room participants by the depth sensor; 
 obtaining the captured images by the images sensor; 
 performing an image processing of the captured images from the image sensor to virtually identify each of the in-room participants; 
 determining a desired virtual position for each of the in-room participants in the augmented view; 
 determining a scaling factor for a desired virtual size of each of the in-room participants; 
 performing a virtual scaling of each of the in-room participants to the desired virtual size in the augmented view; and 
 performing a virtual positioning of each of the in-room participants in the augmented view.