Mixed reality system, program, mobile terminal device, and method

Abstract

One or more embodiments of the invention is a mixed reality system for displaying a mixed-reality image on a display, the system including a mobile terminal device having the display and a photographing device. The system includes a plurality of feature point sets arranged in the prescribed real space. The mobile terminal device recognizes each of the feature point sets photographed by the photographing device, determines a viewpoint position of a virtual camera, in a virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, based on arrangement position and posture information for each of the feature point sets and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, and generates a mixed-reality image in which an image of the virtual object is superimposed on a photographed image of the real space.

1 . A mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising a mobile terminal device having the display and a photographing device that photographs the real space, wherein:
 the mixed reality system   includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and   the mobile terminal device   stores data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, stores arrangement position and posture information in the virtual space for each of the feature point sets arranged in the prescribed real space,   recognizes each of the feature point sets photographed by the photographing device,   determines a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of the arrangement position and posture information for each of the feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and   on the basis of the data of the first virtual object, the data of the second virtual object, and the viewpoint position, generates a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the real space.   
     
     
         2 . A mixed reality system according to  claim 1 , wherein, in the case where the first virtual object and the second virtual object overlap each other in a field-of-view range of the virtual camera when the mixed-reality image is generated, if a depth distance of the second virtual object from the viewpoint position is larger than that of the first virtual object, the mobile terminal device generates a mixed-reality image in which the image of the second virtual object with overlapping parts removed is superimposed. 
     
     
         3 . A mixed reality system according to  claim 1 , wherein
 the mixed reality system includes five or more of said feature point sets, and   the mobile terminal device, at prescribed time intervals, recognizes each of the feature point sets photographed by the photographing device, determines the viewpoint position, and generates the mixed-reality image.   
     
     
         4 . A mixed reality system according to  claim 1 , wherein the mixed reality system includes four of said feature point sets arranged in the prescribed real space, and the individual feature point sets have positional relationships corresponding to four vertex positions of a virtual rectangle in the real space and are arranged away from each other by a predefined distance in a vertical direction and a predefined distance in a horizontal direction. 
     
     
         5 . A mixed reality system according to  claim 4 , wherein one or a plurality of the feature point sets are securely attached to each of a plurality of rod-like bodies installed in a horizontally-isolated manner. 
     
     
         6 . A mixed reality system according to  claim 1 , wherein each of the feature point sets is securely attached to a transmission member formed of a light transmitting material. 
     
     
         7 . A mixed reality system according to  claim 1 , wherein each of the feature point sets is securely attached by using a wire. 
     
     
         8 . A mixed reality system according to  claim 1 , wherein the feature point sets are posters, paintings, pictures, ArUco markers, AR markers, two-dimensional codes, or data matrices. 
     
     
         9 . A non-transitory computer readable medium storing a program executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the real space, wherein:
 the mixed reality system   includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and   the program causes the mobile terminal device to execute   a step of recognizing each of the feature point sets photographed by the photographing device,   a step of determining a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of arrangement position and posture information in the virtual space in which each of the feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and   a step of generating a mixed-reality image in which an image of a second virtual object according to the viewpoint position is superimposed on the photographed image of the real space, on the basis of data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, data of the second virtual object, in the virtual space, that does not correspond to the real object, and the viewpoint position.   
     
     
         10 . A mobile terminal device in a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the real space, wherein:
 the mixed reality system   includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and   the mobile terminal device   stores data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, stores arrangement position and posture information in the virtual space for each of the feature point sets arranged in the prescribed real space,   recognizes each of the feature point sets photographed by the photographing device,   determines a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of the arrangement position and posture information for each of the feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and   on the basis of the data of the first virtual object, the data of the second virtual object, and the viewpoint position, generates a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the real space.   
     
     
         11 . A method executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the real space, wherein:
 the mixed reality system   includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and   the method includes   a step of recognizing each of the feature point sets photographed by the photographing device,   a step of determining a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of arrangement position and posture information in the virtual space in which each of the feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and   a step of generating a mixed-reality image in which an image of a second virtual object according to the viewpoint position is superimposed on the photographed image of the real space, on the basis of data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, data of the second virtual object, in the virtual space, that does not correspond to the real object, and the viewpoint position.   
     
     
         12 . A mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising a server and a mobile terminal device having the display and a photographing device that photographs the real space, wherein:
 the mixed reality system   includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships;   the server   stores data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, stores arrangement position and posture information in the virtual space for each of the feature point sets arranged in the prescribed real space; and   the mobile terminal device   recognizes each of the feature point sets photographed by the photographing device,   determines a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of the arrangement position and posture information for each of the feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and   on the basis of the data of the first virtual object, the data of the second virtual object, and the viewpoint position, generates a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the real space.   
     
     
         13 . A mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising a mobile terminal device having the display and a photographing device that photographs the real space, wherein:
 the mixed reality system   includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and   the mobile terminal device   stores data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, stores arrangement position and posture information in the virtual space for each of the feature point sets arranged in the prescribed real space,   recognizes each of the feature point sets photographed by the photographing device,   determines position and posture information of the first virtual object on the basis of the arrangement position and posture information for each of the feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and   on the basis of the data of the first virtual object and the data of the second virtual object, generates a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the real space.   
     
     
         14 . A non-transitory computer readable medium storing a program executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the real space, wherein:
 the mixed reality system   includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and   the program causes the mobile terminal device to execute   a step of recognizing each of the feature point sets photographed by the photographing device,   a step of determining position and posture information of a first virtual object, obtained in advance, that corresponds to a real object present in the prescribed real space and that defines the virtual space, on the basis of arrangement position and posture information in the virtual space in which each of the feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and   a step of generating a mixed-reality image in which an image of a second virtual object according to a viewpoint position is superimposed on the photographed image of the real space, on the basis of data of the first virtual object and data of the second virtual object, in the virtual space, that does not correspond to the real object.