Display input device and method for controlling same

Abstract

A display input device includes a head mount display and a main apparatus. The head mount display includes an imager, a transmissive display, a sight line detector, a processor, and an HMD communicator. The main apparatus includes an operation panel including a display panel and an operation portion, a main-apparatus communicator, and a controller. The processor causes the transmissive display to display a virtual item button for selecting an item. When an item belonging to a lowest hierarchical layer is selected, the controller causes the display panel to display a setting-value setting screen. The operation portion accepts setting of a setting value.

What is claimed is: 
     
         1 . A display input device comprising:
 a head mount display; and   a main apparatus,   wherein the head mount display comprises
 an imager, 
 a transmissive display which, while transmitting external light, causes a virtual image to be visually recognized, 
 a sight line detector for detecting a sight line of a user, 
 a processor which controls display on the transmissive display, and 
 an HMD communicator, 
   wherein the main apparatus comprises
 an operation panel which includes a display panel and an operation portion, 
 a main-apparatus communicator which communicates with the HMD communicator, and 
 a controller which controls display on the display panel, 
   wherein the processor
 causes the transmissive display to display a virtual item button for item selection while the imager is capturing an image of the operation panel, 
 recognizes, based on an output of the sight line detector, the virtual item button selected, and 
 recognizes, as a selected item, an item corresponding to the virtual item button selected, 
   wherein the item is one of a plurality of kinds of items which are classified in a hierarchical structure, and   wherein, when the selected item belongs to a lowest hierarchical layer,
 the processor causes the HMD communicator to transmit, to the main-apparatus communicator, a first notification indicating the selected item, 
 the controller, based on the first notification, causes the display panel to display a setting-value setting screen for setting a setting value for the selected item, and 
 the operation portion accepts setting of the setting value. 
   
     
     
         2 . The display input device according to  claim 1 ,
 wherein, when the selected item does not belong to the lowest hierarchical layer, the processor
 causes the transmissive display to display an item display image indicating the selected item, and 
 causes the transmissive display to display, around the item display image, the virtual item button corresponding to the item classified as belonging to a hierarchical layer immediately below a hierarchical layer to which the selected item belongs. 
   
     
     
         3 . The display input device according to  claim 1 ,
 wherein, while the transmissive display is displaying the virtual item button, the controller causes the display panel to display a single-color screen.   
     
     
         4 . The display input device according to  claim 1 ,
 wherein the processor causes the transmissive display to display a message telling that it is necessary to operate the operation portion of the main apparatus to execute a job.   
     
     
         5 . The display input device according to  claim 4 ,
 wherein, when the operation portion has accepted setting of the setting value,
 the controller causes the main-apparatus communicator to transmit, to the HMD communicator, a second notification indicating that the setting value has been set, and 
 the processor causes the transmissive display to keep displaying the message for a period from when the item belonging to the lowest hierarchical layer is selected to when the second notification is received. 
   
     
     
         6 . The display input device according to  claim 1 ,
 wherein, when the operation portion has accepted setting of the setting value,
 the controller causes the main-apparatus communicator to transmit, to the HMD communicator, a second notification indicating that the setting value has been set, and 
 the processor keeps the transmissive display from displaying a virtual button for a period from when the virtual item button corresponding to the item classified as belonging to the lowest hierarchical layer is selected to when the second notification is received. 
   
     
     
         7 . The display input device according to  claim 1 ,
 wherein the processor causes the transmissive display to display the virtual item button such that an angle between any two adjacent lines each connecting a center of a display region of the head mount display and a center of the virtual item button is equal to an angle obtained by dividing an angle of 360° by a number of the virtual item button displayed.   
     
     
         8 . The display input device according to  claim 1 ,
 wherein the processor causes the transmissive display to display the virtual item button to be aligned in one or both of a longitudinal direction and a lateral direction.   
     
     
         9 . The display input device according to  claim 1 ,
 wherein the controller causes the display panel to display a home screen for selecting a job kind, and   wherein, while the imager is capturing an image of the home screen, the processor
 causes the transmissive display to display a virtual job kind button for selecting a job kind, 
 recognizes, based on an output of the sight line detector, the virtual job kind button selected, and 
 causes the transmissive display to display the virtual item button corresponding to the item which is related to a selected job kind and classified as belonging to a highest hierarchical layer. 
   
     
     
         10 . The display input device according to  claim 1 ,
 wherein, on recognizing that capturing of an image of the operation panel has been started, the processor causes the HMD communicator to transmit a request for information about a button displayed on the display panel,   wherein the controller causes the main-apparatus communicator to transmit, to the HMD communicator, a response including information indicating a kind of the button displayed on the display panel, and   wherein the processor, based on the response, causes the transmissive display to virtually display the button.   
     
     
         11 . A method for controlling a display input device, the method comprising:
 including a head mount display and a main apparatus in a display input device;   causing a virtual image to be visually recognized using the head mount display while the head mount display transmitting external light;   detecting a sight line of a user using the head mount display;   providing the main apparatus with an operation panel which includes a display panel and an operation portion;   controlling display on the display panel using the main apparatus;   displaying, on the head mount display, a virtual item button for selecting an item while an imager of the head mount display is capturing an image of the operation panel;   recognizing, based on a detected sight line, the virtual item button selected;   recognizing, as a selected item, an item corresponding to the virtual item button selected;   the item being one of a plurality of kinds of items which are classified in a hierarchical structure;   the head mount display transmitting, to the main apparatus, a first notification indicating the selected item, when the selected item belongs to a lowest hierarchical layer;   the display panel displaying, based on the first notification, a setting-value setting screen for setting a setting value for the selected item; and   accepting setting of the setting value.