Modification of software behavior in run time

Abstract

When executed by one or more processors in a computer system, an application program may provide, to an electronic device, image information that specifies a user interface for display on the electronic device, and may receive, from the electronic device, information that specifies user-interface activity of the user. Furthermore, when executed by the one or more processors, the modification program may identify a state of the application program. Then, the modification program may selectively modify the image information based on the identified state to change the user interface without changing source code for the application program or a compiled executable of the application program, and may provide the modified image information to the electronic device for display. Subsequently, based on information that specifies a user-interface activity of the user that is associated with the modified user interface, the modification program may selectively perform an operation.

1 - 20 . (canceled) 
     
     
         21 . A method for using a virtual machine to unify different rendering techniques, of two or more applications, into a single rendering path, the method comprising:
 injecting a wrapper between A) the two or more applications and B) an operating system of the virtual machine, wherein the two or more applications provide display output to one or more windows managed by the operating system of the virtual machine;   obtaining, via the wrapper, a first window handle for a first window output to by a first application, of the two or more applications, wherein the output of the first application uses a first rendering technique;   capturing first contents of the first window by causing the wrapper to request a first surface associated with the first window handle;   overlaying first additional display information on the captured first contents; and   sending the captured first contents, with the overlaid first additional display information, to a rendering system for presentation on a display.   
     
     
         22 . The method of  claim 21  further comprising:
 obtaining, via the wrapper, a second window handle for a second window output to by a second application, of the two or more applications, wherein the output of the second application uses a second rendering technique different from the first rendering technique; 
 capturing second contents of the second window by causing the wrapper to request a second surface associated with the second window handle; 
 overlaying second additional display information on the captured second contents; and 
 sending the captured second contents, with the overlaid second additional display information, to the rendering system for presentation on the display. 
 
     
     
         23 . The method of  claim 22 , wherein the wrapper accesses the first surface and the second surface, in the virtual machine, using the same, single rendering technique. 
     
     
         24 . The method of  claim 23 , wherein the wrapper accesses the first surface and the second surface using a version of DirectX as the single rendering technique. 
     
     
         25 . The method of  claim 21 , wherein the operating system is a Windows operating system, and wherein the capturing the contents of the first window is performed by requesting surface information from a Windows composition manager of the Windows operating system. 
     
     
         26 . The method of  claim 21 , wherein the capturing the first contents of the first window is performed by obtaining unified contents of a video frame buffer associated with the first application. 
     
     
         27 . The method of  claim 21 , wherein capturing the contents of the first window is performed in a manner that is not specific to the first rendering technique. 
     
     
         28 . The method of  claim 27 , wherein capturing the contents of the first window is not specific to the first rendering technique by the capturing being independent of the first rendering technique and independent of a rendering path used by the first application program. 
     
     
         29 . The method of  claim 21 , wherein the first application program is a video game. 
     
     
         30 . The method of  claim 21 , wherein the method is performed by a cloud-based gaming service. 
     
     
         31 . The method of  claim 21 , wherein the method is performed without incurring additional processing for making additional copies, in main memory or GPU memory, of image output of the first application program. 
     
     
         32 . A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for using a virtual machine to unify different rendering techniques, of two or more applications, into a single rendering path, the process comprising:
 injecting a wrapper between A) the two or more applications and B) an operating system of the virtual machine, wherein the two or more applications provide display output to one or more windows managed by the operating system of the virtual machine;   obtaining, via the wrapper, a first window handle for a first window output to by a first application, of the two or more applications, wherein the output of the first application uses a first rendering technique;   capturing first contents of the first window by causing the wrapper to request a first surface associated with the first window handle;   overlaying first additional display information on the captured first contents; and   sending the captured first contents, with the overlaid first additional display information, to a rendering system for presentation on a display.   
     
     
         33 . The computer-readable storage medium of  claim 32 , wherein the process further comprises:
 obtaining, via the wrapper, a second window handle for a second window output to by a second application, of the two or more applications, wherein the output of the second application uses a second rendering technique different from the first rendering technique;   capturing second contents of the second window by causing the wrapper to request a second surface associated with the second window handle;   overlaying second additional display information on the captured second contents; and   sending the captured second contents, with the overlaid second additional display information, to the rendering system for presentation on the display.   
     
     
         34 . The computer-readable storage medium of  claim 33 , wherein the wrapper accesses the first surface and the second surface, in the virtual machine, using the same, single rendering technique. 
     
     
         35 . The computer-readable storage medium of  claim 34 , wherein the wrapper accesses the first surface and the second surface using a version of DirectX as the single rendering technique. 
     
     
         36 . The computer-readable storage medium of  claim 32 , wherein the operating system is a Windows operating system, and wherein the capturing the contents of the first window is performed by requesting surface information from a Windows composition manager of the Windows operating system. 
     
     
         37 . A computing system for using a virtual machine to unify different rendering techniques, of two or more applications, into a single rendering path, the computing system comprising:
 one or more processors; and   one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
 injecting a wrapper between A) the two or more applications and B) an operating system of the virtual machine, wherein the two or more applications provide display output to one or more windows managed by the operating system of the virtual machine; 
 obtaining, via the wrapper, a first window handle for a first window output to by a first application, of the two or more applications, wherein the output of the first application uses a first rendering technique; 
 capturing first contents of the first window by causing the wrapper to request a first surface associated with the first window handle; 
 overlaying first additional display information on the captured first contents; and 
 sending the captured first contents, with the overlaid first additional display information, to a rendering system for presentation on a display. 
   
     
     
         38 . The computing system of  claim 37 , wherein the capturing the first contents of the first window is performed by obtaining unified contents of a video frame buffer associated with the first application. 
     
     
         39 . The computing system of  claim 37 , wherein capturing the contents of the first window is independent of the first rendering technique and is independent of a rendering path used by the first application program. 
     
     
         40 . The computing system of  claim 37 , wherein the method is performed without incurring additional processing for making additional copies, in main memory or GPU memory, of image output of the first application program.