Offloading Visual Frames to a Gateway Device

Abstract

Implementations offload visual frames from a client device to a gateway or edge device for processing. The client device can receive streaming visual frames and a request to process the visual frames using a data service. The client device can offload visual frames to a gateway or edge device preloaded with a resource that corresponds to the requested data service. After the gateway or edge device processes the visual frames using the resource, the processed visual frames can be returned to the client device. In implementations, the offload device and client device are situated in a network such that a latency for the offload communications supports real-time video display. A gateway device manager can locate a gateway connected to the client device, and resources can be deployed (or the gateway can be prepopulated) so that the device can perform gateway services and edge processing services for the client device.

I/We claim: 
     
         1 . A computer-readable storage medium storing instructions that, when executed by a computing system, cause a wireless device to offload visual frame processing to edge devices, the process comprising:
 sending, by the wireless device, visual frames to a first edge device over a wireless network, wherein the first edge device provides an edge data service to the wireless device and is configured to process the visual frames using a workload resource and return the processed visual frames, to the wireless device over the wireless network;   providing a movement metric for the wireless device,
 wherein the movement metric for the wireless device is used to determine a predicted location of the wireless device, 
 wherein a second edge device is selected, according to the predicted location, for providing the edge data service to the wireless device; and 
 wherein, in response to the selection of the second edge device, edge data services are initialized at the second edge device using workload resources; 
   performing, when the wireless device has moved to the predicted location, a handover of the edge data service from the first edge device to the second edge device;   following the handover, offloading additional visual frames to the second edge device, wherein the second edge device processes the additional visual frames and returns the processed additional visual frames to the wireless device over the wireless network; and   displaying content based on the processed additional visual frames.   
     
     
         2 . The computer-readable storage medium of  claim 1 , wherein the selecting the second edge device further comprises:
 comparing a service criteria for the edge data service to service metrics for multiple candidate edge devices; and   selecting the second edge device from among the multiple candidate edge devices according to the comparison.   
     
     
         3 . The computer-readable storage medium of  claim 2 , wherein the service metrics comprise one or more of: a latency parameter, a processor utilization, an edge resource listing, or a combination thereof. 
     
     
         4 . The computer-readable storage medium of  claim 2 , wherein:
 the multiple candidate edge devices comprise at least the second edge device and an additional edge device,   a latency parameter for the second edge device is lower than a latency parameter for the additional edge device, and   the selecting the second edge device is based on a determination of the latency parameter for the second edge device being lower than the latency parameter for the additional edge device.   
     
     
         5 . The computer-readable storage medium of  claim 2 , wherein:
 the multiple candidate edge devices comprise at least the second edge device and an additional edge device,   an edge resource listing for the second edge device matches the service criteria while an edge resource listing for the additional edge device fails to match the service criteria, and   the selecting the second edge device is based on determinations of the edge resource listing for the second edge device matching the service criteria and the edge resource listing for the additional edge device failing to match the service criteria.   
     
     
         6 . The computer-readable storage medium of  claim 1 , wherein, during performance of the handover, the wireless device offloads a first portion of visual frames to the first edge device and a second portion of visual frames to the second edge device. 
     
     
         7 . The computer-readable storage medium of  claim 6 , wherein the first portion of visual frames and the second portion of visual frames comprise one or more redundant visual frames. 
     
     
         8 . The computer-readable storage medium of  claim 7 , wherein an amount of redundant frames within the first portion of visual frames and the second portion of visual frames is determined based on a service criteria for the edge data service. 
     
     
         9 . The computer-readable storage medium of  claim 1 , wherein at least one workload resources at the first edge device, when used to process the offloaded visual frames:
 performs object tracking for objects in the visual frames,   generates overlays, masks, images, or three-dimensional volumes for augmenting the visual frames,   performs artificial reality video processing for the processed visual frames,   performs three-dimensional mapped environment video processing for the processed visual frames, or   any combination thereof.   
     
     
         10 . A method for a wireless device to offload visual frame processing to edge devices, the method comprising:
 sending, by the wireless device, visual frames to a first edge device over a wireless network, wherein the first edge device provides an edge data service to the wireless device and is configured to process the visual frames using a workload resource and return the processed visual frames, to the wireless device over the wireless network;   providing a movement metric for the wireless device,
 wherein the movement metric for the wireless device is used to determine a predicted location of the wireless device, 
 wherein a second edge device is selected, according to the predicted location, for providing the edge data service to the wireless device; and 
 wherein, in response to the selection of the second edge device, edge data services are initialized at the second edge device using workload resources; and 
   performing, when the wireless device has moved to the predicted location, a handover of the edge data service from the first edge device to the second edge device.   
     
     
         11 . The method of  claim 10  further comprising:
 following the handover, offloading additional visual frames to the second edge device, wherein the second edge device processes the additional visual frames and returns the processed additional visual frames to the wireless device over the wireless network; and 
 displaying content based on the processed additional visual frames. 
 
     
     
         12 . The method of  claim 10 , wherein the selecting the second edge device further comprises:
 comparing a service criteria for the edge data service to service metrics for multiple candidate edge devices; and   selecting the second edge device from among the multiple candidate edge devices according to the comparison.   
     
     
         13 . The method of  claim 12 , wherein the service metrics comprise one or more of: a latency parameter, a processor utilization, an edge resource listing, or a combination thereof. 
     
     
         14 . The method of  claim 12 , wherein:
 the multiple candidate edge devices comprise at least the second edge device and an additional edge device,   a latency parameter for the second edge device is lower than a latency parameter for the additional edge device, and   the selecting the second edge device is based on a determination of the latency parameter for the second edge device being lower than the latency parameter for the additional edge device.   
     
     
         15 . The method of  claim 12 , wherein:
 the multiple candidate edge devices comprise at least the second edge device and an additional edge device,   an edge resource listing for the second edge device matches the service criteria while an edge resource listing for the additional edge device fails to match the service criteria, and   the selecting the second edge device is based on determinations of the edge resource listing for the second edge device matching the service criteria and the edge resource listing for the additional edge device failing to match the service criteria.   
     
     
         16 . A computing system, in a wireless device, for offloading visual frame processing to edge devices, the computing system comprising:
 one or more processors; and   one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
 sending, by the wireless device, visual frames to a first edge device over a wireless network, wherein the first edge device provides an edge data service to the wireless device and is configured to process the visual frames using a workload resource and return the processed visual frames, to the wireless device over the wireless network; 
 providing a movement metric for the wireless device,
 wherein the movement metric for the wireless device is used to determine a predicted location of the wireless device, 
 wherein a second edge device is selected, according to the predicted location, for providing the edge data service to the wireless device; and 
 wherein, in response to the selection of the second edge device, edge data services are initialized at the second edge device using workload resources; and 
 
 performing, when the wireless device has moved to the predicted location, a handover of the edge data service from the first edge device to the second edge device. 
   
     
     
         17 . The computing system of  claim 16 , wherein, during performance of the handover, the wireless device offloads a first portion of visual frames to the first edge device and a second portion of visual frames to the second edge device. 
     
     
         18 . The computing system of  claim 17 , wherein the first portion of visual frames and the second portion of visual frames comprise one or more redundant visual frames. 
     
     
         19 . The computing system of  claim 18 , wherein an amount of redundant frames within the first portion of visual frames and the second portion of visual frames is determined based on a service criteria for the edge data service. 
     
     
         20 . The computing system of  claim 16 , wherein at least one workload resources at the first edge device, when used to process the offloaded visual frames:
 performs object tracking for objects in the visual frames,   generates overlays, masks, images, or three-dimensional volumes for augmenting the visual frames,   performs artificial reality video processing for the processed visual frames,   performs three-dimensional mapped environment video processing for the processed visual frames, or   any combination thereof.