Detecting latency anomalies from pipeline components in cloud-based systems

Abstract

A method, computer readable medium, and system are disclosed for monitoring a pipeline to detect anomalies such as unusual latency associated with a particular stage. Each stage of the pipeline is configured to update metadata associated with content being processed by inserting a time stamp into the metadata when processing of the content is completed by the stage. The server device can collect the metadata from the last stage of the pipeline and analyze the metadata in order to generate metrics for the pipeline, including a residual latency and/or a gain for each stage of the pipeline. In an embodiment, the content is a frame of video to be displayed on a client device after being rendered by a server device, such as through a streaming service (e.g., a video game streaming service). The server device can adjust the pipeline based on the metrics to improve performance.

What is claimed is: 
     
         1 . A system for monitoring latency variability in a pipeline, the system comprising:
 a server device including one or more processors to:
 receive, from a client device, metadata corresponding to content processed by a plurality of stages of the pipeline, wherein the plurality of stages of the pipeline includes at least one stage implemented using the server device and at least one additional stage implemented using the client device; 
 generate, for two or more stages of the plurality of stages of the pipeline, a metric calculated based on the metadata; and 
 adjust a parameter associated with at least one stage of the pipeline based on the metrics for the two or more stages. 
   
     
     
         2 . The system of  claim 1 , wherein the content comprises a plurality of frames streamed to the client device during a session, and wherein the one or more processors are further to:
 calculate, for each frame of the plurality of frames, a gain for each stage of the pipeline;   compare, for each frame of the plurality of frames, a total residual latency to a first threshold value to identify anomalous frames;   calculate, for each stage of the pipeline, an aggregate value by summing a number of anomalous frames in the plurality of frames where the gain corresponding to the stage in the anomalous frame is greater than a second threshold value; and   generate a visual representation based on the metrics, wherein the metric for a particular stage is calculated based on the aggregate value for the particular stage.   
     
     
         3 . The system of  claim 2 , wherein the gain for a particular stage is calculated based on an expected frame duration and a difference between a time stamp corresponding to the particular stage for the current frame and a time stamp corresponding to the particular stage for a previous frame. 
     
     
         4 . The system of  claim 3 , wherein the pipeline generates the plurality of frames in accordance with a variable frame rate, and wherein the expected frame duration is set in accordance with a target frame rate. 
     
     
         5 . The system of  claim 2 , wherein the metric for a particular stage is calculated by multiplying a rate of the gain for the particular stage exceeding the second threshold value by a mean of the gain for the particular stage over a duration of the session. 
     
     
         6 . The system of  claim 1 , wherein the content comprises a frame of video rendered in response to a feedback signal received from the client device. 
     
     
         7 . The system of  claim 6 , wherein the frame of video is rendered in accordance with a ray-tracing algorithm. 
     
     
         8 . The system of  claim 6 , wherein the feedback signal includes at least one signal selected from the group consisting of:
 controller input; or   sensor data from a head-mounted display.   
     
     
         9 . The system of  claim 1 , wherein adjusting the parameter causes an encoding stage or a decoding stage of the pipeline to use a different encoding configuration or a different codec for encoding or decoding the content. 
     
     
         10 . The system of  claim 1 , wherein the information corresponding to each stage of the pipeline comprises a time stamp that indicates a time when the processing of the content by the stage is complete. 
     
     
         11 . The system of  claim 1 , wherein each stage includes one or more components and at least one stage includes a plurality of components, and wherein the metadata includes information corresponding to each stage of the pipeline, and the information for a particular stage of the pipeline comprises a time stamp for each component of the particular stage that indicates a time when the processing of the content by the component is complete. 
     
     
         12 . The system of  claim 1 , wherein the pipeline includes at least one stage selected from the group consisting of: a frame capture stage, an encoding stage, a forward error correction (FEC) stage, a transmit stage, a network stage, a decoding stage, a resynchronization stage, or a display stage. 
     
     
         13 . The system of  claim 1 , wherein the client device is configured to train a neural network for an autonomous vehicle, and wherein the server device is operable to generate simulated camera signals, representative of camera signals captured by the autonomous vehicle, in response to control inputs received from the autonomous vehicle. 
     
     
         14 . A method, comprising:
 receiving, at a server device from a client device, metadata corresponding to content processed by a plurality of stages of a pipeline, wherein the plurality of stages of the pipeline includes at least one stage implemented using the server device and at least one additional stage implemented using the client device;   generating, for two or more stages of the plurality of stages of the pipeline, a metric calculated based on the metadata; and   adjusting a parameter associated with at least one stage of the pipeline based on the metrics for the two or more stages.   
     
     
         15 . The method of  claim 14 , wherein the content comprises a plurality of frames streamed to the client device during a session, the method further comprising:
 calculating, for each frame of the plurality of frames, a gain for each stage of the pipeline;   comparing, for each frame of the plurality of frames, a total residual latency to a first threshold value to identify anomalous frames;   calculating, for each stage of the pipeline, an aggregate value by summing a number of anomalous frames in the plurality of frames where the gain corresponding to the stage in the anomalous frame is greater than a second threshold value; and   generating a visual representation based on the metrics, wherein the metric for a particular stage is calculated based on the aggregate value for the particular stage.   
     
     
         16 . The method of  claim 15 , wherein the gain for a particular stage is calculated based on an expected frame duration and a difference between a time stamp corresponding to the particular stage for the current frame and a time stamp corresponding to the particular stage for a previous frame. 
     
     
         17 . The method of  claim 15 , wherein the metric for a particular stage is calculated by multiplying a rate of the gain for the particular stage exceeding the second threshold value by a mean of the gain for the particular stage over a duration of the session. 
     
     
         18 . The method of  claim 14 , wherein the content comprises a frame of video rendered in response to a feedback signal received from the client device, and wherein the feedback signal includes at least one signal selected from the group consisting of:
 controller input; or   sensor data from a head-mounted display.   
     
     
         19 . The method of  claim 14 , wherein the information corresponding to each stage of the pipeline comprises a time stamp that indicates a time when the processing of the content by the stage is complete. 
     
     
         20 . A processor comprising:
 one or more processing units to receive, at a server device from a client device, metadata corresponding to content processed using a plurality of stages of a pipeline, generate a metric calculated based on the metadata for two or more stages of the pipeline, and adjust a parameter associated with at least one stage of the pipeline based on the metrics generated for the two or more stages, wherein the plurality of stages of the pipeline includes at least one stage implemented using the server device and at least one additional stage implemented using the client device.