Sensor-Action Fusion System for Optimising Sensor Measurement Collection from Multiple Sensors

Abstract

The embodiments described herein aim to improve environmental sensing by providing a computationally efficient and accurate means for fusing sensor data and using this fused data to control sensors to focus on areas that would most reduce the uncertainty in the sensing system. In this way, the system can direct sensors to focus on the most important areas and features within the environment in order to provide the most effective sensor data (e.g. for use by a control system). The methods described herein make use of multi-agent sensor-action fusion. The methods are multi-agent in that a set of machine learning agents are trained in order to control the sensors to focus on the most important features and regions. The embodiments implement sensor-action fusion in that sensor fusion is performed in order to obtain a combined view of the environment and this combined view is utilised to determine the most appropriate actions.

1 . A method for controlling the configuration of one or more sensors based on information shared between a plurality of sensors, the method comprising:
 establishing a hierarchical network of nodes comprising at least a first level comprising a plurality of child nodes and a second level comprising one or more parent nodes, wherein each of the child nodes is assigned to a corresponding sensor and each of the one or more parent nodes is assigned to a corresponding group of child nodes to combine sensor data from the corresponding group of child nodes;   at each parent node:
 receiving, from each of the child nodes in the corresponding group of child nodes for the parent node, sensor data for the sensor corresponding to that child node, the sensor data occupying a corresponding sensor feature space for the child node; 
 encoding the received sensor data to form an encoded combination of sensor data by mapping the received sensor data to a latent space for the parent node; 
 decoding the encoded combination of sensor data to map, for each of one or more of the child nodes of the corresponding group, the encoded combination of sensor data to the sensor feature space for the child node to form a corresponding decoded combination of sensor data; and 
 sending each decoded combination of sensor data to the child node corresponding to the sensor feature space for that decoded combination of sensor data; and 
   at each child node that receives a decoded combination of sensor data:
 determining an action for updating a configuration of the corresponding sensor based on the received decoded combination of sensor data; and 
 issuing an instruction to adjust the configuration of the corresponding sensor in accordance with the action. 
   
     
     
         2 . The method of  claim 1  further comprising at each child node:
 receiving one or more sensor measurements from the sensor corresponding to the child node; 
 encoding the one or more sensor measurements to compress the one or more sensor measurements by mapping the one or more sensor measurements onto a sensor feature space for the child node to form the sensor data, the sensor feature space being a latent space; and 
 sending the sensor data to the parent node corresponding to the child node. 
 
     
     
         3 . The method of  claim 1  wherein each child node that receives a decoded combination of sensor data implements an agent for determining the action that is biased towards selecting an action that achieves one or more of reducing prediction error or focusing on one or more predefined features of interest. 
     
     
         4 . The method of  claim 3  wherein each child node that receives a decoded combination of sensor data implements a classifier configured to identify the one or more predefined features of interest within the corresponding decoded combination of sensor data and bias the agent towards an action that focuses on the one or more predefined features of interest. 
     
     
         5 . The method of  claim 3  wherein each agent:
 determines predicted sensor data based on the decoded combination of sensor data; 
 determines a prediction error based on the predicted sensor data; and 
 is biased towards an action that minimises a cost function comprising the prediction error. 
 
     
     
         6 . The method of  claim 5  wherein each agent:
 determines predicted sensor data based on the combination of sensor data, determines the prediction error based on the predicted sensor data and determines a gradient of the prediction error over the action space for the node; and 
 is biased towards determining an action that minimises a cost function comprising the gradient of the prediction error. 
 
     
     
         7 . The method of  claim 1  wherein the action comprises one or more of:
 adjusting a resolution of the corresponding sensor; 
 adjusting a focus of the corresponding sensor; and 
 directing the corresponding sensors to sense an updated region. 
 
     
     
         8 . The method of  claim 7  wherein each child node is implemented in a corresponding processor connected directly to the corresponding sensor for that child node. 
     
     
         9 . The method of  claim 8  wherein the second level is implemented in a second set of one or more processors and wherein the processors for the first level communicate the sensor data to the one or more processors for the second level. 
     
     
         10 . The method of  claim 1  further comprising:
 at each child node that receives a decoded combination of sensor data, determining a bandwidth action to adjust a size of the sensor feature space for the child node based on the received decoded combination of sensor data and adjusting the size of the sensor feature space in accordance with the action. 
 
     
     
         11 . The method of  claim 10  wherein the bandwidth action is biased towards a bandwidth action that reduces the size of the sensor feature space but is biased away from an action that increases a prediction error for the child node 
     
     
         12 . A node for controlling the configuration of a sensor based on information shared between a plurality of sensors, the node comprising a processor configured to:
 receive one or more sensor measurements from the sensor;   encode the one or more sensor measurements to compress the one or more sensor measurements by mapping the one or more sensor measurements onto latent space for the node to form encoded sensor data;   send the encoded sensor data to a parent node for combination with further encoded sensor data from one or more other sensors of the plurality of sensors;   receive from the parent node a combination of sensor data comprising a combination of the encoded sensor data for the node and the further encoded sensor data from the one or more other sensors mapped to the latent space of the node;   determine an action for updating a configuration of the sensor based on the combination of sensor data; and   issue an instruction to adjust the configuration of the sensor in accordance with the action.   
     
     
         13 . The node of  claim 12  wherein the node is biased towards selecting an action that achieves one or more of reducing prediction error or focusing on one or more predefined features of interest. 
     
     
         14 . The node of  claim 13  wherein the processor is configured to implement a classifier configured to identify one or more predefined features of interest within the combination of sensor data and bias the node towards an action that focuses on the one or more predefined features of interest. 
     
     
         15 . The node of  claim 13  wherein:
 the processor is configured to determine predicted sensor data based on the combination of sensor data and determine the prediction error based on the predicted sensor data; and 
 the processor is biased towards selecting an action that minimises a cost function comprising the prediction error. 
 
     
     
         16 . The node of  claim 13  wherein:
 the processor is configured to determine predicted sensor data based on the combination of sensor data, determine the prediction error based on the predicted sensor data and determine a gradient of the prediction error over the action space for the node; and 
 the processor is biased towards determining an action that minimises a cost function comprising the gradient of the prediction error. 
 
     
     
         17 . The node of  claim 12  wherein the action comprises one or more of:
 adjusting a resolution of the sensor; 
 adjusting a focus of the sensor; and 
 directing the sensor to sense an updated region. 
 
     
     
         18 . The node of  claim 12  wherein the processor is configured to be connected directly to the sensor for receiving the sensor data. 
     
     
         19 . The node of  claim 12  wherein the processor is configured to determine a bandwidth action for adjusting the size of the latent space based on the combination of sensor data, wherein the processor is biased towards a bandwidth action that reduces the size of the latent space but is biased away from an action that increases a prediction error for the node. 
     
     
         20 . The node of  claim 12  wherein the action is determined using a reinforcement learning agent in accordance with parameters of the agent and wherein the processor is configured to update the parameters of the agent to reduce a cost function based on one or more of a prediction error, a gradient of the prediction error over an action space for the node, and a weighting towards one or more predefined features of interest. 
     
     
         21 . A parent node for combining sensor data from multiple sensors for use by one or more child nodes in controlling the configuration of one or more of the sensors, the parent node comprising a processor configured to:
 receive, from each of a group of child nodes, sensor data for a sensor corresponding to the child node, the sensor data occupying a corresponding sensor feature space for the child node;   encode the received sensor data to form an encoded combination of sensor data by mapping the received sensor data to a latent space for the parent node;   decode the encoded combination of sensor data to map, for each of one or more of the child nodes of the group, the encoded combination of sensor data to the sensor feature space for the child node to form a corresponding decoded combination of sensor data; and   send each decoded combination of sensor data to the child node corresponding to the sensor feature space for that decoded combination of sensor data to enable the child node to determine and issue an action for updating a configuration of the corresponding sensor for that child node based on the corresponding combination of sensor data.   
     
     
         22 . A computing system comprising one or more processors configured to implement the method of  claim 1 . 
     
     
         23 . A non-transitory computer readable medium comprising computer executable instructions that, when executed by a processor, cause the processor to implement the method of  claim 1 .