Systems and methods for a computer understanding multi modal data streams

Abstract

Systems and methods for understanding (imputing meaning to) multi modal data streams may be used in intelligent surveillance and allow a) real-time integration of streaming data from video, audio, infrared and other sensors; b) processing of the results of such integration to obtain understanding of the situation as it unfolds; c) assessing the level of threat inherent in the situation; and d) generating of warning advisories delivered to appropriate recipients as necessary for mitigating the threat. The system generates understanding of the system by creating and manipulating models of the situation as it unfolds. The creation and manipulation involve “neuronal packets” formed in mutually constraining associative networks of four basic types. The process is thermodynamically driven, striving to produce a minimal number of maximally stable models. Obtaining such models is experienced as grasping, or understanding the input stream (objects, their relations and the flow of changes).

1 - 25 . (canceled) 
     
     
         26 . A method of obtaining understanding of multimodal data streams by dynamically optimizing allocation of neuronal resources to data elements in the stream wherein resources are drawn from neuronal pool on which four temporal constraints are defined: pool's longevity limit (life span), neuronal recuperation period, mobilization duration, and link decay period. 
     
     
         27 . The method of claim  25  wherein the survival minimum energy inflow is defined on the pool and resource optimization seeks to maintain energy inflow at or above the survival minimum for the duration of the life span. 
     
     
         28 . The method of claim  25  wherein dynamic resource optimization seeks to maximize internally generated negentropy in the neuronal pool computed as a function of the relative increase of the connected structure in the pool and decrease in the frequency of structure modifications in the course of allocations. (or as a function of the number and stability (degree of invariance) of the unifying models constructed in the pool). 
     
     
         29 . The method of claim  25  wherein optimization involves packet adjustments including a) selectively dissociating some neuronal packets from the rest of the virtual network, b) unfolding dissociated packets and re-distributing neurons, and c) enfolding the resulting packets. 
     
     
         30 . The method of claim  25  wherein models are adjusted by inserting/removing links and/or nodes in the networks, adjusting link weights, expanding/shrinking packets, other in order to improve the model's predictive and retrodictive performance and thus achieve a higher degree of situation understanding. 
     
     
         31 . The method of claim  25  wherein adjustment operations by the Control Module engage control neurons and involve inhibiting/activating individual neurons, changing response characteristics (probabilities) of individual neurons, re-distributing neurons between packets, re-distributing packets between entities. 
     
     
         32 . The method of claim  25  wherein formation of neuronal packets and coordination of packet firing (coordinated rotation of packet vectors) in the neuronal pool are subject to temporal constraints including:
 longevity limit θ 1  determining the time period during which neurons remain capable of firing (all firing ceases after θ 1 ); 
 recuperation period θ 2  (having fired, neuron can resume firing no sooner than after θ 2 ), 
 mobilization (recruitment) rate θ 3  (the average number of neurons or neuronal packets across the pool that can commence firing in a unit time); 
 connectivity decay period θ 4  (inter-neuron connection strength (weight) decays to zero in the absence of co-firing during θ 4 ). 
 
     
     
         33 . The method of claim  25  wherein the system for understanding multimodal data streams is applied for situational control of autonomous vehicles comprising a) processing sensor information from sensors mounted on the vehicle, b) applying information to form situation models defining objects in the vehicle surrounds and relations between the said objects comprising behavior of the objects and the mutual behavior constraints, c) applying the model to predict likely events resulting from behavior changes and d) computing control signals for the autonomous vehicle based on the said predictions. 
     
     
         34 . Neuronal pool, comprising groups of identical type neurons (minicolumns) such that
 a) neuronal packets are formed of neurons mobilized from different minicolumns;   b) functional distance in the neuronal space between neuronal packets A and B is determined by the degree of overlap accounting for the number of neuron types present in A but not in B, the number of neuron type present in B but not in A, and the number of neuron types present in both A and B; and   c) the degree of similarity between objects represented by neuronal packets A and B is determined by functional distance between A and B.