Intelligent Digital Camera having Deep Learning Accelerator and Random Access Memory

Abstract

Systems, devices, and methods related to a Deep Learning Accelerator and memory are described. For example, a digital camera may be configured to execute instructions with matrix operands and configured with: a housing; a lens; an image sensor positioned behind the lens to generate image data of a field of view of the digital camera; random access memory to store instructions executable by the Deep Learning Accelerator and store matrices of an Artificial Neural Network; a transceiver; and a controller configured to generate, and communicate using the transceiver to a separate computer, a description of an item or event in the field of view captured in the image data, based on an output of the Artificial Neural Network receiving the image data as an input. The separate computer may selectively request a portion of image data from the digital camera based on the processing of the description.

What is claimed is: 
     
         1 . A device, comprising:
 an image sensor;   random access memory;   at least one processing unit operable to execute instructions and configured, via first data representative of attributes of an artificial neural network and second data representative of the instructions executable by the at least one processing unit, to implement matrix computations of the artificial neural network;   a controller coupled with the image sensor and the random access memory and configured to:
 write image data generated by the image sensor into the random access memory as an input to the artificial neural network, causing the at least one processing unit to implement the matrix computations of the artificial neural network to generate an output of the artificial neural network responsive to the input; and 
 generate, based on the output of the artificial neural network, third data representative of a description of an item or event captured in the image data. 
   
     
     
         2 . The device of  claim 1 , further comprising:
 a transceiver operable to communicate with a computer system, wherein the controller is further configured to communicate to the computer system, using the transceiver, the third data representative of the description.   
     
     
         3 . The device of  claim 1 , further comprising:
 an integrated circuit package configured to enclose the image sensor, the random access memory, the at least one processing unit and the controller.   
     
     
         4 . The device of  claim 3 , wherein the instructions are configured with matrix operands. 
     
     
         5 . The device of  claim 4 , wherein the output of the artificial neural network includes an identification, classification or category of an object, person, or feature, and a location and size of the object, person, or feature; and the description is based on the identification, classification or category, and based on the location and size. 
     
     
         6 . The device of  claim 5 , wherein the output of the artificial neural network includes an identification of an event associated with the object, person, or feature; and the description includes the identification of the event. 
     
     
         7 . The device of  claim 6 , wherein the controller is further configured to generate representative images of the object, person, or feature, extracted based on the output of the artificial neural network. 
     
     
         8 . The device of  claim 7 , wherein the controller is configured to provide the description together with the representative images. 
     
     
         9 . The device of  claim 7 , wherein the controller is configured to provide the representative images in response to a request to the device regarding the description. 
     
     
         10 . The device of  claim 7 , wherein the controller is configured to selectively store image data based on the output of the artificial neural network. 
     
     
         11 . A method, comprising:
 configuring a device, having at least one processing unit operable to execute instructions to implement matrix computations of an artificial neural network, by storing in a memory:
 first data representative of attributes of the artificial neural network; and 
 second data representative of the instructions executable by the at least one processing unit; 
   generating, by an image sensor of the device, image data;   writing, by a controller coupled with the image sensor in the device, the image data into the memory as an input to the artificial neural network, causing the at least one processing unit to execute the instructions;   executing, by the at least one processing unit the instructions to implement the matrix computations of the artificial neural network and generate an output of the artificial neural network responsive to the input; and   generating, based on the output of the artificial neural network, third data representative of a description of an item or event captured in the image data.   
     
     
         12 . The method of  claim 11 , further comprising:
 communicating, by a transceiver of the device, the third data representative of the description to a computer system;   wherein the instructions are configured with matrix operands.   
     
     
         13 . The method of  claim 12 , wherein the output of the artificial neural network includes an identification, classification or category of an object, person, or feature, and a location and size of the object, person, or feature; and the description is based on the identification, classification or category, and based on the location and size. 
     
     
         14 . The method of  claim 13 , wherein the output of the artificial neural network includes an identification of an event associated with the object, person, or feature; and the description includes the identification of the event. 
     
     
         15 . The method of  claim 14 , further comprising:
 generating, by the controller, representative images of the object, person, or feature, extracted based on the output of the artificial neural network.   
     
     
         16 . The method of  claim 15 , further comprising:
 providing the description together with the representative images; or   providing the representative images in response to a request to the device regarding the description; or   any combination thereof.   
     
     
         17 . The method of  claim 15 , further comprising:
 selecting to store image data based on the output of the artificial neural network.   
     
     
         18 . An apparatus, comprising:
 a lens;   an image sensor configured behind the lens to generate image data;   memory configured to store data of an artificial neural network;   at least one processing unit configured to execute instructions configured to implement computations of the artificial neural network having the data stored in the memory; and   wherein the image sensor is configured to write the image data into the memory as an input to the artificial neural network;   wherein the at least one processing unit is configured to execute the instructions to implement the computations of the artificial neural network to generate an output responsive to the input; and   wherein the apparatus is configured to generate a description of an item or event captured in the image data.   
     
     
         19 . The apparatus of  claim 18 , wherein the instructions are configured with matrix operands; and the apparatus further comprises:
 a transceiver, wherein the apparatus is configured to provide the description to a computer system using the transceiver.   
     
     
         20 . The apparatus of  claim 18 , wherein the memory is a random access memory includes non-volatile memory configured to store the data of the artificial neural network and the instructions.