Multimedia processing chip, electronic device, and method for dynamic-image processing

Abstract

A multimedia processing chip, an electronic device, and a method for dynamic-image processing are provided. The multimedia processing chip includes an image signal processor (ISP) and a neural network processing unit (NPU). The ISP is configured to collect status information of image data. The NPU is configured to process the image data with a neural network algorithm. The multimedia processing chip is configured to perform pre-processing on the image data and transmits the status information and the pre-processed image data to an application processing chip.

What is claimed is: 
     
         1 . A multimedia processing chip, comprising:
 an image signal processor (ISP) configured to collect status information of image data; and   a neural-network processing unit (NPU) configured to process the image data with a neural-network algorithm,   at least one of the ISP and the NPU being configured to perform pre-processing on the image data, and the multimedia processing chip is configured to transmit the status information and the image data subjected to the pre-processing to an application processing (AP) chip.   
     
     
         2 . The multimedia processing chip of  claim 1 , wherein the pre-processing comprises a first pre-processing and a second pre-processing, and wherein:
 the ISP is configured to perform the first pre-processing on the image data; and   the NPU is configured to perform the second pre-processing on the image data subjected to the first pre-processing.   
     
     
         3 . The multimedia processing chip of  claim 2 , wherein the pre-processing further comprises bit-width adjustment, and the ISP is configured to:
 perform the bit-width adjustment on the image data subjected to the second pre-processing, such that a bit width of the image data subjected to the bit-width adjustment is the same as a bit width for processing image data by the AP chip.   
     
     
         4 . The multimedia processing chip of  claim 2 , wherein the first pre-processing comprises at least one of bad pixel correction (BPC), linearization, or black level correction (BLC). 
     
     
         5 . The multimedia processing chip of  claim 4 , wherein the first pre-processing further comprises image cropping and/or image down-scaling. 
     
     
         6 . The multimedia processing chip of  claim 1 , wherein the pre-processing comprises bit-width adjustment, and the ISP is configured to perform the bit-width adjustment on the image data subjected to the neural-network algorithm, such that a bit width of the image data subjected to the bit-width adjustment is the same as a bit width for processing image data by the AP chip. 
     
     
         7 . The multimedia processing chip of  claim 1 , wherein the image data comprises dynamic-image data, wherein the neural-network algorithm used by the NPU to process the dynamic-image data comprises at least one of:
 a nightscape algorithm, a high-dynamic range (HDR) algorithm, a blur algorithm, a denoising algorithm, a super-resolution algorithm, or a semantic segmentation algorithm.   
     
     
         8 . The multimedia processing chip of  claim 7 , wherein the multimedia processing chip is configured to process the dynamic-image data in real time, and transmit the dynamic-image data processed to the AP chip in real time. 
     
     
         9 . The multimedia processing chip of  claim 1 , wherein the image data comprises static-image data, wherein the neural-network algorithm used by the NPU to process the static-image data comprises at least one of:
 a nightscape algorithm, an HDR algorithm, a blur algorithm, a denoising algorithm, a super-resolution algorithm, or a semantic segmentation algorithm.   
     
     
         10 . The multimedia processing chip of  claim 1 , wherein the multimedia processing chip is configured to process the image data offline, wherein the image data comprises static-image data or dynamic-image data. 
     
     
         11 . The multimedia processing chip of  claim 1 , wherein the image data is raw image data. 
     
     
         12 . The multimedia processing chip of  claim 1 , wherein the status information comprises at least one of automatic exposure (AE) status information, automatic white balance (AWB) status information, or automatic focus (AF) status information. 
     
     
         13 . The multimedia processing chip of  claim 12 , wherein the status information further comprises lens shade correction (LSC) status information. 
     
     
         14 . An electronic device, comprising:
 a multimedia processing chip comprising:
 an image signal processor (ISP) configured to collect status information of image data; and 
 a neural-network processing unit (NPU) configured to process the image data with a neural-network algorithm, 
 at least one of the ISP and the NPU being configured to perform pre-processing on the image data, and the multimedia processing chip being configured to transmit the status information and the image data subjected to the pre-processing to an application processing (AP) chip; and 
   wherein the AP chip configured to obtain, from the multimedia processing chip, a result of the pre-processing and the status information collected, and perform post-processing on the result of the pre-processing based on the status information.   
     
     
         15 . The electronic device of  claim 14 , wherein the status information comprises at least one of automatic focus (AF) status information, automatic white balance (AWB) status information, or automatic exposure (AE) status information, and the AP chip is configured to:
 calculate a focus parameter based on the AF status information and configure the focus parameter to a camera of the electronic device;   calculate a white-balance parameter based on the AWB status information and perform white balance on the result of the pre-processing based on the white-balance parameter; and   calculate an exposure parameter based on the AE status information, and configure the exposure parameter to the camera of the electronic device, or configure the exposure parameter after compensation to the camera of the electronic device.   
     
     
         16 . The electronic device of  claim 15 , wherein the AF status information comprises phase detection auto focus (PDAF) status information and contrast detection auto focus (CDAF) status information, and
 wherein the ISP of the multimedia processing chip is configured to:
 obtain the CDAF status information by processing the image data with a preset algorithm; and 
 extract the PDAF status information from the image data, and 
   the AP chip is further configured to:
 calculate a CDAF parameter based on the CDAF status information, and 
   configure the CDAF parameter to the camera of the electronic device; and
 calculate a PDAF parameter based on the PDAF status information, and configure the PDAF parameter to the camera of the electronic device. 
   
     
     
         17 . The electronic device of  claim 15 , wherein the status information further comprises lens shade correction (LSC) status information, and the AP chip is further configured to:
 calculate an LSC parameter based on the LSC status information, and perform lens shading correction on the result of the pre-processing based on the LSC parameter.   
     
     
         18 . A method for dynamic-image processing, comprising:
 obtaining, by a multimedia processing chip, dynamic-image data;   collecting, by the multimedia processing chip, status information of the dynamic-image data;   performing, by the multimedia processing chip, pre-processing on the dynamic-image data according to the dynamic-image data;   transmitting, by the multimedia processing chip to an AP chip, the status information collected and the image data subjected to the pre-processing by the multimedia processing chip; and   performing according to the status information, by the AP chip, post-processing on the dynamic-image data subjected to the pre-processing.   
     
     
         19 . The method of  claim 18 , wherein performing the pre-processing on the dynamic-image data comprises:
 optimizing the dynamic-image data; and   processing, with a neural-network algorithm, the dynamic-image data subjected to the optimizing.   
     
     
         20 . The method of  claim 18 , wherein the status information comprises at least one of automatic focus (AF) status information, automatic white balance (AWB) status information, or automatic exposure (AE) status information, and performing according to the status information, by the AP chip, the post-processing on the dynamic-image data subjected to the pre-processing comprises:
 calculating a focus parameter based on the AF status information and configuring the focus parameter to a camera;   calculating a white-balance parameter based on the AWB status information and performing white balance on a result of the pre-processing based on the white-balance parameter; and   calculating an exposure parameter based on the AE status information, and configuring the exposure parameter to the camera, or configuring the exposure parameter after compensation to the camera.