Memory management for video decoding

Abstract

Techniques and tools described herein help manage memory efficiently during video decoding, especially when multiple video clips are concurrently decoded. For example, with clip-adaptive memory usage, a decoder determines first memory usage settings expected to be sufficient for decoding of a video clip. The decoder also determines second memory usage settings known to be sufficient for decoding of the clip. During decoding, memory usage is initially set according to the first settings. Memory usage is adaptively increased during decoding, subject to theoretical limits in the second settings. With adaptive early release of side information, the decoder can release side information memory for a picture earlier than the decoder releases image plane memory for the picture. The decoder can also adapt memory usage for decoded transform coefficients depending on whether the coefficients are for intra-coded blocks or inter-coded blocks, and also exploit the relative sparseness of non-zero coefficient values.

1 .- 17 . (canceled) 
     
     
         18 . One or more tangible computer-readable media storing computer-executable instructions for causing a computing system programmed thereby to perform video decoding operations comprising:
 receiving at least part of a bitstream for a video clip, wherein the video clip includes at least one picture encoded using at least some inter-picture coding;   determining first memory usage settings for decoding intra-coded content of the at least one picture;   determining second memory usage settings for decoding inter-coded content of the at least one picture; and   decoding the video clip, including adapting memory usage based at least in part on the first memory usage settings and the second memory usage settings.   
     
     
         19 . The one or more computer-readable media of  claim 18 , wherein the decoding the video clip further includes:
 allocating a first buffer according to the first memory usage settings;   in the first buffer, storing entropy decoded transform coefficients for intra-coded blocks of a given picture of the at least one picture;   allocating a second buffer according to the second memory usage settings, wherein the first buffer is smaller than the second buffer; and   in the second buffer, storing entropy decoded transform coefficients for inter-coded blocks of the given picture.   
     
     
         20 . The one or more computer-readable media of  claim 19 , wherein the adapting memory usage includes, during the decoding the video clip:
 dynamically growing size for the first buffer by adding to one or more fragment buffers for the first buffer; and   dynamically growing size for the second buffer by adding to one or more fragment buffers for the second buffer.   
     
     
         21 . The one or more computer-readable media of  claim 19 , wherein the first buffer and the second buffer are allocated from heap memory. 
     
     
         22 . The one or more computer-readable media of  claim 19 , wherein the first buffer is allocated from a first memory pool, and wherein the second buffer is allocated from a second memory pool different than the first memory pool. 
     
     
         23 . The one or more computer-readable media of  claim 18 , wherein the first memory usage settings include a first buffer size, and wherein the second memory usage settings include a second buffer size different than the first buffer size. 
     
     
         24 . The one or more computer-readable media of  claim 18 , wherein the decoding the video clip further includes:
 allocating a fragment buffer for a first buffer;   allocating a fragment buffer for a second buffer;   in the fragment buffer for the first buffer, storing entropy decoded transform coefficients for intra-coded blocks, including storing a position, absolute value, and sign per non-zero coefficient, but not storing any zero coefficient values, among the entropy decoded transform coefficients for the intra-coded blocks; and   in the fragment buffer for the second buffer, storing entropy decoded transform coefficients for inter-coded blocks, including storing a position, absolute value, and sign per non-zero coefficient, but not storing any zero coefficient values, among the entropy decoded transform coefficients for the inter-coded blocks.   
     
     
         25 . The one or more computer-readable media of  claim 24 , wherein size of the fragment buffer for the first buffer is same as size of the fragment buffer for the second buffer. 
     
     
         26 . The one or more computer-readable media of  claim 24 , wherein the adapting memory usage includes:
 checking whether buffer size is sufficient for the first buffer and, if not, dynamically adding another fragment buffer to the first buffer; and   checking whether buffer size is sufficient for the second buffer and, if not, dynamically adding another fragment buffer to the second buffer.   
     
     
         27 . A computer system comprising:
 a buffer configured to store at least part of a bitstream for a video clip, wherein the video clip includes at least one picture encoded using at least some inter-picture coding;   a video decoder configured to:
 determine first memory usage settings for decoding intra-coded content of the at least one picture; 
 determine second memory usage settings for decoding inter-coded content of the at least one picture; and 
 decode the video clip, wherein the video decoder is configured to adapt memory usage based at least in part on the first memory usage settings and the second memory usage settings. 
   
     
     
         28 . The computer system of  claim 27 , wherein the video decoder is further configured to:
 allocate a first buffer according to the first memory usage settings;   store, in the first buffer, entropy decoded transform coefficients for intra-coded blocks of a given picture of the at least one picture;   allocate a second buffer according to the second memory usage settings, wherein the first buffer is smaller than the second buffer; and   store, in the second buffer, entropy decoded transform coefficients for inter-coded blocks of the given picture.   
     
     
         29 . The computer system of  claim 28 , wherein the video decoder is configured to allocate the first buffer and the second buffer from heap memory. 
     
     
         30 . The computer system of  claim 28 , wherein, to adapt memory usage, the video decoder is configured to:
 dynamically grow size for the first buffer by adding to one or more fragment buffers for the first buffer; and   dynamically grow size for the second buffer by adding to one or more fragment buffers for the second buffer.   
     
     
         31 . The computer system of  claim 27 , wherein the video decoder is further configured to:
 allocate a fragment buffer for a first buffer;   allocate a fragment buffer for a second buffer;   store, in the fragment buffer for the first buffer, entropy decoded transform coefficients for intra-coded blocks, including storing a position, absolute value, and sign per non-zero coefficient, but not storing any zero coefficient values, among the entropy decoded transform coefficients for the intra-coded blocks; and   store, in the fragment buffer for the second buffer, entropy decoded transform coefficients for inter-coded blocks, including storing a position, absolute value, and sign per non-zero coefficient, but not storing any zero coefficient values, among the entropy decoded transform coefficients for the inter-coded blocks.   
     
     
         32 . The computer system of  claim 31 , wherein, to adapt memory usage, the video decoder is configured to:
 check whether buffer size is sufficient for the first buffer and, if not, dynamically add another fragment buffer to the first buffer; and   check whether buffer size is sufficient for the second buffer and, if not, dynamically add another fragment buffer to the second buffer.   
     
     
         33 . In a computer system that implements a video decoder, a method comprising:
 receiving at least part of a bitstream for a video clip, wherein the video clip includes at least one picture encoded using at least some inter-picture coding;   determining first memory usage settings for decoding intra-coded content of the at least one picture;   determining second memory usage settings for decoding inter-coded content of the at least one picture; and   decoding the video clip, including adapting memory usage based at least in part on the first memory usage settings and the second memory usage settings.   
     
     
         34 . The method of  claim 33 , wherein the decoding the video clip further includes:
 allocating a first buffer according to the first memory usage settings;   in the first buffer, storing entropy decoded transform coefficients for intra-coded blocks of a given picture of the at least one picture;   allocating a second buffer according to the second memory usage settings, wherein the first buffer is smaller than the second buffer; and   in the second buffer, storing entropy decoded transform coefficients for inter-coded blocks of the given picture;   and wherein the adapting memory usage includes, during the decoding the video clip:
 dynamically growing size for the first buffer by adding to one or more fragment buffers for the first buffer; and 
 dynamically growing size for the second buffer by adding to one or more fragment buffers for the second buffer. 
   
     
     
         35 . The method of  claim 33 , wherein the decoding the video clip further includes:
 allocating a fragment buffer for a first buffer;   allocating a fragment buffer for a second buffer;   in the fragment buffer for the first buffer, storing entropy decoded transform coefficients for intra-coded blocks, including storing a position, absolute value, and sign per non-zero coefficient, but not storing any zero coefficient values, among the entropy decoded transform coefficients for the intra-coded blocks; and   in the fragment buffer for the second buffer, storing entropy decoded transform coefficients for inter-coded blocks, including storing a position, absolute value, and sign per non-zero coefficient, but not storing any zero coefficient values, among the entropy decoded transform coefficients for the inter-coded blocks;   and wherein the adapting memory usage includes:
 checking whether buffer size is sufficient for the first buffer and, if not, dynamically adding another fragment buffer to the first buffer; and 
 checking whether buffer size is sufficient for the second buffer and, if not, dynamically adding another fragment buffer to the second buffer. 
   
     
     
         36 . The method of  claim 33 , wherein:
 determining third memory usage settings different than the first memory usage settings and the second memory usage settings, the third memory usage settings indicating one or more theoretical limits on memory usage according to a standard or format specification for decoding the video clip; and   during the decoding the video clip, adapting memory usage based at least in part on the third memory usage settings, wherein the memory usage is initially set according to the first memory usage settings and the second memory usage settings, and wherein the memory usage is increased during the decoding subject to the one or more theoretical limits in the third memory usage settings.   
     
     
         37 . The method of  claim 33 , wherein the decoding comprises, for each of the at least one picture:
 allocating first memory for storing side information associated with the picture;   allocating second memory for storing reconstructed image plane information associated with the picture;   upon determination that the picture is no longer to be used for reference in motion compensation during subsequent decoding, releasing the first memory; and   upon deletion of the picture from a decoded picture buffer, releasing the second memory.