Video deblocking
Abstract
Video deblocking can be implemented in video content processing and delivery environments when displaying decompressed/decoded video information. Discontinuities are identified and smoothed to reduce blockiness, for example at macroblock boundaries in conjunction with Moving Picture Expert Group (MPEG)-based schemes. In exemplary implementation(s), a client device detects discontinuities at or near block boundaries, determines which discontinuities are artificial, and smooths at least those discontinuities that are determined to be artificial. These actions may be accomplished using (i) one or more Laplacian of Gaussian (LoG) operations as applied to both an image and a quantization matrix and (ii) windowed comparisons between values and a threshold resulting from the LoG operations. Alternatively, these actions may be accomplished using a spatio-temporally varying filter. Additionally, in an MPEG context, intra image units may be deblocked using the former approach while non-intra image units may be deblocked using the latter approach.
Claims
exact text as granted — not AI-modified1 . A method for deblocking video, comprising:
detecting one or more discontinuities in proximity to block boundaries of an image; determining whether any of the one or more discontinuities are artificial discontinuities using the image and a quantization matrix; and smoothing the one or more discontinuities that are determined to be artificial discontinuities from the action of determining.
2 . The method as recited in claim 1 , wherein the action of detecting one or more discontinuities in proximity to block boundaries of an image comprises:
detecting the one or more discontinuities in proximity to the block boundaries of the image by applying a Laplacian of Gaussian (LoG) operation on the image.
3 . The method as recited in claim 1 , wherein the action of determining whether any of the one or more discontinuities are artificial discontinuities using the image and a quantization matrix comprises:
determining whether any of the one or more discontinuities are artificial discontinuities using the image and a scaled quantization matrix.
4 . The method as recited in claim 1 , wherein the action of detecting one or more discontinuities in proximity to block boundaries of an image comprises:
detecting the one or more discontinuities in proximity to the block boundaries of the image by applying a Laplacian of Gaussian (LoG) operation on the image to produce a plurality of LoG image values; and wherein the action of determining whether any of the one or more discontinuities are artificial discontinuities using the image and a quantization matrix comprises:
applying the LoG operation on the quantization matrix to produce a plurality of LoG quantization matrix values;
determining a threshold responsive to the plurality of LoG quantization matrix values;
comparing at least a subset of the plurality of LoG image values to the threshold; and
labeling a discontinuity of the one or more discontinuities as an artificial discontinuity if a number of the at least a subset of the plurality of LoG image values that are greater than the threshold is not greater than a predetermined fraction.
5 . The method as recited in claim 1 , wherein the action of smoothing the one or more discontinuities that are determined to be artificial discontinuities from the action of determining comprises:
filtering the artificial discontinuities using at least one of a Gaussian filter and a tent filter.
6 . The method as recited in claim 1 , wherein the actions of detecting, determining, and smoothing are performed by a client device of a television-based entertainment environment.
7 . The method as recited in claim 1 , wherein the actions of detecting and determining are performed with respect to windows of a predetermined pixel width that surround the block boundaries of the image.
8 . The method as recited in claim 1 , further comprising:
determining whether any of the one or more discontinuities are real discontinuities using the image and the quantization matrix; and smoothing the one or more discontinuities that are determined to be real discontinuities.
9 . The method as recited in claim 8 , wherein the action of detecting one or more discontinuities in proximity to block boundaries of an image comprises:
detecting the one or more discontinuities in proximity to the block boundaries of the image by applying a Laplacian of Gaussian (LoG) operation on the image to produce a plurality of LoG image values; and wherein the action of determining whether any of the one or more discontinuities are real discontinuities using the image and the quantization matrix comprises:
applying the LoG operation on the quantization matrix to produce a plurality of LoG quantization matrix values;
determining a threshold responsive to the plurality of LoG quantization matrix values;
comparing at least a subset of the plurality of LoG image values to the threshold; and
labeling a discontinuity of the one or more discontinuities as a real discontinuity if a number of the at least a subset of the plurality of LoG image values that are greater than the threshold is greater than a predetermined fraction.
10 . The method as recited in claim 8 , wherein the action of smoothing the one or more discontinuities that are determined to be real discontinuities comprises:
visually adjusting the real discontinuities by moving pixel values toward each other.
11 . A method for providing video deblocking, comprising:
applying a Laplacian of Gaussian (LoG) operation on an image to produce a first plurality of values, the image being previously encoded using a quantization matrix; applying the LoG operation on the quantization matrix to produce a second plurality of values; determining a threshold responsive to the second plurality of values; comparing at least a portion of the first plurality of values to the threshold; ascertaining a number of the at least a portion of the first plurality of values that exceed the threshold; and if the number is less than a predetermined fraction, then labeling a discontinuity that is associated with the at least a portion of the first plurality of values as an artificial discontinuity.
12 . The method as recited in claim 11 , further comprising:
if the number is greater than the predetermined fraction, then labeling the discontinuity that is associated with the at least a portion of the first plurality of values as a real discontinuity.
13 . The method as recited in claim 12 , further comprising:
performing a visual adjustment procedure on the real discontinuity.
14 . The method as recited in claim 11 , wherein the at least a portion of the first plurality of values corresponds to pixels of the image that are proximate to at least one macroblock boundary.
15 . The method as recited in claim 11 , wherein the quantization matrix comprises a scaled quantization matrix.
16 . The method as recited in claim 11 , wherein the action of determining a threshold responsive to the second plurality of values comprises:
selecting a percentage of an absolute maximum value of the second plurality of values as the threshold.
17 . The method as recited in claim 11 , further comprising:
performing a filter smoothing procedure on the artificial discontinuity.
18 . One or more electronically-accessible media comprising electronically-executable instructions that, when executed, direct an electronic device to perform the method as recited in claim 11 .
19 . An arrangement capable of deblocking video, comprising:
detecting means for detecting one or more discontinuities at macroblock boundaries of a video image; determining means for determining whether the one or more discontinuities include one or more artificial discontinuities and whether the one or more discontinuities include one or more real discontinuities; and smoothing means for smoothing the one or more discontinuities in dependence on whether each discontinuity of the one or more discontinuities comprises an artificial discontinuity or a real discontinuity.
20 . The arrangement as recited in claim 19 , wherein the arrangement comprises a client device of a television-based entertainment environment.
21 . The arrangement as recited in claim 19 , wherein the detecting means utilizes a Laplacian of Gaussian (LoG) operation on the video image and the determining means utilizes the LoG operation on a quantization matrix.
22 . The arrangement as recited in claim 21 , wherein the determining means includes selecting means for selecting a discontinuity threshold from a plurality of results of the LoG operation on the quantization matrix.
23 . The arrangement as recited in claim 21 , wherein the quantization matrix comprises a scaled quantization matrix.
24 . The arrangement as recited in claim 19 , wherein the smoothing means includes visual adjustment means for visually adjusting each discontinuity that comprises a real discontinuity and filter smoothing means for filter smoothing each discontinuity that comprises an artificial discontinuity with at least one smoothing filter.
25 . The arrangement as recited in claim 24 , wherein the at least one smoothing filter comprises at least one of a tent filter and a Gaussian filter.
26 . A client device for a television-based entertainment environment, the client device configured to perform actions comprising:
detecting one or more discontinuities at macroblock boundaries of a video image; determining whether the one or more discontinuities include one or more artificial discontinuities and whether the one or more discontinuities include one or more real discontinuities; smoothing the one or more artificial discontinuities using a filter smoothing procedure; and smoothing the one or more real discontinuities using a visual adjustment procedure.
27 . The client device as recited in claim 26 , wherein the client device is further configured to detect the one or more discontinuities by applying a Laplacian of Gaussian (LoG) operation on the video image.
28 . The client device as recited in claim 26 , wherein the client device is further configured to determine whether the one or more discontinuities include one or more artificial discontinuities and whether the one or more discontinuities include one or more real discontinuities using a Laplacian of Gaussian (LoG) operation on a quantization matrix.
29 . One or more electronically-accessible media comprising instructions that, when executed, direct an electronic device to:
detect one or more discontinuities in proximity to block boundaries of an image, the image having been previously quantized using a quantization matrix; determine whether any of the one or more discontinuities are artificial discontinuities using the image and the quantization matrix; and smooth the one or more discontinuities that are determined to be artificial discontinuities using a filter smoothing procedure; wherein the instructions for the detection and the determination entail performing at least one Laplacian of Gaussian (LoG) operation.
30 . The one or more electronically-accessible media as recited in claim 29 , wherein the one or more electronically-accessible media comprise at least one memory of a client device of a television-based entertainment environment.
31 . The one or more electronically-accessible media as recited in claim 29 , wherein the quantization matrix comprises a scaled quantization matrix.
32 . The one or more electronically-accessible media as recited in claim 29 , further comprising instructions that, when executed, direct the electronic device to:
smooth one or more discontinuities that are real discontinuities using a visual adjustment procedure.
33 . A method for deblocking video, comprising:
detecting one or more discontinuities in proximity to block boundaries of an image; determining whether any of the one or more discontinuities are artificial discontinuities; and smoothing the one or more discontinuities that are determined to be artificial discontinuities; wherein the actions of detecting, determining, and smoothing are effectuated using a spatio-temporally varying filter.
34 . The method as recited in claim 33 , wherein the spatio-temporally varying filter smooths artificial discontinuities to a greater extent in the vicinity of the block boundaries.
35 . The method as recited in claim 33 , wherein the image is one image in a group of images; and wherein the spatio-temporally varying filter smooths artificial discontinuities to a greater extent in images that are relatively upstream in the group of images and smooths artificial discontinuities to a lesser extent in images that are relatively downstream in the group of images.
36 . The method as recited in claim 35 , wherein the image comprises a picture that has been compressed and decompressed in accordance with a Moving Pictures Expert Group (MPEG)-compliant scheme; and wherein the group of images comprises a group of pictures (GOP).
37 . The method as recited in claim 33 , wherein the spatio-temporally varying filter includes at least one temporal variable that is dependent on a group of pictures (GOP) position number (GPN).
38 . The method as recited in claim 33 , wherein the spatio-temporally varying filter includes a plurality of filter zones between two block boundaries of the image.
39 . A method for deblocking video, comprising:
ascertaining block boundaries of an image unit; ascertaining a position of the image unit in a group of images; and smoothing one or more discontinuities of the image unit responsive to individual locations of the one or more discontinuities relative to the block boundaries of the image unit and responsive to the position of the image unit in the group of images.
40 . The method as recited in claim 39 , wherein the image unit comprises at least one of a frame and a macroblock.
41 . The method as recited in claim 39 , wherein the action of smoothing one or more discontinuities of the image unit responsive to individual locations of the one or more discontinuities relative to the block boundaries of the image unit comprises smoothing the one or more discontinuities in increasing amounts as the individual locations of the one or more discontinuities approach the block boundaries.
42 . The method as recited in claim 39 , wherein the action of smoothing one or more discontinuities of the image unit responsive to the position of the image unit in the group of images comprises smoothing the one or more discontinuities in decreasing amounts as the position of the image unit in the group of images increases.
43 . The method as recited in claim 39 , wherein the actions of ascertaining and smoothing are performed by a client device of a television-based entertainment environment.
44 . One or more electronically-accessible media comprising electronically-executable instructions that, when executed, direct an electronic device to perform the method as recited in claim 39 .
45 . An arrangement that is capable of deblocking video, comprising:
ascertaining means for ascertaining block boundaries of an image unit and a position of the image unit in a group of images; and smoothing means for smoothing one or more discontinuities of the image unit (i) responsive to individual locations of the one or more discontinuities relative to the block boundaries of the image unit and (ii) responsive to the position of the image unit in the group of images.
46 . The arrangement as recited in claim 45 , wherein the ascertaining means comprises a Moving Pictures Expert Group (MPEG)-compliant decoder.
47 . The arrangement as recited in claim 45 , wherein the smoothing means comprises a spatial filtering means for spatial filtering that includes a plurality of filtering zones for application across the image unit; and wherein filtering zones of the plurality of filtering zones that overlap the block boundaries of the image unit filter more strongly than filtering zones of the plurality of filtering zones that are located away from the block boundaries.
48 . The arrangement as recited in claim 45 , wherein the smoothing means comprises a temporal filtering means for temporal filtering that includes a filtering modifier that varies depending on how far downstream the position of the image unit is from an intra image unit in the group of images.
49 . The arrangement as recited in claim 45 , wherein the arrangement comprises a client device for a television-based entertainment environment.
50 . A client device for a television-based entertainment environment, the client device configured to perform actions comprising:
spatially filtering an image having a plurality of blocks, each block of the plurality of blocks having a middle and at least two boundaries; the spatial filtering orchestrated such that filtering increases near the at least two boundaries and decreases near the middle of each block of the plurality of blocks; and modifying the spatial filtering using a temporally-varying variable, the temporally-varying variable established such that the spatial filtering is greater for images near the beginning of groups of images and reduced for images near the end of groups of images.
51 . The client device as recited in claim 50 , wherein the spatial filtering smooths at least artificial discontinuities.
52 . The client device as recited in claim 50 , wherein the temporally-varying variable may be determined from a formula using a group of images position number.
53 . The client device as recited in claim 50 , wherein the temporally-varying variable may be determined from a set of temporally-varying variables using a group of images position number.
54 . The client device as recited in claim 50 , wherein the client device is further configured to perform actions comprising:
receiving video data via an interface; and decoding the video data into video information that includes the image.
55 . The client device as recited in claim 50 , wherein the client device is further configured to perform an action comprising:
forwarding the image, which has been spatially filtered as modified by the temporally-varying variable, to a video screen.
56 . One or more electronically-accessible media comprising instructions that, when executed, direct an electronic device to:
spatially filter an image having a plurality of blocks, each block of the plurality of blocks having a middle and at least two boundaries; the spatial filtering orchestrated such that filtering increases near the at least two boundaries and decreases near the middle of each block of the plurality of blocks; and modify the spatial filtering using a temporally-varying variable, the temporally-varying variable established such that the spatial filtering is greater for images near the beginning of groups of images than for images near the end of groups of images.
57 . The one or more electronically-accessible media as recited in claim 56 , further comprising instructions that, when executed, direct the electronic device to:
apply the spatial filtering on discontinuities of the image.
58 . The one or more electronically-accessible media as recited in claim 56 , further comprising instructions that, when executed, direct the electronic device to:
receive video data via an interface; decode the video data into video information that includes the image; and forward the image, which has been spatially filtered as modified by the temporally-varying variable, to a video screen.
59 . A method for providing video deblocking, comprising:
detecting a discontinuity in proximity to a block boundary of video information; determining whether the discontinuity is an artificial discontinuity; and if so, smoothing the artificial discontinuity on both sides of the block boundary.
60 . The method as recited in claim 59 , wherein the action of detecting a discontinuity in proximity to a block boundary of video information comprises:
detecting one or more discontinuities at four different block boundaries of a block of the video information.
61 . The method as recited in claim 59 , wherein the action of detecting a discontinuity in proximity to a block boundary of video information comprises:
detecting one or more discontinuities in a window around the block boundary of a block of the video information.
62 . The method as recited in claim 59 , wherein the block boundary comprises a macroblock boundary; and
wherein the action of detecting a discontinuity in proximity to a block boundary of video information comprises:
detecting one or more discontinuities in proximity to the macroblock boundary of a macroblock of the video information.
63 . The method as recited in claim 59 , wherein the action of detecting a discontinuity in proximity to a block boundary of video information comprises:
applying a Laplacian of Gaussian (LoG) operation on an image of the video information.
64 . The method as recited in claim 59 , wherein the action of determining whether the discontinuity is an artificial discontinuity comprises:
applying a Laplacian of Gaussian (LoG) operation on a quantization matrix that has been used to de-quantize the video information.
65 . The method as recited in claim 59 , wherein the action of determining whether the discontinuity is an artificial discontinuity comprises:
applying a Laplacian of Gaussian (LoG) operation on a quantization matrix that is in the time domain.
66 . The method as recited in claim 59 , wherein the action of determining whether the discontinuity is an artificial discontinuity comprises:
applying a Laplacian of Gaussian (LoG) operation on a quantization matrix to produce a plurality of values; analyzing the plurality of values to determine a real discontinuity threshold responsive to at least one maximum absolute value of the plurality of values.
67 . The method as recited in claim 59 , wherein the action of determining whether the discontinuity is an artificial discontinuity comprises:
comparing a first plurality of values that result from applying a Laplacian of Gaussian (LoG) operation on an image of the video information to a threshold that is determined responsive to a second plurality of values that result from applying the LoG operation on a quantization matrix.
68 . The method as recited in claim 67 , wherein the quantization matrix comprises a scaled quantization matrix.
69 . The method as recited in claim 59 , wherein the action of determining whether the discontinuity is an artificial discontinuity comprises:
determining that the discontinuity is an artificial discontinuity if a predetermined fraction of a first plurality of values that result from applying a Laplacian of Gaussian (LoG) operation on an image of the video information are less than a threshold that is determined responsive to a second plurality of values that result from applying the LoG operation on a quantization matrix.
70 . The method as recited in claim 59 , wherein the action of determining whether the discontinuity is an artificial discontinuity comprises:
determining that the discontinuity is not an artificial discontinuity if a predetermined fraction of a first plurality of values that result from applying a Laplacian of Gaussian (LoG) operation on an image of the video information are greater than a threshold that is determined responsive to a second plurality of values that result from applying the LoG operation on a quantization matrix.
71 . The method as recited in claim 70 , further comprising:
visually adjusting the discontinuity that is not an artificial discontinuity.
72 . The method as recited in claim 59 , wherein the action of smoothing the artificial discontinuity on both sides of the block boundary comprises:
applying at least one smoothing filter to the artificial discontinuity.
73 . The method as recited in claim 72 , wherein the action of applying at least one smoothing filter to the artificial discontinuity comprises:
applying at least one of a Gaussian filter and a tent filter to the artificial discontinuity.
74 . The method as recited in claim 59 , further comprising:
decoding compressed video data to produce the video information.
75 . The method as recited in claim 59 , wherein the action of smoothing the artificial discontinuity on both sides of the block boundary comprises:
smoothing pixels of the video information; and further comprising:
forwarding the video information for display on a screen.
76 . The method as recited in claim 59 , wherein the actions of detecting, determining, and smoothing are performed by a client device of a television-based entertainment environment.
77 . The method as recited in claim 59 , wherein the actions of detecting, determining, and smoothing comprise applying a spatio-temporally varying filter.
78 . The method as recited in claim 59 , wherein the actions of detecting, determining, and smoothing comprise applying a spatio-temporally varying filter that varies spatially across a block of the video information and temporally along a group of pictures.
79 . The method as recited in claim 78 , wherein the spatial variation is such that filtering increases near block boundaries of the block and decreases near the middle of the block.
80 . The method as recited in claim 78 , wherein the temporal variation is such that filtering decreases as pictures progress downstream along the group of pictures.
81 . The method as recited in claim 59 , wherein, when an intra image unit of the video information is under consideration,
the action of detecting comprises applying a Laplacian of Gaussian (LoG) operation on the intra image unit to produce a first plurality of values; the action of determining comprises comparing the first plurality of values to a threshold that is determined responsive to a second plurality of values that result from applying the LoG operation on a quantization matrix; and the action of smoothing comprises applying at least one of a Gaussian filter and a tent filter to the artificial discontinuity on both sides of the block boundary.
82 . The method as recited in claim 59 , wherein, when a non-intra image unit of the video information is under consideration,
the actions of detecting, determining, and smoothing comprise applying a spatio-temporally varying filter that varies spatially across the non-intra image unit and temporally along a group of images of the video information.
83 . One or more electronically-accessible media comprising electronically-executable instructions that, when executed, direct an electronic device to perform the method as recited in claim 59 .
84 . A method for deblocking video, comprising:
determining whether an image unit of video information is an intra image unit or a non-intra image unit, the image unit having at least two boundaries; if the image unit is determined to be an intra image unit, then performing actions comprising:
detecting one or more discontinuities at the at least two boundaries of the image unit;
determining whether the one or more discontinuities include one or more artificial discontinuities and whether the one or more discontinuities include one or more real discontinuities;
smoothing the one or more artificial discontinuities, if any, using a filter smoothing procedure; and
smoothing the one or more real discontinuities, if any, using a visual adjustment procedure; and
if the image unit is determined to be a non-intra image unit, then performing actions comprising:
ascertaining a position of the image unit in a group of images; and
smoothing one or more discontinuities of the image unit responsive (i) to individual locations of the one or more discontinuities relative to the at least two boundaries of the image unit and (ii) to the position of the image unit in the group of images.
85 . The method as recited in claim 84 , wherein the image unit comprises at least one of a frame and a macroblock.
86 . The method as recited in claim 84 , wherein the method is performed by a client device of a television-based entertainment environment.
87 . The method as recited in claim 84 , wherein the group of images comprises a group of pictures (GOP) in accordance with a Moving Pictures Expert Group (MPEG)-compliant encoding/decoding scheme.
88 . The method as recited in claim 84 , wherein the action of determining whether the one or more discontinuities include one or more artificial discontinuities and whether the one or more discontinuities include one or more real discontinuities comprises applying a Laplacian of Gaussian (LoG) operation on a quantization matrix, the quantization matrix previously being used to quantize the image unit.
89 . The method as recited in claim 84 , wherein the action of smoothing one or more discontinuities of the image unit responsive (i) to individual locations of the one or more discontinuities relative to the at least two boundaries of the image unit and (ii) to the position of the image unit in the group of images comprises smoothing the one or more discontinuities to a greater extent the closer the individual locations are to a boundary of the at least two boundaries and to a lesser extent the further the non-intra image unit is from an immediately-previous intra image unit.
90 . The method as recited in claim 84 , wherein the non-intra image unit comprises at least one of a predicted image unit and a bi-directional image unit.
91 . A method for deblocking video, comprising:
analyzing intra macroblocks of video information to detect one or more discontinuities and to determine whether the one or more discontinuities include one or more artificial discontinuities and whether the one or more discontinuities include one or more real discontinuities; and applying a spatio-temporally varying filter to non-intra macroblocks of the video information.
92 . The method as recited in claim 91 , further comprising:
determining whether a macroblock is an intra macroblock or a non-intra macroblock based on whether a frame that contains the macroblock is an intra frame or a non-intra frame, respectively.
93 . The method as recited in claim 91 , further comprising:
determining whether a macroblock is an intra macroblock or a non-intra macroblock on a macroblock-by-macroblock basis.
94 . The method as recited in claim 91 , further comprising:
smoothing the one or more artificial discontinuities, if any, using a filter smoothing procedure.
95 . The method as recited in claim 91 , further comprising:
smoothing the one or more real discontinuities, if any, using a visual adjustment procedure.
96 . The method as recited in claim 91 , wherein the action of analyzing intra macroblocks of video information comprises:
detecting the one or more discontinuities by using a Laplacian of Gaussian (LoG) operation on an image of the video information.
97 . The method as recited in claim 91 , wherein the action of analyzing intra macroblocks of video information comprises:
determining whether the one or more discontinuities include one or more artificial discontinuities and whether the one or more discontinuities include one or more real discontinuities by using a Laplacian of Gaussian (LoG) operation on a quantization matrix, the quantization matrix previously used to quantize an image of the video information.
98 . The method as recited in claim 97 , wherein the action of determining whether the one or more discontinuities include one or more artificial discontinuities and whether the one or more discontinuities include one or more real discontinuities by using a Laplacian of Gaussian (LoG) operation on a quantization matrix comprises:
determining whether the one or more discontinuities include one or more real discontinuities by using the LoG operation on a scaled quantization matrix.
99 . The method as recited in claim 91 , wherein the action of applying a spatio-temporally varying filter comprises:
increasing filtering for pixels proximate to macroblock boundaries of the non-intra macroblocks of the video information; and decreasing filtering for pixels distant from the macroblock boundaries of the non-intra macroblocks of the video information.Join the waitlist — get patent alerts
Track US2003235250A1 — get alerts on status changes and closely related new filings.
We store only your email — no account needed. See our privacy policy.