Learning method, learning device for detecting objectness by detecting bottom lines and top lines of nearest obstacles and testing method, testing device using the same

Abstract

A method for learning parameters of CNNs capable of identifying objectnesses by detecting bottom lines and top lines of nearest obstacles in an input image is provided. The method includes steps of: a learning device, (a) instructing a first CNN to generate first encoded feature maps and first decoded feature maps, and instructing a second CNN to generate second encoded feature maps and second decoded feature maps; (b) generating first and second obstacle segmentation results respectively representing where the bottom lines and the top lines are estimated as being located per each column, by referring to the first and the second decoded feature maps respectively; (c) estimating the objectnesses by referring to the first and the second obstacle segmentation results; (d) generating losses by referring to the objectnesses and their corresponding GTs; and (f) backpropagating the losses, to thereby learn the parameters of the CNNs.

What is claimed is: 
     
       1. A method for learning parameters of CNNs capable of identifying one or more objectnesses by detecting one or more bottom lines and one or more top lines of one or more nearest obstacles in at least one input image, comprising steps of:
 (a) a learning device, if the input image is acquired as a training image, (i) instructing one or more convolutional layers of a first CNN to sequentially generate one or more first encoded feature maps from the input image, and instructing one or more deconvolutional layers of the first CNN to sequentially generate one or more first decoded feature maps from the first encoded feature maps, and (ii) instructing one or more convolutional layers of a second CNN to sequentially generate one or more second encoded feature maps from the input image, and instructing one or more deconvolutional layers of the second CNN to sequentially generate one or more second decoded feature maps from the second encoded feature maps; 
 (b) the learning device, on condition that each cell of a grid with a plurality of rows and a plurality of columns is generated by dividing the first and the second decoded feature maps, with respect to a first direction and a second direction, wherein the first direction is in a direction of the rows of the first and the second decoded feature maps and the second direction in a direction of the columns thereof, (i) generating at least one first obstacle segmentation result representing each of first specific rows, where each of the bottom lines of each of the nearest obstacles is estimated as being located per each of the columns, by referring to at least one feature of at least one specific first decoded feature map among the first decoded feature maps, and (ii) generating at least one second obstacle segmentation result representing each of second specific rows, where each of the top lines of each of the nearest obstacles is estimated as being located per each of the columns, by referring to at least one feature of at least one specific second decoded feature map among the second decoded feature maps; 
 (c) the learning device estimating the objectnesses corresponding to the nearest obstacles on a road by referring to the first obstacle segmentation result and the second obstacle segmentation result; 
 (d) the learning device generating one or more losses by referring to the objectnesses and their corresponding GTs wherein each of the GTs includes information on locations of first exact rows where each of the bottom lines of the nearest obstacles is truly positioned per each of the columns, and on locations of second exact rows where each of the top lines of the nearest obstacles is truly positioned per each of the columns; and 
 (f) the learning device backpropagating the losses, to thereby learn the parameters of the CNN. 
 
     
     
       2. The method of  claim 1 , wherein, at the step of (c), the learning device further estimates one or more boundaries of the road by referring to the first obstacle segmentation result and the second obstacle segmentation result, and
 wherein, at the step of (d), the learning device generates the losses by referring the boundaries, the objectnesses, their corresponding GTs. 
 
     
     
       3. The method of  claim 2 , wherein, at the step of (c), if each difference of each distance between (i) each of the first specific rows where each of the bottom lines of each of the obstacles is estimated as located per each of the columns and (ii) each of the second specific rows where each of the top lines of each of the obstacles is estimated as located per each of the columns is detected as less than a predetermined threshold by referring to the first obstacle segmentation result and the second obstacle segmentation result, the learning device determines that the boundaries are present in the input image, and
 wherein if each difference of each distance between (i) each of the first specific rows per each of the columns and (ii) each of the second specific rows per each of the columns is detected as larger than the predetermined threshold, the learning device determines that the objectnesses corresponding to the nearest obstacles are present on the road in the input image. 
 
     
     
       4. The method of  claim 1 , wherein the step of (b) includes steps of:
 (b1) the learning device, supposing that each cell of the grid is generated by dividing the first and the second decoded feature maps with respect to the first direction by first intervals and with respect to the second direction by second intervals, concatenating each of features of each of the rows per each of the columns in a direction of a channel, to thereby generate at least one first reshaped feature map and at least one second reshaped feature map; and 
 (b2) the learning device (i) generating the first obstacle segmentation result by checking each of estimated positions of each of the bottom lines of each of the obstacles on concatenated channels for each of the columns on the first reshaped feature map, wherein the first obstacle segmentation result generated by a first softmax operation which normalizes each value corresponding to each channel per each of the columns, and (ii) generating the second obstacle segmentation result by checking each of estimated positions of each of the top lines of each of the obstacles starting upward from each of the bottom lines on concatenated channels for each of the columns on the second reshaped feature map, wherein the second obstacle segmentation result is generated by a second softmax operation which normalizes each value corresponding to each channel per each of the columns. 
 
     
     
       5. The method of  claim 1 , wherein each of the columns includes one or more pixels in the first direction, and each of the rows includes one or more pixels in the second direction. 
     
     
       6. The method of  claim 1 , wherein the GTs include information representing on which row each of the bottom lines and each of the top lines of each of the nearest obstacles are truly located among the rows, per each of the columns, resulting from dividing the input image into N c  rows, and wherein the first obstacle segmentation result and the second obstacle segmentation result include information representing on which row each of the bottom lines and each of the top lines of each of the obstacles are estimated as located among the rows, per each of the columns, resulting from dividing the input image into N c  rows. 
     
     
       7. The method of  claim 1 , wherein, at the step of (b), a third obstacle segmentation result is generated such that the third obstacle segmentation result includes information on the first and the second specific rows with top two highest values for each of the columns of concatenated features, wherein the concatenated features are generated by using features of the first and the second decoded feature maps, and wherein the third obstacle segmentation result is generated by a third softmax operation which normalizes each value corresponding to each of the rows per each of the columns of the concatenated features. 
     
     
       8. A method for testing CNNs capable of identifying one or more objectnesses for training by detecting one or more bottom lines and one or more top lines of one or more nearest obstacles for testing in at least one test image, comprising steps of:
 (a) a testing device acquiring the test image, on condition that a learning device has performed processes of (i) instructing one or more convolutional layers of a first CNN to sequentially generate one or more first encoded feature maps for training from at least one input image acquired as a training image, and instructing one or more deconvolutional layers of the first CNN to sequentially generate one or more first decoded feature maps for training from the first encoded feature maps for training, and instructing one or more convolutional layers of a second CNN to sequentially generate one or more second encoded feature maps for training from the input image, and instructing one or more deconvolutional layers of the second CNN to sequentially generate one or more second decoded feature maps for training from the second encoded feature maps for training, (ii) assuming that each cell of a grid with a plurality of rows and a plurality of columns is generated by dividing the first and the second decoded feature maps for training, with respect to a first direction and a second direction, wherein the first direction is in a direction of the rows of the first and the second decoded feature maps for training and the second direction is in a direction of the columns thereof, generating at least one first obstacle segmentation result for training representing each of first specific rows for training, where each of the bottom lines of each of one more nearest obstacles for training is estimated as being located per each of the columns, by referring to at least one feature for training of at least one specific first decoded feature map for training among the first decoded feature maps for training, and generating at least one second obstacle segmentation result for training representing each of second specific rows for training, where each of the top lines of each of the nearest obstacles for training is estimated as being located per each of the columns, by referring to at least one feature for training of at least one specific second decoded feature map for training among the second decoded feature maps for training, (iii) estimating the objectnesses for training corresponding to the nearest obstacles for training on a road by referring to the first obstacle segmentation result for training and the second obstacle segmentation result for training, (iv) generating one or more losses by referring to the objectnesses for training and their corresponding GTs wherein each of the GTs includes information on locations of first exact rows where each of the bottom lines of the nearest obstacles for training is truly positioned per each of the columns, and on locations of second exact rows where each of the top lines of the nearest obstacles for training is truly positioned per each of the columns, and (v) backpropagating the losses, to thereby learn the parameters of the CNNs; 
 (b) the testing device (i) instructing the convolutional layers of the first CNN to sequentially generate one or more first encoded feature. maps for testing from the test image, and instructing the deconvolutional layers of the first CNN to sequentially generate one or more first decoded feature maps for testing from the first encoded feature maps for testing, and (ii) instructing the convolutional layers of the second CNN to sequentially generate one or more second encoded feature maps for testing from the test image, and instructing the deconvolutional layers of the second CNN to sequentially generate one or more second decoded feature maps for testing from the second encoded feature maps for testing; 
 (c) the testing device, assuming that each cell of a grid with a plurality of rows and a plurality of columns is generated by dividing the first and the second decoded feature maps for testing, with respect to a first direction and a second direction, wherein the first direction is in a direction of the rows of the first and the second decoded feature maps for testing and the second direction is in a direction of the columns thereof, (i) generating at least one first obstacle segmentation result for testing representing each of first specific rows for testing, where each of the bottom lines of each of the nearest obstacles for testing is estimated as being located per each of the columns, by referring to at least one feature for testing of at least one specific first decoded feature map for testing among the first decoded feature maps for testing, and (ii) generating at least one second obstacle segmentation result for testing representing each of second specific rows for testing, where each of the top lines of each of the nearest obstacles for testing is estimated as being located per each of the columns, by referring to at least one feature for testing of at least one specific second decoded feature map for testing among the second decoded feature maps for testing; and 
 (d) the testing device estimating the objectnesses for testing corresponding to the nearest obstacles for testing on a road by referring to the first obstacle segmentation result for testing and the second obstacle segmentation result for testing. 
 
     
     
       9. The method of  claim 8 , wherein, at the step of (d), the testing device further estimates one or more boundaries for testing of the road by referring to the first obstacle segmentation result for testing and the second obstacle segmentation result for testing, and
 wherein, at the process of (iv), the learning device has generated the losses by referring to one or more boundaries for training, the objectnesses for training, and their corresponding GTs. 
 
     
     
       10. The method of  claim 9 , wherein, at the step of (d), if each difference of each distance between (i) each of the first specific rows for testing where each of the bottom lines of each of the obstacles for testing is estimated as located per each of the columns and (ii) each of the second specific rows for testing where each of the top lines of each of the obstacles for testing is estimated as located per each of the columns is detected as less than a predetermined threshold by referring to the first obstacle segmentation result for testing and the second obstacle segmentation result for testing, the testing device determines that the boundaries for testing are present in the test image, and
 wherein, if each difference of each distance between (i) each of the first specific rows for testing per each of the columns and (ii) each of the second specific rows for testing per each of the columns is detected as larger than the predetermined threshold, the testing device determines that the objectnesses for testing corresponding to the nearest obstacles for testing are present on the road in the test image. 
 
     
     
       11. The method of  claim 8 , wherein the step of (c) includes steps of:
 (c1) the testing device, supposing that each cell of the grid is generated by the first and the second decoded feature maps for testing with respect to the first direction by first intervals and with respect the second direction by second intervals, concatenating each of features for testing of each of the rows per each of the columns in a direction of a channel, to thereby generate at least one first reshaped feature map for testing and at least one second reshaped feature map for testing; and 
 (c2) the testing device (i) generating the first obstacle segmentation result for testing by checking each of estimated positions of each of the bottom lines of each of the obstacles for testing on concatenated channels for each of the columns on the first reshaped feature map for testing, wherein the first obstacle segmentation result for testing is generated by a first softmax operation which normalizes each value corresponding to each channel per each of the columns, and (ii) generating the second obstacle segmentation result for testing by checking each of estimated positions of each of the top lines of each of the obstacles for testing starting upward from each of the bottom lines on concatenated channels for each of the columns on the second reshaped feature map for testing, wherein the second obstacle segmentation result for testing is generated by a second softmax operation which normalizes each value corresponding to each channel per each of the columns. 
 
     
     
       12. The method of  claim 8 , wherein each of the columns includes one or more pixels in the first direction, and each of the rows includes one or more pixels in the second direction. 
     
     
       13. The method of  claim 8 , wherein the GTs include information representing on which row each of the bottom lines and each of the top lines of each of the nearest obstacles for training are truly located among the rows, per each of the columns, resulting from dividing the input image into N c  rows, and
 wherein the first obstacle segmentation result for testing and the second obstacle segmentation result for testing include information representing on which row each of the bottom lines and each of the top lines of each of the obstacles for testing are estimated as located among the rows, per each of the columns, resulting from dividing the test image into N c  rows. 
 
     
     
       14. The method of  claim 8 , wherein, at the step of (c), a third obstacle segmentation result for testing generated such that the third obstacle segmentation result for testing includes information on the first and the second specific rows for testing with top two highest values for each of the columns of concatenated features for testing, wherein the concatenated features for testing are generated by using features for testing of the first and the second decoded feature maps for testing, and wherein the third obstacle segmentation result for testing is generated by a third softmax operation which normalizes each value corresponding to each of the rows per each of the columns of the concatenated features for testing. 
     
     
       15. A learning device for learning parameters of CNNs capable of identifying one or more objectnesses by detecting one or more bottom lines and one or more top lines of one or more nearest obstacles in at least one input image, comprising:
 a communication part for acquiring the input image as a training image; and 
 a processor for performing processes of (I) (i) instructing one or more convolutional layers of a first CNN to sequentially generate one or more first encoded feature maps from the input image, and instructing one or more deconvolutional layers of the first CNN to sequentially generate one or more first decoded feature maps from the first encoded feature maps, and (ii) instructing one or more convolutional layers of a second CNN to sequentially generate one or more second encoded feature maps from the input image, and instructing one or more deconvolutional layers of the second CNN to sequentially generate one or more second decoded feature maps from the second encoded feature maps, (II) on condition that each cell of a grid with a plurality of rows and a plurality of columns is generated by dividing the first and the second decoded feature maps, with respect to a first direction and a second direction, wherein the first direction is in a direction of the rows of the first and the second decoded feature maps and the second direction is in a direction of the columns thereof, (i) generating at least one first obstacle segmentation result representing each of first specific rows, where each of the bottom lines of each of the nearest obstacles is estimated as being located per each of the columns, by referring to at least one feature of at least one specific first decoded feature map among the first decoded feature maps, and (ii) generating at least one second obstacle segmentation result representing each of second specific rows, where each of the top lines of each of the nearest obstacles is estimated as being located per each of the columns, by referring to at least one feature of at least one specific second decoded feature map among the second decoded feature maps, (III) estimating the objectnesses corresponding to the nearest obstacles on a road by referring to the first obstacle segmentation result and the second obstacle segmentation result, (IV) generating one or more losses by referring to the objectnesses and their corresponding GTs wherein each of the GTs includes information on locations of first exact rows where each of the bottom lines of the nearest obstacles is truly positioned per each of the columns, and on locations of second exact rows where each of the top lines of the nearest obstacles is truly positioned per each of the columns, and (V) backpropagating the losses, to thereby learn the parameters of the CNN. 
 
     
     
       16. The learning device of  claim 15 , wherein, at the process of (III), the processor further estimates one or more boundaries of the road by referring to the first obstacle segmentation result and the second obstacle segmentation result, and
 wherein, at the process of (IV), the processor generates the losses by referring to the boundaries, the objectnesses, and their corresponding GTs. 
 
     
     
       17. The learning device of  claim 16 , wherein, at the process of (III), if each difference of each distance between (i) each of the first specific rows where each of the bottom lines of each of the obstacles is estimated as located per each of the columns and (ii) each of the second specific rows where each of the top lines of each of the obstacles is estimated as located per each of the columns is detected as less than a predetermined threshold by referring to the first obstacle segmentation result and the second obstacle segmentation result, the processor determines that the boundaries are present in the input image, and
 wherein, if each difference of each distance between (i) each of the first specific rows per each of the columns and (ii) each of the second specific rows per each of the columns is detected as larger than the predetermined threshold, the processor determines that the objectnesses corresponding to the nearest obstacles are present on the road in the input image. 
 
     
     
       18. The learning device of  claim 15 , wherein the process of (II) includes processes of:
 (II-1) supposing that each cell of the grid is generated by dividing the first and the second decoded feature maps with respect to the first direction by first intervals and with respect to the second direction by second intervals, concatenating each of features of each of the rows per each of the columns in a direction of a channel, to thereby generate at least one first reshaped feature map and at least one second reshaped feature map; and 
 (II-2) (i) generating the first obstacle segmentation result by checking each of estimated positions of each of the bottom lines of each of the obstacles on concatenated channels for each of the columns on the first reshaped feature map, wherein the first obstacle segmentation result is generated by a first softmax operation which normalizes each value corresponding to each channel per each of the columns, and (ii) generating the second obstacle segmentation result by checking each of estimated positions of each of the top lines of each of the obstacles starting upward from each of the bottom lines on concatenated channels for each of the columns on the second reshaped feature map, wherein the second obstacle segmentation result is generated by a second softmax operation which normalizes each value corresponding to each channel per each of the columns. 
 
     
     
       19. The learning device of  claim 15 , wherein each of the columns includes one or more pixels in the first direction, and each of the rows includes one or more pixels in the second direction. 
     
     
       20. The learning device of  claim 15 , wherein the GTs include information representing on which row each of the bottom lines and each of the top lines of each of the nearest obstacles are truly located among the rows, per each of the columns, resulting from dividing the input image into N c  rows, and wherein the first obstacle segmentation result and the second obstacle segmentation result include information representing on which row each of the bottom lines and each of the top lines of each of the obstacles are estimated as located among the rows, per each of the columns, resulting from dividing the input image into N c  rows. 
     
     
       21. The learning device of  claim 15 , wherein, at the process of (II), a third obstacle segmentation result is generated such that the third obstacle segmentation result includes information on the first and the second specific rows with top two highest values for each of the columns of concatenated features, wherein the concatenated features are generated by using features of the first and the second decoded feature maps, and wherein the third obstacle segmentation result is generated by a third softmax operation which normalizes each value corresponding to each of the rows per each of the columns of the concatenated features. 
     
     
       22. A testing device for testing CNNs capable of identifying one or more objectnesses for training by detecting one or more bottom lines and one or more top lines of one or more nearest obstacles for testing in at least one test image, comprising:
 a communication part for acquiring the test image, on condition that a learning device has performed processes of (1) instructing one or more convolutional layers of a first CNN to sequentially generate one or more first encoded feature maps for training from at least one input image acquired as a training image, and instructing one or more deconvolutional layers of the first CNN to sequentially generate one or more first decoded feature maps for training from the first encoded feature maps for training, and instructing one or more convolutional layers of a second CNN to sequentially generate one or more second encoded feature maps for training from the input image, and instructing one or more deconvolutional layers of the second CNN to sequentially generate one or more second decoded feature maps for training from the second encoded feature maps for training, (2) assuming that each cell of a grid with a plurality of rows and a plurality of columns is generated by dividing the first and the second decoded feature maps for training, with respect to a first direction and a second direction, wherein the first direction is in a direction of the rows of the first and the second decoded feature maps for training and the second direction is in a direction of the columns thereof, generating at least one first obstacle segmentation result for training representing each of first specific rows for training, where each of the bottom lines of each of one or more nearest obstacles for training is estimated as being located per each of the columns, by referring to at least one feature for training of at least one specific first decoded feature map for training among the first decoded feature maps for training, and generating at least one second obstacle segmentation result for training representing each of second specific rows for training, where each of the top lines of each of the nearest obstacles for training is estimated as being located per each of the columns, by referring to at least one feature for training of at least one specific second decoded feature map for training among the second decoded feature maps for training, (3) estimating the objectnesses for training corresponding to the nearest obstacles for training on a road by referring to the first obstacle segmentation result for training and the second obstacle segmentation result for training, (4) generating one or more losses by referring to the objectnesses for training and their corresponding GTs wherein each of the GTs includes information on locations of first exact rows where each of the bottom lines of the nearest obstacles for training is truly positioned per each of the columns, and on locations of second exact rows where each of the top lines of the nearest obstacles for training is truly positioned per each of the columns, and (5) backpropagating the losses, to thereby learn the parameters of the CNNs; and 
 a processor for performing processes of (I) (i) instructing the convolutional layers of the first CNN to sequentially generate one or more first encoded feature maps for testing from the test image, and instructing the deconvolutional layers of the first CNN to sequentially generate one or more first decoded feature maps for testing from the first encoded feature maps for testing, and (ii) instructing the convolutional layers of the second CNN to sequentially generate one or more second encoded feature maps for testing from the test image, and instructing the deconvolutional layers of the second CNN to sequentially generate one or more second decoded feature maps for testing from the second encoded feature maps for testing, (II) assuming that each cell of a grid with a plurality of rows and a plurality of columns is generated by dividing the first and the second decoded feature maps for testing, with respect to a first direction and a second direction, wherein the first direction is in a direction of the rows of the first and the second decoded feature maps for testing and the second direction is it a direction of the columns thereof, (i) generating at least one first obstacle segmentation result for testing representing each of first specific rows for testing, where each of the bottom lines of each of the nearest obstacles for testing is estimated as being located per each of the columns, by referring to at least one feature for testing of at least one specific first decoded feature map for testing among the first decoded feature maps for testing, and (ii) generating at least one second obstacle segmentation result for testing representing each of second specific rows for testing, where each of the top lines of each of the nearest obstacles for testing is estimated as being located per each of the columns, by referring to at least one feature for testing of at least one specific second decoded feature map for testing among the second decoded feature maps for testing, and (III) estimating the objectnesses for testing corresponding to the nearest obstacles for testing on a road by referring to the first obstacle segmentation result for testing and the second obstacle segmentation result for testing. 
 
     
     
       23. The testing device of  claim 22 , wherein, at the process of (III), the processor further estimates one or more boundaries for testing of the road by referring to the first obstacle segmentation result for testing and the second obstacle segmentation result for testing, and
 wherein, at the process of (4), the learning device has generated the losses by referring to one or more boundaries for training, the objectnesses for training, and their corresponding GTs. 
 
     
     
       24. The testing device of  claim 23 , wherein at the process of (III), if each difference of each distance between (i) each of the first specific rows for testing where each of the bottom lines of each of the obstacles for testing is estimated as located per each of the columns and (ii) each of the second specific rows for testing where each of the top lines of each of the obstacles for testing is estimated as located per each of the columns is detected as less than a predetermined threshold by referring to the first obstacle segmentation result for testing and the second obstacle segmentation result for testing, the processor determines that the boundaries for testing are present in the test image, and
 wherein, if each difference of each distance between (i) each of the first specific rows for testing per each of the columns and (ii) each of the second specific rows for testing per each the columns is detected as larger than the predetermined threshold, the processor determines that the objectnesses for testing corresponding to the nearest obstacles for testing are present on the road in the test image. 
 
     
     
       25. The testing device of  claim 22 , wherein the process of (II) includes processes of:
 (II-1) supposing that each cell of the grid is generated by dividing the first and the second decoded feature maps for testing with respect to the first direction by first intervals and with respect to the second direction by second intervals, concatenating each of features for testing of each of the rows per each of the columns in a direction of a channel, to thereby generate at least one first reshaped feature map for testing and at least one second reshaped feature map for testing; and 
 (II-2) (i) generating the first obstacle segmentation result for testing by checking each of estimated positions of each of the bottom lines of each of the obstacles for testing on concatenated channels for each of the columns on the first reshaped feature map for testing, wherein the first obstacle segmentation result for testing is generated by a first softmax operation which normalizes each value corresponding to each channel per each of the columns, and (ii) generating the second obstacle segmentation result for testing by checking each of estimated positions of each of the top lines of each of the obstacles for testing starting upward from each of the bottom lines on concatenated channels for each of the columns on the second reshaped feature map for testing, wherein the second obstacle segmentation result for testing is generated by a second softmax operation which normalizes each value corresponding to each channel per each of the columns. 
 
     
     
       26. The testing device of  claim 22 , wherein each of the columns includes one or more pixels in the first direction, and each of the rows includes one or more pixels in the second direction. 
     
     
       27. The testing device of  claim 22 , wherein the GTs include information representing on which row each of the bottom lines and each of the top lines of each of the nearest obstacles for training are truly located among the rows, per each of the columns, resulting from dividing the input image into N c  rows, and
 wherein the first obstacle segmentation result for testing and the second obstacle segmentation result for testing include information representing on which row each of the bottom lines and each of the top lines of each of the obstacles for testing are estimated as located among the rows, per each of the columns, resulting from dividing the test image into N c  rows. 
 
     
     
       28. The testing device of  claim 22 , wherein, at the process of (II), a third obstacle segmentation result for testing is generated such that the third obstacle segmentation result for testing includes information on the first and the second specific rows for testing with top two highest values for each of the columns of concatenated features for testing, wherein the concatenated features for testing are generated by using features for testing of the first and the second decoded feature maps for testing, and wherein the third obstacle segmentation result for testing is generated by a third softmax operation which normalizes each value corresponding to each of the rows per each of the columns of the concatenated features for testing.