System and method for providing a decision basis for controlling a robotic arm, computer program and non-volatile data carrier

Abstract

A system and method for providing a decision basis for controlling a robotic arm to perform at least one action relating to a milk-producing animal, wherein a camera registers three-dimensional image data of a milking location that includes a reference object and a rotating platform upon which an animal stands with its hind legs facing the camera, and a control unit checks if an entry window for a robotic arm can be found in the image data by searching for the reference object in the image data, and if the reference object is found the control unit searches for an acceptable obstacle-free volume in the image data, this volume being located within an allowed space relative to the reference object.

The invention claimed is: 
     
       1. A system for controlling a robotic arm to perform at least one action relating to a milk-producing animal ( 100 ), the system comprising:
 a camera ( 110 ) configured to register three-dimensional image data (D img3D ) representing a milking location comprising a rotating platform ( 130 ) that receives said animal ( 100 ) such that the animal stands with its hind legs (LH, RH) facing the camera ( 110 ); and 
 a control unit ( 120 ) comprising at least one processor and a memory unit, the memory unit having stored therein instructions that, when executed by the processor, cause the processor to receive the image data (D img3D ) from the camera ( 110 ), process the image data (D img3D ), and control the robotic arm, 
 wherein the milking location also comprises a reference object (R 1 ) that forms part of a structure arranged on a stationary part ( 131 ) of the milking location separate from the rotating platform ( 130 ), said structure being configured to prevent the hind legs (LH, RH) from reaching outside of a safety zone for the milking location, and 
 wherein the at least one processor in the control unit ( 120 ) is programmed to:
 determine whether an entry window for the robotic arm is in the image data (D img3D ) by first searching for the reference object (R 1 ) in the image data (D img3D ), 
 if the reference object (R 1 ) is found in the image data (D img3D ), search for an acceptable obstacle-free volume (V) in the image data (D img3D ), where said acceptable obstacle-free volume (V) is defined as:
 being located within an allowed space (S) relative to the reference object (R 1 ), 
 comprising a continuous area (A) that faces the camera ( 110 ) and exceeds a threshold size, and 
 starting at a predefined position (P) and extending at least a primary distance (d OK ) in a depth direction away from the camera ( 110 ), and 
 
 if the acceptable obstacle-free volume (V) is found in the image data (D img3D ), control the robotic arm to perform at least one action relating to a milk-producing animal ( 100 ). 
 
 
     
     
       2. The system according to  claim 1 , wherein the at least one processor in the control unit ( 120 ) is further programmed so that if the reference object (R 1 ) is not found in the image data (D img3D ), the at least one processor obtains information about the reference object (R 1 ) from stored image data (D img3D ) representing the milking location, said stored image data (D img3D ) registered by the camera ( 110 ) at an earlier occasion when the milking location was positioned in front of the camera ( 110 ). 
     
     
       3. The system according to  claim 1 , wherein:
 the reference object (R 1 ) comprises at least a part of a physical entity extending laterally in parallel with a plane of rotation for the rotating platform ( 130 ); 
 the camera ( 110 ) is arranged with each of a position, an orientation, and a field of view such that the three-dimensional image data (D img3D ), in a lateral direction, comprises a width of at least one milking stall and, in a longitudinal direction, comprises the allowed space (S) and the reference object (R 1 ); and 
 the reference object (R 1 ) extends across a central part of the field of view in the lateral direction. 
 
     
     
       4. The system according to  claim 1 , wherein, upon said second searching finding the acceptable obstacle-free volume (V) in the image data (D img3D ), the at least one processor of the control unit ( 120 ) is programmed to:
 check whether at least one contour (C 1 , C 2 , C 3 ) delimiting the continuous area (A) fulfills at least one shape requirement, and 
 upon fulfilling the at least one shape requirement, classify said acceptable obstacle-free volume (V) as the entry window for the robotic arm. 
 
     
     
       5. The system according to  claim 4 , wherein, upon said second searching finding the acceptable obstacle-free volume (V) in the image data (D img3D ), the at least one processor of the control unit ( 120 ) is programmed to:
 search for an udder (U) of said animal ( 100 ) in the allowed space (S) of the image data (D img3D ) using the reference object (R 1 ) as a first boundary condition and using the rotating platform ( 130 ) as a second boundary condition. 
 
     
     
       6. The system according to  claim 4 , wherein a first shape requirement of the at least one shape requirement defines a maximally acceptable degree of jaggedness of the at least one contour (C 1 , C 2 , C 3 ). 
     
     
       7. The system according to  claim 6 , wherein a second shape requirement of the at least one shape requirement defines a maximally acceptable degree of irregularity of the continuous area (A). 
     
     
       8. The system according to  claim 4 , wherein a first shape requirement of the at least one shape requirement defines a maximally acceptable degree of irregularity of the continuous area (A). 
     
     
       9. The system according to  claim 1 , wherein the searching for said acceptable obstacle-free volume (V) comprises filtering out information in the image data (D img3D ) which information represents objects located closer to the camera ( 110 ) than a first threshold distance (d 1 ) and farther away from the camera ( 110 ) than a second threshold distance (d 2 ), the first and second threshold distances (d 1 , d 2 ) being separated from one another by the primary distance (d OK ). 
     
     
       10. The system according to  claim 9 , wherein the predefined position (P) is located at the first threshold distance (d 1 ) from the camera ( 110 ). 
     
     
       11. The system according to  claim 10 , wherein the first threshold distance (d 1 ) is zero. 
     
     
       12. The system according to  claim 1 , wherein the primary distance (d OK ) is defined in relation to at least one measure of the rotating platform ( 130 ). 
     
     
       13. The system according to  claim 1 , wherein the searching for said acceptable obstacle-free volume (V) comprises filtering out information in the image data (D img3D ), said information representing intensities of light outside of an acceptable range of intensity. 
     
     
       14. The system according to  claim 1 ,
 wherein the milking location further comprises at least one auxiliary reference object (R 11 , R 2 ) of the milking location whose position is known relative to the position of the camera ( 110 ), and 
 the at least one processor in the control unit ( 120 ) is programmed to search for the at least one auxiliary reference object (R 11 , R 2 ) in the image data (D img3D ) when performing the checking as to whether the entry window is in the image data (D img3D ). 
 
     
     
       15. A method for controlling a robotic arm to perform at least one action relating to a milk-producing animal ( 100 ), the method comprising:
 registering, via a camera ( 110 ), three-dimensional image data (D img3D ) representing a milking location comprising a rotating platform ( 130 ) that receives said animal ( 100 ) such that the animal stands with its hind legs (LH, RH) facing the camera ( 110 ), the milking location also comprising a reference object (R 1 ) that forms part of a structure arranged on a stationary part ( 131 ) of the milking location separate from the rotating platform ( 130 ), said structure being configured to prevent the hind legs (LH, RH) from reaching outside of a safety zone for the milking location; 
 determining whether an entry window for the robotic arm is in the image data (D img3D ) by first searching for the reference object (R 1 ) in the image data (D img3D ), 
 if the reference object (R 1 ) is found in the image data (D img3D ), searching for an acceptable obstacle-free volume (V) in the image data (D img3D ), where said acceptable obstacle-free volume (V) is defined as:
 being located within an allowed space (S) relative to the reference object (R 1 ), 
 comprising a continuous area (A) that faces the camera ( 110 ) and exceeds a threshold size, 
 starting at a predefined position (P) and extending at least a primary distance (d OK ) in a depth direction away from the camera ( 110 ), and 
 
 if the acceptable obstacle-free volume (V) is found in the image data (D img3D ), controlling the robotic arm to perform at least one action relating to a milk-producing animal ( 100 ). 
 
     
     
       16. The method according to  claim 15 , wherein, when said first searching does not find the reference object (R 1 ) in the image data (D img3D ), the checking proceeds with:
 obtaining information about the reference object (R 1 ) from stored image data (D img3D ) representing the milking location, said stored image data (D img3D ) registered by the camera ( 110 ) at an earlier occasion when the milking location was positioned in front of the camera ( 110 ). 
 
     
     
       17. The method according to  claim 16 , wherein the searching for said acceptable obstacle-free volume (V) comprises filtering out information in the image data (D img3D ) which information represents objects located closer to the camera ( 110 ) than a first threshold distance (d 1 ) and farther away from the camera ( 110 ) than a second threshold distance (d 2 ), the first and second threshold distances (d 1 , d 2 ) being separated from one another by the primary distance (d OK ). 
     
     
       18. The method according to  claim 17 , wherein the predefined position (P) is located at the first threshold distance (d 1 ). 
     
     
       19. The method according to  claim 18 , wherein the first threshold distance (d 1 ) is zero. 
     
     
       20. The method according to  claim 16 , wherein the primary distance (d OK ) is defined in relation to at least one measure of the rotating platform ( 130 ). 
     
     
       21. The method according to  claim 16 , wherein the searching for said acceptable obstacle-free volume (V) comprises filtering out in-formation in the image data (D img3D ), said information representing intensities of light outside of an acceptable range of intensity. 
     
     
       22. The method according to  claim 16 ,
 wherein the milking location comprises at least one auxiliary reference object (R 11 , R 2 ) of the milking location whose position is known relative to the position of the camera ( 110 ), and 
 the method further comprises:
 searching for the at least one auxiliary reference object (R 11 , R 2 ) in the image data (D img3D ) when performing the checking as to whether the entry window is in the image data (D img3D ). 
 
 
     
     
       23. A non-transitory computer-readable data recording medium having recorded thereon a computer program ( 127 ) comprising software that, upon execution by at least one processor, causes the at least one processor to execute the method according to  claim 16 . 
     
     
       24. The method according to  claim 15 , wherein:
 the reference object (R 1 ) comprises at least a part of a physical entity extending laterally in parallel with a plane of rotation for the rotating platform ( 130 ); 
 the three-dimensional image data (D img3D ) is registered from such a position, an orientation, and a field of view such that the three-dimensional image data (D img3D ), in a lateral direction, comprises a width of at least one milking stall and, in a longitudinal direction, comprises the allowed space (S) and the reference object (R 1 ); and 
 the reference object (R 1 ) extends across a central part of the field of view in the lateral direction. 
 
     
     
       25. The method according to  claim 15 , wherein, upon said searching finding the acceptable obstacle-free volume (V) in the image data (D img3D ), the method proceeds with:
 checking whether at least one contour (C 1 , C 2 , C 3 ) delimiting the continuous area (A) fulfills at least one shape requirement, and 
 upon fulfilling the at least one shape requirement, classifying said acceptable obstacle-free volume (V) as the entry window for the robotic arm. 
 
     
     
       26. The method according to  claim 25 , wherein, upon said searching finding the acceptable obstacle-free volume (V) in the image data (D img3D ), the method proceeds with:
 searching for an udder (U) of said animal ( 100 ) in the allowed space (S) of the image data (D img3D ) using the reference object (R 1 ) as a first boundary condition and using the rotating platform ( 130 ) as a second boundary condition. 
 
     
     
       27. The method according to  claim 25 , wherein a first shape requirement of the at least one shape requirement defines a maximally acceptable degree of jaggedness of the at least one contour (C 1 , C 2 , C 3 ). 
     
     
       28. The method according to  claim 25 , wherein a first shape requirement of the at least one shape requirement defines a maximally acceptable degree of irregularity of the continuous area (A). 
     
     
       29. A system for controlling a robotic arm to perform at least one action relating to a milk-producing animal ( 100 ), the system comprising:
 a camera ( 110 ) configured to register three-dimensional image data (D img3D ) representing a milking location comprising a rotating platform ( 130 ) that receives said animal ( 100 ) such that the animal stands with its hind legs (LH, RH) facing the camera ( 110 ); and 
 a control unit ( 120 ) comprising at least one processor and a memory unit, the memory unit having stored therein instructions that, when executed by the processor, cause the processor to receive the image data (D img3D ) from the camera ( 110 ), and process the image data (D img3D ) to derive the decision basis (DB), 
 wherein the milking location also comprises a reference object (R 1 ) that forms part of a structure arranged on a stationary part ( 131 ) of the milking location separate from the rotating platform ( 130 ), said structure being configured to prevent the hind legs (LH, RH) from reaching outside of a safety zone for the milking location, and 
 wherein the at least one processor in the control unit ( 120 ) is programmed to:
 determine whether an entry window for the robotic arm is in the image data (D img3D ) by first searching for the reference object (R 1 ) in the image data (D img3D ), 
 if the reference object (R 1 ) is found in the image data (D img3D ), searching for an acceptable obstacle-free volume (V) in the image data (D img3D ), where said acceptable obstacle-free volume (V) is defined as:
 being located within an allowed space (S) relative to the reference object (R 1 ), 
 comprising a continuous area (A) that faces the camera ( 110 ) and exceeds a threshold size, 
 starting at a predefined position (P) and extending at least a primary distance (d OK ) in a depth direction away from the camera ( 110 ), and 
 
 if the acceptable obstacle-free volume (V) is found in the image data (D img3D ), controlling the robotic arm to perform at least one action relating to a milk-producing animal ( 100 ), 
 
 wherein:
 the reference object (R 1 ) comprises at least a part of a physical entity extending laterally in parallel with a plane of rotation for the rotating platform ( 130 ); 
 the camera ( 110 ) is arranged with each of a position, an orientation, and a field of view such that the three-dimensional image data (D img3D ), in a lateral direction, comprises a width of at least one milking stall and, in a longitudinal direction, comprises the allowed space (S) and the reference object (R 1 ); and 
 the reference object (R 1 ) extends across a central part of the field of view in the lateral direction.