Online recognition of robots

Abstract

Robots accessing a server are identified by allocating an identity tag to a user accessing data stored on the web server in order to identify that user; monitoring the requests made to the server over time by the user identified by the tag; and predicting whether the identified user is a robot based upon one or more properties of the monitored requests predetermined to signify automation of the process of generating the requests.

1 . A method of identifying robots which are accessing a server, the method comprising: 
 allocating an identity tag to a user accessing data stored on the web server in order to identify that user;    monitoring the requests made to the server over time by the user identified by the tag;    and predicting whether the identified user is a robot based upon one or more properties of the monitored requests predetermined to signify automation of the process of generating the requests.    
     
     
         2 . A method according to  claim 1  in which the prediction of a user as a robot is based upon one or more of the following properties of the requests made by a user: 
 (a) the time between requests for data made by an identified user,  
 (b) the order in which data from the web server is requested by an identified user; and  
 (c) the number of requests made by a user in a given period of time.  
 
     
     
         3 . The method of  claim 2  in which all three of the properties (a), (b) and (c) are used together to identify robots.  
     
     
         4 . A method according to an preceding claim in which the step of allocating an identity tag to a user comprise identifying the network address of the user.  
     
     
         5 . A method according to  claim 4  in which the identity tag is the same as the network address.  
     
     
         6 . A method according to  claim 2  in which property (a) at least is determined and in which the method includes the step of identifying a user as a robot when the time between requests for data is shorter than a predetermined minimum time.  
     
     
         7 . A method according to  claim 6  in which an average of the time between two or three or more subsequent requests is taken and the method makes a decision based upon the average time taken between requests.  
     
     
         8 . A method according to  claim 2  in which property (b) at least is determined and in which the method includes the step of identifying a user as a robot when the user requests data in a systematic manner which is to an extent independent of the content of the data.  
     
     
         9 . A method according to  claim 8  in which the requests are considered systematic where a user is systematically requesting every piece of data linked in a web page from the top of a requested page to the bottom, or from the bottom of a requested page to the top.  
     
     
         10 . A method according to any preceding claim which further comprises storing the request information (user identity, request time and request content) in a database, and subsequently analysing the data in the database to identify robots.  
     
     
         11 . A robot identification system for use in combination with a networked server, the system comprising: 
 an identification means adapted to identify a user requesting data from the networked server and allocate an identity tag to that user,    a request monitoring means adapted to monitor the requests made to the server over time by the user identified by the tag, and    a robot identifying means adapted to identify if the identified user is a robot based upon one or more properties of the requests made by that user that have been monitored by the request monitoring means wherein the one or more properties are predetermined to signify automation of the process of generating the requests.    
     
     
         12 . A robot identification system according to  claim 11  which includes a timer or counter which counts the elapsed time between subsequent requests made by a user.  
     
     
         13 . A robot identification system according to  claim 11  or  claim 12  comprising a web server or on a separate server which is connected to a web server to which the user is making requests programmed according to an appropriate computer program.  
     
     
         14 . A robot identification system according to  claim 11  or  12  in which the server comprises a processor which intercepts requests from users prior to passing the requests on to a web server.  
     
     
         15 . A robot identification system according to any one of  claim 11  to  14  in which the request monitoring means is adapted to monitor: 
 (a) the time between requests for data made by an identified user,  
 (b) the order in which data from the web server is requested by an identified user; and  
 (c) the number of requests made by a user in a given period of time.  
 
     
     
         16 . A robot identification system according to  claim 15  which monitors all of (a), (b) and (c) to identify robots.  
     
     
         17 . A robot identification system according to  claim 15  or  claim 16  in which an area of memory is provided into which the identity tags, inter-request times and total number of requests made are recorded and the identification means is adapted to process the information stored in this area of memory to determine if a user is a robot.  
     
     
         18 . A computer program which when running on a processor is adapted to cause the processor to: 
 (i) allocate an identity tag to a user accessing data stored on the web server in order to identify that user,    (ii) monitor the requests made to the server over time by the user identified by the tag, and    (iii) predict whether the identified user is a robot based upon one or more properties of the monitored requests wherein the one or more properties are predetermined to signify automation of the process of generating the requests.    
     
     
         19 . A data carrier which carries a computer program which when running on a processor is adapted to cause the processor to: 
 (i) allocate an identity tag to a user accessing data stored on the web server in order to identify that user,    (ii) monitor the requests made to the server over time by the user identified by the tag, and    (iii) predict whether the identified user is a robot based upon one or more properties of the monitored requests wherein the one or more properties are predetermined to signify automation of the process of generating the requests.