Identifying bots

Abstract

A method of identifying if a web client has browser capabilities. An originating machine receives a web page request from the web client. The originating machine generates a page request id (PRID) and a script which, when executed by a web client with a browser, regenerates a PRID, and embeds the script in a response. The originating machine sends the response to the web client tor the web client to process and, if the web client is capable, to execute the embedded script, thereby to regenerate a PRID, and to return the regenerated PRID to the originating machine. The originating machine compares the returned regenerated PRID with the generated PRID, a match indicating that the web client has browser capabilities.

1 . A method of identifying if a web client has browser capabilities, comprising:
 an originating machine receiving a web page request from the web client;   the originating machine generating a page request id (PRID) and a script which, when executed by a web client with a browser, regenerates a PRID, and embedding the script in a response;   the originating machine sending the response to the web client for the web client to process and, if the web client is capable, to execute the embedded script, thereby to regenerate a PRID, and to return the regenerated PRID to the originating machine;   the originating machine or a controller machine comparing the returned regenerated PRID with the generated PRID, a match indicating that the web client has browser capabilities.   
     
     
         2 . A method according to  claim 1  wherein the PRID comprises a combination of numbers and/or characters. 
     
     
         3 . A method according to  claim 1  or  claim 2  wherein the PRID is generated using a server-side code such as PHP, .Net framework or Python. 
     
     
         4 . A method according to  claim 1  or  claim 2  wherein the PRID is generated using a web-server filter (such as ISAPI) or a network appliance. 
     
     
         5 . A method according to any of  claims 1  to  4  where the embedded script comprises a sequence of functions or sub-scripts. 
     
     
         6 . A method according to  claim 5  wherein the controller machine produces banks of functions or sub-scripts for the originating machine, and the controller machine updates the banks on the originating machine from time-to-time. 
     
     
         7 . A method according to any of  claims 1  to  4  wherein the embedded script comprises a portion of a document which can be interpreted by a browser and rendered into a document object model, and client side script which is used to query the document and the document object model and yield known characters or strings in return. 
     
     
         8 . A method according to  claim 7  wherein the embedded script yields different PRIDs on different browsers, depending on the way they run scripts and render documents into a document object model, and in which the different results serve to further identify the browser type and version and validate them against the HTTP headers, including User-Agent. 
     
     
         9 . A method according to any preceding claim wherein the response comprises the requested page and the script is embedded within the content of the requested page. 
     
     
         10 . A method according to any preceding claim wherein the regenerated PRID is returned with the next page request or in a separate request designated for the purpose. 
     
     
         11 . A method according to any preceding claim wherein the originating machine and further comprises storage means for storing generated PRIDs and an indication of returned regenerated PRIDs. 
     
     
         12 . A method according to any preceding claim wherein the originating machine further sends the generated PRIDs and the returned PRIDs to the controller machine which comprises storage means for storing the generated and returned PRIDs and computes matches and match ratios per IP address and session. 
     
     
         13 . A method according to any preceding claim wherein, in the event that in excess of a predetermined number and/or ratio of PRIDs are generated in response to page requests from the same IP address without the web client returning regenerated PRIDs or returning only deformed PRIDs, the web client is suspected to be a bot. 
     
     
         14 . A method according to  claim 13  wherein a suspected bot or a chosen IP or session is set a challenge, such as a CAPTCHA, that it must satisfy before a requested web page is delivered. 
     
     
         15 . A method according to  claim 14  wherein the challenge comprises:
 the originating machine generating a PRID and a script which, when executed by a web client with a browser, regenerates a PRID and reproduces the previous page request, and embeds the script in a response; 
 the originating machine sending the response to the web client for the web client to process and, if it is capable, to execute the embedded script, thereby to regenerate a PRID and the previous page request, and to return the regenerated PRID and the previous page request to the originating machine; 
 the originating machine comparing the returned regenerated PRID with the generated PRID and, in response to a match, sending the previously requested page. 
 
     
     
         16 . A method according to  claim 14  wherein the challenge comprises additional functions to condition the re-submission of the page request with the embedded PRID on further tests that ensure that the page is being viewed and displayed on a web browser and is not run in the background on an iFrame, a child window or a browser running as a background service. 
     
     
         17 . A method according to any of  claims 13  to  16  wherein page requests from a suspected bot are either blocked or restricted according to rules, optionally held on a rule engine. 
     
     
         18 . A method according to any preceding claim wherein the originating machine comprises a web server or anything in front (to the web client side) of a web server, such as a logical or physical third party server or any logical or physical network machine. 
     
     
         19 . A method according any preceding claim wherein the controller machine performs one or more other of the steps of the method. 
     
     
         20 . An originating machine for executing the method according to any of  claims 1  to  19 . 
     
     
         21 . A controller machine for executing the method according to any of  claims 1  to  19 . 
     
     
         22 . A computer program product comprising computer code which when run on an originating machine and/or a controller machine causes it to execute the method according to any of  claims 1  to  19 .