System and method for url fetching retry mechanism

Abstract

A method for overcoming intermittent, temporary, or other fetching failures by using multiple attempts for retrieving a content from a web server to a client device is disclosed. The URL fetching may use direct or non-direct fetching schemes, or a combination thereof. The non-direct fetching method may use intermediate devices, such as proxy server, Data-Center proxy server, tunnel devices, or any combination thereof. Upon sensing a failure of a fetching action, the action is repeated using the same or different parameters or attributes, such as by using different intermediate devices, selected based on different parameters or attributes, such as different countries. The repetitions are limited to a pre-defined maximum number or attempts. The fetching attempts may be performed by the client device, by an intermediate device in a non-direct fetching scheme, or a combination thereof. Various fetching schemes may be used sequentially until the content is retrieved.

1 . A method for fetching a content over the Internet by a client device using a first device, the content is stored in a web server and is identified by a Uniform Resource Locator (URL), the method comprising:
 receiving, by the first device from the client device, a URL request;   fetching N times, wherein N>1 or N=1, by the first device from the web server, a first response, until the first response is determined to be a proper response, wherein each one of the N times fetchings consists of, or comprises:   sending, by the first device to the web server, a first message that comprises the URL request;   receiving, by the first device from the web server, the first response in response to the sending of the first message to the web server; and   checking and determining whether the first response is a proper response that comprises the content;   responsive to determining that the first response is a proper response, sending, by the first device to the client device, the content; and   responsive to determining that the first response is not a proper response, sending, by the first device to the client device, an error message.   
     
     
         2 . The method according to  claim 1 , wherein N is equal to, or more than, 1, 2, 3, 4, 5, 7, 10, 15, 20, 25, 30, 40, 50, or 100. 
     
     
         3 . The method according to  claim 2 , wherein N is less than 2, 3, 4, 5, 7, 10, 15, 20, 25, 30, 40, 50, 100, 150, or 200. 
     
     
         4 . The method according to  claim 1 , wherein the communication over the Internet between the client device and the first device, or between the first device and the web server, is based on, uses, or is compatible with, Hypertext Transfer Protocol (HTTP) or Hypertext Transfer Protocol Secure (HTTPS) protocol or connection, wherein the client device serves as a HTTP or HTTPS client, wherein the first device serves as an HTTP or HTTPS client, or wherein the first device serves as an HTTP or HTTPS server, and wherein the first message or the first response is an HTTP or HTTPS message. 
     
     
         5 . The method according to  claim 1 , wherein the content comprises, or consists of, a web-page or a web-site that. includes, consists of, or comprises, a part or whole of files, text, numbers, audio, voice, multimedia, video, images, music, or computer program. 
     
     
         6 . The method according to  claim 1 , wherein the checking of the response comprises identifying and checking a HTTP status code that is received by the first device in response to the sending of the URL request. 
     
     
         7 . The method according to  claim 6 , wherein the first response is determined as a proper response responsive to a status code of 2xx. 
     
     
         8 . The method according to  claim 7 , wherein the response is determined as not being a proper response responsive to a status code of 4xx or 5xx. 
     
     
         9 . The method according to  claim 8 , wherein the response is determined as not being a proper response responsive to a status code of HTTP 404 error message. 
     
     
         10 . The method according to  claim 1 , wherein the checking of the response comprises using a timeout mechanism. 
     
     
         11 . The method according to  claim 10 , wherein the response is determined as not being a proper response in response to not receiving a proper response after elapsed defined time period after an initiation of the fetching. 
     
     
         12 . The method according to  claim 1 , wherein the checking if an URL redirection is identified, and wherein the response is determined as not being a proper response in response to detecting the URL redirection. 
     
     
         13 . The method according to  claim 12 , wherein the URL redirection is identified by checking that a HTTP status code is 3xx Redirection. 
     
     
         14 . The method according to  claim 1 , for use with a criterion, wherein the checking of the response comprises verifying if the content received satisfy the criterion. 
     
     
         15 . The method according to  claim 14 , wherein the criterion relates to a feature, characteristic, or type, of a received information. 
     
     
         16 . The method according to  claim 14 , wherein the criterion comprises a value, and wherein the response is determined as not being a proper response in response to comparing a received information feature, characteristic, or type, to the value. 
     
     
         17 . The method according to  claim 16 , wherein the criterion comprises a value of a size of a file, and wherein the response is determined as not being a proper response in response to comparing the received information size to the value. 
     
     
         18 . The method according to  claim 1 , wherein more than 2 fetchings are performed, and wherein the same first message is sent by the first device to the web server in all of the fetchings. 
     
     
         19 . The method according to  claim 1 , wherein more than 2 fetchings are performed, and wherein at least two different first messages are sent by the first device to the web server in the fetchings. 
     
     
         20 . The method according to  claim 19 , wherein more than 2 fetchings are performed, and wherein a distinct first message is sent by the first device to the web server in each of the fetchings. 
     
     
         21 . The method according to  claim 20 , for use with a list of IP addresses stored in the first device, and wherein each of the fetchings further comprising selecting, by the first device, an IP address from the list, and wherein the sending by the first device to the web server of the first message comprises using the selected IP address as a source IP address. 
     
     
         22 . The method according to  claim 21 , wherein the list comprises at least 10, 20, 50, 100, 200, 500, 1,000, 2,000, 5,000, 10,000, 20,000, 50,000, 100,000, 200,000, 500,000, 1,000,000, 2,000,000, 5,000,000, or 10,000,000 IP addresses. 
     
     
         23 . The method according to  claim 21 , for use with a group of devices, wherein each of the IP addresses is an IP address of a device in the group, and wherein the sending by the first device to the web server of the first message comprises:
 sending, by the first device to a device in the group that is addressed by the selected IP address, the first message;   sending, by the selected device to the web server, the first message;   receiving, by the selected device from the web server, the first response to the sent first message; and   sending, by the selected device to the first device, the first response.   
     
     
         24 . The method according to  claim 23 , wherein at least one of, or each one of, the devices in the group is a client device. 
     
     
         25 . The method according to  claim 24 , further for use with a virtualization, wherein at least one of, or each one of, the devices in the group is a client device that consists of, comprises, is part of, or is integrated with, a server device that virtualizes a client device addressed by the selected IP address. 
     
     
         26 . The method according to  claim 24 , wherein at least one of, or each one of, the devices in the group is a client device in a client/server architecture that is housed in a single enclosure that is a hand-held enclosure or a portable enclosure, wherein at least one of, or each one of, the devices in the group is a client device that is integrated in part or entirely in an appliance, or wherein at least one of, or each one of, the devices in the group consists of, comprises, is integrated with, or is part of, a wearable device that is wearable on a person. 
     
     
         27 . The method according to  claim 26 , wherein at least one of, or each one of, the devices in the group is a client device that consists of, comprises, is part of, or is integrated with, a notebook computer, a laptop computer, a media player, a Digital Still Camera (DSC), a Digital video Camera (DVC or digital camcorder), a Personal Digital Assistant (PDA), a cellular telephone, a digital camera, a video recorder, or a smartphone. 
     
     
         28 . The method according to  claim 27 , wherein at least one of, or each one of, the devices in the group is a client device that consists of, comprises, is part of, or is integrated with, a smartphone that comprises, or is based on, an Apple iPhone 6 or a Samsung Galaxy S6. 
     
     
         29 . The method according to  claim 24 , wherein at least one of, or each one of, the devices in the group is a client device that is operative for storing, operating, or using an operating system. 
     
     
         30 . The method according to  claim 29 , wherein the operating system is a mobile operating system. 
     
     
         31 . The method according to  claim 30 , wherein the mobile operating system is based on, or comprises, Android version 2.2 (Froyo), Android version 2.3 (Gingerbread), Android version 4.0 (Ice Cream Sandwich), Android Version 4.2 (Jelly Bean), Android version 4.4 (KitKat), Apple iOS version 3, Apple iOS version 4, Apple iOS version 5, Apple iOS version 6, Apple iOS version 7, Microsoft Windows® Phone version 7, Microsoft Windows® Phone version 8, Microsoft Windows® Phone version 9, or Blackberry® operating system. 
     
     
         32 . The method according to  claim 21 , wherein the selection is based on, or uses, load balancing. 
     
     
         33 . The method according to  claim 21 , wherein the selection is based on, or uses, random selection. 
     
     
         34 . The method according to  claim 33 , wherein random selection uses, or is based on, one or more random numbers generated by a random number generator. 
     
     
         35 . The method according to  claim 34 , wherein the random number generator is hardware based. 
     
     
         36 . The method according to  claim 35 , wherein the random number generator is using thermal noise, shot noise, nuclear decaying radiation, photoelectric effect, or quantum phenomena. 
     
     
         37 . The method according to  claim 34 , wherein the random number generator is software based. 
     
     
         38 . The method according to  claim 37 , wherein the random number generator is based on executing an algorithm for generating pseudo-random numbers. 
     
     
         39 . The method according to  claim 21 , wherein the selection is based on, or uses, an estimated geographical location of the client device or of the web server. 
     
     
         40 . The method according to  claim 39 , further comprising estimating the geographical location of the client device or of the web server using geolocation. 
     
     
         41 . The method according to  claim 40 , wherein the geolocation is based on IP geolocation. 
     
     
         42 . The method according to  claim 40 , wherein the geolocation is based on W3C Geolocation Application Programming Interface (API). 
     
     
         43 . The method according to  claim 39 , wherein each IP address is selected based on estimated as being associated to being in the same area as the client device or the web server. 
     
     
         44 . The method according to  claim 43 , wherein each IP address is selected based on estimated as being in the same continent, country, state, region, city, postal/zip code, latitude, longitude, or Timezone as the client device or the web server. 
     
     
         45 . The method according to  claim 43 , wherein each IP address is selected based on being a recent one to be selected, or based on being the least recent to be selected. 
     
     
         46 . The method according to  claim 21 , wherein the selection comprises manually selecting each IP address by a user. 
     
     
         47 . The method according to  claim 46 , further comprising displaying to the user the IP addresses in the list for each fetching, and manually selecting, by the user, an IP address from the list. 
     
     
         48 . The method according to  claim 1 , wherein the first device or the client device comprises, is part of, or consists of, a client device. 
     
     
         49 . The method according to  claim 48 , wherein the first device is housed in a single enclosure that is a hand-held enclosure or a portable enclosure. 
     
     
         50 . The method according to  claim 49 , wherein the first device consists of, comprises, is part of, or is integrated with, a notebook computer, a laptop computer, a media player, a Digital Still Camera (DSC), a Digital video Camera (DVC or digital camcorder), a Personal Digital Assistant (PDA), a cellular telephone, a digital camera, a video recorder, or a smartphone. 
     
     
         51 . The method according to  claim 50 , wherein the first device consists of, comprises, is part of, or is integrated with, a smartphone that comprises, or is based on, an Apple iPhone 6 or a Samsung Galaxy S6. 
     
     
         52 . The method according to  claim 48 , further comprising storing, operating, or using an operating system, by the first device. 
     
     
         53 . The method according to  claim 52 , wherein the operating system is a mobile operating system. 
     
     
         54 . The method according to  claim 53 , wherein the mobile operating system is based on, or comprises, Android version 2.2 (Froyo), Android version 2.3 (Gingerbread), Android version 4.0 (Ice Cream Sandwich), Android Version 4.2 (Jelly Bean), Android version 4.4 (KitKat), Apple iOS version 3, Apple iOS version 4, Apple iOS version 5, Apple iOS version 6, Apple iOS version 7, Microsoft Windows® Phone version 7, Microsoft Windows® Phone version 8, Microsoft Windows® Phone version 9, or Blackberry® operating system. 
     
     
         55 . The method according to  claim 1 , wherein the first device or the client device is integrated in part or entirely in an appliance. 
     
     
         56 . The method according to  claim 55 , wherein a primary functionality of the appliance is associated with food storage, handling, or preparation. 
     
     
         57 . The method according to  claim 56 , wherein a primary function of the appliance is heating food, and wherein the appliance is a microwave oven, an electric mixer, a stove, an oven, or an induction cooker. 
     
     
         58 . The method according to  claim 56 , wherein the appliance is a refrigerator, a freezer, a food processor, a dishwasher, a food blender, a beverage maker, a coffeemaker, or an iced-tea maker. 
     
     
         59 . The method according to  claim 55 , wherein a primary function of the appliance is associated with environmental control, and the appliance consists of, or is part of, an HVAC system. 
     
     
         60 . The method according to  claim 59 , wherein a primary function of the appliance is associated with temperature control, and wherein the appliance is an air conditioner or a heater. 
     
     
         61 . The method according to  claim 55 , wherein a primary function of the appliance is associated with cleaning, wherein the primary function is associated with clothes cleaning, and the appliance is a washing machine or a clothes dryer, or wherein the appliance is a vacuum cleaner. 
     
     
         62 . The method according to  claim 55 , wherein a primary function of the appliance is associated with water control or water heating. 
     
     
         63 . The method according to  claim 55 , wherein the appliance is an answering machine, a telephone set, a home cinema method, a HiFi method, a CD or DVD player, an electric furnace, a trash compactor, a smoke detector, a light fixture, or a dehumidifier. 
     
     
         64 . The method according to  claim 55 , wherein the appliance is a battery-operated portable electronic device, and the appliance is a notebook, a laptop computer, a media player, a cellular phone, a Personal Digital Assistant (PDA), an image processing device, a digital camera, a video recorder, or a handheld computing device. 
     
     
         65 . The method according to  claim 55 , wherein the integration involves sharing a component. 
     
     
         66 . The method according to  claim 65 , wherein the integration involves housing in same enclosure, sharing same processor, or mounting onto same surface. 
     
     
         67 . The method according to  claim 65 , wherein the integration involves sharing a same connector. 
     
     
         68 . The method according to  claim 67 , wherein the connector is a power connector for connecting to a power source, and wherein the integration involves sharing the same connector for being powered from same power source, or wherein the integration involves sharing same power supply. 
     
     
         69 . The method according to  claim 1 , wherein the first device or the client device, consists of, comprises, is integrated with, or is part of, a wearable device that is wearable on a person. 
     
     
         70 . The method according to  claim 69 , wherein the wearable device is wearable on an organ of the person head. 
     
     
         71 . The method according to  claim 70 , wherein the organ is an eye, ear, face, cheek, nose, mouth, lip, forehead, or chin. 
     
     
         72 . The method according to  claim 69 , wherein the wearable device is constructed to have a form substantially similar to, is constructed to have a shape allowing mounting or wearing identical or similar to, or is constructed to have a form to at least in part substitute for, headwear, eyewear, or earpiece. 
     
     
         73 . The method according to  claim 72 , wherein the headwear consists of, structured as, or comprises, a bonnet, a cap, a crown, a fillet, a hair cover, a hat, a helmet, a hood, a mask, a turban, a veil, or a wig. 
     
     
         74 . The method according to  claim 72 , wherein the eyewear consists of, structured as, or comprises, glasses, sunglasses, a contact lens, a blindfold, or a goggle. 
     
     
         75 . The method according to  claim 72 , wherein the earpiece consists of, structured as, or comprises, a hearing aid, a headphone, a headset, or an earplug. 
     
     
         76 . The method according to  claim 69 , wherein the wearable device is shaped for permanently or releseably being attachable to, or be part of, a clothing piece of a person. 
     
     
         77 . The method according to  claim 76 , wherein the attaching uses taping, gluing, pinning, enclosing, encapsulating, a pin, or a latch and hook clip. 
     
     
         78 . The method according to  claim 77 , wherein the clothing piece is a top, bottom, or full-body underwear, or a headwear, a footwear, an accessory, an outwear, a suit, a dress, a skirt, or a top. 
     
     
         79 . The method according to  claim 69 , wherein the wearable device further comprises an annular member defining an aperture therethrough that is sized for receipt therein of a part of a human body. 
     
     
         80 . The method according to  claim 1 , further comprising storing, operating, or using, by the first device or by the client device, a client operating system. 
     
     
         81 . The method according to  claim 80 , wherein the client operating system consists of, comprises, or is based on, one out of Microsoft Windows 7, Microsoft Windows XP, Microsoft Windows 8, Microsoft Windows 8.1, Linux, and Google Chrome OS. 
     
     
         82 . The method according to  claim 80 , wherein the client operating system is a Real-Time Operating System (RTOS). 
     
     
         83 . The method according to  claim 82 , wherein the RTOS comprises FreeRTOS, SafeRTOS, QNX, VxWorks, or Micro-Controller Operating Systems (μC/OS). 
     
     
         84 . The method according to  claim 1 , wherein the first device consists of, comprises, is part of, or is integrated with, a server device. 
     
     
         85 . The method according to  claim 84 , wherein the server device consists of, includes, is part of, or is integrated with, a proxy server. 
     
     
         86 . The method according to  claim 85 , wherein the proxy server consists of, includes, is part of, or is integrated with, an HTTP proxy server, a web-proxy server, a caching proxy, an open-source caching proxy server, a cloud-based proxy server, an open proxy server, a forwarding proxy server, a reverse proxy server, a transparent proxy server, a non-transparent proxy server, an anonymous proxy server, a translation proxy server, a SOCKS proxy server, a CGI web proxy server, a suffix proxy server, an I2P anonymous proxy server, a DNS proxy server, or any combination thereof. 
     
     
         87 . The method according to  claim 1 , further for use with a virtualization, wherein the first device or the client device consists of, comprises, is part of, or is integrated with, a server device that virtualizes a client device addressed by the selected IP address. 
     
     
         88 . The method according to  claim 87 , wherein the client device virtualization executed as part of a Virtual Machine (VM). 
     
     
         89 . The method according to  claim 88 , for use with an host computer that implements the VM, wherein the method further comprising executing, by the host computer, a hypervisor or a Virtual Machine Monitor (VMM). 
     
     
         90 . The method according to  claim 88 , wherein the virtualization includes, is based on, or uses, full virtualization, para-virtualization, or hardware assisted virtualization. 
     
     
         91 . The method according to  claim 1 , wherein the communication over the Internet between the client device and the first device, or between the first device and the web server, is based on, uses, or is compatible with, Socket Secure (SOCKS) protocol or connection, wherein one device serves as an SOCKS server and the other device serves as an SOCKS client. 
     
     
         92 . The method according to  claim 91 , wherein the SOCKS protocol or connection is according to, based on, or is compatible with, SOCKS4, SOCKS4a, or SOCKS5, or wherein the SOCKS protocol or connection is according to, based on, or is compatible with, IETF RFC 1928, IETF RFC 1929, IETF RFC 1961, or IETF RFC 3089. 
     
     
         93 . The method according to  claim 1 , wherein the communication over the Internet between the client device and the first device, or between the first device and the web server, is based on, uses, or is compatible with, WebSocket (ws) or WebSocket Secure (wss) protocol or connection. 
     
     
         94 . A non-transitory computer readable medium containing computer instructions that, when executed by a computer processor, cause the processor to perform at least part of the steps of  claim 1 . 
     
     
         95 . A non-transitory computer readable medium containing computer instructions that, when e executed by a computer processor, cause the processor to perform all of the steps of  claim 1 . 
     
     
         96 . The method according to  claim 1 , wherein at least part of steps of  claim 1  are included in a Software Development Kit (SDK) that is provided as a non-transitory computer readable medium containing computer instructions, and wherein the method further comprising installing the SDK. 
     
     
         97 . The method according to  claim 1 , for use with a web browser, further comprising:
 identifying, by the client device, the URL request by the web browser;   sending, by the client device to the first device, the URL request; and   receiving, by the client device, the content or the error message from the first device.   
     
     
         98 . The method according to  claim 97 , further comprising using, by the web server in the client device, the content in response to receiving of the content from the first device. 
     
     
         99 . The method according to  claim 98 , further comprising displaying, by the web server in the client device, the content to a user. 
     
     
         100 . The method according to  claim 97 , further comprising, by the client device, notifying to a user of the client device, or displaying a notification to the user, in response to receiving of the error message from the first device. 
     
     
         101 . The method according to  claim 97 , further comprising storing, operating, or using, by the client device or the first device, the web browser. 
     
     
         102 . The method according to  claim 101 , wherein the web browser consists of, comprises of, or is based on, Microsoft Internet Explorer, Google Chrome, Opera™, or Mozilla Firefox®. 
     
     
         103 . The method according to  claim 101 , wherein the web browser is a mobile web browser. 
     
     
         104 . The method according to  claim 103 , wherein the mobile web browser consists of, comprises of, or is based on, Safari, Opera Mini™, or Android web browser. 
     
     
         105 . The method according to  claim 97 , wherein at least one of the steps is performed as part of a plug-in or an extension integrated with the web browser. 
     
     
         106 . The method according to  claim 105 , wherein at least one of the steps performed by the client device is integrated with the web browser in a form of a plug-in or an extension. 
     
     
         107 . The method according to  claim 105 , wherein the identifying of the URL uses a plug-in or an extension to the web browser. 
     
     
         108 . The method according to  claim 105 , wherein the integration is by hooking to the web browser, wherein the integration is in a filter driver form, or, wherein the web browser and the steps are communicating using an Inter-Process Communication (IPC). 
     
     
         109 . The method according to  claim 108 , wherein the identifying of the URL uses hooking to the web browser, a filter driver form, or using an Inter-Process Communication (IPC). 
     
     
         110 . The method according to  claim 108 , wherein the IPC is using a file sharing, a signal, a socket, a pipe, a message queue, a shared memory, a semaphore, or memory mapped file. 
     
     
         111 . The method according to  claim 108 , wherein the IPC is using a clipboard, a Component Object Model (COM), a data copy, a DDE protocol, or mailslots. 
     
     
         112 . The method according to  claim 1 , for use with a web browser, further comprising:
 identifying, by the client device, the URL request by the web browser;   sending, by the client device to the first device, the URL request; and   receiving, by the client device, the content or the error message from the first device.   
     
     
         113 . The method according to  claim 1 , for use with a web browser, further comprising:
 identifying, by the client device, the URL request by the web browser;   fetching M times, wherein M>1 or M=1, by the client device, a second response, until the second response is determined not to be the error message, wherein each one of the M times fetchings comprises:   sending, by the client device to the first device, the URL request;   receiving, by the client device from the first device, a second response; and   checking and determining whether the second response is a proper response that comprises the content or whether the second response is the error message; and   responsive to determining that the second response is a proper response, using, by the web browser, the content.   
     
     
         114 . The method according to  claim 113 , wherein M is equal to, or more than, 1, 2, 3, 4, 5, 7, 10, 15, 20, 25, 30, 40, 50, or 100. 
     
     
         115 . The method according to  claim 114 , wherein M is less than 2, 3, 4, 5, 7, 10, 15, 20, 25, 30, 40, 50, 100, 150, or 200. 
     
     
         116 . The method according to  claim 113 , wherein more than 2 fetchings out of the M fetchings are performed, and wherein a same message is sent by the client device to the first device in all of the fetchings. 
     
     
         117 . The method according to  claim 113 , wherein more than 2 fetchings out of the M fetchings are performed, and wherein at least two different messages are sent by the client device to the first device in the fetchings. 
     
     
         118 . The method according to  claim 113 , wherein more than 2 fetchings out of the M fetchings are performed, and wherein a distinct message is sent by the client device to the first device in each of the fetchings. 
     
     
         119 . The method according to  claim 118 , for use with a list of IP addresses, and wherein each of the M fetchings further comprising selecting, by the client device, an IP address from the list, and wherein the sending by the client device to the first device comprises using the selected IP address as a source IP address. 
     
     
         120 . The method according to  claim 119 , wherein the list comprises at least 10, 20, 50, 100, 200, 500, 1,000, 2,000, 5,000, 10,000, 20,000, 50,000, 100,000, 200,000, 500,000, 1,000,000, 2,000,000, 5,000,000, or 10,000,000 IP addresses. 
     
     
         121 . The method according to  claim 119 , wherein each IP address is manually selected by a user. 
     
     
         122 . The method according to  claim 121 , further comprising for each fetching displaying to the user the IP addresses in the list, and selecting, by the user, an IP address from the list. 
     
     
         123 . The method according to  claim 1 , further comprising extracting the URL using SSL sniffing. 
     
     
         124 - 481 . (canceled)