Mobile Web Scraping

Abstract

Methods, systems and computer program products implementing data aggregation using distributed Web scraping are disclosed. A mobile device can scrape one or more target sites to collect data from accounts of a particular user. The scraping can occur under scraping conditions as specified by the user. The scraping conditions can include conditions based on time, power, bandwidth, usage, or any combination of the above. The scraping conditions can ensure that the scraping occurs at time that is most convenient to the user, e.g., when sufficient bandwidth is available to the mobile device or the mobile device is not performing other tasks. The mobile device can upload the scraped data to a data aggregation server under submission conditions as specified by the user. The data aggregation server can aggregate the scraped data, enrich the aggregated data, and provide the enriched data to the user through Web access.

What is claimed is: 
     
         1 . A system comprising:
 a data aggregation server configured to perform operations comprising
 receiving, for each user of a plurality of users, authorization to aggregate account data for a plurality of different accounts of the user, 
 issuing, for each mobile device of a plurality of mobile devices with each mobile device associated with a different respective user of the plurality of users, a request to scrape account data for one or more of the different accounts of the respective user, 
 receiving, from each mobile device of the plurality of mobile devices, the scraped account data for the one or more of the different accounts of the respective user for the mobile device, and 
 generating, for each user of the plurality of users and using the scraped account data corresponding to the user, a report that includes aggregated account data for the plurality of different accounts of the user; and 
   the plurality of mobile devices, wherein each mobile device is associated with a different respective user and is configured to perform operations comprising
 receiving, from the data aggregation server, a request to scrape account data for one or more different accounts of the respective user associated with the mobile device, 
 determining, from a target database, a respective target site for each of the one or more different accounts of the respective user, 
 in response to receiving the request, scraping the account data from the one or more target sites, including providing respective user credentials to each target site, navigating pages of each target site, and retrieving the account data from the pages, and 
 providing the scraped account data for the one or more different accounts of the respective user to the data aggregation server. 
   
     
     
         2 . The system of  claim 1 , wherein a request to scrape account data for a plurality of different accounts of a particular user is issued to a particular mobile device associated with the particular user according to a schedule specified by the particular user. 
     
     
         3 . The system of  claim 1 , wherein providing the scraped account data for the one or more different accounts of the respective user to the data aggregation server comprises determining that a data submission condition is satisfied. 
     
     
         4 . The system of  claim 3 , wherein the data submission condition includes at least one of a time condition, a power condition, a bandwidth condition, or a device usage condition. 
     
     
         5 . The system of  claim 3 , wherein each data submission condition is specified on a respective mobile device of a respective user. 
     
     
         6 . The system of  claim 1 , wherein scraping the account data from the one or more target sites includes determining that a data scraping condition is satisfied. 
     
     
         7 . The system of  claim 6 , wherein data scraping condition included at least one of a time condition, a power condition, a bandwidth condition, or a device usage condition. 
     
     
         8 . The method of  claim 6 , wherein the data scraping condition specifies whether the scraping should occur after wakeup or before sleep. 
     
     
         9 . The method of  claim 6 , wherein the data scraping condition specifies that the scraping shall occur when a Wi Fi connection is present. 
     
     
         10 . The method of  claim 6 , wherein each condition is based on a user specified timing mode, and automatic timing mode, and a mixed timing mode. 
     
     
         11 . The system of  claim 6 , wherein each data scraping condition is specified on a respective mobile device of a respective user. 
     
     
         12 . The system of  claim 11 , wherein the request is obtained by a scraping scheduler of the mobile device of the user. 
     
     
         13 . The system of  claim 12 , wherein the request is sent from the data aggregation server according to a schedule specified by the user of the mobile client device. 
     
     
         14 . The system of  claim 1 , wherein a content of the report corresponding to a particular user generated by the data aggregation server is specified by the particular user through a user interface provided by the data aggregation server. 
     
     
         15 . The system of  claim 1 , wherein security features of a particular target site prevent the data aggregation server from accessing the target site. 
     
     
         16 . The system of  claim 1 , further comprising an intermediate storage server configured to perform operations comprising:
 receiving scraped account data of a user at a first time; and   providing the scraped account data of the users to an agent of the data aggregation server at a second time.   
     
     
         17 . The system of  claim 1 , wherein the data aggregation server is configured to perform operations further comprising:
 providing, for each user of the plurality of users, the generated report corresponding to the user to the mobile device associated with the user for presentation.