Smart near-real-time folder scan based on a breadth first search

Abstract

In response to a folder event received for a first folder, a first work item is dequeued from an ID queue and metadata of the first folder, and immediate children of the first folder, is fetched and enqueued as work items in a metadata queue. If further first folder children remain to be scanned, the first work item is updated with child IDs for each immediate child of the first folder that is a folder, and it is inserted into the ID queue. In a second pass, a child ID is dequeued and metadata of immediate children of the folder associated with the child ID is fetched and enqueued as work items in the metadata queue. The second pass is repeated for all child IDs in the updated work item. This process is repeated for each generation of children of the first folder or until a specified limit is met.

What is claimed is: 
     
         1 . A system in a server, comprising:
 a processor; and   a memory device that stores program code for execution by the processor, the program code comprising:
 a queue manager that dequeues, from a first persistent queue, a first work item associated with a folder event indicative of an action performed on a first folder; and 
 an API manager that fetches first metadata of the first folder and of immediate children of the first folder; 
 wherein the queue manager enqueues, for the first folder and each immediate child of the first folder, the fetched first metadata in a second persistent queue in a second pass; 
 the API manager fetches second metadata for additional children of the first folder; and 
 the queue manager enqueues the fetched second metadata in the second persistent queue. 
   
     
     
         2 . The system of  claim 1 , wherein to enqueue, for the first folder and each immediate child of the first folder, the fetched first metadata in a second persistent queue, the queue manager:
 creates a second work item corresponding to each of the first folder and the immediate children of the first folder that comprises the respective first metadata; and   enqueues each second work item in the second persistent queue.   
     
     
         3 . The system of  claim 2 , wherein the system provides each second work item for detection of possible policy violation. 
     
     
         4 . The system of  claim 1 , wherein the API manager fetches metadata for a number of children of the first folder up to a set number of children. 
     
     
         5 . The system of  claim 4 , wherein the API manager dynamically adjusts the set number of children of the first folder for which to fetch metadata. 
     
     
         6 . The system of  claim 1 , wherein the API manager fetches metadata for a set depth of generations of children of the first folder. 
     
     
         7 . The system of  claim 1 , wherein the queue manager:
 inserts, in an internal queue of the first work item, a child folder identifier for each folder of the children of the first folder; and   enqueues the updated first work item into the first persistent queue.   
     
     
         8 . A method in a server, comprising:
 receiving from a cloud-based file system a folder event indicative of an action performed on a first folder, the folder event including at least a folder identifier (ID) of the first folder;   dequeuing a first work item associated with the folder event from a first persistent queue;   fetching first metadata of the first folder and of immediate children of the first folder;   enqueuing, for the first folder and each immediate child of the first folder, the fetched first metadata in a second persistent queue;   fetching second metadata for additional children of the first folder; and   enqueuing the fetched second metadata in the second persistent queue.   
     
     
         9 . The method of  claim 8 , wherein said enqueuing, for the first folder and each immediate child of the first folder, the fetched first metadata in a second persistent queue comprises:
 creating a second work item corresponding to each of the first folder and the immediate children of the first folder that comprises the respective first metadata; and   enqueuing each second work item in the second persistent queue.   
     
     
         10 . The method of  claim 9 , further comprising:
 providing each second work item for detection of possible policy violation.   
     
     
         11 . The method of  claim 8 , further comprising:
 setting a number of children of the first folder for which to fetch metadata; and   wherein said fetching first metadata and said second metadata comprises:
 fetching metadata for a number of children of the first folder up to the set number. 
   
     
     
         12 . The method of  claim 11 , further comprising:
 dynamically adjusting the set number of children of the first folder for which to fetch metadata.   
     
     
         13 . The method of  claim 8 , further comprising:
 setting a depth of generations of children of the first folder for which to fetch metadata; and   wherein said fetching first metadata and said second metadata comprises:
 fetching metadata for the set depth of generations of children of the first folder. 
   
     
     
         14 . The method of  claim 8 , further comprising:
 inserting, in an internal queue of the first work item, a child folder identifier for each folder of the children of the first folder; and   enqueuing the updated first work item into the first persistent queue.   
     
     
         15 . A computer-readable storage medium having program code recorded thereon that when executed by a processor causes the processor to perform a method comprising:
 receiving from a cloud-based file system a folder event indicative of an action performed on a first folder, the folder event including at least a folder identifier (ID) of the first folder;   dequeuing a first work item associated with the folder event from a first persistent queue;   fetching first metadata of the first folder and of immediate children of the first folder;   enqueuing, for the first folder and each immediate child of the first folder, the fetched first metadata in a second persistent queue;   fetching second metadata for additional children of the first folder; and   enqueuing the fetched second metadata in the second persistent queue.   
     
     
         16 . The computer-readable storage medium of  claim 15 , wherein said enqueuing, for the first folder and each immediate child of the first folder, the fetched first metadata in a second persistent queue comprises:
 creating a second work item corresponding to each of the first folder and the immediate children of the first folder that comprises the respective first metadata; and   enqueuing each second work item in the second persistent queue.   
     
     
         17 . The computer-readable storage medium of  claim 16 , further comprising:
 providing each second work item for detection of possible policy violation.   
     
     
         18 . The computer-readable storage medium of  claim 15 , further comprising:
 setting a number of children of the first folder for which to fetch metadata; and   wherein said fetching first metadata and said second metadata comprises:
 fetching metadata for a number of children of the first folder up to the set number. 
   
     
     
         19 . The computer-readable storage medium of  claim 18 , further comprising:
 dynamically adjusting the set number of children of the first folder for which to fetch metadata.   
     
     
         20 . The computer-readable storage medium of  claim 15 , further comprising:
 setting a depth of generations of children of the first folder for which to fetch metadata; and   wherein said fetching first metadata and said second metadata comprises:
 fetching metadata for the set depth of generations of children of the first folder.