Model globalization for long document summarization

Abstract

A summarization system includes: K embedding modules configured to: receive K blocks of text, respectively, of a document to be summarized; and generate K first representations based on the K blocks of text, respectively, where K is an integer greater than 2; a first propagation module configured to generate second representations based on the K first representations; a second propagation module configured to generate third representations based on the second representations; an output module configured to select ones of the K blocks based on the third representations; and a summary module configured to generate a summary of the document from text of the selected ones of the K blocks.

1 . A summarization system, comprising:
 K embedding modules configured to:
 receive K blocks of text, respectively, of a document to be summarized; and 
 generate K first representations based on the K blocks of text, respectively, 
 where K is an integer greater than 2; 
   a first propagation module configured to generate second representations based on the K first representations;   a second propagation module configured to generate third representations based on the second representations, the third representations including vectors;   an output module configured to select ones of the K blocks based on the third representations,   wherein the output module includes a neural network and a softmax function configured to classify each of the K blocks, based on the third representations, as being either a first state or a second state and is configured to select the ones of the K blocks based on the classifications; and   a summary module configured to generate a summary of the document from text of the selected ones of the K blocks.   
     
     
         2 . The summarization system of  claim 1  wherein the document to be summarized includes at least 1024 tokens. 
     
     
         3 . (canceled) 
     
     
         4 . The summarization system of  claim 1  wherein the neural network includes a feed forward neural network. 
     
     
         5 . The summarization system of  claim 1  wherein the first propagation module includes:
 K transformer modules having a transformer architecture and configured to receive the K first representations and to generate fourth representations based on the K first representations; 
 a recurrent neural network module configured to receive K first ones of the fourth representations and to generate K fifth representations based on the K first ones of the fourth representations; and 
 K feed forward neural network modules configured to generate K sixth representations based on the K fifth representations, respectively, 
 wherein the second representations include (a) second ones of the fourth representations and (b) the K sixth representations. 
 
     
     
         6 . The summarization system of  claim 5  wherein the recurrent neural network module is a bidirectional gate recurrent unit module. 
     
     
         7 . The summarization system of  claim 5  wherein the fourth representations are a group consisting of (a) the first K ones of the fourth representations and (b) the second ones of the fourth representations. 
     
     
         8 . The summarization system of  claim 5  wherein the second propagation module includes:
 K second transformer modules having the transformer architecture and configured to receive the second representations and to generate seventh representations based on the sixth representations; 
 a second recurrent neural network module configured to receive K first ones of the seventh representations and to generate K eighth representations based on the K first ones of the seventh representations; and 
 K second feed forward neural network modules configured to generate K ninth representations based on the K eighth representations, respectively, 
 wherein the third representations include (a) second ones of the seventh representations and (b) the K eighth representations. 
 
     
     
         9 . The summarization system of  claim 5  wherein the K first ones of the fourth representations are K representations of classification (CLS) tokens of the K blocks, respectively. 
     
     
         10 . The summarization system of  claim 1  wherein the K first representations each include a token embedding, a segment embedding, and a positional encoding, the positional encoding representing a position of the token in its one of the K blocks. 
     
     
         11 . The summarization system of  claim 1  further comprising a third propagation module configured to generate fourth representations based on the third representations,
 wherein the output module is configured to select the ones of the K blocks based on the fourth representations. 
 
     
     
         12 . The summarization system of  claim 1  wherein the summary module is configured to store the summary in memory in association with the document. 
     
     
         13 . The summarization system of  claim 1  wherein the summary module is configured to transmit the summary to a computing device via a network. 
     
     
         14 . The summarization system of  claim 1  wherein the K blocks each include at least one sentence of text from the document. 
     
     
         15 . The summarization system of  claim 1  wherein each of the first and second propagation modules includes (a) K transformer modules configured to independently determine dependencies locally within the K blocks of text and (b) a bi-directional recurrent neural network module configured to process output of the K transformer modules and to determine dependencies globally within the document. 
     
     
         16 . A query system, comprising:
 a search module configured to receive a query as input and obtain search results based on the query; and   the summarization system of  claim 1 , wherein the summarization module is configured to produce summaries of the search results for the search module,   wherein the search module is further configured to output the search results and the summaries of the search results produced by the summarization module.   
     
     
         17 . A summarization system, comprising:
 K means for:
 receiving K blocks of text, respectively, of a document to be summarized; and 
 generating K first representations based on the K blocks of text, respectively, 
 where K is an integer greater than 2; 
   a means for generating second representations based on the K first representations;   a means for generating third representations based on the second representations, the third representations including vectors;   a means for selecting ones of the K blocks based on the third representations wherein the means for selecting ones of the K blocks being for classifying each of the K blocks, based on the third representations, as being either a first state or a second state and selecting the ones of the K blocks based on the classifications; and   a means for generating a summary of the document from text of the selected ones of the K blocks.   
     
     
         18 . A summarization method, comprising:
 receiving K blocks of text, respectively, of a document to be summarized;   generating K first representations based on the K blocks of text, respectively,   where K is an integer greater than 2;   generating second representations based on the K first representations;   generating third representations based on the second representations, the third representations including vectors;   selecting ones of the K blocks based on the third representations,   the selecting ones of the K blocks including:
 classifying each of the K blocks, based on the third representations, as being either a first state or a second state; and 
 selecting the ones of the K blocks based on the classifications; and 
   generating a summary of the document from text of the selected ones of the K blocks.   
     
     
         19 . (canceled) 
     
     
         20 . The summarization method of  claim 18  wherein the K first representations each include a token embedding, a segment embedding, and a positional encoding, the positional encoding representing a position of the token in its one of the K blocks. 
     
     
         21 . The summarization method of  claim 18  further comprising generating fourth representations based on the third representations,
 wherein selecting ones of the K blocks includes selecting the ones of the K blocks based on the fourth representations. 
 
     
     
         22 . The summarization method of  claim 18  wherein the K blocks each include at least one sentence of text from the document. 
     
     
         23 . A method comprising:
 receiving a query as input;   obtaining search results based on the query; and   performing the summarization method of  claim 18  to produce summaries of the search results; and   outputting the search results and the summaries of the search results.