System and method for natural language processing with pretrained language models

Abstract

A computer-implemented system and method and for learning an entity-independent representation are disclosed. The method may include: receiving an input text; identifying named entities in the input text; replacing the named entities in the input text with entity markers; parsing the input text into a plurality of tokens; generating a plurality of token embeddings based on the plurality of tokens; generating a plurality of positional embeddings based on the respective position of each of the plurality of tokens within the input text; generating a plurality of token type embeddings based on the plurality of tokens and the one or more named entities in the input text; and processing the plurality of token embeddings, the plurality of positional embeddings, and the plurality of token type embeddings using a transformer neural network model to generate a hidden state vector for each of the plurality of tokens in the input text.

What is claimed is: 
     
         1 . A computer-implemented method for learning an entity-independent representation, the method comprising:
 receiving an input text;   identifying one or more named entities in the input text;   replacing the identified one or more named entities in the input text with one or more entity markers, each of the one or more entity markers corresponding to a respective named entity in the one or more identified named entities;   parsing the input text including the one or more entity markers into a plurality of tokens;   generating a plurality of token embeddings based on the plurality of tokens;   generating a plurality of positional embeddings based on the respective position of each of the plurality of tokens within the input text;   generating a plurality of token type embeddings based on the plurality of tokens and the one or more named entities in the input text; and   processing the plurality of token embeddings, the plurality of positional embeddings, and the plurality of token type embeddings using a transformer neural network model (“the transformer model”) to generate a hidden state vector for each of the plurality of tokens in the input text.   
     
     
         2 . The method of  claim 1 , wherein each token embedding for a respective token in the plurality of tokens comprises a vector representation of fixed dimensions for the respective token. 
     
     
         3 . The method of  claim 1 , wherein when a token in the plurality of tokens is not a named entity, the corresponding token type embedding comprises a first type value;
 wherein when a token in the plurality of tokens is a named entity, the corresponding token type embedding comprises a type value that is different from the first type value;   and wherein each unique named entity within the plurality of tokens has a unique type value for the corresponding token type embedding.   
     
     
         4 . The method of  claim 1 , wherein the input text comprises a sentence and each token comprises a word in the sentence. 
     
     
         5 . The method of  claim 4 , wherein parsing the input text into the plurality of tokens comprises:
 adding a first token representing a beginning of the sentence before a first word of the sentence;   adding a second token representing an end of the sentence after a last word of the sentence; and   generating the plurality of tokens including the first token and the second token.   
     
     
         6 . The method of  claim 1 , wherein the transformer model comprises an encoder block, the encoder block comprising a plurality of layers, and each of the plurality of layers comprises a multi-head self-attention mechanism and a feed forward network. 
     
     
         7 . The method of  claim 6 , wherein the transformer model is trained based on a masked language modeling to predict masked words in an input sentence. 
     
     
         8 . The method of  claim 7 , wherein the transformer model is trained to optimize a consistency loss L c . 
     
     
         9 . The method of  claim 8 , wherein the consistency loss L c  is based on:
     L   c =(KL( P∥Q )+KL( Q∥P ))/2,   where P is a probability distribution over a vocabulary during a forward pass on a training sentence, Q is a probability distribution over the vocabulary during a forward pass on a sentence based on the training sentence with entities in the training sentence replaced with entity markers, and KL is a Kullback-Leibler divergence.   
     
     
         10 . The method of  claim 1 , wherein the transformer model is trained to optimize a semantics loss L sem . 
     
     
         11 . The method of  claim 10 , wherein the semantics loss L sem  is based on:
     L   sem =MSE( S 1 CLS   ,S 2 CLS ),   where S1 CLS  represents a last layer output of the transformer model corresponding to a CLS token for a training sentence, S2 CLS  represents a last layer output of the transformer model corresponding to a CLS token for a sentence based on the training sentence with entities in the training sentence replaced with entity markers, and MSE is the Mean Squared Error Loss.   
     
     
         12 . The method of  claim 1 , wherein the transformer model is trained to optimize an overall loss based on:
     L   t =α(MLM( S 1)+MLM( S 2))+β L   c   +γL   sem  
   where α, β and γ are hyperparameters, S1 is a training sentence, L c  is a consistency loss, L sem  is a semantics loss, and MLM is a masked language modeling loss.   
     
     
         13 . The method of  claim 1 , wherein the transformer model is trained on a commonsense reasoning downstream task. 
     
     
         14 . The method of  claim 1 , wherein the transformer model is trained on a sentiment analysis downstream task. 
     
     
         15 . A computer system for learning an entity-independent representation, the system comprising:
 a processor; and   a memory in communication with the processor, the memory storing instructions that when executed, cause the processor to perform:
 receive an input text; 
 identify one or more named entities in the input text; 
 replace the identified one or more named entities in the input text with one or more entity markers, each of the one or more entity markers corresponding to a respective named entity in the one or more identified named entities; 
 parse the input text including the one or more entity markers into a plurality of tokens; 
 generate a plurality of token embeddings based on the plurality of tokens; 
 generate a plurality of positional embeddings based on the respective position of each of the plurality of tokens within the input text; 
 generate a plurality of token type embeddings based on the plurality of tokens and the one or more named entities in the input text; and 
 process the plurality of token embeddings, the plurality of positional embeddings, and the plurality of token type embeddings using a transformer neural network model (“the transformer model”) to generate a hidden state vector for each of the plurality of tokens in the input text. 
   
     
     
         16 . The system of  claim 15 , wherein each token embedding for a respective token in the plurality of tokens comprises a vector representation of fixed dimensions for the respective token. 
     
     
         17 . The system of  claim 15 , wherein when a token in the plurality of tokens is not a named entity, the corresponding token type embedding comprises a first type value; wherein when a token in the plurality of tokens is a named entity, the corresponding token type embedding comprises a type value that is different from the first type value; and wherein each unique named entity within the the plurality of tokens has a unique type value for the corresponding token type embedding. 
     
     
         18 . The system of  claim 15 , wherein the input text comprises a sentence and each token comprises a word in the sentence. 
     
     
         19 . The system of  claim 18 , wherein parsing the input text into the plurality of tokens comprises:
 adding a first token representing a beginning of the sentence before a first word of the sentence;   adding a second token representing an end of the sentence after a last word of the sentence; and   generating the plurality of tokens including the first token and the second token.   
     
     
         20 . The system of  claim 15 , wherein the transformer model comprises an encoder block, the encoder block comprising a plurality of layers, and each of the plurality of layers comprises a multi-head self-attention mechanism and a feed forward network. 
     
     
         21 . The system of  claim 20 , wherein the transformer model is trained based on a masked language modeling to predict masked words in an input sentence. 
     
     
         22 . The system of  claim 21 , wherein the transformer model is trained to optimize a consistency loss L c . 
     
     
         23 . The system of  claim 22 , wherein the consistency loss L c  is based on:
     L   c =(KL( P∥Q )+KL( Q∥P ))/2,   where P is a probability distribution over a vocabulary during a forward pass on a training sentence, Q is a probability distribution over the vocabulary during a forward pass on a sentence based on the training sentence with entities in the training sentence replaced with entity markers, and KL is a Kullback-Leibler divergence.   
     
     
         24 . The system of  claim 15 , wherein the transformer model is trained to optimize a semantics loss L sem . 
     
     
         25 . The system of  claim 24 , wherein the semantics loss L sem  is based on:
     L   sem =MSE( S 1 CLS   ,S 2 CLS ),   where S1 CLS  represents a last layer output of the transformer model corresponding to a CLS token for a training sentence, S2 CLS  represents a last layer output of the transformer model corresponding to a CLS token for a sentence based on the training sentence with entities in the training sentence replaced with entity markers, and MSE is the Mean Squared Error Loss.   
     
     
         26 . The system of  claim 15 , wherein the transformer model is trained to optimize an overall loss based on:
     L   t =α(MLM( S 1)+MLM( S 2))+β L   c   +γL   sem  
   where α, β and γ are hyperparameters, S1 is a training sentence, L c  is a consistency loss, L sem  is a semantics loss, and MLM is a masked language modeling loss.   
     
     
         27 . The system of  claim 15 , wherein the transformer model is trained on a commonsense reasoning downstream task. 
     
     
         28 . The system of  claim 15 , wherein the transformer model is trained on a sentiment analysis downstream task. 
     
     
         29 . A non-transitory computer-readable medium having computer executable instructions stored thereon for execution by one or more computing devices, the instructions, when executed, cause the one or more computing devices to:
 receive an input text;   identify one or more named entities in the input text;   replace the identified one or more named entities in the input text with one or more entity markers, each of the one or more entity markers corresponding to a respective named entity in the one or more identified named entities;   parse the input text including the one or more entity markers into a plurality of tokens;   generate a plurality of token embeddings based on the plurality of tokens;   generate a plurality of positional embeddings based on the respective position of each of the plurality of tokens within the input text;   generate a plurality of token type embeddings based on the plurality of tokens and the one or more named entities in the input text; and   process the plurality of token embeddings, the plurality of positional embeddings, and the plurality of token type embeddings using a transformer neural network model to generate a hidden state vector for each of the plurality of tokens in the input text.