Generating word embeddings with a word embedder and a character embedder neural network models

Abstract

The technology disclosed provides a so-called “joint many-task neural network model” to solve a variety of increasingly complex natural language processing (NLP) tasks using growing depth of layers in a single end-to-end model. The model is successively trained by considering linguistic hierarchies, directly connecting word representations to all model layers, explicitly using predictions in lower tasks, and applying a so-called “successive regularization” technique to prevent catastrophic forgetting. Three examples of lower level model layers are part-of-speech (POS) tagging layer, chunking layer, and dependency parsing layer. Two examples of higher level model layers are semantic relatedness layer and textual entailment layer. The model achieves the state-of-the-art results on chunking, dependency parsing, semantic relatedness and textual entailment.

What is claimed is: 
     
         1 . A method for encoding words, the method comprising:
 receiving, at a word embedder implemented on a processor and trained to create a word embedding space, input words;   mapping, by the word embedder, a word from the input words into a word embedding space, to produce a word embedding vector;   processing, by a character embedder implemented on the processor, character substrings of the word in the input words, the character substrings having multiple substring lengths;   mapping, by the character embedder, the character substrings into corresponding intermediate vectors, the intermediate vectors representing positions of the character substrings in a character embedding space;   identifying unique character substrings from the character substrings;   combining, by the character embedder, intermediate vectors from the unique character substrings to produce a character embedding vector; and   generating, using a word embedding processor, an embedding vector for the word by combining the word embedding vector and the character embedding vector, wherein the word that was previously mapped into the word embedding space is represented by the word embedding vector and the character embedding vector and the word previously not mapped into the word embedding space is represented by the character embedding vector.   
     
     
         2 . The method of  claim 1 , wherein the word embedding vector includes an embedding of the word previously mapped in the word embedding space. 
     
     
         3 . The method of  claim 1 , further comprising:
 the word embedder mapping the word previously not mapped into the word embedding space to the word embedding vector that is a reserved word embedding vector for unknown words.   
     
     
         4 . The method of  claim 1 , further comprising:
 combining the intermediate vectors from the unique character substrings by element wise averaging to produce the character substring vector.   
     
     
         5 . The method of  claim 1 , further comprising:
 combining the intermediate vectors from the unique character substrings by element-wise selection of maximum values from the intermediate vectors.   
     
     
         6 . The method of  claim 1 , wherein the substring lengths are up to four characters. 
     
     
         7 . The method of  claim 1 , further comprising:
 training, using a skip-gram model, the word embedder to create a word embedding matrix with the word embedding space.   
     
     
         8 . The method of  claim 1 , further comprising:
 training, using a continuous bag-of-words (CBOW) model the word embedder to create a word embedding matrix with the word embedding space.   
     
     
         9 . The method of  claim 1 , further comprising:
 projecting the intermediate vectors of the unique character substrings into a space of dimensionality that equals dimensionality of the word embedding vector.   
     
     
         10 . A multi-layer neural network system that processes words in an input sentence, including words not previously mapped into a word embedding space, the system comprising:
 a memory configured to store at least one neural network for a word embedder and a substring embedder;   a processor coupled to the memory and configured to:
 map, by the word embedder, a word from the input sentence into the word embedding space, to produce a word embedding vector; 
 process, by a character embedder, character substrings of the word in the input sentence at multiple scales of substring length; 
 map, by the character embedder, the character substrings into corresponding intermediate vectors, the intermediate vectors representing positions of the character substrings in a character embedding space; 
 combine, by the character embedder, intermediate vectors from unique character substrings of the character substrings to produce a character embedding vector; and 
 generate an embedding vector for the word by combining the word embedding vector and the character embedding vector, wherein the word that is a previously recognized word is represented by the word embedding vector and the character embedding vector and the word not previously mapped into the word embedding space is represented by the character embedding vector. 
   
     
     
         11 . The multi-layer neural network system of  claim 10 , wherein the word embedding vector includes an embedding of the word previously mapped in the word embedding space. 
     
     
         12 . The system of  claim 10 , wherein the processor is further configured to:
 map, using the word embedder, the word previously not mapped into the word embedding space into the word embedding vector that is a reserved word embedding vector for unknown words.   
     
     
         13 . The system of  claim 10 , wherein the processor is further configured to:
 combine the intermediate vectors from the unique character substrings by element wise averaging to produce the character embedding vector.   
     
     
         14 . The system of  claim 10 , wherein the processor is further configured to:
 combine the intermediate vectors from the unique character substrings by element-wise selection of maximum values from the intermediate vectors.   
     
     
         15 . The system of  claim 10 , wherein the multiple scales of substring length include up to four characters. 
     
     
         16 . The system of  claim 10 , wherein the processor is further configured to:
 train, using a skip-gram model, the word embedder to create a word embedding matrix with the word embedding space.   
     
     
         17 . The system of  claim 10 , wherein the processor is further configured to:
 train, using a continuous bag-of-words (CBOW) model the word embedder to create a word embedding matrix with the word embedding space.   
     
     
         18 . The system of  claim 10 , wherein the processor is further configured to:
 project the intermediate vectors into a space of dimensionality that equals dimensionality of the word embedding vector.   
     
     
         19 . A non-transitory computer readable medium having instructions stored thereon, that when executed by a processor, cause the processor to perform operations, the operations comprising:
 receiving, at a word embedder, input words;   mapping, by the word embedder, a word from the input words into a word embedding space, to produce a word embedding vector when the word was previously mapped to the word embedding space or mapping the word into the word embedding vector as a reserved word when the word was not previously mapped into the word embedding space;   processing, by a character embedder, character substrings of the word in the input words at multiple scales of substring length;   mapping, by the character embedder, the character substrings into corresponding intermediate vectors, the intermediate vectors representing positions of the character substrings in a character embedding space;   combining, by the character embedder, intermediate vectors from unique character substrings to produce a character embedding vector; and   generating, using a word embedding processor, an embedding vector for the word by combining the word embedding vector and the character embedding vector.   
     
     
         20 . The non-transitory computer readable medium of  claim 19 , wherein the word embedder and the character embedder are neural network models trained using a skip-gram model.