Learning graph representations using hierarchical transformers for content recommendation

Abstract

Knowledge graphs can greatly improve the quality of content recommendation systems. There is a broad variety of knowledge graphs in the domain including clicked user-ad graphs, clicked query-ad graphs, keyword-display URL graphs etc. A hierarchical Transformer model learns entity embeddings in knowledge graphs. The model consists of two different Transformer blocks where the bottom block generates relation-dependent embeddings for the source entity and its neighbors, and the top block aggregates the outputs from the bottom block to produce the target entity embedding. To balance the information from contextual entities and the source entity itself, a masked entity model (MEM) task is combined with a link prediction task in model training.

1 - 20 . (canceled) 
     
     
         21 . A computer-implemented method of completing an incomplete triplet in a knowledge graph comprising:
 receiving a source entity-relation pair information from the knowledge graph;   receiving neighborhood entity-relation pair information from the knowledge graph;   capturing interaction information for the source entity-relation pair information and the neighborhood entity-relation pair information;   providing link predictions for the incomplete triplet based on the interaction information;   selecting one of the link predictions to be a target node for the incomplete triplet; and   adding the target node to the incomplete triplet in the knowledge graph.   
     
     
         22 . The computer-implemented method of  claim 21 , further comprising:
 determining neighborhood relational information from the knowledge graph; and   converting the neighborhood relational information into the neighborhood entity-relation pair information.   
     
     
         23 . The computer-implemented method of  claim 21 , wherein providing the link predictions for the incomplete triplet includes:
 aggregating the interaction information for the source entity-relation pair information and the interaction information for the neighborhood entity-relation pair information; and   providing target entity predictions based on the aggregated interaction information.   
     
     
         24 . The computer-implemented method of  claim 21 , further comprising:
 converting the incomplete triplet from the knowledge graph to the source entity-relation pair information, wherein the incomplete triplet is missing one of a subject or an object.   
     
     
         25 . The computer-implemented method of  claim 21 , wherein selecting the one of the link predictions to be a target node for the incomplete triplet comprises:
 ranking the link predictions based on a plausibility score; and   selecting the highest ranked link prediction to be the target node for the incomplete triplet.   
     
     
         26 . The computer-implemented method of  claim 21 , wherein the source entity-relation pair information further comprises a token embedding, a source embedding, and a predicate embedding. 
     
     
         27 . The computer-implemented method of  claim 26 , wherein the token embedding is a classification token. 
     
     
         28 . The computer-implemented method of  claim 26 , wherein providing link predictions further comprises:
 providing a token for each link prediction, wherein the token comprises an aggregation of the source embedding and the predicate embedding; and   determining the plausibility score for the link prediction based on the token.   
     
     
         29 . The computer-implemented method of  claim 21 , wherein the knowledge graph comprises a plurality of nodes connected by edges, wherein each of the plurality of nodes comprises an entity and each of the edges represents a relationship between two of the plurality of entities. 
     
     
         30 . A non-transitory computer-readable medium storing instructions for completing an incomplete triplet in a knowledge graph, the instructions when executed by one or more processors of a computing device, cause the computing device to:
 receive a source entity-relation pair information from the knowledge graph;   receive neighborhood entity-relation pair information from the knowledge graph;   capture interaction information for the source entity-relation pair information and the neighborhood entity-relation pair information;   provide link predictions for the incomplete triplet based on the interaction information;   select one of the link predictions to be a target node for the incomplete triplet; and   add the target node to the incomplete triplet in the knowledge graph.   
     
     
         31 . The non-transitory computer-readable medium of  claim 30 , wherein the instructions when executed by the one or more processors further cause the computing device to:
 determine neighborhood relational information from the knowledge graph; and   convert the neighborhood relational information into the neighborhood entity-relation pair information.   
     
     
         32 . The non-transitory computer-readable medium of  claim 30 , wherein to provide the link predictions for the incomplete triplet includes to:
 aggregate the interaction information for the source entity-relation pair information and the interaction information for the neighborhood entity-relation pair information; and   provide target entity predictions based on the aggregated interaction information.   
     
     
         33 . The non-transitory computer-readable medium of  claim 30 , wherein the instructions when executed by the one or more processors further cause the computing device to:
 convert the incomplete triplet from the knowledge graph to the source entity-relation pair information, wherein the incomplete triplet is missing one of a subject or an object.   
     
     
         34 . The non-transitory computer-readable medium of  claim 30 , wherein to select the one of the link predictions to be a target node for the incomplete triplet further comprises to:
 rank the link predictions based on a plausibility score; and   select the highest ranked link prediction to be the target node for the incomplete triplet.   
     
     
         35 . The non-transitory computer-readable medium of  claim 30 , wherein the source entity-relation pair information further comprises a token embedding, a source embedding, and a predicate embedding. 
     
     
         36 . The non-transitory computer-readable medium of  claim 30 , wherein to provide link predictions further comprises to:
 provide a token for each link prediction, wherein the token comprises an aggregation of the source embedding and the predicate embedding; and   determine the plausibility score for the link prediction based on the token.   
     
     
         36 . The non-transitory computer-readable medium of  claim 30 , wherein the knowledge graph comprises a plurality of nodes connected by edges, wherein each of the plurality of nodes comprises an entity and each of the edges represents a relationship between two of the plurality of entities. 
     
     
         37 . A system for completing an incomplete triplet in a knowledge graph, the system comprising:
 a processor;   memory storing computer-executable instructions, which when executed, cause the system to:
 receive a source entity-relation pair information from the knowledge graph; 
 receive neighborhood entity-relation pair information from the knowledge graph; 
 capture interaction information for the source entity-relation pair information and the neighborhood entity-relation pair information; 
 provide link predictions for the incomplete triplet based on the interaction information; 
 select one of the link predictions to be a target node for the incomplete triplet; and 
 add the target node to the incomplete triplet in the knowledge graph. 
   
     
     
         38 . The system of  claim 37 , wherein the plurality of instructions, when executed, further cause the system to:
 convert the incomplete triplet from the knowledge graph to the source entity-relation pair information, wherein the incomplete triplet is missing one of a subject or an object.   
     
     
         39 . The system of  claim 37 , wherein the plurality of instructions, when executed, further cause the system to:
 determine neighborhood relational information from the knowledge graph; and   convert the neighborhood relational information into the neighborhood entity-relation pair information.   
     
     
         40 . The system of  claim 37 , wherein to select the one of the link predictions to be a target node for the incomplete triplet further comprises to:
 rank the link predictions based on a plausibility score; and   select the highest ranked link prediction to be the target node for the incomplete triplet.