Automatic content-based append detection

Abstract

Automatic append includes: identifying, based at least in part on contents of a first data set comprising a first plurality of columns and contents of a second data set comprising a second plurality of columns, a plurality of matching columns and a plurality of non-matching columns. The matching columns comprise one or more columns among the first plurality of columns; and corresponding one or more matching columns among the second plurality of columns. The non-matching columns comprise: one or more columns among the first plurality of columns that do not match with any columns among the second plurality of columns; and one or more columns among the second plurality of columns that do not match with any columns among the first plurality of columns. Automatic append further includes obtaining a user specification of a first one or more non-matching columns to be appended to a second one or more non-matching columns, the first one or more non-matching columns and the second one or more non-matching columns being selected among the plurality of non-matching columns; and appending the first data set and the second data set according to at least the identified plurality of matching columns and the user specification.

What is claimed is: 
     
         1 . A method, comprising:
 identifying, based at least in part on contents of a first data set comprising a first plurality of columns and contents of a second data set comprising a second plurality of columns:
 a plurality of matching columns comprising:
 one or more columns among the first plurality of columns; and 
 corresponding one or more matching columns among the second plurality of columns, wherein the one or more columns among the first plurality of columns and the corresponding one or more matching columns among the second plurality of columns have at least some matching content; and 
 
 a plurality of non-matching columns comprising:
 one or more columns among the first plurality of columns that do not match with any columns among the second plurality of columns; and 
 one or more columns among the second plurality of columns that do not match with any columns among the first plurality of columns; 
 
 obtaining a specification of: 
 a first one or more non-matching columns to be appended to a second one or more non-matching columns, the first one or more non-matching columns and the second one or more non-matching columns being selected among the plurality of non-matching columns; 
 a change to the plurality of matching columns; or 
 both; and 
   appending at least a portion of the first data set and at least a portion of the second data set according to the plurality of matching columns and the specification.   
     
     
         2 . The method of  claim 1 , wherein the identification of the plurality of matching columns and the plurality of non-matching columns includes:
 extracting features of contents in the first plurality of columns of the first data set to generate a first plurality of results and extracting features of contents in the second plurality of columns of the second data set to generate a second plurality of results; and   clustering the first plurality of results and the second plurality of results based on the extracted features.   
     
     
         3 . The method of  claim 2 , wherein the clustering of the first plurality of results and the second plurality of results includes performing a K-means based clustering technique. 
     
     
         4 . The method of  claim 2 , further comprising identifying among clustering results one or more non-matching columns, one or more clusters with matching pairs, and one or more clusters with tied matching columns. 
     
     
         5 . The method of  claim 4 , further comprising performing a pattern matching operation on the one or more clusters with the tied matching columns to identify one or more additional clusters with matching pairs. 
     
     
         6 . The method of  claim 5 , wherein the pattern matching operation is implemented as a TOPEI-based pattern matching operation. 
     
     
         7 . The method of  claim 5 , further comprising performing a title matching operation on one or more remaining clusters with the tied matching columns to identify one or more additional clusters with untied matching columns. 
     
     
         8 . The method of  claim 2 , wherein the features that are extracted for a column include one or more of: number of spaces in cells of the column, number of punctuations in the cells of the column, average length of values in the cells of the column, variance of values in the cells of the column, total number of words in the cells of the column, average number of words in the cells of the column, and/or number of symbol type transitions in the cells of the column. 
     
     
         9 . The method of  claim 1 , further comprising outputting the plurality of non-matching columns to be displayed. 
     
     
         10 . The method of  claim 1 , further comprising causing a selection interface to be provided to a user, and the selection interface being configured for the user to: select a first column among non-matching columns of the first data set to be appended to a second column among non-matching columns of the second data set, select a first column among non-matching columns of the second data set to be appended to a second column among non-matching columns of the first data set, or both. 
     
     
         11 . The method of  claim 1 , wherein the plurality of matching columns are identified based at least in part on a plurality of N-gram feature vectors. 
     
     
         12 . The method of  claim 1 , further comprising:
 determining N-grams of entries in the first data set and in the second data set;   forming a plurality of matrices based at least in part on the N-grams of the entries in the first data set and in the second data set;   determining, based at least in part on the plurality of matrices, a first plurality of N-gram feature vectors corresponding to the first plurality of columns and a second plurality of N-gram feature vectors corresponding to the second plurality of columns; and   comparing one or more vectors in the first plurality of N-gram feature vectors with one or more vectors in the second plurality of N-gram feature vectors to determine the matching columns.   
     
     
         13 . The method of  claim 12 , wherein the comparing of the one or more vectors in the first plurality of N-gram feature vectors with the one or more vectors in the second plurality of N-gram feature vectors to determine the matching columns includes computing cosine similarities. 
     
     
         14 . The method of  claim 12 , wherein the comparing of the one or more vectors in the first plurality of N-gram feature vectors with the one or more vectors in the second plurality of N-gram feature vectors includes projecting the one or more vectors in the first plurality of N-gram feature vectors and the one or more vectors in the second plurality of N-gram feature vectors in a vector space. 
     
     
         15 . A system, comprising:
 one or more processors configured to:
 identify, based at least in part on contents of a first data set comprising a first plurality of columns and contents of a second data set comprising a second plurality of columns:
 a plurality of matching columns comprising:
 one or more columns among the first plurality of columns; and 
 corresponding one or more matching columns among the second plurality of columns, wherein the one or more columns among the first plurality of columns and the corresponding one or more matching columns among the second plurality of columns have at least some matching content; and 
 
 a plurality of non-matching columns comprising:
 one or more columns among the first plurality of columns that do not match with any columns among the second plurality of columns; and 
 one or more columns among the second plurality of columns that do not match with any columns among the first plurality of columns; 
 
 
 obtain a specification of:
 a first one or more non-matching columns to be appended to a second one or more non-matching columns, the first one or more non-matching columns and the second one or more non-matching columns being selected among the plurality of non-matching columns; 
 a change to the plurality of matching columns; or 
 
 both; and 
 append at least a portion of the first data set and at least a portion of the second data set according to the plurality of matching columns and the specification; and 
   one or more memories coupled to the one or more processors and configured to provide the one or more processors with instructions.   
     
     
         16 . The system of  claim 15 , wherein to identify the plurality of matching columns and the plurality of non-matching columns includes to:
 extract features of contents in the first plurality of columns of the first data set to generate a first plurality of results and extract features of contents in the second plurality of columns of the second data set to generate a second plurality of results; and   cluster the first plurality of results and the second plurality of results based on the extracted features.   
     
     
         17 . The system of  claim 16 , wherein to cluster the first plurality of results and the second plurality of results includes to perform a K-means based clustering technique. 
     
     
         18 . The system of  claim 16 , wherein the one or more processors are further configured to identify among clustering results one or more non-matching columns, one or more clusters with matching pairs, and one or more clusters with tied matching columns. 
     
     
         19 . The system of  claim 18 , wherein the one or more processors are further configured to perform a pattern matching operation on the one or more clusters with the tied matching columns to identify one or more additional clusters with matching pairs. 
     
     
         20 . The system of  claim 19 , wherein the pattern matching operation is implemented as a TOPEI-based pattern matching operation.