Computer-implemented system and method for tagged and rectangular data processing

Abstract

A computer-implemented system and method for tagged data and rectangular data conversions. The system and method receive tagged input data that is in a non-rectangular format and that uses a hierarchical arrangement of tags to indicate data relationships. The tagged input data is displayed in a graphical interface, and the graphical interface is used to create a mapping specification from the tagged input data.

It is claimed: 
     
       1. A computer-implemented method, system for transforming a first data file, including a hierarchically arranged tagged input data structure, to an output data structure in rectangular format, using a mapping specification in a second data file for use in a procedure, the system comprising:
 receiving, using one or more processors, tagged input data that is in a non-rectangular format, wherein the tagged input data is organized such that a hierarchical arrangement of tags indicates data relationships between one or more parent tags and one or more child tags, and wherein each parent tag is associated with one or more child tags; 
 using the data relationships to determine rectangularization characteristics for the tagged input data; 
 displaying the tagged input data in a graphical interface; 
 receiving, through the graphical interface, an interaction with the tagged input data; 
 one or more processors; 
 one or more computer readable storage mediums containing instructions to cause the one or more processors to perform operations including:
 generating a the second data file, wherein the second data file includes the mapping specification from the tagged input data, wherein the mapping specification is generated based upon the rectangularization characteristics and the interaction, and, wherein the mapping specification identifies one or more tables;: 
 includes one or more data directives that indicate relationships between data in the first data file, wherein the data directives are used to parse content from the first data file and to format the parsed content into rectangular formatted data, wherein the data directives include one or more XPath specifications specifying data items and locations associated with the data items that are to be extracted from hierarchically arranged tagged input data for use in generating output data in rectangular format, 
 is based on rectangularization characteristics for the hierarchically arranged tagged input data, wherein the rectangularization characteristics are based on one or more data relationships between one or more parent tags and one or more child tags, wherein each parent tag is associated with two or more child tags and wherein each association between a parent tag and two or more child tags forms a single corresponding table specific to that association, and  
 generatingidentifies a single table corresponding to each data relationship between a parent tag and one or more child tags, wherein the format of the single table is generateddetermined using the mapping specificationdata directives; 
 
 receiving rectangular formatted input data that uses columns and rows to indicate data relationships; 
 generating tagged output data using the rectangular formatted input data and the mapping specification, wherein the tagged output data is in a non-rectangular format, wherein the tagged output data uses a hierarchical arrangement of tags to indicate data relationships between one or more parent tags and one or more child tags, and wherein each parent tag is associated with one or more child tags; and 
 receiving a set of program instructions including:
 an indication of a location of the first data file, the first data file including a hierarchically arranged tagged input data structure in non-rectangular format configured to be transformed into rectangular format;   an indication of a location of the second data file, the second data file different than the first data file;   a libname statement identifying the first data file, the second data file, and an XML libname engine; and   an indication of a procedure to process a data file and an indication of the data file to process, wherein the procedure requires data in rectangular format and the indication of the data file to process indicates the first data file in non-rectangular format;   
 using the indication of the location of the first data file to access the first data file; 
 displaying the tagged input data in a graphical interface; 
 receiving, through the graphical interface, an interaction with the tagged input data; 
 transforming the first data file to an output data structure in rectangular format compatible with the procedure, by:
 parsing the hierarchically arranged tagged input data in accordance with the data directives, including the one or more XPath specifications; 
 generating the output data structure in rectangular format by assembling the parsed hierarchically arranged tagged input data into rectangular format; and 
 
 generating an output of the procedure using the output data structure in rectangular format; and 
 
 coordinating bi-directional communication between a software application that uses a hierarchical data format and a software application that uses a rectangularized data format, wherein coordinating bi-directional communication includes using the single table and the tagged output data structure. 
 
     
     
       2. The computer-implemented method of claim  1  17, wherein a generating the mapping specification second data file further includes identifying repeating patterns in the tagged input data. 
     
     
       3. The computer-implemented method of  claim 1 , further comprising:
 creating one or more XPath locators, wherein an XPath locator identifies a place to extract data items from the tagged input data.   
     
     
       4. The computer-implemented method of claim  1  17, further comprising:
 using a mapping engine to read the mapping specification and prepare an XPath list, wherein the XPath list specifies particular data items for extraction from the tagged input data. 
 
     
     
       5. The computer-implemented method of claim  1  17, further comprising:
 recording metadata associated with the tagged input data; and 
 using the recorded metadata to provide one or more data storage locations by projecting one or more tables. 
 
     
     
       6. The computer-implemented method of claim  1  17, wherein the graphical interface includes one or more tabs, the one or more tabs including at least one of a properties tab, a format tab, a processing tab, an ordinal tab, and a validation tab. 
     
     
       7. The computer implemented method of  claim 6 , further comprising:
 using the one or more tabs to specify the one or more data directives. 
 
     
     
       8. The computer implemented method of  claim 7 , further comprising:
 using the one or more data directives to process one or more tagged data items of the tagged input data into rectangular formatted data; and 
 using the one or more data directives to process rectangular formatted input data into one or more tagged data items of tagged output data. 
 
     
     
       9. A system, comprising:
 one or more processors; 
 one or more computer readable storage mediums containing instructions to cause the one or more processors to perform operations including: non-transitory computer program product for transforming a first data file, including a hierarchically arranged tagged input data structure, to an output data structure in rectangular format, using a mapping specification in a second data file for use in a procedure, the non-transitory computer program product tangibly embodied in a non-transitory machine readable storage medium, including instructions operable to cause a data processing apparatus to: 
 receiving tagged input data that is in a non-rectangular format, wherein the tagged input data is organized such that a hierarchical arrangement of tags indicates data relationships between one or more parent tags and one or more child tags, and wherein each parent tag is associated with one or more child tags; 
 using the data relationships to determine rectangularization characteristics for the tagged input data; 
 displaying the tagged input data in a graphical interface; 
 receiving, through the graphical interface, an interaction with the tagged input data; 
 generating a generate the second data file, wherein the second data file includes the mapping specification from the tagged input data, wherein the mapping specification is generated based upon the rectangularization characteristics and the interaction, and, wherein the mapping specification identifies one or more tables;: 
 includes one or more data directives that indicate relationships between data in the first data file, wherein the data directives are used to parse content from the first data file and to format the parsed content into rectangular formatted data, wherein the data directives include one or more XPath specifications specifying data items and locations associated with the data items that are to be extracted from hierarchically arranged tagged input data for use in generating output data in rectangular format, 
 is based on rectangularization characteristics for the hierarchically arranged tagged input data, wherein the rectangularization characteristics are based on one or more data relationships between one or more parent tags and one or more child tags, wherein each parent tag is associated with two or more child tags and wherein each association between a parent tag and two or more child tags forms a single corresponding table specific to that association, and 
 
 generatingidentifies a single table corresponding to each data relationship between a parent tag and one or more child tags, wherein the format of the single table is generateddetermined using the mapping specificationdata directives; 
 receiving rectangular formatted input data that uses columns and rows to indicate data relationships; 
 generating tagged output data using the rectangular formatted input data and the mapping specification, wherein the tagged output data is in a non-rectangular format, wherein the tagged output data uses a hierarchical arrangement of tags to indicate data relationships between one or more parent tags and one or more child tags, and wherein each parent tag is associated with one or more child tags; and 
 receive a set of program instructions including:
 an indication of a location of the first data file, the first data file including a hierarchically arranged tagged input data structure in non-rectangular format configured to be transformed into rectangular format;   an indication of a location of the second data file, the second data file different than the first data file;   a libname statement identifying the first data file, the second data file, and an XML libname engine; and   an indication of a procedure to process a data file and an indication of the data file to process, wherein the procedure requires data in rectangular format and the indication of the data file to process indicates the first data file in non-rectangular format;   
 use the indication of the location of the first data file to access the first data file; 
 display the tagged input data in a graphical interface; 
 receive, through the graphical interface, an interaction with the tagged input data; 
 transform the first data file to an output data structure in rectangular format compatible with the procedure, by:
 parsing the hierarchically arranged tagged input data in accordance with the data directives, including the one or more XPath specifications; 
 generating the output data structure in rectangular format by assembling the parsed hierarchically arranged tagged input data into rectangular format; and 
 
 generating an output of the procedure using the output data structure in rectangular format; and 
 coordinatingcoordinate bi-directional communication between an application that uses a hierarchical data format and a software application that uses a rectangularized data format, wherein coordinating bi-directional communication includes using the single table and the tagged output data structure. 
 
     
     
       10. The system of claim  9  1, wherein the one or more computer readable storage mediums further comprise instructions to cause the one or more processors to perform operations including generating the mapping specification second data file by identifying repeating patterns in the tagged input data. 
     
     
       11. The system of  claim 9 , wherein the one or more computer readable storage mediums further comprise instructions to cause the one or more processors to perform operations including creating one or more XPath locators, wherein an XPath locator identifies a place to extract data items from the tagged input data. 
     
     
       12. The system of claim  9  1, wherein the one or more computer readable storage mediums further comprise instructions to cause the one or more processors to perform operations including using a mapping engine to read the mapping specification and prepare an XPath list, wherein the XPath list specifies particular data items for extraction from the tagged input data. 
     
     
       13. The system of claim  9  1, wherein the one or more computer readable storage mediums further comprise instructions to cause the one or more processors to perform operations including:
 recording metadata associated with the tagged input data; and 
 using the recorded metadata to provide one or more data storage locations by projecting one or more tables. 
 
     
     
       14. The system of claim  9  1, wherein the graphical interface includes one or more tabs, the one or more tabs including at least one of a properties tab, a format tab, a processing tab, an ordinal tab, and a validation tab. 
     
     
       15. The system of  claim 14 , wherein the one or more tabs are configured to specify the one or more data directives. 
     
     
       16. The system of  claim 15 , wherein the one or more data directives are configured to process one or more tagged data items of the tagged input data into rectangular formatted data, and wherein the one or more data directives are configured to process rectangular formatted input data into one or more tagged data items of tagged output data. 
     
     
       17. A non-transitory computer program product for transforming a first data file, including a hierarchically arranged tagged input data structure, to an output data structure in rectangular format, using a mapping specification in a second data file for use in a procedure, the non-transitory computer program product tangibly embodied in a non-transitory machine readable storage medium, including instructions operable to cause a data processing apparatus to:
 receive tagged input data that is in a non-rectangular format, wherein the tagged input data is organized such that a hierarchical arrangement of tags indicates data relationships between one or more parent tags and one or more child tags, and wherein each parent tag is associated with one or more child tags; 
 use the data relationships to determine rectangularization characteristics for the tagged input data; 
 display the tagged input data in a graphical interface; 
 receive, through the graphical interface, an interaction with the tagged input data; 
 generate a the second data file, wherein the second data file includes the mapping specification from the tagged input data, wherein the mapping specification is generated based upon the rectangularization characteristics and the interaction, and, wherein the mapping specification identifies one or more tables: 
 includes one or more data directives that indicate relationships between data in the first data file, wherein the data directives are used to parse content from the first data file and to format the parsed content into rectangular formatted data, wherein the data directives include one or more XPath specifications specifying data items and locations associated with the data items that are to be extracted from hierarchically arranged tagged input data for use in generating output data in rectangular format, 
 is based on rectangularization characteristics for the hierarchically arranged tagged input data, wherein the rectangularization characteristics are based on one or more data relationships between one or more parent tags and one or more child tags, wherein each parent tag is associated with two or more child tags and wherein each association between a parent tag and two or more child tags forms a single corresponding table specific to that association, and 
 
 generateidentifies a single table corresponding to each data relationship between a parent tag and one or more child tags, wherein the format of the single table is generateddetermined using the mapping specificationdata directives; 
 receive rectangular formatted input data that uses columns and rows to indicate data relationships; 
 generate tagged output data using the rectangular formatted input data and the mapping specification, wherein the tagged output data is in a non-rectangular format, wherein the tagged output data uses a hierarchical arrangement of tags to indicate data relationships between one or more parent tags and one or more child tags, and wherein each parent tag is associated with one or more child tags; and 
 receive a set of program instructions including:
 an indication of a location of the first data file, the first data file including a hierarchically arranged tagged input data structure in non-rectangular format configured to be transformed into rectangular format;   an indication of a location of the second data file, the second data file different than the first data file;   a libname statement identifying the first data file, the second data file, and an XML libname engine; and   an indication of a procedure to process a data file and an indication of the data file to process, wherein the procedure requires data in rectangular format and the indication of the data file to process indicates the first data file in non-rectangular format;   
 use the indication of the location of the first data file to access the first data file; 
 display the tagged input data in a graphical interface; 
 receive, through the graphical interface, an interaction with the tagged input data; 
 transform the first data file to an output data structure in rectangular format compatible with the procedure, by:
 parsing the hierarchically arranged tagged input data in accordance with the data directives, including the one or more XPath specifications; 
 generating the output data structure in rectangular format by assembling the parsed hierarchically arranged tagged input data into rectangular format; and 
 
 generating an output of the procedure using the output data structure in rectangular format; and 
 coordinate bi-directional communication between an a software application that uses a hierarchical data format and a software application that uses a rectangularized data format, wherein coordinating bi-directional communication includes using the single table and the tagged output data structure. 
 
     
     
       18. The computer program product of claim  17  9, further comprising instructions to generate the mapping specification  second data file by identifying repeating patterns in the tagged input data. 
     
     
       19. The computer program product of  claim 17 , further comprising instructions to create one or more XPath locators, wherein an XPath locator is configured to identify a place to extract data items from the tagged input data. 
     
     
       20. The computer program product of claim  17  9, further comprising instructions to use a mapping engine to read the mapping specification and prepare an XPath list, wherein the XPath list is configured to specify particular data items for extraction from the tagged input data. 
     
     
       21. The computer program product of claim  17  9, further comprising instructions to:
 record metadata associated with the tagged input data; and 
 use the recorded metadata to provide one or more data storage locations by projecting one or more tables. 
 
     
     
       22. The computer program product of claim  17  9, wherein the graphical interface includes one or more tabs, the one or more tabs including at least one of a properties tab, a format tab, a processing tab, an ordinal tab, and a validation tab. 
     
     
       23. The computer program product of  claim 22 , wherein the one or more tabs are configured to specify the one or more data directives. 
     
     
       24. The computer program product of  claim 23 , wherein the one or more data directives are configured to process one or more tagged data items of the tagged input data into rectangular formatted data, and wherein the one or more data directives are configured to process rectangular formatted input data into one or more tagged data items of the tagged output data. 
     
     
       25. A system for transforming a first data file, including a hierarchically arranged tagged input data structure, to an output data structure in rectangular format, using a mapping specification in a second data file for use in a procedure, the system comprising:
 one or more processors;   a computer-readable storage medium containing instructions configured to cause the one or more processors to perform operations, including:   forming the second data file, wherein the second data file includes a mapping specification, wherein the mapping specification:   includes one or more data directives, which indicate relationships between data in the first data file, wherein the data directives are used to parse content from the first data file and to format the parsed content into rectangular formatted data, wherein the data directives include one or more XPath specifications specifying data items and locations associated with the data items that are to be extracted from the hierarchically arranged tagged input data for use in generating output data in rectangular format; and   is based on rectangularization characteristics for the hierarchically arranged tagged input data, wherein the rectangularization characteristics are based on one or more data relationships between one or more parent tags and one or more child tags, wherein each parent tag is associated with two or more child tags and wherein each association between a parent tag and two or more child tags forms a single corresponding table specific to that association, and   identifies one or more tables that will be generated from the hierarchically arranged tagged input data, and wherein each association between a parent tag and two or more child tags forms a single corresponding table specific to that association;   receiving a set of program instructions including:   an indication of a location of the first data file, the first data file including a hierarchically arranged tagged input data structure in non-rectangular format configured to be transformed into rectangular format, wherein the tagged input data is in an eXtensible Markup Language (XML) format;   an indication of a location of the second data file, the second data file different than the first data file;   a libname statement identifying the first data file, the second data file, and an XML libname engine; and   an indication of a procedure to process a data file and an indication of the data file to process, wherein the procedure requires data in rectangular format and the indication of the data file to process indicates the first data file in non-rectangular format;   using the indication of the location of the first data file to access the first data file;   transforming the first data file to an output data structure in rectangular format compatible with the procedure, by:   parsing the hierarchically arranged tagged input data in accordance with the data directives, including the one or more XPath specifications; and   generating the output data structure in rectangular format by assembling the parsed hierarchically arranged tagged input data into rectangular format; and   generating an output of the procedure using the output data structure in rectangular format.   
     
     
       26. The system of claim 25, the computer-readable storage medium further containing instructions configured to cause the one or more processors to perform operations, including:
 checking encoding contained in the mapping specification to ensure the encoding conforms to a tagged input data.   
     
     
       27. The system of claim 25, the computer-readable storage medium further containing instructions configured to cause the one or more processors to perform operations, including:
 recording metadata, the metadata facilitating data conversion from source data formats to destination data formats, column length alignment, and data buffering prioritization for projection.   
     
     
       28. The system of claim 27, the computer-readable storage medium further containing instructions configured to cause the one or more processors to perform operations, including:
 projecting one or more tables and one or more columns.   
     
     
       29. The system of claim 25, wherein the tagged input data includes one or more pcdata fields within the tags of the tagged input data, and wherein the tagged input data includes one or more attributes within the tags of the tagged input data. 
     
     
       30. The system of claim 25, wherein the one or more data directives include a hierarchical arrangement of tags indicating one or more relationships among conversion operations and attributes corresponding to the one or more data directives. 
     
     
       31. The system of claim 30, wherein the one or more data relationships provide information about how the conversion operations and the attributes corresponding to the one or more data directives are used to automatically parse the tagged input data. 
     
     
       32. The system of claim 25, wherein the one or more data directives include one or more definitions. 
     
     
       33. The system of claim 32, wherein the one or more definitions include a column length. 
     
     
       34. The system of claim 32, wherein the one or more definitions include a column data type. 
     
     
       35. The system of claim 32, wherein the one or more definitions include a column type. 
     
     
       36. The system of claim 32, wherein the one or more definitions include acceptable values for a column. 
     
     
       37. The system of claim 32, wherein the one or more definitions include an indication that a value for a data item within the tagged input data is retained until a new value for the data item is encountered in the tagged input data. 
     
     
       38. The system of claim 32, wherein the one or more definitions include a selection criterion for filtering one or more data items contained within the tagged input data. 
     
     
       39. The system of claim 32, wherein the one or more definitions include a string function. 
     
     
       40. The system of claim 32, wherein the one or more definitions include an axes function. 
     
     
       41. The system of claim 32, wherein the one or more definitions include a node-related function. 
     
     
       42. The system of claim 25, wherein a default value is specified and applied when a value for a data item within the tagged input data is missing. 
     
     
       43. The system of claim 25, wherein column content of the output data structure is generated based upon one or more data items contained within the tagged input data. 
     
     
       44. The system of claim 25, wherein an XML parser is used to automatically parse the tagged input data based upon the one or more data directives. 
     
     
       45. The system of claim 25, wherein the tagged input data includes one or more tagged hierarchical paths, and wherein a specific tagged hierarchical path uses an XML path descriptor to identify which of the tags in the tagged input data is to be used as an indicator. 
     
     
       46. A computer-program product, tangibly embodied in a machine-readable storage medium, including instructions configured to cause a data processing apparatus to:
 forming the second data file, wherein the second data file includes a mapping specification, wherein the mapping specification:   includes one or more data directives, which indicate relationships between data in the first data file, wherein the data directives are used to parse content from the first data file and to format the parsed content into rectangular formatted data, wherein the data directives include one or more XPath specifications specifying data items and locations associated with the data items that are to be extracted from the hierarchically arranged tagged input data for use in generating output data in rectangular format; and   is based on rectangularization characteristics for the hierarchically arranged tagged input data, wherein the rectangularization characteristics are based on one or more data relationships between one or more parent tags and one or more child tags, wherein each parent tag is associated with two or more child tags and wherein each association between a parent tag and two or more child tags forms a single corresponding table specific to that association, and   identifies one or more tables that will be generated from the hierarchically arranged tagged input data, and wherein each association between a parent tag and two or more child tags forms a single corresponding table specific to that association;   receive a set of program instructions including:   an indication of a location of the first data file, the first data file including a hierarchically arranged tagged input data structure in non-rectangular format configured to be transformed into rectangular format, wherein the tagged input data is in an eXtensible Markup Language (XML) format;   an indication of a location of the second data file, the second data file different than the first data file;   a libname statement identifying the first data file, the second data file, and an XML libname engine; and   an indication of a procedure to process a data file and an indication of the data file to process, wherein the procedure requires data in rectangular format and the indication of the data file to process indicates the first data file in non-rectangular format;   use the indication of the location of the first data file to access the first data file;   transform the first data file to an output data structure in rectangular format compatible with the procedure, by:   parsing the hierarchically arranged tagged input data in accordance with the data directives, including the one or more XPath specifications; and   generating the output data structure in rectangular format by assembling the parsed hierarchically arranged tagged input data into rectangular format; and   generate an output of the procedure using the output data structure in rectangular format.   
     
     
       47. The computer-program product of claim 46, further comprising instructions configured to cause a data processing apparatus to:
 check encoding contained in the mapping specification to ensure the encoding conforms to a tagged input data.   
     
     
       48. The computer-program product of claim 46, further comprising instructions configured to cause a data processing apparatus to:
 record metadata, the metadata facilitating data conversion from source data formats to destination data formats, column length alignment, and data buffering prioritization for projection.   
     
     
       49. The computer-program product of claim 48, further comprising instructions configured to cause a data processing apparatus to:
 project one or more tables and one or more columns.   
     
     
       50. The computer-program product of claim 46, wherein the tagged input data includes one or more pcdata fields within the tags of the tagged input data, and wherein the tagged input data includes one or more attributes within the tags of the tagged input data. 
     
     
       51. The computer-program product of claim 46, wherein the one or more data directives include a hierarchical arrangement of tags indicating one or more relationships among conversion operations and attributes corresponding to the one or more data directives. 
     
     
       52. The computer-program product of claim 51, wherein the one or more data relationships provide information about how the conversion operations and the attributes corresponding to the one or more data directives are used to automatically parse the tagged input data. 
     
     
       53. The computer-program product of claim 46, wherein the one or more data directives include one or more definitions. 
     
     
       54. The computer-program product of claim 53, wherein the one or more definitions include a column length. 
     
     
       55. The computer-program product of claim 53, wherein the one or more definitions include a column data type. 
     
     
       56. The computer-program product of claim 53, wherein the one or more definitions include a column type. 
     
     
       57. The computer-program product of claim 53, wherein the one or more definitions include acceptable values for a column. 
     
     
       58. The computer-program product of claim 53, wherein the one or more definitions include an indication that a value for a data item within the tagged input data is retained until a new value for the data item is encountered in the tagged input data. 
     
     
       59. The computer-program product of claim 53, wherein the one or more definitions include a selection criterion for filtering one or more data items contained within the tagged input data. 
     
     
       60. The computer-program product of claim 53, wherein the one or more definitions include a string function. 
     
     
       61. The computer-program product of claim 53, wherein the one or more definitions include an axes function. 
     
     
       62. The computer-program product of claim 53, wherein the one or more definitions include a node-related function. 
     
     
       63. The computer-program product of claim 46, wherein a default value is specified and applied when a value for a data item within the tagged input data is missing. 
     
     
       64. The computer-program product of claim 46, wherein column content of the output data structure is generated based upon one or more data items contained within the tagged input data. 
     
     
       65. The computer-program product of claim 46, wherein an XML parser is used to automatically parse the tagged input data based upon the one or more data directives. 
     
     
       66. The computer-program product of claim 46, wherein the tagged input data includes one or more tagged hierarchical paths, and wherein a specific tagged hierarchical path uses an XML path descriptor to identify which of the tags in the tagged input data is to be used as an indicator. 
     
     
       67. A data structure conversion method to transform a data structure into a format compatible with one or more procedures, the method comprising:
 receiving a set of program instructions including:   an indication of a location of a first data file, the first data file including a hierarchically arranged tagged input data structure in non-rectangular format configured to be transformed into rectangular format;   an indication of a location of a second data file different than the first data file, wherein the second data file includes a mapping specification used to transform the hierarchically arranged tagged input data structure into rectangular format;   a libname statement identifying the first data file, the second data file, and an XML libname engine; and   an indication of a procedure to process a data file and an indication of the data file to process, wherein the procedure requires data in rectangular format and the indication of the data file to process indicates the first data file in non-rectangular format;   using the indication of the location of the first data file to access the first data file; using the indication of the location of the second data file, accessing the second data file, the second data file including the mapping specification, wherein the mapping specification:   includes one or more data directives that indicate relationships between data in the first data file, wherein the data directives are used to parse content from the first data file and to format the parsed content into rectangular formatted data, wherein the data directives include one or more XPath specifications specifying data items and locations associated with the data items that are to be extracted from the hierarchically arranged tagged input data for use in generating output data in rectangular format,   is based on rectangularization characteristics for the hierarchically arranged tagged input data, wherein the rectangularization characteristics are based on one or more data relationships between one or more parent tags and one or more child tags, wherein each parent tag is associated with two or more child tags and wherein each association between a parent tag and two or more child tags forms a single corresponding table specific to that association, and   identifies one or more tables to be generated from the hierarchically arranged tagged input data, wherein the format of the one or more tables is determined using the data directives;   transforming the first data file to an output data structure in rectangular format compatible with the procedure, by:   parsing the hierarchically arranged tagged input data in accordance with the data directives, including the one or more XPath specifications;   generating the output data structure in rectangular format by assembling the parsed hierarchically arranged tagged input data into rectangular format; and   generating an output of the procedure using the output data structure in rectangular format.   
     
     
       68. The method of claim 67, further comprising:
 checking encoding contained in the mapping specification to ensure the encoding conforms to a data hierarchy of the tagged input data.   
     
     
       69. The method of claim 67, further comprising:
 recording metadata, the metadata facilitating data conversion from source data formats to destination data formats, column length alignment, and data buffering prioritization for projection.   
     
     
       70. The method of claim 69, further comprising:
 projecting one or more tables and one or more columns.   
     
     
       71. The method of claim 67, wherein the tagged input data is in an eXtensible Markup Language (XML) format. 
     
     
       72. The method of claim 67, wherein the tagged input data includes one or more pcdata fields within the tags of the tagged input data, and wherein the tagged input data includes one or more attributes within the tags of the tagged input data. 
     
     
       73. The method of claim 67, wherein the one or more data directives include a hierarchical arrangement of tags indicating one or more relationships among conversion operations and attributes corresponding to the one or more data directives. 
     
     
       74. The method of claim 73, wherein the one or more data relationships provide information about how the conversion operations and the attributes corresponding to the one or more data directives are used to automatically parse the tagged input data. 
     
     
       75. The method of claim 67, wherein the one or more data directives include one or more definitions. 
     
     
       76. The method of claim 75, wherein the one or more definitions include a column length. 
     
     
       77. The method of claim 75, wherein the one or more definitions include a column data type. 
     
     
       78. The method of claim 75, wherein the one or more definitions include a column type. 
     
     
       79. The method of claim 75, wherein the one or more definitions include acceptable values for a column. 
     
     
       80. The method of claim 75, wherein the one or more definitions include an indication that a value for a data item within the tagged input data is retained until a new value for the data item is encountered in the tagged input data. 
     
     
       81. The method of claim 75, wherein the one or more definitions include a selection criterion for filtering one or more data items contained within the tagged input data. 
     
     
       82. The method of claim 75, wherein the one or more definitions include a string function. 
     
     
       83. The method of claim 75, wherein the one or more definitions include an axes function. 
     
     
       84. The method of claim 75, wherein the one or more definitions include a node-related function. 
     
     
       85. The method of claim 67, wherein a default value is specified and applied when a value for a data item within the tagged input data is missing. 
     
     
       86. The method of claim 67, wherein column content of the output data structure is generated based upon one or more data items contained within the tagged input data. 
     
     
       87. The method of claim 67, wherein an XML parser is used to automatically parse the tagged input data based upon the one or more data directives. 
     
     
       88. The method of claim 67, wherein the tagged input data includes one or more tagged hierarchical paths, and wherein a specific tagged hierarchical path uses an XML path descriptor to identify which of the tags in the tagged input data is to be used as an indicator. 
     
     
       89. A system, comprising:
 one or more processors;   a computer-readable storage medium containing instructions configured to cause the one or more processors to perform operations, including:   receiving a set of program instructions including:   an indication of a location of a first data file, the first data file including a hierarchically arranged tagged input data structure in non-rectangular format configured to be transformed into rectangular format;   an indication of a location of a second data file different than the first data file, wherein the second data file includes a mapping specification used to transform the hierarchically arranged tagged input data structure into rectangular format;   a libname statement identifying the first data file, the second data file, and an XML libname engine; and   an indication of a procedure to process a data file and an indication of the data file to process, wherein the procedure requires data in rectangular format and the indication of the data file to process indicates the first data file in non-rectangular format;   using the indication of the location of the first data file to access the first data file; using the indication of the location of the second data file, accessing the second data file, the second data file including the mapping specification, wherein the mapping specification:   includes one or more data directives that indicate relationships between data in the first data file, wherein the data directives are used to parse content from the first data file and to format the parsed content into rectangular formatted data, wherein the data directives include one or more XPath specifications specifying data items and locations associated with the data items that are to be extracted from the hierarchically arranged tagged input data for use in generating output data in rectangular format,   is based on rectangularization characteristics for the hierarchically arranged tagged input data, wherein the rectangularization characteristics are based on one or more data relationships between one or more parent tags and one or more child tags, wherein each parent tag is associated with two or more child tags and wherein each association between a parent tag and two or more child tags forms a single corresponding table specific to that association, and   identifies one or more tables to be generated from the hierarchically arranged tagged input data, wherein the format of the one or more tables is determined using the data directives;   transforming the first data file to an output data structure in rectangular format compatible with the procedure, by:   parsing the hierarchically arranged tagged input data in accordance with the data directives, including the one or more XPath specifications;   generating the output data structure in rectangular format by assembling the parsed hierarchically arranged tagged input data into rectangular format; and   generating an output of the procedure using the output data structure in rectangular format.   
     
     
       90. The system of claim 89, the computer-readable storage medium further containing instructions configured to cause the one or more processors to perform operations, including:
 checking encoding contained in the mapping specification to ensure the encoding conforms to a data hierarchy of the tagged input data.   
     
     
       91. The system of claim 89, the computer-readable storage medium further containing instructions configured to cause the one or more processors to perform operations, including:
 recording metadata, the metadata facilitating data conversion from source data formats to destination data formats, column length alignment, and data buffering prioritization for projection.   
     
     
       92. The system of claim 91, the computer-readable storage medium further containing instructions configured to cause the one or more processors to perform operations, including:
 projecting one or more tables and one or more columns.   
     
     
       93. The system of claim 89, wherein the tagged input data is in an eXtensible Markup Language (XML) format. 
     
     
       94. The system of claim 89, wherein the tagged input data includes one or more pcdata fields within the tags of the tagged input data, and wherein the tagged input data includes one or more attributes within the tags of the tagged input data. 
     
     
       95. The system of claim 89, wherein the one or more data directives include a hierarchical arrangement of tags indicating one or more relationships among conversion operations and attributes corresponding to the one or more data directives. 
     
     
       96. The system of claim 95, wherein the one or more data relationships provide information about how the conversion operations and the attributes corresponding to the one or more data directives are used to automatically parse the tagged input data. 
     
     
       97. The system of claim 89, wherein the one or more data directives include one or more definitions. 
     
     
       98. The system of claim 97, wherein the one or more definitions include a column length. 
     
     
       99. The system of claim 97, wherein the one or more definitions include a column data type. 
     
     
       100. The system of claim 97, wherein the one or more definitions include a column type. 
     
     
       101. The system of claim 97, wherein the one or more definitions include acceptable values for a column. 
     
     
       102. The system of claim 97, wherein the one or more definitions include an indication that a value for a data item within the tagged input data is retained until a new value for the data item is encountered in the tagged input data. 
     
     
       103. The system of claim 97, wherein the one or more definitions include a selection criterion for filtering one or more data items contained within the tagged input data. 
     
     
       104. The system of claim 97, wherein the one or more definitions include a string function. 
     
     
       105. The system of claim 97, wherein the one or more definitions include an axes function. 
     
     
       106. The system of claim 97, wherein the one or more definitions include a node-related function. 
     
     
       107. The system of claim 89, wherein a default value is specified and applied when a value for a data item within the tagged input data is missing. 
     
     
       108. The system of claim 89, wherein column content of the output data structure is generated based upon one or more data items contained within the tagged input data. 
     
     
       109. The system of claim 89, wherein an XML parser is used to automatically parse the tagged input data based upon the one or more data directives. 
     
     
       110. The system of claim 89, wherein the tagged input data includes one or more tagged hierarchical paths, and wherein a specific tagged hierarchical path uses an XML path descriptor to identify which of the tags in the tagged input data is to be used as an indicator. 
     
     
       111. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to:
 receive a set of program instructions including:   an indication of a location of a first data file, the first data file including a hierarchically arranged tagged input data structure in non-rectangular format configured to be transformed into rectangular format;   an indication of a location of a second data file different than the first data file, wherein the second data file includes a mapping specification used to transform the hierarchically arranged tagged input data structure into rectangular format;   a libname statement identifying the first data file, the second data file, and an XML libname engine; and   an indication of a procedure to process a data file and an indication of the data file to process, wherein the procedure requires data in rectangular format and the indication of the data file to process indicates the first data file in non-rectangular format;   use the indication of the location of the first data file to access the first data file; use the indication of the location of the second data file, accessing the second data file, the second data file including the mapping specification, wherein the mapping specification:   includes one or more data directives that indicate relationships between data in the first data file, wherein the data directives are used to parse content from the first data file and to format the parsed content into rectangular formatted data, wherein the data directives include one or more XPath specifications specifying data items and locations associated with the data items that are to be extracted from the hierarchically arranged tagged input data for use in generating output data in rectangular format,   is based on rectangularization characteristics for the hierarchically arranged tagged input data, wherein the rectangularization characteristics are based on one or more data relationships between one or more parent tags and one or more child tags, wherein each parent tag is associated with two or more child tags and wherein each association between a parent tag and two or more child tags forms a single corresponding table specific to that association, and   identifies one or more tables to be generated from the hierarchically arranged tagged input data, wherein the format of the one or more tables is determined using the data directives;   transform the first data file to an output data structure in rectangular format compatible with the procedure, by:   parsing the hierarchically arranged tagged input data in accordance with the data directives, including the one or more XPath specifications;   generating the output data structure in rectangular format by assembling the parsed hierarchically arranged tagged input data into rectangular format; and   generate an output of the procedure using the output data structure in rectangular format.   
     
     
       112. The computer-program product of claim 111, further comprising instructions configured to cause a data processing apparatus to:
 check encoding contained in the mapping specification to ensure the encoding conforms to a data hierarchy of the tagged input data.   
     
     
       113. The computer-program product of claim 111, further comprising instructions configured to cause a data processing apparatus to:
 record metadata, the metadata facilitating data conversion from source data formats to destination data formats, column length alignment, and data buffering prioritization for projection.   
     
     
       114. The computer-program product of claim 113, further comprising instructions configured to cause a data processing apparatus to:
 project one or more tables and one or more columns.   
     
     
       115. The computer-program product of claim 111, wherein the tagged input data is in an eXtensible Markup Language (XML) format. 
     
     
       116. The computer-program product of claim 111, wherein the tagged input data includes one or more pcdata fields within the tags of the tagged input data, and wherein the tagged input data includes one or more attributes within the tags of the tagged input data. 
     
     
       117. The computer-program product of claim 111, wherein the one or more data directives include a hierarchical arrangement of tags indicating one or more relationships among conversion operations and attributes corresponding to the one or more data directives. 
     
     
       118. The computer-program product of claim 117, wherein the one or more data relationships provide information about how the conversion operations and the attributes corresponding to the one or more data directives are used to automatically parse the tagged input data. 
     
     
       119. The computer-program product of claim 111, wherein the one or more data directives include one or more definitions. 
     
     
       120. The computer-program product of claim 119, wherein the one or more definitions include a column length. 
     
     
       121. The computer-program product of claim 119, wherein the one or more definitions include a column data type. 
     
     
       122. The computer-program product of claim 119, wherein the one or more definitions include a column type. 
     
     
       123. The computer-program product of claim 119, wherein the one or more definitions include acceptable values for a column. 
     
     
       124. The computer-program product of claim 119, wherein the one or more definitions include an indication that a value for a data item within the tagged input data is retained until a new value for the data item is encountered in the tagged input data. 
     
     
       125. The computer-program product of claim 119, wherein the one or more definitions include a selection criterion for filtering one or more data items contained within the tagged input data. 
     
     
       126. The computer-program product of claim 119, wherein the one or more definitions include a string function. 
     
     
       127. The computer-program product of claim 119, wherein the one or more definitions include an axes function. 
     
     
       128. The computer-program product of claim 119, wherein the one or more definitions include a node-related function. 
     
     
       129. The computer-program product of claim 111, wherein a default value is specified and applied when a value for a data item within the tagged input data is missing. 
     
     
       130. The computer-program product of claim 111, wherein column content of the output data structure is generated based upon one or more data items contained within the tagged input data. 
     
     
       131. The computer-program product of claim 111, wherein an XML parser is used to automatically parse the tagged input data based upon the one or more data directives. 
     
     
       132. The computer-program product of claim 111, wherein the tagged input data includes one or more tagged hierarchical paths, and wherein a specific tagged hierarchical path uses an XML path descriptor to identify which of the tags in the tagged input data is to be used as an indicator. 
     
     
       133. The method of claim 67, wherein determining the second data file includes receiving the second data file or generating the second data file. 
     
     
       134. The method of claim 67, wherein a first pass through the first data set is used to access the first data file and transform the first data file to an output data structure in rectangular format, and a second pass through the first data set is used to generate the output of the procedure using the output data structure in rectangular format. 
     
     
       135. The method of claim 67, wherein the procedure is invoked by a SAS proc statement. 
     
     
       136. The method of claim 67, wherein the mapping specification comprises mapping specification controls including one or more format tags and one or more informat tags that indicate characteristics of the output data structure. 
     
     
       137. The system of claim 89, wherein determining the second data file includes receiving the second data file or generating the second data file. 
     
     
       138. The system of claim 89, wherein a first pass through the first data set is used to access the first data file and transform the first data file to an output data structure in rectangular format, and a second pass through the first data set is used to generate the output of the procedure using the output data structure in rectangular format. 
     
     
       139. The system of claim 89, wherein the procedure is invoked by a SAS proc statement. 
     
     
       140. The system of claim 89, wherein the mapping specification comprises mapping specification controls including one or more format tags and one or more informat tags that indicate characteristics of the output data structure. 
     
     
       141. The computer-program product of claim 111, wherein determining the second data file includes receiving the second data file or generating the second data file. 
     
     
       142. The computer-program product of claim 111, wherein a first pass through the first data set is used to access the first data file and transform the first data file to an output data structure in rectangular format, and a second pass through the first data set is used to generate the output of the procedure using the output data structure in rectangular format. 
     
     
       143. The computer-program product of claim 111, wherein the procedure is invoked by a SAS proc statement. 
     
     
       144. The computer-program product of claim 111, wherein the mapping specification comprises mapping specification controls including one or more format tags and one or more informat tags that indicate characteristics of the output data structure.