Storing and loading data in an array-based computing environment

Abstract

A schema that enables a user to store data generated in an array-based computing environment in a scientific data file is disclosed. The schema may provide a mapping between the data types of the array-based computing environment and the data types of the scientific data file format. The schema may also apply when data is loaded from the scientific data file into the array-based computing environment. When the data is stored in the scientific data file, the file contains descriptions of the data as the variables of the array-based computing environment so that the data in the file can be loaded into the array-based computing environment without additional user input. The loaded variables (name and value) are identical to their state before the data is stored in the file.

1 . A computer-implemented method for representing data from an array-based computing language, the method comprising the steps of:
 providing data in the array based computing language;   mapping types of the data in the array based computing language to a scientific data format; and   allowing for sharing of values of fields in the scientific data format.   
   
   
       2 . The method of  claim 1 , further comprising the step of:
 storing data in a scientific data format.   
   
   
       3 . The method of  claim 2 , wherein the data is stored in a repository including at least one of a file, a database, memory and storage. 
   
   
       4 . The method of  claim 3  wherein the file has a .MAT extension. 
   
   
       5 . The method of  claim 3 , wherein the data is encrypted when stored in the repository. 
   
   
       6 . The method of  claim 1 , wherein the data comprises time series data encapsulated in an object. 
   
   
       7 . The method of  claim 1 , wherein at least a subset of the array-based computing language comprises MATLAB®-compatible commands. 
   
   
       8 . The method of  claim 1 , wherein the scientific data format file comprises a Hierarchical Data Format, Version 5 (HDF5) file. 
   
   
       9 . The method of  claim 1 , wherein the data comprises variables having a property specifying a format for storing the variables. 
   
   
       10 . The method of  claim 1 , further comprising the step of:
 loading data from an HDF file created from data in the array-based computing language into a workspace of a computing environment.   
   
   
       11 . The method of  claim 10 , wherein a portion of the data locked for reading or writing, wherein the locked portion is specific to the data that are stored or loaded. 
   
   
       12 . The method of  claim 10 , wherein more than one read process read different portions of the data and builds an in-memory representation of the data. 
   
   
       13 . The method of  claim 10 , wherein a state of the loaded data is substantially the same as a state of the data in the workspace before the data is stored in the scientific data format file. 
   
   
       14 . The method of  claim 1 , wherein the step of mapping comprises the step of:
 reversing a dimension of the data to store the data in the scientific data format file.   
   
   
       15 . The method of  claim 1 , wherein the step of mapping comprises the step of:
 determining whether the data is to be stored in a little or big endian representation.   
   
   
       16 . The method of  claim 1 , wherein the step of mapping comprises the step of:
 mapping a sparse array as a group in the scientific data format.   
   
   
       17 . The method of  claim 16 , wherein the sparse array is mapped as three datasets, each dataset having one dimensional dataspace. 
   
   
       18 . The method of  claim 16 , wherein the sparse array is represented in canonical form and stored as one dataset having one dimensional dataspace. 
   
   
       19 . The method of  claim 1 , wherein the step of mapping comprises the step of:
 mapping an empty array as a dataset with an attribute indicating that the array is empty.   
   
   
       20 . The method of  claim 1 , wherein the step of mapping comprises the step of:
 mapping a data type as a dataset of references that point to child nodes of a top level group.   
   
   
       21 . The method of  claim 20 , wherein the data type is a cell array and wherein each element of the cellarray is stored in one of the child nodes. 
   
   
       22 . The method of  claim 1 , wherein the step of mapping further comprises mapping at least one user defined data type to the scientific data format. 
   
   
       23 . The method of  claim 1 , wherein the step of mapping further comprises mapping one or more data types to the scientific data format. 
   
   
       24 . The method of  claim 23 , where at least one data type is at least one of: a struct, a function handle, an opaque type, and an object oriented data type. 
   
   
       25 . A system for storing or loading data in an array-based computing environment, the system comprising:
 a workspace for containing data generated in the array-based computing environment,   a repository for storing the data in a scientific data format; and   a schema for mapping types of the data generated in the array-based computing environment to corresponding data types of the scientific data format, wherein values of fields shared in the scientific data format.   
   
   
       26 . The system of  claim 25 , further comprising:
 a plurality of schemas for mapping types of the data in the array based computing environment to a plurality of scientific data formats; and   a user interface for enabling a user to select one of the plurality of schemas.   
   
   
       27 . The system of  claim 25 , further comprising:
 an API for enabling a user to access the data in the scientific data format.   
   
   
       28 . The system of  claim 25 , wherein the repository comprises at least one of a file, a database, memory and storage. 
   
   
       29 . The system of  claim 25 , wherein the data comprises time series data encapsulated in an object. 
   
   
       30 . The system of  claim 25 , wherein at least a subset of the array-based computing language comprises MATLAB®-compatible commands. 
   
   
       31 . The system of  claim 25 , wherein the scientific data format file comprises a Hierarchical Data Format, Version 5 (HDF5) file. 
   
   
       32 . The system of  claim 25 , wherein the data comprises variables having a property specifying a format for storing the variables. 
   
   
       33 . The system of  claim 25 , wherein the workspace comprises named and hierarchical workspaces. 
   
   
       34 . A medium holding computer executable instructions for representing data from an array-based computing language, comprising:
 providing data in the array based computing language;   mapping types of the data in the array based computing language to a scientific data format; and   allowing for sharing of values of fields in the scientific data format.   
   
   
       35 . The medium of  claim 34 , further comprising:
 storing data in a scientific data format.   
   
   
       36 . The medium of  claim 35 , wherein the data is stored in a repository including at least one of a file, a database, memory and storage. 
   
   
       37 . The medium of  claim 36  wherein the file has a .MAT extension. 
   
   
       38 . The medium of  claim 36 , wherein the data is encrypted when stored in the repository. 
   
   
       39 . The medium of  claim 34 , wherein the data comprises time series data encapsulated in an object. 
   
   
       40 . The medium of  claim 34 , wherein at least a subset of the array-based computing language comprises MATLAB®-compatible commands. 
   
   
       41 . The medium of  claim 34 , wherein the scientific data format file comprises a Hierarchical Data Format, Version 5 (HDF5) file. 
   
   
       42 . The medium of  claim 34 , wherein the data comprises variables having a property specifying a format for storing the variables. 
   
   
       43 . The medium of  claim 34 , further comprising:
 loading data from an HDF file created from data in the array-based computing language into a workspace of a computing environment.   
   
   
       44 . The medium of  claim 43 , wherein a portion of the data locked for reading or writing, wherein the locked portion is specific to the data that are stored or loaded. 
   
   
       45 . The medium of  claim 43 , wherein more than one read process read different portions of the data and builds an in-memory representation of the data. 
   
   
       46 . The medium of  claim 43 , wherein a state of the loaded data is substantially the same as a state of the data in the workspace before the data is stored in the scientific data format file. 
   
   
       47 . The medium of  claim 34 , wherein a dimension of the data is reversed to store the data in the scientific data format file. 
   
   
       48 . The medium of  claim 34 , wherein it is determined whether the data is to be stored in a little or big endian representation. 
   
   
       49 . The medium of  claim 34 , wherein a sparse array is mapped as a group in the scientific data format. 
   
   
       50 . The medium of  claim 49 , wherein the sparse array is mapped as three datasets, each dataset having one dimensional dataspace. 
   
   
       51 . The medium of  claim 49 , wherein the sparse array is represented in canonical form and stored as one dataset having one dimensional dataspace. 
   
   
       52 . The medium of  claim 34 , wherein an empty array is mapped as a dataset with an attribute indicating that the array is empty. 
   
   
       53 . The medium of  claim 34 , wherein a data type is mapped as a dataset of references that point to child nodes of a top level group. 
   
   
       54 . The medium of  claim 53 , wherein the data type is a cell array and wherein each element of the cellarray is stored in one of the child nodes. 
   
   
       55 . The medium of  claim 34 , wherein at least one user defined data type is mapped to the scientific data format. 
   
   
       56 . The medium of  claim 34 , wherein one or more data types are mapped to the scientific data format. 
   
   
       57 . The medium of  claim 56 , where at least one data type is at least one of: a struct, a function handle, an opaque type, and an object oriented data type. 
   
   
       58 . A system for storing or loading data, the system comprising:
 an array-based computing environment for providing data,   a schema for mapping types of the data generated in the array-based computing environment to corresponding data types of the scientific data format file; and   a storage unit for storing the data in a scientific data format file, wherein values of fields shared in the scientific data format.   
   
   
       59 . The system of  claim 58 , wherein the storage unit is internal or external to the array-based computing environment.