Data Array Manipulation

Abstract

Data array manipulation is described. In an embodiment, concurrent access to a multi-dimensional data array stored on a storage device is enabled by providing separate computational elements with access to a model of the data array for processing the data and consequently request changes to the model. The data array is updated in accordance with the changes, and notification of the changes is provided to the other computational elements concurrently accessing the model. In another embodiment, a data interface apparatus is provided that comprises a storage interface that generates a model of the data array, and an application interface that provides access to the model to the computational element for processing. The application interface receives changes to the model resulting from the processing, and a command to commit the changes to the data array. The storage interface then writes the changes to the data array as an atomic operation.

1 . A computer-implemented method of providing concurrent access to a multi-dimensional data array stored on a storage device, comprising:
 providing a first computational element and at least one further computational element with access to a model of the multi-dimensional data array, the access being provided from a storage service in communication with the storage device and executed on a processor;   receiving, at the storage service, a request message from the first computational element comprising a change to the model;   updating the data array stored on the storage device in accordance with the change to the model; and   responsive to updating the data array, transmitting a notification message comprising the change to the model from the storage service to the at least one further computational element.   
     
     
         2 . A method according  claim 1 , wherein the model comprises a set of operations to manipulate data values from at least a portion of the data array and a shape descriptor for the data values. 
     
     
         3 . A method according  claim 1 , wherein the model comprises data values presented in a predefined format that is compatible with the first computational element and the at least one further computational element. 
     
     
         4 . A method according  claim 1 , wherein the step of updating the data array comprises converting the change to the model to an equivalent change to the data array at a data interface program executed on the processor, and storing the change to the data array on the storage device using an atomic storage operation. 
     
     
         5 . A method according  claim 4 , wherein the step of updating the data array further comprises determining whether the change to the data array was stored successfully. 
     
     
         6 . A method according  claim 5 , wherein the step of updating the data array further comprises, responsive to determining that the change to the data array was stored successfully, transmitting a success message from the storage service to first computational element. 
     
     
         7 . A method according  claim 4 , further comprising the step of generating the model of the data array at the data interface program executed on the processor and providing the storage service with access to the model of the data array. 
     
     
         8 . A method according  claim 1 , wherein the request message comprising the change to the model is received from the first computational element via a data interface program arranged to communicate with the first computational element and the storage service. 
     
     
         9 . A method according  claim 1 , wherein the first computational element and the at least one further computational element are: different computer-executable applications; separate instances of the same computer-executable application; or concurrently executed threads of the same computer-executable application. 
     
     
         10 . A data interface apparatus for updating a plurality of multi-dimensional data arrays stored on a storage device, comprising:
 a storage interface arranged to communicate with the storage device and generate a model of the data arrays stored on the storage device; and   an application interface arranged to communicate with the storage interface and a computational element, provide the computational element with access to the model for processing, receive and store a plurality of changes to the model from the computational element resulting from the processing, and receive a command from the computational element to commit the changes to the data arrays,   wherein the storage interface is further arranged to write the changes to the data arrays responsive to receiving the command as an atomic operation such that if the changes are not successfully written to the storage device the data arrays are reverted to their state prior to writing the changes.   
     
     
         11 . An apparatus according to  claim 10 , wherein the application interface is further arranged to determine whether changes comply with at least one predefined constraint responsive to receiving the command from the computational element. 
     
     
         12 . An apparatus according to  claim 11 , wherein the predefined constraint is a shared dimension between two or more of the data arrays. 
     
     
         13 . An apparatus according to  claim 10 , wherein the application interface is arranged to provide the computational element with access to the model by presenting a predefined application programming interface to the computation element irrespective of a file format in which the data array is stored on the storage device. 
     
     
         14 . An apparatus according to  claim 10 , wherein the plurality of changes to the model comprise at least one of: a change to a data value within the model; an addition of a data element to the model; an addition of a data row to the model; an addition of a data column to the model; an addition of a multidimensional slice to the model; a deletion of a data element from the model; a deletion of a data row from the model; a deletion of a data column from the model; and deletion of a multidimensional slice from the model. 
     
     
         15 . An apparatus according  claim 10 , wherein the storage interface is arranged to generate the model of the data arrays stored on the storage device in a first file format. 
     
     
         16 . An apparatus according  claim 15 , wherein the apparatus further comprises a second storage interface arranged to generate a further model of further data arrays stored on the storage device in a second file format, and the application interface is further arranged to provide the computational element with access to the further model for processing, receive and store a plurality of changes to the further model from the computational element resulting from the processing, and receive a further command from the computational element to commit the further model changes to the further data arrays, and wherein the second storage interface is further arranged to write the further model changes to the further data arrays responsive to receiving the further command as an atomic operation such that if the changes are not successfully written to the storage device the further data arrays are reverted to their state prior to writing the changes. 
     
     
         17 . An apparatus according  claim 10 , wherein the application interface is arranged to transmit a failure message to the computational element if the changes are not successfully written to the storage device. 
     
     
         18 . An apparatus according  claim 10 , wherein the application interface is arranged to transmit a success message to the computational element if the changes are successfully written to the storage device. 
     
     
         19 . An apparatus according  claim 18 , wherein the application interface is arranged to delete the stored changes if the changes are successfully written to the storage device. 
     
     
         20 . A data processing method for concurrently processing a plurality of multi-dimensional data arrays stored on a storage device, comprising:
 generating a model of the data arrays stored on the storage device at a data interface in communication with the storage device and executed on a processor;   providing a first application program and at least one further application program executed on the processor with access to the model of the data arrays from a storage service in communication with the data interface and executed on the processor;   concurrently processing the model at the first application program and at the least one further application program;   receiving, at the storage service, a request message from the first application program comprising a change to the model;   converting, at the data interface, the change to the model to an equivalent change to the data arrays at a data interface program executed on the processor, and storing the change to the data arrays on the storage device using an atomic storage operation; and   responsive to storing the change to the data arrays, transmitting a notification message comprising the change to the model from the storage service to the at least one further application program.