Handling And Processing Of Massive Numbers Of Processing Instructions In Real Time

Abstract

A system is designed for processing instructions in real time during a session. This system comprises: a preloader for obtaining reference data relating to the instructions, the reference data indicating the current values of each specified resource account data file, and the preloader being arranged to read the reference data for a plurality of received instructions in parallel from a master database; an enriched instruction queue for queuing the instructions together with their respective preloaded reference data; an execution engine for determining sequentially whether each received instruction can be executed under the present values of the relevant resource account files and for each executable instruction to generate an updating command; and an updater, responsive to the updating command from the execution engine (for updating the master database with the results of each executable instruction, the operation of the plurality of updaters being decoupled from the operation of the execution engine.

1 . A system for processing and handling very high numbers of processing instructions in real time during a processing session, each processing instruction specifying data files relating to two different entities and the quantity and type of resources to be exchanged between those files, the system comprising:
 a plurality of preloaders, each preloader being arranged to obtain reference data relating to the instructions, the plurality of preloaders being arranged to operate in parallel to read the reference data from a master database and to output the reference data together with the instruction message to an execution engine;   the execution engine being arranged to execute each instruction message sequentially, the execution engine using the preloaded reference information to determine whether the instruction message can be executed under the present condition of the relevant resource account files rather than by using any master database access;   a high-speed local memory for storing a current condition table arranged to be updated with the results of the executed instructions on the files to establish an updated real-time position of the relevant files; and   a recycling module for storing a list of failed processing instructions in the high-speed local memory and for carrying out a recycling procedure where the failed processing instructions are presented for re-execution after an update, of the resource account files in the table identified by that failed processing instruction, has occurred.   
     
     
         2 . A system according to  claim 1 , wherein the recycling module is arranged to store the failed instructions in order of their priority and to present the highest priority instructions first for re-execution during the recycling procedure. 
     
     
         3 . A system according to  claim 1 , wherein the recycling module is arranged to reserve the current value of a resource account file for an instruction in the list and to use this reserved amount in fulfilling the requirements of the instruction during a recycling procedure. 
     
     
         4 . A system according to  claim 1 , wherein the recycling module is arranged to present failed processing instructions for re-execution a predetermined maximum number of times, and if the processing instruction has still not been executed, the recycling module is arranged to cancel the failed instruction. 
     
     
         5 . A system according to  claim 1 , further comprising an initial input queue arranged to receive real-time instructions and batch instructions from a plurality of different sources and to queue the instructions for input to the preloaders. 
     
     
         6 . A system according to  claim 5 , wherein the initial input queue comprises a plurality of initial input queues each one of the plurality being arranged to provide a dedicated input to a corresponding one of the plurality of preloaders. 
     
     
         7 . A system according to  claim 6 , further comprising a routing framework, the routing framework being arranged to distribute an instruction to the plurality of initial input queues operating concurrently. 
     
     
         8 . A system according to  claim 7 , wherein the routing framework is arranged to assign to an instruction an instruction name describing the type of information the instruction relates to and a workload balancer key which provides the unique key identifying the instruction. 
     
     
         9 . A system according to  claim 8 , wherein the routing framework comprises a workload balancer arranged in combination with the routing framework to select one of the initial input queues by use of the received instruction name and workload balancer key. 
     
     
         10 . A system according to  claim 9 , wherein the workload balancer is arranged to link a given workload balancer key always to the same specific destination to ensure that all instructions relating to a specific resource account data file are always routed to the same one of the plurality of queues. 
     
     
         11 . A system according to  claim 5 , wherein the initial input queue is arranged to assign a priority to each incoming instruction. 
     
     
         12 . A system according to  claim 1 , wherein the execution engine is arranged to determine whether the current instruction can be executed under the present values of the relevant resource account files by referring to a stored set of predetermined rules. 
     
     
         13 . A system according to  claim 1 , wherein the execution engine comprises a time-stamping module arranged to determine the time of receipt of the processing instruction and to create a timestamp and apply the same to the processing instruction. 
     
     
         14 . A system according to  claim 1 , further comprising an enriched instruction queue for queuing a plurality of the processing instructions together with their respective preloaded reference data. 
     
     
         15 . A system according  claim 14 , further comprising a pacing module for determining the processing instruction throughput of the execution engine and applying wait states to the queue to slow down the speed of loading of processing instructions to less than the throughput of the execution engine. 
     
     
         16 . A system according  claim 1 , further comprising a plurality of updaters, responsive to the updating command from the execution engine for updating the master database with the results of each executable instruction, the operation of the plurality of updaters being decoupled from the operation of the execution engine. 
     
     
         17 . A system according  claim 16 , further comprising a reporting module for reporting the outcome of the execution of one of the received instructions, the reporting module being arranged to output the updating command for each instruction which it has been determined can be executed and a notification instruction for each failed instruction. 
     
     
         18 . The system according to  claim 17 , wherein the reporting module is arranged to store temporarily a plurality of updating commands until a commit event occurs, and to output the plurality of stored updating commands on occurrence of the commit event, wherein the commit event represents the completion of one physical unit of work by the execution engine. 
     
     
         19 . The system according to  claim 17 , further comprising a finality queue arranged to receive and queue the updating command and the notification instruction from the reporting module. 
     
     
         20 . A method for processing and handling very high numbers of processing instruction in real time during a processing session, each processing instruction specifying data files relating to two different entities and the quantity and type of resources to be exchanged between those files, the method comprising:
 instantiating a plurality of preloaders, each preloader being arranged to obtain and preload reference data relating to the instructions, the plurality of preloaders being arranged to operate in parallel to read the reference data from a master database and to output the reference data together with the instruction message to an execution engine;   executing each instruction message at the execution engine sequentially using the preloaded reference information to determine whether the instruction message can be executed under the present condition of the relevant resource account files rather than by using any master database access;   storing a current condition table in a high-speed local memory of the execution engine, the current condition table being arranged to be updated with the results of the executed instructions on the files to establish an updated real-time position of the relevant files;   storing a list of failed processing instructions in the high-speed local memory; and   implementing a recycling procedure where the failed processing instructions are presented for re-execution after an update, of the resource account files in the current condition table identified by that failed processing instruction, has occurred.