System and method for message processing and routing

Abstract

A message routing system that allows applications at either end of the system to run as-is without modification. The system functions in a multithreaded environment and is capable of handling complex routing rules and message transformation. It is also capable of learning and executing new routing rules and message transformations in formats previously unrecognized by the system. The system enables precise and reliable logging of messages throughout processing and supports publication of enterprise-wide broadcast messages. The system further preferably employs cooperating inbound and outbound transport processes for consuming, routing, processing, safely storing and publishing messages in batches of logical units of work to ensure that the logical units of work are not lost in system transactions. The system also preferably utilizes a replay server for preserving and replaying messages that might otherwise fail to reach their intended destinations.

1 . A computerized message routing system comprising: 
 (a) router means, said router means including means for consuming messages from a publisher, means for publishing the messages to at least one subscriber, and means for publishing the messages to a replay server; and    (b) a replay server for storing all messages published by said router means and for republishing certain ones of the messages to a subscriber on demand of the subscriber.    
     
     
         2 . The system of  claim 1  wherein said replay server republishes messages directly to the subscriber.  
     
     
         3 . The system of  claim 1  wherein said replay server republishes messages to said router means for delivery by said router means to the subscriber.  
     
     
         4 . A method for recovering messages that fail to reach their intended destinations in a computerized message routing system, the method comprising the steps of: 
 (a) storing all messages published by a router on a replay server; and    (b) republishing certain ones of the messages from the replay server to a subscriber on demand of the subscriber.    
     
     
         5 . The method of  claim 4  wherein step (b) comprises republishing certain ones of the messages by the replay server directly to a subscriber.  
     
     
         6 . The method of  claim 4  wherein step (b) comprises republishing certain ones of the messages by the replay server to the router for delivery by the router to the subscriber.  
     
     
         7 . The method of  claim 4  further comprising encoding the messages republished by the replay server such that the subscriber recognizes the messages as republished rather than originally published messages.  
     
     
         8 . A computerized message routing system comprising: 
 consumer process means for consuming messages from a publisher and for writing the messages to at least one file;    publisher process means for reading messages that have been written by said consumer process means to said at least one file and for publishing the messages to at least one subscriber; and    a file system in communication with said consumer process means and said publisher process means, said file system comprising: 
 said at least one file, wherein said at least one file stores messages written from said consumer process means in batches; and  
 means for maintaining write and read offsets for message batches that are written to said at least one file by said consumer process means and that are read from said at least one file by said publisher process means, whereby the write and read offsets enable data to be persisted in said at least one file such that duplicate messages are not written by said consumer process means to said at least one file or published by said publisher process means to the at least one subscriber in the event message recovery is required.  
   
     
     
         9 . The system of  claim 8  wherein the write and read offsets include: 
 an END offset written by said consumer process means to said means for maintaining offsets for a batch of messages stored in said at least one file; and    a START offset written by said publisher process means to said means for maintaining offsets for a batch of messages read from said at least one file.    
     
     
         10 . The system of  claim 9  wherein said START offset precedes said END offset for a batch of messages.  
     
     
         11 . A method for preventing duplicate publication of data in a computerized message routing system comprising: 
 consuming messages from a publisher and writing the messages in batches to at least one file;    reading messages from the at least one file and publishing the messages to at least one subscriber; and    maintaining write and read offsets for message batches that are written to and read from the at least one file, whereby the write and read offsets enable data to be persisted in the at least one file such that duplicate messages cannot be written to the at least one file or published to the at least one subscriber in the event message recovery is required.    
     
     
         12 . The method of  claim 11  wherein the step of maintaining write and read offsets includes: 
 writing an END offset for a batch of messages stored in said at least one file; and    writing a START offset for a batch of messages read from said at least one file.    
     
     
         13 . The method of  claim 11  wherein the said START offset precedes said END offset for a batch of messages.  
     
     
         14 . A method for expanding the messaging processing capability of a computerized message routing system comprising a message handler that performs routing of messages from a publisher to endpoint subscribers, pre-routing transformation of the messages prior to routing of the messages to the endpoint subscribers and post-routing transformation of the messages after routing of the messages to the endpoint subscribers, said method comprising the steps of: 
 providing the message handler with endpoint message transformation procedures performed by a new group of endpoint subscribers that desire to receive messages in a format previously unrecognized by the message handler; and    when all members of the new group of endpoint subscribers have made their endpoint message transformation procedures known to the message handler, automatically performing by the message handler the endpoint message transformation procedures formerly performed by the new group of endpoint subscribers as post-routing message transformation and delivering messages to the new group of endpoint subscribers in the format previously unrecognized by the message handler.    
     
     
         15 . A computerized message routing system comprising; 
 at least one inbound transport in communication with at least one consumer process that is operable to run at least one consumer thread for each inbound message topic; and    at least one outbound transport in communication with at least one publisher process that is operable to run at least one publisher thread for each inbound message topic,    wherein the at least one consumer process communicates with the at least one publisher process via at least one message file,    wherein the number of consumer processes equals the number of inbound transports,    wherein the number of publisher processes equals the number of outbound transports,    wherein the maximum number of consumer threads equals the number of inbound message topics,    wherein the maximum number of publisher threads equals the number of inbound message topics, and    wherein the number of message files equals the number of inbound topics times the number of outbound transports.    
     
     
         16 . A method for operating a computerized message routing system, said method comprising the steps of: 
 providing at least one inbound transport in communication with at least one consumer process that is operable to run at least one consumer thread for each inbound message topic;    providing at least one outbound transport in communication with at least one publisher process that is operable to run at least one publisher thread for each inbound message topic; and    communicating the at least one consumer process with the at least one publisher process via at least one message file,    wherein the number of consumer processes equals the number of inbound message transports,    wherein the number of publisher processes equals the number of outbound message transports,    wherein the maximum number of consumer threads equals the number of inbound message topics,    wherein the maximum number of publisher threads equals the number of inbound message topics, and    wherein the number of message files equals the number of inbound topics times the number of outbound transports.    
     
     
         17 . A method for operating a computerized message routing system, said method comprising the steps of: 
 (a) consuming a message from a message bus of an inbound messaging node;    (b) invoking an introspection module based on a subject on which the message has been published to the inbound node;    (c) examining the contents of the message;    (d) extracting at least one routing key from the message based on the contents of the message;    (e) examining the at least one routing key;    (f) identifying a routing tag based on the at least one routing key;    (g) evaluating the routing tag to determine whether the routing tag is bound to one or both of an outbound subject and a routing rule; and, either    (h) if the routing tag is bound to an outbound subject, then publishing the message to a message bus of an outbound messaging node, or    (i) if the routing tag is bound to a routing rule or a routing rule and an outbound subject, then extracting at least one routing key based on the routing rule and repeating steps (g), (h) and (i) until the message is published to a message bus of an outbound messaging node.    
     
     
         18 . A computerized message routing system comprising: 
 (a) router means, said router means including: 
 (i) consumer process means for consuming messages from a publisher and for writing the messages to at least one file, and  
 (ii) publisher process means for reading messages from said at least one file that have been written by said consumer process means to said at least one file, for publishing the messages to at least one subscriber and for publishing the messages to a replay server;  
 (iii) a file system in communication with said consumer process means and said publisher process means, said file system comprising:  
 said at least one file, wherein said at least one file stores messages written from said consumer process means in batches; and  
 means for maintaining write and read offsets for message batches that are written to said at least one file by said consumer process means and that are read from said at least one file by said publisher process means, whereby the write and read offsets enable data to be persisted in said at least one file such that duplicate messages are not written by said consumer process means to said at least one file or published by said publisher process means to the at least one subscriber in the event message recovery is required; and  
   (b) a replay server for storing all messages published by said publisher process means and for republishing certain ones of the messages to a subscriber on demand of the subscriber.    
     
     
         19 . The system of  claim 18  wherein said replay server republishes messages directly to the subscriber.  
     
     
         20 . The system of  claim 18  wherein said replay server republishes messages to said router means for delivery by said router means to the subscriber.  
     
     
         21 . The system of  claim 18  wherein the write and read offsets include: 
 an END offset written by said consumer process means to said means for maintaining offsets for a batch of messages stored in said at least one file; and    a START offset written by said publisher process means to said means for maintaining offsets for a batch of messages read from said at least one file.    
     
     
         22 . The system of  claim 21  wherein said START offset precedes said END offset for a batch of messages.  
     
     
         23 . A computerized message routing system comprising: 
 (a) router means, said router means including: 
 (i) at least one consumer process means for consuming messages from a publisher and for writing the messages to at least one file;  
 (ii) at least one publisher process means for reading messages from said at least one file, for publishing the messages to at least one subscriber, and for publishing the messages to a replay server;  
 (iii) a file system in communication with said at least one consumer process means and said at least one publisher process means, said file system comprising:  
 said at least one file, wherein said at least one file stores messages written from said at least one consumer process means in batches; and  
   (b) means for maintaining write and read offsets for message batches that are written to said at least one file by said at least one consumer process means and that are read from said at least one file by said at least one publisher process means, whereby the write and read offsets enable data to be persisted in said at least one file such that duplicate messages are not written by said at least one consumer process means to said at least one file or published by said at least one publisher process means to the at least one subscriber in the event message recovery is required;    (c) at least one inbound transport in communication with said at least one consumer process means, wherein said at least one consumer process means is operable to run at least one consumer thread for each inbound message topic; and    (d) at least one outbound transport in communication with said at least one said publisher process means wherein said at least one publisher process means is operable to run at least one publisher thread for each inbound message topic,    wherein said at least one consumer process means communicates with said at least one publisher process means via said at least one file,    wherein the number of consumer processes means equals the number of inbound transports,    wherein the number of publisher processes means equals the number of outbound transports,    wherein the maximum number of consumer threads equals the number of inbound message topics,    wherein the maximum number of publisher threads equals the number of inbound message topics, and    wherein the number of said at least one file equals the number of inbound topics times the number of outbound transports.    
     
     
         24 . The system of  claim 23  wherein the write and read offsets include: 
 an END offset written by said at least one consumer process means to said means for maintaining offsets for a batch of messages stored in said at least one file; and    a START offset written by said at least one publisher process means to said means for maintaining offsets for a batch of messages read from said at least one file.    
     
     
         25 . The system of  claim 24  wherein said START offset precedes said END offset for a batch of messages.  
     
     
         26 . A computerized message routing system comprising; 
 (a) at least one inbound transport in communication with at least one consumer process that is operable to run at least one consumer thread for each inbound message topic; and    (b) at least one outbound transport in communication with at least one publisher process, said at least one publisher process being operable to run at least one publisher thread for each inbound message topic and to publish messages to at least one subscriber and to a replay server,    wherein the at least one consumer process communicates with the at least one publisher process via at least one message file,    wherein the number of consumer processes equals the number of inbound transports,    wherein the number of publisher processes equals the number of outbound transports,    wherein the maximum number of consumer threads equals the number of inbound message topics,    wherein the maximum number of publisher threads equals the number of inbound message topics, and    wherein the number of message files equals the number of inbound topics times the number of outbound transports; and    (c) a replay server for storing all messages published by said at least one publisher process and for republishing certain ones of the messages to a subscriber on demand of the subscriber.    
     
     
         27 . The system of  claim 26  wherein said replay server republishes messages directly to the subscriber.  
     
     
         28 . The system of  claim 26  wherein said replay server republishes messages to said router means for delivery by said router means to the subscriber.  
     
     
         29 . A computerized message routing system comprising: 
 (a) router means, said router means including: 
 (i) at least one consumer process means for consuming messages from a publisher and for writing the messages to at least one file, and  
 (ii) at least one publisher process means for reading messages from said at least one file that have been written by said consumer process means to said at least one file, for publishing the messages to at least one subscriber and for publishing the messages to a replay server;  
 (iii) a file system in communication with said at least one consumer process means and said at least one publisher process means, said file system comprising:  
 said at least one file, wherein said at least one file stores messages written from said at least one consumer process means in batches; and  
 means for maintaining write and read offsets for message batches that are written to said at least one file by said at least one consumer process means and that are read from said at least one file by said at least one publisher process means, whereby the write and read offsets enable data to be persisted in said at least one file such that duplicate messages are not written by said at least one consumer process means to said at least one file or published by said at least one publisher process means to the at least one subscriber in the event message recovery is required;  
   (b) at least one inbound transport in communication with said at least one consumer process means, wherein said at least one consumer process means is operable to run at least one consumer thread for each inbound message topic;    (c) at least one outbound transport in communication with said at least one said publisher process means wherein said at least one publisher process means is operable to run at least one publisher thread for each inbound message topic,    wherein said at least one consumer process means communicates with said at least one publisher process means via said at least one file,    wherein the number of consumer processes means equals the number of inbound transports,    wherein the number of publisher processes means equals the number of outbound transports,    wherein the maximum number of consumer threads equals the number of inbound message topics,    wherein the maximum number of publisher threads equals the number of inbound message topics, and    wherein the number of said at least one file equals the number of inbound topics times the number of outbound transports; and    (d) a replay server for storing all messages published by said at least one publisher process means and for republishing certain ones of the messages to a subscriber on demand of the subscriber.    
     
     
         30 . The system of  claim 29  wherein the write and read offsets include: 
 an END offset written by said at least one consumer process means to said means for maintaining offsets for a batch of messages stored in said at least one file; and    a START offset written by said at least one publisher process means to said means for maintaining offsets for a batch of messages read from said at least one file.    
     
     
         31 . The system of  claim 30  wherein said START offset precedes said END offset for a batch of messages.  
     
     
         32 . The system of  claim 29  wherein said replay server republishes messages directly to the subscriber.  
     
     
         33 . The system of  claim 29  wherein said replay server republishes messages to said router means for delivery by said router means to the subscriber.