Tuning fraud-detection rules using machine learning

Abstract

Techniques are disclosed relating to tuning fraud-detection rules using machine learning. In some embodiments, a server system may maintain rule information indicative of a plurality of fraud-detection rules for a transaction system. For example, in some embodiments, the server system may implement a fraud-detection service for the transaction system. In some embodiments, the server system may select a first rule to update, where the first rule includes one or more evaluation criteria and one or more corresponding user-defined threshold values. The server system may apply a machine learning algorithm to training data associated with the transaction system to generate an updated version of the first rule. The server system may then compare a performance of the first rule and the updated version of the first rule and, based on that performance, determine whether to suggest the updated version of the first rule to the transaction system.

What is claimed is: 
     
         1 . A method, comprising:
 maintaining, by a computer system, rule information indicative of a plurality of fraud-detection rules for a transaction system, wherein a given one of the plurality of fraud-detection rules specifies:
 a plurality of evaluation criteria used to evaluate a level of risk for transactions associated with the transaction system, and 
 for each of the plurality of evaluation criteria, one or more corresponding threshold values; 
   performing, by the computer system, tuning operations to tune a first one of the plurality of fraud-detection rules, wherein the tuning operations include:
 retrieving metadata associated with the first fraud-detection rule, wherein the metadata includes a first evaluation criterion and an existing threshold value; 
 applying a selected algorithm to transaction data to determine an updated threshold value for the first evaluation criterion; and 
 evaluating a performance of the first fraud-detection rule using the updated threshold value; and 
   based on the evaluating, sending, by the computer system to the transaction system, a suggested update to the first fraud-detection rule.   
     
     
         2 . The method of  claim 1 , wherein the applying the selected algorithm includes:
 generating a decision tree based on the transaction data, wherein the transaction data includes final fraud decisions for previous transactions associated with the transaction system; and   traversing the decision tree, based on the first fraud-detection rule, to determine the updated threshold value for the first evaluation criterion.   
     
     
         3 . The method of  claim 1 , further comprising:
 receiving, by the computer system from the transaction system, an acceptance of the suggested update to the first fraud-detection rule; and   updating, by the computer system, the rule information for the first fraud-detection rule to specify the updated threshold value as a corresponding threshold value for the first evaluation criterion.   
     
     
         4 . The method of  claim 3 , further comprising:
 prior to the receiving the acceptance of the suggested update to the first fraud-detection rule, receiving, by the computer system from the transaction system, a request to test the suggested update;   in response to the request, applying, by the computer system, the first fraud-detection rule using the updated threshold value to transaction data associated with the transaction system over a predetermined time period;   based on the applying, determining, by the computer system, test results for the first fraud-detection rule using the updated threshold value; and   sending, by the computer system to the transaction system, test data that includes:
 the test results for the first fraud-detection rule using the updated threshold value over the predetermined time period; and 
 actual performance results of the first fraud-detection rule using the existing threshold value over the predetermined time period. 
   
     
     
         5 . The method of  claim 1 , wherein the suggested update to the first fraud-detection rule includes a suggested second evaluation criteria and corresponding second threshold value. 
     
     
         6 . The method of  claim 1 , wherein the evaluating the performance of the first fraud-detection rule using the updated threshold value includes:
 based on one or more performance metrics, comparing a performance of the first fraud-detection rule using the existing threshold value to the performance of the first fraud-detection rule using the updated threshold value; and   in response to the performance of the first fraud-detection rule using the updated threshold value exceeding a predetermined threshold, sending the suggested update to the first fraud-detection rule to the transaction system.   
     
     
         7 . The method of  claim 6 , wherein the one or more performance metrics include at least one of precision, recall, an f1 score, an f2 score, and an f0.5 score. 
     
     
         8 . The method of  claim 1 , further comprising:
 initiating, by the computer system, the tuning operations in response to detecting a decrease in a performance of the first fraud-detection rule using the existing threshold value.   
     
     
         9 . A non-transitory, computer-readable medium having instructions stored thereon that are capable of execution by a computer system to perform operations comprising:
 maintaining rule information indicative of a plurality of fraud-detection rules for a transaction system, wherein a given one of the plurality of fraud-detection rules specifies:
 a plurality of evaluation criteria used to evaluate a level of risk for transactions associated with the transaction system, and 
 for each of the plurality of evaluation criteria, one or more corresponding threshold values; 
   performing tuning operations to tune a first one of the plurality of fraud-detection rules, wherein the tuning operations include:
 retrieving metadata associated with the first fraud-detection rule, wherein the metadata includes a first evaluation criterion and an existing threshold value; 
 applying a selected algorithm to transaction data to determine an updated threshold value for the first evaluation criterion; and 
 evaluating a performance of the first fraud-detection rule using the updated threshold value; and 
   based on the evaluating, sending, to the transaction system, a suggested update to the first fraud-detection rule.   
     
     
         10 . The non-transitory, computer-readable medium of  claim 9 , wherein the applying the selected algorithm includes:
 generating a decision tree based on the transaction data, wherein the transaction data includes final fraud decisions for previous transactions associated with the transaction system; and   traversing the decision tree, based on the first fraud-detection rule, to determine the updated threshold value for the first evaluation criterion.   
     
     
         11 . The non-transitory, computer-readable medium of  claim 9 , wherein the operations further comprise:
 receiving, by the computer system from the transaction system, an acceptance of the suggested update to the first fraud-detection rule; and   updating, by the computer system, the rule information for the first fraud-detection rule to specify the updated threshold value as a corresponding threshold value for the first evaluation criterion.   
     
     
         12 . The non-transitory, computer-readable medium of  claim 11 , wherein the operations further comprise:
 prior to the receiving the acceptance of the suggested update to the first fraud-detection rule, receiving, from the transaction system, a request to test the suggested update;   in response to the request, applying the first fraud-detection rule using the updated threshold value to transaction data associated with the transaction system over a predetermined time period;   based on the applying, determining test results for the first fraud-detection rule using the updated threshold value; and   sending, to the transaction system, test data that includes:
 the test results for the first fraud-detection rule using the updated threshold value over the predetermined time period; and 
 actual performance results of the first fraud-detection rule using the existing threshold value over the predetermined time period. 
   
     
     
         13 . The non-transitory, computer-readable medium of  claim 9 , wherein the suggested update to the first fraud-detection rule includes a suggested second evaluation criteria and corresponding second threshold value. 
     
     
         14 . The non-transitory, computer-readable medium of  claim 9 , the evaluating the performance of the first fraud-detection rule using the updated threshold value includes:
 based on one or more performance metrics, comparing a performance of the first fraud-detection rule using the existing threshold value to the performance of the first fraud-detection rule using the updated threshold value; and   in response to the performance of the first fraud-detection rule using the updated threshold value exceeding a predetermined threshold, sending the suggested update to the first fraud-detection rule to the transaction system.   
     
     
         15 . The non-transitory, computer-readable medium of  claim 9 , wherein the operations further comprise:
 initiating the tuning operations in response to detecting a decrease in a performance of the first fraud-detection rule using the existing threshold value.   
     
     
         16 . A system, comprising:
 a non-transitory memory storing instructions; and   a processor configured to execute the instructions to cause the system to:
 select a first rule to update, wherein the first rule is one of a set of rules used by the system to implement a fraud-detection service for a transaction system, wherein the first rule includes one or more evaluation criteria and one or more corresponding user-defined threshold values; 
 apply a machine learning algorithm to training data associated with the transaction system to generate an updated version of the first rule; 
 compare a performance of the first rule and the updated version of the first rule; 
 determine to suggest the updated version of the first rule to the transaction system based on the comparing; and 
 in response to receiving an acceptance of the updated version of the first rule, update the set of rules used to implement the fraud-detection service for the transaction system. 
   
     
     
         17 . The system of  claim 16 , wherein, to apply the machine learning algorithm, executing the instructions further causes the system to:
 generate a decision tree based on the training data, wherein the training data includes final fraud decisions for previous transactions associated with the transaction system; and   traverse the decision tree, based on the first rule, to determine an updated one or more threshold values for the one or more evaluation criteria.   
     
     
         18 . The system of  claim 16 , wherein executing the instructions further causes the system to:
 in response to a request to test the updated version of the first rule, apply the updated version of the first rule to transaction data associated with the transaction system over a predetermined time period;   determine test results for the updated version of the first rule; and   send, to the transaction system, test data that includes:
 the test results for the updated version of the first rule over the predetermined time period; and 
 actual performance results of the first rule over the predetermined time period. 
   
     
     
         19 . The system of  claim 16 , wherein the updated version of the first rule includes one or more updated threshold values for the one or more evaluation criteria. 
     
     
         20 . The system of  claim 16 , wherein executing the instructions further causes the system to:
 determine to apply the machine learning algorithm based on the performance of the first rule falling below a predetermined performance threshold.