Determining estimated delivery time of items obtained from a warehouse for users of an online concierge system to reduce probabilities of delivery after the estimated delivery time

Abstract

An online concierge system displays an interface to a user identifying an estimated time of arrival for an order. To generate the estimated time of arrival for the order, the online concierge system trains a prediction engine to predict delivery time based on a predicted selection time for a shopper to select the order for fulfillment and predicted travel time for the shopper to deliver items of the order to a location identified by the order. The online concierge system generates a policy optimization model that computes an adjustment for the predicted delivery time. The adjustment is determined by solving a stochastic optimization problem with a constraint on a probability of the order being delivered after the estimated time of arrival. The predicted delivery time combined with the adjustment determines the estimated time of delivery displayed to the user to balance between minimizing late deliveries and wait times.

1 . A computer-implemented method comprising:
 at a computer system comprising at least one processor and memory:   receiving an order at an online concierge system from a user, the order including one or more items and is associated with a delivery address;   generating a predicted total fulfillment time of the order, the predicted total fulfillment time comprising a time for a shopper to accept fulfillment of the order on behalf of the user and a time the one or more items are delivered to the delivery address;   generating an adjusted total fulfillment time by a machine learning model to increase a probability that the order will be fulfilled within the predicted total fulfillment time, wherein the adjusted total fulfillment time is generated from the predicted total fulfillment time based on a plurality of factors including a first factor of an accuracy of the predicted total and a second factor of the probability that the order will be fulfilled within the predicted total fulfillment time; and   displaying the adjusted total fulfillment time to the user in an interface in conjunction with information identifying the order.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein generating the predicted total fulfillment time of the order comprises:
 identifying one or more characteristics of the order;   predicting the time for the shopper to accept fulfillment of the order; and   generating a predicted delivery time based on the one or more characteristics of the order.   
     
     
         3 . The computer-implemented method of  claim 2 , wherein the one or more characteristics of the order includes a number of items in the order, a distance between a warehouse identified in the order and the delivery address, a value of the order, or an amount of compensation the user for fulfilling the order. 
     
     
         4 . The computer-implemented method of  claim 1 , wherein generating the predicted total fulfillment time of the order comprises:
 determining a travel time for the shopper delivering items from a warehouse identified by the order to the delivery address; and   determining a predicted delivery time as a combination of the travel time and a delivery specific selection time.   
     
     
         5 . The computer-implemented method of  claim 1 , wherein generating the predicted total fulfillment time of the order is by a second machine learning model that is trained on historical rates at which shoppers decide to select orders within a geographic region that includes the delivery address. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein the interface further displays an option for the user to select fulfillment of the order by another time of arrival outside of the adjusted total fulfillment time. 
     
     
         7 . The computer-implemented method of  claim 1 , wherein the time for a shopper to accept fulfillment of the order on behalf of the user is determined based on one or more of: a rate at which the online concierge system receives orders including locations within the delivery address, a rate at which shoppers select orders including locations within the delivery address, and a number of orders including locations within the delivery address received by the online concierge system and not selected for fulfillment by shoppers. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein training of the machine learning model comprises:
 setting a threshold probability of the order being fulfilled after the predicted total fulfillment time of the order,   imposing a constraint that a probability of the order being fulfilled after the predicted delivery time does not exceed the threshold probability, and   solving a stochastic optimization problem with the constraint that involves the first factor and the second factor.   
     
     
         9 . A non-transitory computer-readable medium configured to store code comprising instructions, the instructions, when executed by one or more processors, cause the one or more processors to:
 receive an order at an online concierge system from a user, the order including one or more items and is associated with a delivery address;   generate a predicted total fulfillment time of the order, the predicted total fulfillment time comprising a time for a shopper to accept fulfillment of the order on behalf of the user and a time the one or more items are delivered to the delivery address;   generate an adjusted total fulfillment time by a machine learning model to increase a probability that the order will be fulfilled within the predicted total fulfillment time, wherein the adjusted total fulfillment time is generated from the predicted total fulfillment time based on a plurality of factors including a first factor of an accuracy of the predicted total and a second factor of the probability that the order will be fulfilled within the predicted total fulfillment time; and   display the adjusted total fulfillment time to the user in an interface in conjunction with information identifying the order.   
     
     
         10 . The non-transitory computer-readable medium of  claim 9 , wherein the instructions for generating the predicted total fulfillment time of the order comprises instructions for:
 identifying one or more characteristics of the order;   predicting the time for the shopper to accept fulfillment of the order; and   generating a predicted delivery time based on the one or more characteristics of the order.   
     
     
         11 . The non-transitory computer-readable medium of  claim 10 , wherein the one or more characteristics of the order includes a number of items in the order, a distance between a warehouse identified in the order and the delivery address, a value of the order, or an amount of compensation the user for fulfilling the order. 
     
     
         12 . The non-transitory computer-readable medium of  claim 9 , wherein the instructions for generating the predicted total fulfillment time of the order comprises instructions for:
 determining a travel time for the shopper delivering items from a warehouse identified by the order to the delivery address; and   determining a predicted delivery time as a combination of the travel time and a delivery specific selection time.   
     
     
         13 . The non-transitory computer-readable medium of  claim 9 , wherein generating the predicted total fulfillment time of the order is by a second machine learning model that is trained on historical rates at which shoppers decide to select orders within a geographic region that includes the delivery address. 
     
     
         14 . The non-transitory computer-readable medium of  claim 9 , wherein the interface further displays an option for the user to select fulfillment of the order by another time of arrival outside of the adjusted total fulfillment time. 
     
     
         15 . The non-transitory computer-readable medium of  claim 9 , wherein the time for a shopper to accept fulfillment of the order on behalf of the user is determined based on one or more of: a rate at which the online concierge system receives orders including locations within the delivery address, a rate at which shoppers select orders including locations within the delivery address, and a number of orders including locations within the delivery address received by the online concierge system and not selected for fulfillment by shoppers. 
     
     
         16 . The non-transitory computer-readable medium of  claim 9 , wherein training of the machine learning model comprises:
 setting a threshold probability of the order being fulfilled after the predicted total fulfillment time of the order,   imposing a constraint that a probability of the order being fulfilled after the predicted delivery time does not exceed the threshold probability, and   solving a stochastic optimization problem with the constraint that involves the first factor and the second factor.   
     
     
         17 . A system comprising:
 one or more processors; and   memory configured to store code comprising instructions, the instructions, when executed by the one or more processors, cause the one or more processors to:
 receive an order at an online concierge system from a user, the order including one or more items and is associated with a delivery address; 
 generate a predicted total fulfillment time of the order, the predicted total fulfillment time comprising a time for a shopper to accept fulfillment of the order on behalf of the user and a time the one or more items are delivered to the delivery address; 
 generate an adjusted total fulfillment time by a machine learning model to increase a probability that the order will be fulfilled within the predicted total fulfillment time, wherein the adjusted total fulfillment time is generated from the predicted total fulfillment time based on a plurality of factors including a first factor of an accuracy of the predicted total and a second factor of the probability that the order will be fulfilled within the predicted total fulfillment time; and 
 display the adjusted total fulfillment time to the user in an interface in conjunction with information identifying the order. 
   
     
     
         18 . The system of  claim 17 , wherein the instructions for generating the predicted total fulfillment time of the order comprises instructions for:
 identifying one or more characteristics of the order;   predicting the time for the shopper to accept fulfillment of the order; and   generating a predicted delivery time based on the one or more characteristics of the order.   
     
     
         19 . The system of  claim 18 , wherein the one or more characteristics of the order includes a number of items in the order, a distance between a warehouse identified in the order and the delivery address, a value of the order, or an amount of compensation the user for fulfilling the order. 
     
     
         20 . The system of  claim 17 , wherein the instructions for generating the predicted total fulfillment time of the order comprises instructions for:
 determining a travel time for the shopper delivering items from a warehouse identified by the order to the delivery address; and   determining a predicted delivery time as a combination of the travel time and a delivery specific selection time.