Dynamic randomized controlled testing with consumer electronics devices

Abstract

One embodiment of the present invention sets forth a technique for performing dynamic randomized controlled testing to evaluate design choices for a consumer electronics (CE) device. A plurality of CE devices may communicate with a server system over a data communications network to be assigned a variant of the design choice. The variant may correspond to a variant of a user interface component of the CE device or a functional component of the CE device. Once assigned, feedback may be collected from the plurality of CE devices specifying one or more performance metrics associated with the design variant assigned to a respective CE device.

I claim: 
     
       1. A computer-implemented method for evaluating a design choice for a consumer electronics (CE) device, the method comprising:
 receiving, from each of a plurality of CE devices, a communication over a data communication network; 
 in response to the received communication from each respective CE device:
 determining an eligibility of each of the plurality of CE devices to participate in the dynamic randomized controlled testing, 
 assigning one of a first design variant of the design choice and a second design variant of the design choice to the CE device, and 
 transmitting an indication of the assigned design variant of the design choice to the CE device; and 
 
 collecting feedback from the plurality of CE devices specifying one or more performance metrics associated with the design variant assigned to a respective CE device. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein the design variant provides a server-delivered user interface component transmitted to the CE device. 
     
     
       3. The computer-implemented method of  claim 1 , wherein the design variant provides one or more user interface parameters used to configure a user interface display generated by the CE device. 
     
     
       4. The computer-implemented method of  claim 1 , wherein the design variant provides executable logic used to provide one or more user features to a user of the CE device. 
     
     
       5. The computer-implemented method of  claim 1 , wherein the design variant provides one or more parameters used to select features to enable or disable on the CE device. 
     
     
       6. The computer-implemented method of  claim 1 , further comprising:
 in response to the feedback collected from the plurality of CE devices, determining that the performance metrics of the first design variant cross a specified threshold, reassigning CE devices, of the plurality assigned to receive the first design variant, to receive the second design variant. 
 
     
     
       7. The computer-implemented method of  claim 1 , wherein the CE device is a computing device configured to playback media data stored on a computer-readable storage medium. 
     
     
       8. The computer-implemented method of  claim 1 , wherein the design variant is randomly assigned to each of the plurality of the CE devices. 
     
     
       9. A computer-implemented method for evaluating a design choice for a consumer electronics (CE) device, the method comprising:
 providing a computing device configured to playback an audio-visual stream received over the data communications network; 
 receiving, from each of a plurality of CE devices, a communication over a data communication network; 
 in response to the received communication from each respective CE device:
 assigning one of a first design variant of the design choice and a second design variant of the design choice to the CE device, and 
 transmitting an indication of the assigned design variant of the design choice to the CE device; and 
 
 collecting feedback from the plurality of CE devices specifying one or more performance metrics associated with the design variant assigned to a respective CE device. 
 
     
     
       10. The computer-implemented method of  claim 9 , wherein the design variant provides executable logic used to provide one or more user features to a user of the CE device. 
     
     
       11. The computer-implemented method of  claim 9 , wherein the design variant provides one or more parameters used to select features to enable or disable on the CE device. 
     
     
       12. The computer-implemented method of  claim 9 , further comprising:
 in response to the feedback collected from the plurality of CE devices, determining that the performance metrics of the first design variant cross a specified threshold, reassigning CE devices, of the plurality assigned to receive the first design variant, to receive the second design variant. 
 
     
     
       13. A non-transitory computer-readable storage medium including instructions that, when executed by a processing unit, cause the processing unit to performing an operation for performing dynamic randomized controlled testing to evaluate design choices for a consumer electronics (CE) device, the operation comprising:
 receiving, from each of a plurality of CE devices, a communication over a data communication network; 
 in response to the received communication from each respective CE device:
 determining an eligibility of each of the plurality of CE devices to participate in the dynamic randomized controlled testing, 
 assigning one of a first design variant of the design choice and a second design variant of the design choice to the CE device, and 
 transmitting an indication of the assigned design variant of the design choice to the CE device; and 
 
 collecting feedback from the plurality of CE devices specifying one or more performance metrics associated with the design variant assigned to a respective CE device. 
 
     
     
       14. The computer-readable storage medium of  claim 13 , wherein the design variant comprises one of a server-delivered user interface component transmitted to the CE device and one or more user interface parameters used to configure a user interface display generated by the CE device. 
     
     
       15. The computer-readable storage medium of  claim 13 , wherein the design variant provides one of an executable logic used to provide one or more user features to a user of the CE device and one or more parameters used to select features to enable or disable on the CE device. 
     
     
       16. The computer-readable storage medium of  claim 13 , wherein the operation further comprises:
 in response to the feedback collected from the plurality of CE devices, determining that the performance metrics of the first design variant cross a specified threshold, reassigning CE devices, of the plurality, assigned to receive the first design variant, to receive the second design variant. 
 
     
     
       17. The computer-readable storage medium of  claim 13 , wherein the CE device is a computing device configured to playback an audio-visual stream received over the data communications network. 
     
     
       18. The computer-readable storage medium of  claim 13 , wherein the design variant is randomly assigned to each of the plurality of the CE devices. 
     
     
       19. A system, comprising:
 a memory that includes an application configured to performing dynamic randomized controlled testing to evaluate design choices for a consumer electronics (CE) device; and 
 a processing unit coupled to the memory and configured to:
 receive, from each of a plurality of CE devices, a communication over a data communication network; 
 in response to the received communication from each respective CE device:
 determining an eligibility of each of the plurality of CE devices to participate in the dynamic randomized controlled testing, 
 assign one of a first design variant of the design choice and a second design variant of the design choice to the CE device, and 
 transmit an indication of the assigned design variant of the design choice to the CE device; and 
 
 collect feedback from the plurality of CE devices specifying one or more performance metrics associated with the design variant assigned to a respective CE device. 
 
 
     
     
       20. The system of  claim 19 , wherein the design variant comprises one of a server-delivered user interface component transmitted to the CE device and one or more user interface parameters used to configure a user interface display generated by the CE device. 
     
     
       21. The system of  claim 19 , wherein the design variant provides one of an executable logic used to provide one or more user features to a user of the CE device and one or more parameters used to select features to enable or disable on the CE device. 
     
     
       22. The system of  claim 19 , wherein the processing unit is further configured to:
 in response to the feedback collected from the plurality of CE devices, determine that the performance metrics of the first design variant cross a specified threshold, reassigning CE devices, of the plurality, assigned to receive the first design variant, to receive the second design variant. 
 
     
     
       23. The system of  claim 19 , wherein the CE device provides one of a computing device configured to playback an audio-visual stream received over the data communications network. 
     
     
       24. The system of  claim 19 , wherein the design variant is randomly assigned to each of the plurality of the CE devices.