Evaluation control

Abstract

Systems and methods for evaluation control are disclosed herein. An evaluation control system can be used in evaluating one or several artifacts. The evaluation control system can be used to verify an evaluation and/or to provide evaluation training. The evaluation system can include a processor and a user device. The evaluation system can provide one or several artifacts and evaluation criteria to an evaluator. The evaluation system can receive one or several tags associated with the one or several artifacts and/or the evaluation criteria. The one or several tags can include information that, in connection with the evaluation criteria, support an evaluation and/or score for the one or several artifacts.

What is claimed is: 
     
         1 . A system for verifying the evaluation of subject matter, the system comprising:
 a processor configured to:
 receive a portfolio, wherein the portfolio comprises a compilation of artifacts, wherein the artifacts comprise work product of an evaluatee; 
 provide one of the artifacts to an evaluator; 
 receive a plurality of first indications of an evaluation criteria, wherein the evaluation criteria comprises a plurality of sub-criteria, wherein the first indications of the evaluation criteria identify a first portion of the artifact and a first portion of the evaluation criteria; 
 assign a value to the plurality of indications of the evaluation criteria, wherein the value identifies the source of the indications of the evaluation criteria, wherein the source of the indications of the evaluation criteria is the evaluator; 
 receive a first evaluation of the artifact, wherein the evaluation is based on the evaluation criteria and the first indications of the evaluation criteria; 
 receive a second evaluation of the artifact, second indications of the evaluation criteria, and second associated portions of the artifact; 
 compare the first and second evaluations of the artifact, indications of the evaluation criteria, and the associated portions of the artifact; and 
 provide an indication of the differences between the first and second evaluations of the artifact, indications of the evaluation criteria, and the associated portions of the artifact; and 
   memory configured to store:
 the received portfolio; 
 the first evaluation of the artifact; 
 the second evaluation of the artifact; 
 the second indications of the evaluation criteria and the second associated portions of the artifact; and 
 the result of the comparison of the first and second evaluations of the artifact. 
   
     
     
         2 . The system of  claim 1 , wherein the processor is further configured to select the evaluator. 
     
     
         3 . The system of  claim 1 , wherein the identified portion of the evaluation criteria comprises a sub-criterion 
     
     
         4 . The system of  claim 1 , further comprising a user device configured to:
 display the first evaluation of the artifact;   display the first indications of the evaluation criteria; and   display first associated portions of the artifact.   
     
     
         5 . The system of  claim 4 , wherein the comparison of the first and second evaluations of the artifact further comprises:
 generation of a difference value, wherein the difference value characterizes the degree of difference between the first and second evaluations;   retrieval of an acceptance threshold, wherein the acceptance threshold is a value demarking levels of acceptable and unacceptable differences between the first and second evaluations; and   comparison of the difference value to acceptance threshold.   
     
     
         6 . The system of  claim 5 , further comprising comparing the first indications of the evaluation criteria and first associated portions of the artifact with the second indications of the evaluation criteria and second associated portions of the artifact if the comparison of the difference value to the acceptance threshold indicates an unacceptable level of difference between the first and second evaluations. 
     
     
         7 . The system of  claim 5 , wherein the processor is further configured to generate a final score based on the received plurality of first indications of an evaluation criteria. 
     
     
         8 . A method of verifying the evaluation of subject matter comprising:
 receiving a portfolio, wherein the portfolio comprises a compilation of artifacts, wherein the artifacts comprise work product of an evaluatee;   providing one of the artifacts to an evaluator;   receiving a plurality of first indications of an evaluation criteria, wherein the evaluation criteria comprises a plurality of sub-criteria, wherein the first indications of the evaluation criteria identify a first portion of the artifact and a first portion of the evaluation criteria;   assigning a value to the plurality of indications of the evaluation criteria, wherein the value identifies the source of the indications of the evaluation criteria;   receiving a first evaluation of the artifact, wherein the evaluation is based on the evaluation criteria and the first indications of the evaluation criteria;   providing the first evaluation of the artifact, the first indications of the evaluation criteria, and first associated portions of the artifact;   receiving a second evaluation of the artifact, second indications of the evaluation criteria, and second associated portions of the artifact;   comparing the first and second evaluations of the artifact, indications of the evaluation criteria, and the associated portions of the artifact; and   providing an indication of the differences between the first and second evaluations of the artifact, indications of the evaluation criteria, and the associated portions of the artifact.   
     
     
         9 . The method of  claim 8 , further comprising selecting the evaluator. 
     
     
         10 . The method of  claim 8 , wherein the identified portion of the evaluation criteria comprises a sub-criterion. 
     
     
         11 . The method of  claim 10 , wherein the comparison of the first and second evaluations of the artifact further comprises:
 generating a difference value, wherein the difference value characterizes the degree of difference between the first and second evaluations;   retrieving an acceptance threshold, wherein the acceptance threshold is a value demarking levels of acceptable and unacceptable differences between the first and second evaluations; and   comparing the difference value to acceptance threshold.   
     
     
         12 . The method of  claim 11 , further comprising comparing the first indications of the evaluation criteria and first associated portions of the artifact with the second indications of the evaluation criteria and second associated portions of the artifact if the comparison of the difference value to the acceptance threshold indicates an unacceptable level of difference between the first and second evaluations. 
     
     
         13 . The method of  claim 10 , further comprising generating a final score based on the received plurality of first indications of an evaluation criteria. 
     
     
         14 . A method of training an evaluator comprising:
 generating a portfolio comprising a compilation of artifacts, wherein the artifacts comprise work product of an evaluatee;   receiving evaluation criteria associated with the portfolio, wherein the evaluation criteria comprise a plurality of sub-criteria;   receiving a key, wherein the key comprises a plurality of indications of an evaluation criteria, wherein the indications in the key are the correct indications of an evaluation criteria for a portfolio; wherein the indications of the of the evaluation criteria in the key identify a portion of one of the artifacts and a sub-criteria of the evaluation criteria;   providing the portfolio to a trainee;   receiving a portfolio evaluation, wherein the portfolio evaluation comprises plurality of indications of the evaluation criteria, wherein the indications in the portfolio evaluation are received from the trainee; wherein the indications of the evaluation criteria in the portfolio evaluation identify a portion of one of the artifacts and a sub-criteria of the evaluation criteria;   comparing the key and the portfolio evaluation according to a Boolean function to determine the accuracy of the portfolio evaluation; and   providing an indicator of the accuracy of the portfolio evaluation.   
     
     
         15 . The method of  claim 14 , wherein the comparison of the key and the portfolio evaluation comprises comparing the score of the artifact in the portfolio evaluation with the score of the artifact in the key. 
     
     
         16 . The method of  claim 15 , wherein the comparison of the portfolio evaluation and the key further comprises:
 generating a difference value, wherein the difference value characterizes the degree of difference between the portfolio evaluation and the key;   retrieving an acceptance threshold, wherein the acceptance threshold is a value demarking levels of acceptable and unacceptable differences between the portfolio evaluation and the key; and   comparing the difference value to acceptance threshold.   
     
     
         17 . The method of  claim 14 , wherein the comparison of the key and the portfolio evaluation comprises the comparison of the indications of the evaluation criteria in the portfolio evaluation to the indications of the evaluation criteria in the key. 
     
     
         18 . The method of  claim 15 , further comprising providing additional training material.