Systems and methods for automated aggregated content comment generation

Abstract

Systems and methods for automated aggregated content commenting are disclosed herein. Such a system can include a memory including a content database; an evaluation criteria database; and a comment database. The system can include at least one server. The at least one server can: receive a content identifier; receive a user identifier; provide a representation of the piece of content to the user; receive a comment from the user identifying a portion of the piece of content; generate a portion of the comment database linked to the user and the content; generate an entry in the portion of the comment database linked to the user and the content; and store the comment in the entry in the portion of the comment database.

What is claimed is: 
     
         1 . A system for verifying an evaluation of subject matter, the system comprising:
 memory comprising an evaluation criteria comprising a plurality of sub criteria;   a processor configured to:
 receive a portfolio comprising a compilation of artifacts comprising work product of an evaluatee; 
 provide one of the artifacts to an evaluator; 
 receive a plurality of first indications identifying a first portion of the artifact and a first portion of the evaluation criteria; 
 assign a value to the plurality of indications identifying the evaluator as source of the indications; 
 receive a first evaluation of the artifact based on the evaluation criteria and the first indications; 
 receive a second evaluation of the artifact, second indications, and second associated portions of the artifact; 
 compare the first and second evaluations of the artifact, the first and second indications, and the associated portions of the artifact; and 
 indicate differences between the first and second evaluations of the artifact, between the first and second indications, and the associated portions of the artifact. 
   
     
     
         2 . The system of  claim 1 , wherein the processor is further configured to select the evaluator. 
     
     
         3 . The system of  claim 1 , wherein the identified portion of the evaluation criteria comprises a sub-criterion 
     
     
         4 . The system of  claim 1 , further comprising a user device configured to:
 display the first evaluation of the artifact;   display the first indications; and   display first associated portions of the artifact.   
     
     
         5 . The system of  claim 4 , wherein the comparison of the first and second evaluations of the artifact further comprises:
 generation of a difference value, wherein the difference value characterizes a degree of difference between the first and second evaluations;   retrieval of an acceptance threshold, wherein the acceptance threshold is a value demarking levels of acceptable and unacceptable differences between the first and second evaluations; and   comparison of the difference value to the acceptance threshold.   
     
     
         6 . The system of  claim 5 , further comprising comparing the first indications and first associated portions of the artifact with the second indications and second associated portions of the artifact when the comparison of the difference value to the acceptance threshold indicates an unacceptable level of difference between the first and second evaluations. 
     
     
         7 . The system of  claim 5 , wherein the processor is further configured to automatically generate a final score based on the received plurality of first indications. 
     
     
         8 . The system of  claim 1 , wherein the memory further comprises: the compilation of artifacts; the plurality of first indications; the first evaluation of the artifact, and the second evaluation of the artifact. 
     
     
         9 . The system of  claim 1 , wherein the first indications comprise first tags and wherein the second indications comprise second tags. 
     
     
         10 . The system of  claim 1 , wherein the processor is further configured to provide an intervention based on the differences between the first and second evaluations of the artifact, the first and second indications of the evaluation criteria, and the associated portions of the artifact. 
     
     
         11 . The system of  claim 10 , wherein the intervention comprises remedial training. 
     
     
         12 . A method of verifying an evaluation of subject matter comprising:
 receiving a portfolio comprising a compilation of artifacts comprising work product of an evaluatee;   providing one of the artifacts to an evaluator;   receiving a plurality of first indications identifying a first portion of the artifact and a first portion of an evaluation criteria, wherein the evaluation criteria comprises a plurality of sub criteria;   assigning a value identifying the evaluator as source of the indications to the plurality of indications;   receiving a first evaluation of the artifact based on the evaluation criteria and the first indications;   receiving a second evaluation of the artifact, second indications, and second associated portions of the artifact;   comparing the first and second evaluations of the artifact, the first and second indications, and the associated portions of the artifact; and   providing an indication of differences between the first and second evaluations of the artifact, the first and second indications, and the associated portions of the artifact.   
     
     
         13 . The method of  claim 12 , further comprising selecting the evaluator. 
     
     
         14 . The method of  claim 12 , wherein the identified portion of the evaluation criteria comprises a sub-criterion 
     
     
         15 . The method of  claim 14 , wherein the comparison of the first and second evaluation s of the artifact further comprises:
 generating a difference value, wherein the difference value characterizes a degree of difference between the first and second evaluations;   retrieving an acceptance threshold, wherein the acceptance threshold is a value demarking levels of acceptable and unacceptable differences between the first and second evaluations; and   comparing the difference value to an acceptance threshold.   
     
     
         16 . The method of  claim 15 , further comprising comparing the first indications and first associated portions of the artifact with the second indications and second associated portions of the artifact when the comparison of the difference value to the acceptance threshold indicates an unacceptable level of difference between the first and second evaluations. 
     
     
         17 . The method of  claim 14 , further comprising generating a final score based on the received plurality of first indications. 
     
     
         18 . The method of  claim 12 , wherein the first indications comprise first tags and wherein the second indications comprise second tags. 
     
     
         19 . The method of  claim 12 , further comprising: providing an intervention based on the differences between the first and second evaluations of the artifact, the first and second indications of the evaluation criteria, and the associated portions of the artifact. 
     
     
         20 . The method of  claim 19 , wherein the intervention comprises remedial training.