Classification of Psychiatric Disorder Related Spontaneous Communication Using Large Language Model Embeddings

Abstract

A method includes obtaining a corpus of input text including a plurality of textual utterance sets. Each textual utterance set from the plurality of textual utterance sets includes a respective set of multiple textual utterances originating from a respective one of multiple disorders/conditions. For each corresponding textual utterance: the method also includes processing, using a large language model (LLM), the corresponding textual utterance to generate a respective LLM embedding that represents the corresponding textual utterance in a high dimensional embedding space and processing, using a classification model, the respective LLM embedding to predict a classification label for the corresponding textual utterance, the predicted classification label including one of the multiple disorders/conditions. The method also includes training the classification model based on the classification label predicted for each corresponding textual utterance and the corresponding disorder/condition label paired with each corresponding textual utterance.

What is claimed is: 
     
         1 . A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
 obtaining a corpus of input text comprising a plurality of textual utterance sets, each textual utterance set from the plurality of textual utterance sets comprising a respective set of multiple textual utterances originating from a respective one of multiple disorders/conditions;   for each corresponding textual utterance from each respective textual utterance set:
 processing, using a large language model (LLM), the corresponding textual utterance to generate a respective LLM embedding that represents the corresponding textual utterance in a high dimensional embedding space, wherein the corresponding textual utterance is paired with a corresponding disorder/condition label indicating the respective disorder/condition from which the corresponding textual utterance originated; and 
 processing, using a classification model, the respective LLM embedding to predict a classification label for the corresponding textual utterance, the predicted classification label comprising one of the multiple disorders/conditions; and 
   based on the classification label predicted for each corresponding textual utterance and the corresponding disorder/condition label paired with each corresponding textual utterance, training the classification model.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the operations further comprise:
 receiving regular expression codes indicating a list of terms related to the multiple disorders/conditions; and   for each respective textual utterance set:
 inspecting, using the regular expression codes, the textual utterances from the respective set of multiple textual utterances to identify any textual utterances containing one or more expressions related to the respective disorder/condition among the multiple disorders/conditions from which the respective set of multiple textual utterances originated; and 
 for each corresponding textual utterance in the respective set of multiple textual utterances identified as containing any of the one or more expressions related to the respective disorder/condition, modifying or removing the corresponding textual utterance from the respective textual utterance set. 
   
     
     
         3 . The computer-implemented method of  claim 1 , wherein each textual utterance from each respective set of multiple textual utterances comprises one or more sentences. 
     
     
         4 . The computer-implemented method of  claim 1 , wherein each textual utterance from each respective set of multiple textual utterances comprises a post from an individual to an online forum topic associated with the respective disorder/condition. 
     
     
         5 . The computer-implemented method of  claim 1 , wherein the multiple disorders/conditions comprise multiple common psychiatric disorders/conditions. 
     
     
         6 . The computer-implemented method of  claim 5 , wherein the multiple common psychiatric disorders/conditions comprise schizophrenia, borderline personality disorder, depression, attention deficit hyperactivity disorder, anxiety, post-traumatic stress disorder, and bipolar disorder. 
     
     
         7 . The computer-implemented method of  claim 5 , wherein the multiple common psychiatric disorders/conditions are selected from the group consisting of schizophrenia, borderline personality disorder, depression, attention deficit hyperactivity disorder, anxiety, post-traumatic stress disorder, and bipolar disorder. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein the operations further comprise:
 receiving frequency information associated with the number of textual utterances in each respective textual utterance set,   wherein training the classification model is further based on the frequency information.   
     
     
         9 . The computer-implemented method of  claim 8 , wherein the classification model is trained based on the frequency information by assigning weights to the respective LLM embeddings generated for the textual utterances in each respective textual utterance set that are that are inversely proportional to the number of the textual utterances in each respective textual utterance set. 
     
     
         10 . The computer-implemented method of  claim 1 , wherein the LLM is trained using representational instruction tuning. 
     
     
         11 . The computer-implemented method of  claim 10 , wherein the LLM is further trained using generative instruction tuning. 
     
     
         12 . The computer-implemented method of  claim 1 , wherein the LLM comprises a plurality of sliding window attention (SWA) layers. 
     
     
         13 . The computer-implemented method of  claim 1 , wherein the LLM comprises a plurality of grouped-query attention (GQA) layers. 
     
     
         14 . The computer-implemented method of  claim 1 , wherein, when processing the corresponding textual utterance to generate the respective LLM embedding, the LLM employs bidirectional attention followed by mean pooling of a final hidden state to generate the respective LLM embedding. 
     
     
         15 . The computer-implemented method of  claim 1 , wherein the classification model comprises a multiclass classifier that employs extreme gradient boosting. 
     
     
         16 . The computer-implemented method of  claim 1 , wherein the operations further comprise generating a two-dimensional visualization that represents the LLM embeddings projected in a two-dimensional embedding space. 
     
     
         17 . The computer-implemented method of  claim 1 , wherein the operations further comprise, after training the classification model:
 receiving a textual input of one or more sentences from an individual, the textual input not paired with any disorder/condition label that indicates the respective disorder/condition associated with the corresponding textual utterance;   processing, using the LLM, the textual input to generate a respective LLM embedding that represents the textual input in the high dimensional embedding space; and   processing, using the trained classification model, the respective LLM embedding to predict a classification label for the textual input.   
     
     
         18 . A system comprising:
 data processing hardware; and   memory hardware in communication with the data processing hardware and storing instructions that when executed by the data processing hardware causes the data processing hardware to perform operations comprising:
 obtaining a corpus of input text comprising a plurality of textual utterance sets, each textual utterance set from the plurality of textual utterance sets comprising a respective set of multiple textual utterances originating from a respective one of multiple disorders/conditions; 
 for each corresponding textual utterance from each respective textual utterance set:
 processing, using a large language model (LLM), the corresponding textual utterance to generate a respective LLM embedding that represents the corresponding textual utterance in a high dimensional embedding space, wherein the corresponding textual utterance is paired with a corresponding disorder/condition label indicating the respective disorder/condition from which the corresponding textual utterance originated; and 
 processing, using a classification model, the respective LLM embedding to predict a classification label for the corresponding textual utterance, the predicted classification label comprising one of the multiple disorders/conditions; and 
 
 based on the classification label predicted for each corresponding textual utterance and the corresponding disorder/condition label paired with each corresponding textual utterance, training the classification model. 
   
     
     
         19 . The system of  claim 18 , wherein the operations further comprise:
 receiving regular expression codes indicating a list of terms related to the multiple disorders/conditions; and   for each respective textual utterance set:
 inspecting, using the regular expression codes, the textual utterances from the respective set of multiple textual utterances to identify any textual utterances containing one or more expressions related to the respective disorder/condition among the multiple disorders/conditions from which the respective set of multiple textual utterances originated; and 
 for each corresponding textual utterance in the respective set of multiple textual utterances identified as containing any of the one or more expressions related to the respective disorder/condition, modifying or removing the corresponding textual utterance from the respective textual utterance set. 
   
     
     
         20 . The system of  claim 18 , wherein each textual utterance from each respective set of multiple textual utterances comprises one or more sentences. 
     
     
         21 . The system of  claim 18 , wherein each textual utterance from each respective set of multiple textual utterances comprises a post from an individual to an online forum topic associated with the respective disorder/condition. 
     
     
         22 . The system of  claim 18 , wherein the multiple disorders/conditions comprise multiple common psychiatric disorders/conditions. 
     
     
         23 . The system of  claim 22 , wherein the multiple common psychiatric disorders/conditions comprise schizophrenia, borderline personality disorder, depression, attention deficit hyperactivity disorder, anxiety, post-traumatic stress disorder, and bipolar disorder. 
     
     
         24 . The system of  claim 22 , wherein the multiple common psychiatric disorders/conditions are selected from the group consisting of schizophrenia, borderline personality disorder, depression, attention deficit hyperactivity disorder, anxiety, post-traumatic stress disorder, and bipolar disorder. 
     
     
         25 . The system of  claim 18 , wherein the operations further comprise:
 receiving frequency information associated with the number of textual utterances in each respective textual utterance set,   wherein training the classification model is further based on the frequency information.   
     
     
         26 . The system of  claim 25 , wherein the classification model is trained based on the frequency information by assigning weights to the respective LLM embeddings generated for the textual utterances in each respective textual utterance set that are that are inversely proportional to the number of the textual utterances in each respective textual utterance set. 
     
     
         27 . The system of  claim 18 , wherein the LLM is trained using representational instruction tuning. 
     
     
         28 . The system of  claim 27 , wherein the LLM is further trained using generative instruction tuning. 
     
     
         29 . The system of  claim 18 , wherein the LLM comprises a plurality of sliding window attention (SWA) layers. 
     
     
         30 . The system of  claim 18 , wherein the LLM comprises a plurality of grouped-query attention (GQA) layers. 
     
     
         31 . The system of  claim 18 , wherein, when processing the corresponding textual utterance to generate the respective LLM embedding, the LLM employs bidirectional attention followed by mean pooling of a final hidden state to generate the respective LLM embedding. 
     
     
         32 . The system of  claim 18 , wherein the classification model comprises a multiclass classifier that employs extreme gradient boosting. 
     
     
         33 . The system of  claim 18 , wherein the operations further comprise generating a two-dimensional visualization that represents the LLM embeddings projected in a two-dimensional embedding space. 
     
     
         34 . The system of  claim 18 , wherein the operations further comprise, after training the classification model:
 receiving a textual input of one or more sentences from an individual, the textual input not paired with any disorder/condition label that indicates the respective disorder/condition associated with the corresponding textual utterance;   processing, using the LLM, the textual input to generate a respective LLM embedding that represents the textual input in the high dimensional embedding space; and   processing, using the trained classification model, the respective LLM embedding to predict a classification label for the textual input.