Speech as a Biomarker for Depression

Sanne       Koops; Sanne    G.    Brederoo; Janna    N.    de Boer; Femke    G.    Nadema; Alban    E.    Voppel; Iris    E.    Sommer

Abstract

Background: Depression is a debilitating disorder that at present lacks a reliable biomarker to aid in diagnosis and early detection. Recent advances in computational analytic approaches have opened up new avenues in developing such a biomarker by taking advantage of the wealth of information that can be extracted from a person’s speech.

Objective: The current review provides an overview of the latest findings in the rapidly evolving field of computational language analysis for the detection of depression. We cover a wide range of both acoustic and content-related linguistic features, data types (i.e., spoken and written language), and data sources (i.e., lab settings, social media, and smartphone-based). We put special focus on the current methodological advances with regard to feature extraction and computational modeling techniques. Furthermore, we pay attention to potential hurdles in the implementation of automatic speech analysis.

Conclusion: Depressive speech is characterized by several anomalies, such as lower speech rate, less pitch variability and more self-referential speech. With current computational modeling techniques, such features can be used to detect depression with an accuracy of up to 91%. The performance of the models is optimized when machine learning techniques are implemented that suit the type and amount of data. Recent studies now work towards further optimization and generalizability of the computational language models to detect depression. Finally, privacy and ethical issues are of paramount importance to be addressed when automatic speech analysis techniques are further implemented in, for example, smartphones. Altogether, computational speech analysis is well underway towards becoming an effective diagnostic aid for depression.

Keywords: Computational speech analysis, natural language processing, machine learning, depression, biomarker, categorization, diagnosis.

[1]
World Health Organization. Depression and other common mental disorders: Global health estimates. 2017.  Available from: https://apps.who.int/iris/handle/10665/254610

[2]
American Psychiatric Association. Diagnostic and statistical manual of mental disorders.  (5th ed.), Arlington, VA 2013.  Available from: https://www.psychiatry.org/psychiatrists/practice/dsm

[3]
Mundt JC, Vogel AP, Feltner DE, Lenderking WR. Vocal acoustic biomarkers of depression severity and treatment response. Biol Psychiatry  2012; 72(7): 580-7.
 [http://dx.doi.org/10.1016/j.biopsych.2012.03.015] [PMID: 22541039]

[4]
Thibaut F. Controversies in psychiatry. Dialogues Clin Neurosci  2018; 20(3): 151-2.
 [http://dx.doi.org/10.31887/DCNS.2018.20.3/fthibaut] [PMID: 30581283]

[5]
Walsh CG, Chaudhry B, Dua P, et al. Stigma, biomarkers, and algorithmic bias: Recommendations for precision behavioral health with artificial intelligence. JAMIA Open  2020; 3(1): 9-15.
 [http://dx.doi.org/10.1093/jamiaopen/ooz054] [PMID: 32607482]

[6]
Hall JA, Harrigan JA, Rosenthal R. Nonverbal behavior in clinician-patient interaction. Appl Prev Psychol  1995; 4: 21-37.
 [http://dx.doi.org/10.1016/S0962-1849(05)80049-6]

[7]
Sobin C, Sackeim HA. Psychomotor symptoms of depression. Am J Psychiatry  1997; 154(1): 4-17.
 [http://dx.doi.org/10.1176/ajp.154.1.4] [PMID: 8988952]

[8]
Pinker S. The stuff of thought: Language as a window into human nature. New York: Penguin Group (Viking Press) 2007.

[9]
Kraepelin E. Manic depressive insanity and paranoia. J Nerv Ment Dis  1921; 53: 350.
 [http://dx.doi.org/10.1097/00005053-192104000-00057]

[10]
Wang J, Zhang L, Liu T, Pan W, Hu B, Zhu T. Acoustic differences between healthy and depressed people: A cross-situation study. BMC Psychiatry  2019; 19(1): 300.
 [http://dx.doi.org/10.1186/s12888-019-2300-7] [PMID: 31615470]

[11]
Alghowinem SM, Gedeon T, Goecke R, et al. Interpretation of depression detection models via feature selection methods. IEEE Trans Affect Comput  2020; 1: 1-1.
 [http://dx.doi.org/10.1109/TAFFC.2020.3035535]

[12]
Low DM, Bentley KH, Ghosh SS. Automated assessment of psychiatric disorders using speech: A systematic review. Laryngoscope Investig Otolaryngol  2020; 5(1): 96-116.
 [http://dx.doi.org/10.1002/lio2.354] [PMID: 32128436]

[13]
Cohen AS, McGovern JE, Dinzeo TJ, Covington MA. Speech deficits in serious mental illness: A cognitive resource issue? Schizophr Res  2014; 160(1-3): 173-9.
 [http://dx.doi.org/10.1016/j.schres.2014.10.032] [PMID: 25464920]

[14]
Liu Z, Kang H, Feng L, et al. Speech pause time: A potential biomarker for depression detection. 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). 2017 November 13-16; Kansas City, MO, USA. 
 [http://dx.doi.org/10.1109/BIBM.2017.8217971]

[15]
Jia Y, Liang Y, Zhu T. An analysis of voice quality of Chinese patients with depression.  2019 22nd Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O- COCOSDA) 2019 October, 1-6. 
 [http://dx.doi.org/10.1109/O-COCOSDA46868.2019.9060848]

[16]
Trifu RN, Nemes B, Bodea-Hațegan C, Cozman D. Linguistic indicators of language in major depressive disorder (MDD). An evidence based research. J Evid Based Psychother  2017; 17(1): 105-28.
 [http://dx.doi.org/10.24193/jebp.2017.1.7]

[17]
Tackman AM, Sbarra DA, Carey AL, et al. Depression, negative emotionality, and self-referential language: A multi-lab, multi-measure, and multi-language-task research synthesis. J Pers Soc Psychol  2019; 116(5): 817-34.
 [http://dx.doi.org/10.1037/pspp0000187] [PMID: 29504797]

[18]
Bernard JD, Baddeley JL, Rodriguez BF, et al. Depression, language, and affect: An examination of the influence of baseline depression and affect induction on language. J Lang Soc Psychol  2016; 35: 317-26.
 [http://dx.doi.org/10.1177/0261927X15589186]

[19]
Brockmeyer T, Zimmermann J, Kulessa D, et al. Me, myself, and I: self-referent word use as an indicator of self-focused attention in relation to depression and anxiety. Front Psychol  2015; 6: 1564.
 [http://dx.doi.org/10.3389/fpsyg.2015.01564] [PMID: 26500601]

[20]
Zimmermann J, Wolf M, Bock A, et al. The way we refer to ourselves reflects how we relate to others: Associations between first-person pronoun use and interpersonal problems. J Res Pers  2013; 47: 218-25.
 [http://dx.doi.org/10.1016/j.jrp.2013.01.008]

[21]
Jarrold W, Javitz HS, Krasnow R, et al. Depression and self-focused language in structured interviews with older men. Psychol Rep  2011; 109(2): 686-700.
 [http://dx.doi.org/10.2466/02.09.21.28.PR0.109.5.686-700] [PMID: 22238866]

[22]
Beck AT. Depression clinical, experimental and theoretical aspects. New York: Hoeber 1967.

[23]
Zimmermann J, Brockmeyer T, Hunn M, Schauenburg H, Wolf M. First-person pronoun use in spoken language as a predictor of future depressive symptoms: Preliminary evidence from a clinical sample of depressed patients. Clin Psychol Psychother  2017; 24(2): 384-91.
 [http://dx.doi.org/10.1002/cpp.2006] [PMID: 26818665]

[24]
Mor N, Winquist J. Self-focused attention and negative affect: A meta-analysis. Psychol Bull  2002; 128(4): 638-62.
 [http://dx.doi.org/10.1037/0033-2909.128.4.638] [PMID: 12081086]

[25]
Nolen-Hoeksema S, Wisco BE, Lyubomirsky S. Rethinking Rumination. Perspect Psychol Sci  2008; 3(5): 400-24.
 [http://dx.doi.org/10.1111/j.1745-6924.2008.00088.x] [PMID: 26158958]

[26]
Xu S, Yang Z, Chakraborty D, et al.  Automated Verbal and Non-verbal Speech Analysis of Interviews of Individuals with Schizophrenia and Depression. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); 2019 July 23-7; Berlin, Germany.
 [http://dx.doi.org/10.1109/EMBC.2019.8857071]

[27]
Cummins N, Scherer S, Krajewski J, et al. A review of depression and suicide risk assessment using speech analysis. Speech Commun  2015; 71: 10-49.
 [http://dx.doi.org/10.1016/j.specom.2015.03.004]

[28]
Morales M, Scherer S, Levitan R. A cross-modal review of indicators for depression detection systems. Proceedings of the Fourth Workshop on Computational Linguistics and Clinical Psychology-From Linguistic Signal to Clinical Reality.  Vancouver, BC, Canada. 2017; pp. 1-12.
 [http://dx.doi.org/10.18653/v1/W17-3101]

[29]
Tokuno S. Pathophysiological voice analysis for diagnosis and monitoring of depression. In: Kim YK, Ed.  Understanding Depression.  Singapore: Springer 2018; pp. 83-95.
 [http://dx.doi.org/10.1007/978-981-10-6577-4_6]

[30]
Mendiratta A, Scibelli F, Esposito AM, et al. Automatic detection of depressive states from speech. In: Esposito A, Ed.  Faudez-Zanuy M, Morabito FC, Pasero E Multidisciplinary Approaches to Neural Computing.  Cham: Springer 2018; pp. 301-14.
 [http://dx.doi.org/10.1007/978-3-319-56904-8_29]

[31]
Mundt JC, Snyder PJ, Cannizzaro MS, Chappie K, Geralts DS. Voice acoustic measures of depression severity and treatment response collected via interactive voice response (IVR) technology. J Neurolinguist  2007; 20(1): 50-64.
 [http://dx.doi.org/10.1016/j.jneuroling.2006.04.001] [PMID: 21253440]

[32]
Silva WJ, Lopes L, Galdino MKC, Almeida AA. Voice acoustic parameters as predictors of depression. J Voice  2021; S0892-1997(21): 205-8.
 [http://dx.doi.org/10.1016/j.jvoice.2021.06.018]

[33]
Jiang H, Hu B, Liu Z, et al. Detecting depression using an ensemble logistic regression model based on multiple speech features. Comput Math Methods Med  2018; 2018: 6508319.
 [http://dx.doi.org/10.1155/2018/6508319] [PMID: 30344616]

[34]
Yamamoto M, Takamiya A, Sawada K, et al. Using speech recognition technology to investigate the association between timing-related speech features and depression severity. PLoS One  2020; 15(9): e0238726.
 [http://dx.doi.org/10.1371/journal.pone.0238726] [PMID: 32915846]

[35]
Cannizzaro M, Harel B, Reilly N, Chappell P, Snyder PJ. Voice acoustical measurement of the severity of major depression. Brain Cogn  2004; 56(1): 30-5.
 [http://dx.doi.org/10.1016/j.bandc.2004.05.003] [PMID: 15380873]

[36]
Alim SA, Rashid NKA. Some Commonly Used Speech Feature Extraction Algorithms. London, UK: IntechOpen 2018.

[37]
Eyben F, Weninger F, Gross F, et al. Recent developments in openSMILE, the Munich open-source multimedia feature extractor. MM ’13: Proceedings of the 21st ACM International Conference on Multimedia.  Barcelona, Spain. 2013; pp. 835-.
 [http://dx.doi.org/10.1145/2502081.2502224]

[38]
Asgari M, Shafran I. Improvements to harmonic model for extracting better speech features in clinical applications. Comput Speech Lang  2018; 47: 298-313.
 [http://dx.doi.org/10.1016/j.csl.2017.08.005]

[39]
Khalid S, Khalil T, Nasreen S. A survey of feature selection and feature extraction techniques in machine learning. Science and Information Conference.  2014 August 2-29; London, UK. 2014; pp. 372-8.
 [http://dx.doi.org/10.1109/SAI.2014.6918213]

[40]
Wang X, Kumar A, Shelton CR, Wong BM. Harnessing deep neural networks to solve inverse problems in quantum dynamics: machine-learned predictions of time-dependent optimal control fields. Phys Chem Chem Phys  2020; 22(40): 22889-99.
 [http://dx.doi.org/10.1039/D0CP03694C] [PMID: 32935687]

[41]
Raza A, Bardhan S, Xu L, et al. A machine learning approach for predicting defluorination of per- and Polyfluoroalkyl Substances (PFAS) for their efficient treatment and removal. Environ Sci Technol Lett  2019; 6: 624-9.
 [http://dx.doi.org/10.1021/acs.estlett.9b00476]

[42]
Tasnim M, Stroulia E. Detecting Depression from Voice. In: Meurs MJ, Rudzicz F, Eds.  Advances in Artificial Intelligence Canadian AI 2019. Lecture Notes in Computer ScienceCham: Springer 2019.
 [http://dx.doi.org/10.1007/978-3-030-18305-9_47]

[43]
Alghowinem S, Goecke R, Wagner M, et al. A comparative study of different classifiers for detecting depression from spontaneous speech. IEEE International Conference on Acoustics, Speech and Signal Processing.  2013 May 6-31; BC, Canada. 2013; pp. 8022-6.
 [http://dx.doi.org/10.1109/ICASSP.2013.6639227]

[44]
McGinnis EW, Anderau SP, Hruschak J, et al. Giving voice to vulnerable children: Machine learning analysis of speech detects anxiety and depression in early childhood. IEEE J Biomed Health Inform  2019; 23(6): 2294-301.
 [http://dx.doi.org/10.1109/JBHI.2019.2913590] [PMID: 31034426]

[45]
Jiang H, Hu B, Liu Z, et al. Investigation of different speech types and emotions for detecting depression using different classifiers. Speech Commun  2017; 90: 39-46.
 [http://dx.doi.org/10.1016/j.specom.2017.04.001]

[46]
Liu Z, Wang D, Zhang L, Hu B. Tree for depression recognition in speech. arXiv Prepr  2020; 10(46): 38.

[47]
Zhao Z, Bao Z, Zhang Z, et al. Automatic assessment of depression from speech via a hierarchical attention transfer network and attention autoencoders. IEEE J Sel Top Signal Process  2020; 14: 423-34.
 [http://dx.doi.org/10.1109/JSTSP.2019.2955012]

[48]
Cummins N, Baird A, Schuller BW. Speech analysis for health: Current state-of-the-art and the increasing impact of deep learning. Methods  2018; 151: 41-54.
 [http://dx.doi.org/10.1016/j.ymeth.2018.07.007] [PMID: 30099083]

[49]
Yang L, Jiang D, Sahli H. Feature augmenting networks for improving depression severity estimation from speech signals. IEEE Access  2020; 8: 24033-45.
 [http://dx.doi.org/10.1109/ACCESS.2020.2970496]

[50]
Dubagunta SP, Vlasenko B, Doss MM.  Learning voice source related information for depression detection. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2019 May 12-17; Bringhton, UK; pp. 6525-9.
 [http://dx.doi.org/10.1109/ICASSP.2019.8683498]

[51]
Higuchi M, Tokuno S, Nakamura M, et al. Classification of bipolar disorder, major depressive disorder, and healthy state using voice. Asian J Pharm Clin Res  2018; 11: 89-93.
 [http://dx.doi.org/10.22159/ajpcr.2018.v11s3.30042]

[52]
Espinola CW, Gomes JC, Pereira JMS, et al. Vocal acoustic analysis and machine learning for the identification of schizophrenia. Res Biomed Eng  2021; 37: 33-46.
 [http://dx.doi.org/10.1007/s42600-020-00097-1]

[53]
de Boer JN, Voppel AE, Begemann MJH, Schnack HG, Wijnen F, Sommer IEC. Clinical use of semantic space models in psychiatry and neurology: A systematic review and meta-analysis. Neurosci Biobehav Rev  2018; 93: 85-92.
 [http://dx.doi.org/10.1016/j.neubiorev.2018.06.008] [PMID: 29890179]

[54]
Sugathadasa K, Ayesha B, de Silva N, et al. Synergistic union of word2vec and lexicon for domain specific semantic similarity. 2017 IEEE International Conference on Industrial and Information Systems (ICIIS).  2017 December 15-16; Peradeniya, Sri Lanka. 1-6.
 [http://dx.doi.org/10.1109/ICIINFS.2017.8300343]

[55]
Losada DE, Gamallo P. Evaluating and improving lexical resources for detecting signs of depression in text. Lang Resour Eval  2020; 54: 1-24.
 [http://dx.doi.org/10.1007/s10579-018-9423-1]

[56]
Zhang L, Ghosh R, Dekhil M, Hsu M, Liu B. Combining lexicon-based and learning-based methods for Twitter sentiment analysis.  HP Lab Tech Rep HPL 2011; p. 89.

[57]
Neuman Y, Cohen Y, Assaf D, Kedma G. Proactive screening for depression through metaphorical and automatic text analysis. Artif Intell Med  2012; 56(1): 19-25.
 [http://dx.doi.org/10.1016/j.artmed.2012.06.001] [PMID: 22771201]

[58]
De Choudhury M, Gamon M, Counts S, et al. Predicting depression via social media. Proceedings of the International AAAI Conference on Web and Social Media.  2013 June 28.

[59]
Williamson JR, Godoy E, Cha M, et al. Detecting depression using vocal, facial and semantic communication cues. Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge.  Amsterdam, Netherlands. 2016; pp. 11-8.
 [http://dx.doi.org/10.1145/2988257.2988263]

[60]
Guohou S, Lina Z, Dongsong Z. What reveals about depression level? The role of multimodal features at the level of interview questions. Inf Manage  2020; 57: 103349.
 [http://dx.doi.org/10.1016/j.im.2020.103349]

[61]
Xezonaki D, Paraskevopoulos G, Potamianos A, et al. Affective conditioning on hierarchical attention networks applied to depression detection from transcribed clinical interviews. Interspeech  2020; 2020: 4556-60.
 [http://dx.doi.org/10.21437/Interspeech.2020-2819]

[62]
Coppersmith G, Dredze M, Harman C. Quantifying mental health signals in Twitter. Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality.  Baltimore, Maryland, USA. 2014; pp. 51-60.
 [http://dx.doi.org/10.3115/v1/W14-3207]

[63]
Schwartz HA, Eichstaedt J, Kern M, et al. Towards assessing changes in degree of depression through facebook. Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality.  Baltimore, Maryland, USA. 2014; pp. 118-25.
 [http://dx.doi.org/10.3115/v1/W14-3214]

[64]
Tsugawa S, Kikuchi Y, Kishino F, et al. Recognizing depression from twitter activity. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems.  Seoul, Republic of Korea. 2015; pp. 3187-96.

[65]
Leis A, Ronzano F, Mayer MA, Furlong LI, Sanz F. Detecting signs of depression in tweets in Spanish: Behavioral and linguistic analysis. J Med Internet Res  2019; 21(6): e14199.
 [http://dx.doi.org/10.2196/14199] [PMID: 31250832]

[66]
Guntuku SC, Yaden DB, Kern ML, et al. Detecting depression and mental illness on social media: An integrative review. Curr Opin Behav Sci  2017; 18: 43-9.
 [http://dx.doi.org/10.1016/j.cobeha.2017.07.005]

[67]
Mart\’\inez-Castaño R, Pichel JC, Losada DE. A big data platform for real time analysis of signs of depression in social media. Int J Environ Res Public Health  2020; 17: 4752.
 [http://dx.doi.org/10.3390/ijerph17134752]

[68]
Chancellor S, De Choudhury M. Methods in predictive techniques for mental health status on social media: A critical review. NPJ Digit Med  2020; 3: 43.
 [http://dx.doi.org/10.1038/s41746-020-0233-7] [PMID: 32219184]

[69]
Stasak B, Epps J. Differential performance of automatic speech-based depression classification across smartphones. 2017 Seventh International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). 2017 October 23-26; San Antonio, TX USA. 
 [http://dx.doi.org/10.1109/ACIIW.2017.8272609]

[70]
Braun S, Annovazzi C, Botella C, et al. Assessing chronic stress, coping skills, and mood disorders through speech analysis: A self-assessment ‘voice app’ for laptops, tablets, and smartphones. Psychopathology  2016; 49(6): 406-19.
 [http://dx.doi.org/10.1159/000450959] [PMID: 27842303]

[71]
Cheng PGF, Ramos RM, Bitsch JÁ, et al. Psychologist in a pocket: Lexicon development and content validation of a mobile-based app for depression screening. JMIR Mhealth Uhealth  2016; 4(3): e88.
 [http://dx.doi.org/10.2196/mhealth.5284] [PMID: 27439444]

[72]
Epstein J, Bequette A. Smart phone applications in clinical practice. J Ment Health Couns  2013; 35: 283-95.
 [http://dx.doi.org/10.17744/mehc.35.4.f85k258620765tj4]

[73]
Huang Z, Epps J, Joachim D.  Speech landmark bigrams for depression detection from naturalistic smartphone speech. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2019 May 12-17; Brighton, UK; pp. 5856-60.
 [http://dx.doi.org/10.1109/ICASSP.2019.8682916]

[74]
Huang Z, Epps J, Joachim D.  Exploiting vocal tract coordination using dilated CNNs for depression detection in naturalistic environments. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2020 May 4-8; Barcelona, Spain; pp. 6549-53.
 [http://dx.doi.org/10.1109/ICASSP40776.2020.9054323]

[75]
Huang Z, Epps J, Joachim D, et al. Natural language processing methods for acoustic and landmark event-based features in speech-based depression detection. IEEE J Sel Top Signal Process  2019; 14: 435-48.
 [http://dx.doi.org/10.1109/JSTSP.2019.2949419]

[76]
Di Matteo D, Wang W, Fotinos K, et al. Smartphone-detected ambient speech and self-reported measures of anxiety and depression: Exploratory observational study. JMIR Form Res  2021; 5(1): e22723.
 [http://dx.doi.org/10.2196/22723] [PMID: 33512325]

[77]
Sunyaev A, Dehling T, Taylor PL, Mandl KD. Availability and quality of mobile health app privacy policies. J Am Med Inform Assoc  2015; 22(e1): e28-33.
 [http://dx.doi.org/10.1136/amiajnl-2013-002605] [PMID: 25147247]

[78]
The Guardian. Apple contractors ‘regularly hear confidential details’ on Siri recordings.   Available from: https://www.theguardian.com/technology/2019/jul/26/apple-contractors-regularly-hear-confidential-details-on-siri-recordings

[79]
Shokri R, Stronati M, Song C, et al. Membership inference attacks against machine learning models. 2017 IEEE Symposium on Security and Privacy (SP).  2017 May 22-26; San Jose, CA, USA. 3-18.
 [http://dx.doi.org/10.1109/SP.2017.41]

[80]
McFarlane J, Illes J. Neuroethics at the interface of machine learning and schizophrenia. NPJ Schizophr  2020; 6(1): 18.
 [http://dx.doi.org/10.1038/s41537-020-0108-6] [PMID: 32686681]

[81]
Oomen PP, de Boer JN, Brederoo SG, et al. Characterizing Speech Heterogeneity in Schizophrenia-spectrum disorders. Submitted  

[82]
de Boer JN, Voppel AE, Brederoo SG, Wijnen FNK, Sommer IEC. Language disturbances in schizophrenia: the relation with antipsychotic medication. NPJ Schizophr  2020; 6(1): 24.
 [http://dx.doi.org/10.1038/s41537-020-00114-3] [PMID: 32895389]

Cite As

CNS & Neurological Disorders - Drug Targets

Speech as a Biomarker for Depression

Abstract