Depression is a significant global health burden, affecting over 322 mil- lion people worldwide and projected to surpass cardiovascular disease as the leading cause of disability by 2030. Despite advancements in mental health services, accurate and accessible diagnostic methods remain a crit- ical challenge. Traditional approaches, such as psychiatric consultations, face limitations due to the physician-patient ratio and reliance on subjec- tive self-report scales, which can lead to inaccuracies. Recent research has explored alternative methods, including facial behavior analysis, for objec- tive depression assessment. This approach is cost-effective, non-invasive, and suitable for real-world applications. This study builds upon existing research, such as FacePsy and MoodCapture, by enhancing facial behavior analysis using deep learning techniques. We propose an improved frame- work for depression detection, introducing novel methods to extract and interpret facial indicators associated with depressive symptoms. Through comprehensive experiments, we demonstrate the effectiveness of our ap- proach for real-world applications. Our findings contribute to the advance- ment of AI-driven mental health diagnostics, offering promising avenues for accessible and accurate diagnostic tools. This study bridges theoretical research with practical applications, fostering the development of innova- tive solutions for addressing the mental health crisis
In this project, we evaluated the performance of various machine learning algorithms on data with missing values. The following algorithms were applied:
- AdaBoost
- Bagging
- CatBoost
- GradientBoosting
- LightGBM
- RandomForest
- XGBoost
- LSTM
Two strategies were employed to handle missing data:
- Drop NaN: Rows with missing values were removed.
- Imputed Mean: Missing values were replaced with the mean value of the corresponding feature.
This universal model employ Leave One Participant Out Cross validation to make generelized model
| Algorithm | Full Features (Drop NaN) | Eyes and Smile Probability (Drop NaN) |
|---|---|---|
| AdaBoost | 0.3994 | 0.4073 |
| Bagging | 0.3278 | 0.7093 |
| CatBoost | 0.3812 | 0.7388 |
| GradientBoosting | 0.3914 | 0.5704 |
| LightGBM | 0.3806 | 0.7096 |
| RandomForest | 0.3351 | 0.7046 |
| XGBoost | 0.3800 | 0.5920 |
| LSTM | 0.0688 | 0.0484 |
| Algorithm | Full Features (Imputed Mean) | Eyes and Smile Probability (Imputed Mean) | Head Euler Angle |
|---|---|---|---|
| AdaBoost | 0.3994 | 0.3885 | 0.2013 |
| Bagging | 0.3278 | 0.6995 | 0.1593 |
| CatBoost | 0.3812 | 0.7289 | 0.1835 |
| GradientBoosting | 0.3914 | 0.5613 | 0.1905 |
| LightGBM | 0.3806 | 0.7024 | 0.1853 |
| RandomForest | 0.3351 | 0.6922 | 0.1700 |
| XGBoost | 0.3800 | 0.5806 | 0.1847 |
| LSTM | 0.1337 | 0.0227 | 0.0810 |
| Algorithm | Full Features (Drop NaN) | Eyes and Smile Probability (Drop NaN) |
|---|---|---|
| AdaBoost | 0.5485 | 0.5156 |
| Bagging | 0.5061 | 0.7476 |
| CatBoost | 0.5492 | 0.7845 |
| GradientBoosting | 0.5511 | 0.6173 |
| LightGBM | 0.5487 | 0.7525 |
| RandomForest | 0.5083 | 0.7432 |
| XGBoost | 0.5490 | 0.6538 |
| LSTM | 0.2493 | 0.2202 |
| Algorithm | Full Features (Imputed Mean) | Eyes and Smile Probability (Imputed Mean) | Head Euler Angle |
|---|---|---|---|
| AdaBoost | 0.5485 | 0.5001 | 0.4079 |
| Bagging | 0.5061 | 0.7396 | 0.5996 |
| CatBoost | 0.5492 | 0.7774 | 0.4676 |
| GradientBoosting | 0.5511 | 0.6033 | 0.4393 |
| LightGBM | 0.5487 | 0.7431 | 0.4450 |
| RandomForest | 0.5083 | 0.7307 | 0.5922 |
| XGBoost | 0.5490 | 0.6347 | 0.4748 |
| LSTM | 0.3379 | 0.2414 | 0.2938 |
| Algorithm | Full Features (Drop NaN) | Eyes and Smile Probability (Drop NaN) |
|---|---|---|
| AdaBoost | 0.6112 | 0.4334 |
| Bagging | 0.5130 | 0.6649 |
| CatBoost | 0.5785 | 0.7058 |
| GradientBoosting | 0.5841 | 0.5415 |
| LightGBM | 0.5788 | 0.6726 |
| RandomForest | 0.5037 | 0.6607 |
| XGBoost | 0.5774 | 0.5649 |
| LSTM | 0.5534 | 0.6563 |
| Algorithm | Full Features (Imputed Mean) | Eyes and Smile Probability (Imputed Mean) | Head Euler Angle |
|---|---|---|---|
| AdaBoost | 0.6112 | 0.4255 | 0.4708 |
| Bagging | 0.5130 | 0.6564 | 0.5451 |
| CatBoost | 0.5785 | 0.6978 | 0.4366 |
| GradientBoosting | 0.5841 | 0.5343 | 0.4324 |
| LightGBM | 0.5788 | 0.6663 | 0.4367 |
| RandomForest | 0.5037 | 0.6498 | 0.5113 |
| XGBoost | 0.5774 | 0.5564 | 0.4417 |
| LSTM | 0.5269 | 0.6430 | 0.5193 |
This Hybrid model employ Leave One Participant Day Out Cross validation for testing and Nested Cross validation for training
| Algorithm | Full Features (Drop NaN) | Eyes and Smile Probability (Drop NaN) |
|---|---|---|
| AdaBoost | 0.5302 | 0.4622 |
| Bagging | 0.5107 | 0.5007 |
| CatBoost | 0.4967 | 0.5039 |
| GradientBoosting | 0.5188 | 0.4786 |
| LightGBM | 0.4999 | 0.4971 |
| RandomForest | 0.5012 | 0.4995 |
| XGBoost | 0.5061 | 0.4987 |
| LSTM | 0.4559 | 0.4487 |
| Algorithm | Full Features (Imputed Mean) | Eyes and Smile Probability (Imputed Mean) | Head Euler Angle |
|---|---|---|---|
| AdaBoost | 0.5093 | 0.4663 | 0.5296 |
| Bagging | 0.5240 | 0.4950 | 0.5144 |
| CatBoost | 0.5074 | 0.4942 | 0.5232 |
| GradientBoosting | 0.5054 | 0.4729 | 0.5275 |
| LightGBM | 0.5204 | 0.4910 | 0.5273 |
| RandomForest | 0.5249 | 0.4921 | 0.5153 |
| XGBoost | 0.5140 | 0.4889 | 0.5240 |
| LSTM | 0.5201 | 0.4490 | 0.4506 |
| Algorithm | Full Features (Drop NaN) | Eyes and Smile Probability (Drop NaN) |
|---|---|---|
| AdaBoost | 0.3994 | 0.4073 |
| Bagging | 0.3278 | 0.7093 |
| CatBoost | 0.3812 | 0.7388 |
| GradientBoosting | 0.3914 | 0.5704 |
| LightGBM | 0.3806 | 0.7096 |
| RandomForest | 0.3351 | 0.7046 |
| XGBoost | 0.3800 | 0.5920 |
| LSTM | 0.0688 | 0.0484 |
| Algorithm | Full Features (Imputed Mean) | Eyes and Smile Probability (Imputed Mean) | Head Euler Angle |
|---|---|---|---|
| AdaBoost | 0.3994 | 0.3885 | 0.2013 |
| Bagging | 0.3278 | 0.6995 | 0.1593 |
| CatBoost | 0.3812 | 0.7289 | 0.1835 |
| GradientBoosting | 0.3914 | 0.5613 | 0.1905 |
| LightGBM | 0.3806 | 0.7024 | 0.1853 |
| RandomForest | 0.3351 | 0.6922 | 0.1700 |
| XGBoost | 0.3800 | 0.5806 | 0.1847 |
| LSTM | 0.1337 | 0.0227 | 0.0810 |
| Algorithm | Full Features (Drop NaN) | Eyes and Smile Probability (Drop NaN) |
|---|---|---|
| AdaBoost | 0.4215 | 0.3577 |
| Bagging | 0.3495 | 0.2931 |
| CatBoost | 0.4072 | 0.1647 |
| GradientBoosting | 0.4085 | 0.3554 |
| LightGBM | 0.3965 | 0.2464 |
| RandomForest | 0.3445 | 0.2946 |
| XGBoost | 0.4125 | 0.3391 |
| LSTM | 0.3713 | 0.4099 |
[1] Sharifa Alghowinem, Roland Goecke, Michael Wagner, Gordon Parker, and Michael Breakspear. Head pose and movement analysis as an indicator of depression. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, pages 283–288, 2013.
[2] C. ´A. Casado, M. L. Ca˜nellas, and M. B. L´opez. Depression recognition using remote photoplethysmography from facial videos. IEEE Transactions on Affective Computing, 14(4):3305–3316, 2023.
[3] Yulia E. Chentsova-Dutton, Jeanne L. Tsai, and Ian H. Gotlib. Further evidence for the cultural norm hypothesis: Positive emotion in depressed and control European American and Asian American women. Cultural Diversity and Ethnic Minority Psychology, 16(2):284, 2010.
[4] Prerna Chikersal, Afsaneh Doryab, Michael Tumminia, Daniella K. Villalba, Janine M. Dutcher, Xinwen Liu, Sheldon Cohen, Kasey G. Creswell, Jennifer Mankoff, J. David Creswell, et al. Detecting depression and predicting its onset using longitudinal symptoms captured by passive sensing: A machine learning approach with robust feature selection. ACM Transactions on Computer-Human Interaction (TOCHI), 28(1):1–41, 2021.
[5] J. F. Cohn, T. S. Kruez, I. Matthews, Y. Yang, M. H. Nguyen, M. T. Padilla, and F. De la Torre. Detecting depression from facial actions and vocal prosody. In 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, pages 1–7. IEEE, September 2009.
[6] Jeffrey F. Cohn, Tomas Simon Kruez, Iain Matthews, Ying Yang, Minh Hoai Nguyen, Margara Tejera Padilla, Feng Zhou, and Fernando De la Torre. Detecting depression from facial actions and vocal prosody. In 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, pages 1–7. IEEE, 2009.
[7] GBD 2019 Mental Disorders Collaborators. Global, regional, and national burden of 12 mental disorders in 204 countries and territories, 1990-2019: a systematic analysis for the global burden of disease study 2019. The Lancet. Psychiatry, 9(2):137–150, 2022.
[8] WHO Depression. Other common mental disorders: global health estimates. Geneva: World Health Organization, 24(1), 2017.
[9] GBD 2017 Disease, Injury Incidence, and Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the global burden of disease study 2017. Lancet (London, England), 392(10159):1789–1858, 2018.
[10] GBD 2019 Diseases and Injuries Collaborators. Global burden of 369 diseases and injuries in 204 countries and territories, 1990–2019: a systematic analysis for the global burden of disease study 2019. Lancet (London, England), 396(10258):1204–1222, 2020.
[11] Asma Ahmad Farhan, Chaoqun Yue, Reynaldo Morillo, Shweta Ware, Jin Lu, Jinbo Bi, Jayesh Kamath, Alexander Russell, Athanasios Bamis, and Bing Wang. Behavior vs. introspection: Refining prediction of clinical depression via smartphone sensing data. In 2016 IEEE Wireless Health (WH), pages 1–8. IEEE, 2016.
[12] Hans-Ulrich Fisch, Siegfried Frey, and Hans-Peter Hirsbrunner. Analyzing nonverbal behavior in depression. Journal of Abnormal Psychology, 92(3):307, 1983.
[13] Wolfgang Gaebel and Wolfgang Wölwer. Facial expressivity in the course of schizophrenia and depression. European Archives of Psychiatry and Clinical Neuroscience, 254:335–342, 2004.
[14] J. G. Gehricke and D. Shapiro. Reduced facial expression and social context in major depression: Discrepancies between facial muscle activity and self-reported emotion. Psychiatry Research, 95(2):157–167, 2000.
[15] R. Islam and S. W. Bae. Facepsy: An open-source affective mobile sensing system—analyzing facial behavior and head gesture for depression detection in naturalistic settings. Proceedings of the ACM on Human-Computer Interaction, 8(MHCI):1–32, 2024.
[16] Rahul Islam and Sang Won Bae. Pupilsense: Detection of depressive episodes through pupillary response in the wild. In International Conference on Activity and Behavior Computing, 2024.
[17] Jyoti Joshi, Roland Goecke, Gordon Parker, and Michael Breakspear. Can body expressions contribute to automatic depression analysis? In 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pages 1–7. IEEE, 2013.
[18] K. Koller-Schlaud, A. Ströhle, E. Bärwolf, J. Behr, and J. Rentzsch. EEG frontal asymmetry and theta power in unipolar and bipolar depression. Journal of Affective Disorders, 276:501–510, 2020.
[19] S. Nepal, A. Pillai, W. Wang, T. Griffin, A. C. Collins, M. Heinz, and A. Campbell. Moodcapture: Depression detection using in-the-wild smartphone images. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–18. ACM, May 2024.
[20] Kennedy Opoku Asare, Isaac Moshe, Yannik Terhorst, Julio Vega, Simo Hosio, Harald Baumeister, Laura Pulkki-Råback, and Denzil Ferreira. Mood ratings and digital biomarkers from smartphone and wearable data differentiates and predicts depression status: A longitudinal data analysis. 2022. Preprint or unpublished work, DOI not available.
[21] World Health Organization. The ICD-10 classification of mental and behavioural disorders: clinical descriptions and diagnostic guidelines, volume 1. World Health Organization, 1992.
[22] Paola Pedrelli, Szymon Fedor, Asma Ghandeharioun, Esther Howe, Dawn F. Ionescu, Darian Bhathena, Lauren B. Fisher, Cristina Cusin, Maren Nyer, Albert Yeung, et al. Monitoring changes in depression severity using wearable and mobile sensors. Frontiers in Psychiatry, 11:584711, 2020.
[23] Lawrence Ian Reed, Michael A. Sayette, and Jeffrey F. Cohn. Impact of depression on response to comedy: A dynamic facial coding analysis. Journal of Abnormal Psychology, 116(4):804, 2007.
[24] Babette Renneberg, Katrin Heyn, Rita Gebhard, and Silke Bachmann. Facial expression of emotions in borderline personality disorder and depression. Journal of Behavior Therapy and Experimental Psychiatry, 36(3):183–196, 2005.
[25] Denise M. Sloan, Milton E. Strauss, Stuart W. Quirk, and Martha Sajatovic. Subjective and expressive emotional responses in depression. Journal of Affective Disorders, 46(2):135–141, 1997.
[26] Denise M. Sloan, Milton E. Strauss, and Katherine L. Wisner. Diminished response to pleasant stimuli by depressed women. Journal of Abnormal Psychology, 110(3):488, 2001.
[27] Siyang Song, Shashank Jaiswal, Linlin Shen, and Michel Valstar. Spectral representation of behaviour primitives for depression analysis. IEEE Transactions on Affective Computing, 2020.
[28] Michel Valstar, Björn Schuller, Kirsty Smith, Timur Almaev, Florian Eyben, Jarek Krajewski, Roddy Cowie, and Maja Pantic. Avec 2014: 3D dimensional affect and depression recognition challenge. In Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, pages 3–10, 2014.
[29] A. Vázquez-Romero and A. Gallardo-Antolín. Automatic detection of depression in speech using ensemble convolutional neural networks. Entropy, 22(6):688, 2020.
[30] Rui Wang, Andrew T. Campbell, and Xia Zhou. Using opportunistic face logging from smartphone to infer mental health: Challenges and future directions. In Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers, pages 683–692, 2015.
[31] Y. Xing, N. Rao, M. Miao, Q. Li, Q. Li, X. Chen, and J. Wu. Task-state heart rate variability parameter-based depression detection model and effect of therapy on the parameters. IEEE Access, 7:105701–105709, 2019.