Graduation Year


Document Type




Degree Name

Doctor of Philosophy (Ph.D.)

Degree Granting Department

Computer Science and Engineering

Major Professor

Yu Y. Sun, Ph.D.

Co-Major Professor

Dmitry Goldgof, Ph.D.

Committee Member

Ghada Zamzmi, Ph.D.


Computer Assisted Intervention, Computer Vision, Deep Learning, Machine Learning, Medical Application


Neonates can not express their pain like an adult person. Due to the lacking of proper muscle growth and inability to express non-verbally, it is difficult to understand their emotional status. In addition, if the neonates are under any treatment or left monitored after any major surgeries (post-operative), it is more difficult to understand their pain due to the side effect of medications and the caring system (i.e. intubated, masked face, covered body with blanket, etc.). In a clinical environment, usually, bedside nurses routinely observe the neonate and measure the pain status following any standard clinical pain scale. But current practice is intermittent and subjective.In this dissertation, we investigate developing an automated neonatal monitoring system that can replace the current manual system in a clinical environment. We explore the availability of current neonatal pain datasets and the challenges of neonatal pain data collection. We further collect and present a multimodal neonatal pain dataset, especially focusing on postoperative pain that mimics a real clinical environment. To develop the automated system, at first, we explore the effectiveness of individual modalities (i.e., facial expression, body movement, crying sound, vital signs) and propose novel spatio-temporal methods to analyze neonatal pain. Later, we develop a full multimodal system that is capable of using all modalities together to provide the pain signal. We also want to make sure that the automated system can effectively function in a real clinical environment. In a real clinical system, missing modality is very common due to different clinical factors (i.e., masked face, covered body, etc.) or algorithms' limitations (i.e., low light condition, side face view, etc.). We also propose a multimodal system that can effectively regenerate pseudo-features for missing modalities and continue assessing neonatal pain.