Graduation Year

2021

Document Type

Dissertation

Degree

Ph.D.

Degree Name

Doctor of Philosophy (Ph.D.)

Degree Granting Department

Computer Science and Engineering

Major Professor

Shaun Canavan, Ph.D.

Committee Member

Rangachar Kasturi, Ph.D.

Committee Member

Jeffrey F. Cohn, Ph.D.

Committee Member

Marvin J. Andujar, Ph.D.

Committee Member

Pei-Sung Lin, Ph.D.

Committee Member

Elizabeth Schotter, Ph.D.

Keywords

Action Units, Expression, Physiological Signals

Abstract

Affective computing builds and evaluates systems that can recognize, interpret, and simulate human emotion. It is an interdisciplinary field, which includes computer science, psychology, and many others. For years, human emotion has been studied in psychology but recently has become a prominent field in computer science. Largely, the field of affective computing has been focused on analyzing static facial expressions to recognize human emotions, without taking bias (e.g. gender, data bias), context, or temporal information into account. Psychology has shown the difficulty of analyzing emotions without incorporating this type of information. In this dissertation, we have proposed new approaches to recognizing emotions by incorporating both contextual and temporal information, as well as approaches to mitigate data bias. More specifically, this dissertation has the following theoretical and application-based contributions: (1) The first work to recognize context using temporal dynamics from facial action units; (2) We recognize multiple self-reported emotions using facial expression-based videos; (3) A new approach to mitigate data bias in facial action units is proposed; and (4) Multimodal, temporal fusion of physiological signals and action units are used for emotion recognition. This dissertation has a wide range of applications in the fields including, but not limited to, medicine, security, and entertainment.

Download

Share

COinS