Graduation Year

2025

Document Type

Dissertation

Degree

Ph.D.

Degree Name

Doctor of Philosophy (Ph.D.)

Degree Granting Department

Computer Science and Engineering

Major Professor

Shaun Canavan, Ph.D.

Committee Member

Sudeep Sarkar, Ph.D.

Committee Member

Paul Rosen, Ph.D.

Committee Member

Mauricio Pamplona Segundo, Ph.D.

Committee Member

Alison Salloum, Ph.D.

Committee Member

Achilleas Kourtellis, Ph.D.

Keywords

Facial Expression Recognition, Multimodal, Normalizing Flows, Out-of-Distribution, Representation Learning

Abstract

Affective Computing (AC) is a subdomain of AI that primarily deals with recognizing and interpreting human emotions. This field inherently intersects with psychological studies, as the comprehension of human emotions and behaviors necessitates an understanding of their underlying cognitive processes. One such concepts that lends itself from psychology is context. Roughly speaking, context in AC is defined as any meta information (e.g., environment) that can be utilized to describe the interaction between a user and a model to solve a particular application (e.g., emotion recognition). This doctoral dissertation comprises of two distinct yet interconnected components (Part I and II), the first of which seeks to analyze and study the effect of integrating such contexts in designing AC applications.

In the first part, we investigate the impact of context across diverse affect-based systems, including those designed for reflective thinking, continuous authentication, and post-traumatic stress disorder (PTSD) analysis. Initially, we examine the effect of task-specific classifiers in a unimodal system for reflective thinking prediction and showcase the performance improvement on inclusion of the tasks as context. Next, we shift our focus to multimodal systems for continuous authentication, where we delve into the effects of multiple emotional responses (context) as an indicator for identifying a particular subject across different sessions. We demonstrate that although context inclusion yields improved performance, its impact is superseded by model's capability to perform continuous authentication in a single-session environment. Finally, we illustrate the influence of clinician-focused sessions as context to classify children videos with PTSD. we further extend our study of context by evaluating the inter-dependencies across context and its contribution towards the classification. Concluding with the release of a new context-integrated dataset to further expand the resources available for research in this domain.

The significance of context-aware affect systems is paralleled by the critical role of model interpretability. The aforementioned applications of affect extends to healthcare, biometrics, and education which are considered highly sensitive fields and the interpretation of machine learning model output is of paramount importance due to the potential consequences of erroneous conclusion. To this end, the second part of the dissertation focuses on novel representation learning approaches to interpretable predictions. More specifically, we employ probabilistic likelihood models to learn class-specific distribution to provide exact likelihood values of a data point belonging to a particular class, as opposed to softmax psuedo-probabilities. For instance, our initial work proposes a facial expression recognition (FER) model using normalizing flows to learn latent representations per expression. We showcase that while this approach produces latent representations consistent with model output, it exhibits limitations in real-world classification scenarios. To overcome these constraints, we introduce an innovative loss function that leverages supervised contrastive learning (SCL) to capture class-specific distributions rather than representations. We find that although this novel method enhances classification accuracy, its primary contribution lies in its capacity to detect out-of-distribution data, a crucial yet under-explored application in affective computing. We provide extensive quantitative and qualitative analysis to validate our approach.

Download

Share

COinS