Action, Cognition, and Emotion for Understanding Human Behavior in Daily Life

Three Circular Components for Human Behavior

Human behavior is a complex interplay of three components: actions, cognitions, and emotions1. While they are connected, it is hard to tell what exactly is cause and effect since they do not run independently of each other. For instance, while drinking coffee generally helps to reduce a person’s stress, the stress-relieving effects of coffee may vary from day to day for many reasons. However, understanding human behaviors is challenging because (1) emotions are usually affected by various complex and subtle factors in daily life; (2) only a few labels are available in real-world scenarios, where humans rate their feeling in indoor environments using self-reporting tools. The limited and unbalanced class labels deteriorate the performance of classifiers in a broad range of emotion classification and causality identification problems. The overarching goal of this research is to address these key challenges - to build interactive systems to discover cause and effect relationship in human behaviors, powered by an unprecedented real-world dataset from users in their daily life. Particularly, This work has analyzed multi-modal data along with advanced wearable sensor technology.

Figure 1.(Left Top) Headset style wearable device. IMU, EEG, PPG sensors, and a tiny frontal camera. The location of two EEG electrodes (F3, F4) on the 10-20 international system. (Left Bottom) An overview of my prior work. The proposed wearable device collects a user’s contextual information in situations with frontal images and emotion measured by EEG and PPG signals. Given this information, the predictive affect model detects emotional changes, which are used as an input with frontal images to discover the causal relationship between emotions and situations. (Right) (a) Real world dataset “Affective Lifelog Dataset” D and its subset which contains ground-truth labels. (b) Example images in the dataset D. ( c) Proportion of the subset and the dataset. It implies we have very small amount of ground-truth labels. (d) Distribution of ground-truth labels in valence and arousal labels on the subset.

Designing such systems requires three fundamental research thrusts: (1) Unobtrusive wearable device: recording human affect and action should be unobtrusive when measured in the natural environment. (2) Predictive affective modeling: the systems must be able to build a predictive model of emotions by learning physiological signals; (3) Causal influence model: the system must be able to identify emotional causality in human behaviors. These three thrusts are highly inter-related because a closed loop system which integrates action, cognition, and emotion for inferencing causality is needed for understanding human behaviors in real-world environments. In my prior work, I have studied these thrusts:

  • Unobtrusive wearable device: Everyday technology requires wearable systems to have the unprecedented ability to perform the comfortable, long-term, and in situ assessment of physiological activities. Furthermore, human affect is sophisticated and subtle. it is vulnerable to personal, social, and contextual attributes. The noticeability and visibility of wearable devices could elicit unnecessary and irrelevant emotions. Therefore, recording human affect should be unobtrusive when measured in the natural environment. To design an unnoticeable device, I imitated the design of existing easy-to-use wireless headsets. See Figure 1.(Left Top). The device consists of a tiny PPG sensor and a minimum configuration of a two-channel EEG, the lowest number of channels necessary to learn patterns of lateralized frontal cortex activity involved in human affect. Using this device, my colleagues and I collected a real-world dataset, which consists of physiological signals, accelerometer signals, and frontal images obtained from 16 male and 5 female university students. They wore the device over six hours per day for up to 60 days.2

  • Predictive affect modeling: Predictive ability of emotional changes is a fundamental measure of affective intelligence because it enables the machines to characterize mental activities for recognizing states of feeling. I am particularly interested in physiological changes in brain and heart that influence human behaviors. I have presented a predictive affect model that allows a machine to predict emotional changes in day-to-day activities. The affect model quantifies affective dynamics into a certain degree of emotion ratings with respect to valence and arousal.3 4

Figure 2. Predictive Affect Modeling

  • Causal influence modeling: Emotions are caused by specific events. People experience emotions only when things are out of the ordinary or unusual. Emotions serve an adaptive role in helping people guide social behaviors. I have presented an asymmetric measure to identify the causal relationship between affective contents and human emotion in daily life. The model for the first time allowed users to understand when, what, and how their surroundings affect them unawares in their daily life.2

My prior work provides a computational foundation for affective intelligence for understanding human behaviors. By leveraging this work, I will address the bidirectional causality problem on social behaviors with affective intelligence. In particular, I plan on three goals that align with my research direction: 1) to learn subtle emotional signal or intent of social interactions, 2) to reason about emotional causality in social interactions and to predict social affordances, and 3) to build a user interface that reacts according to affective intelligence on social interaction.

Byung Hyung Kim
Research Assistant Professor

Ph.D. in Computer Science at KAIST. Current research interests include algorithmic transparency, interpretability in affective intelligence, computational emotional dynamics, cerebral asymmetry and the effects of emotion on brain structure for affective computing, brain-computer interface, and assistive and rehabilitative technology.