Continuous Prediction of Spontaneous Affect from Multiple Cues and Modalities in Valence-Arousal Space, IEEE TAC 2011

The work introduced here converges with this recent shift in affect recognition, from recognizing posed expressions in terms of discrete and basic emotion categories, to the recognition of spontaneous expressions in terms of dimensional and continuous descriptions.

Contributions:

• Fuse facial expression, shoulder gesture and speech cues in analysis of human affect.
• Propose an output-associative fusion framework that incorporates correlations and covariances between the emotion dimensions.
• Demonstrate that capturing temporal correlations and remembering the temporally distant events (or storing them in memory) is of utmost importance for continuous affect prediction.

Challenges mentioned: reliability of ground truth, baseline problem, unbalanced data.

Interesting observation:

• Arousal can be much better predicted than valence using audio cues. 
• For valence dimension instead, visual cues (facial expressions and shoulder movements) appear to perform better.