Few-Shot Character Understanding
in Movies as an Assessment to
Meta-Learning of Theory-of-Mind

Mo Yu*1, Qiujing Wang*2, Shunchi Zhang*2,
Yisi Sang3, Kangsheng Pu3, Zekai Wei3, Han Wang1, Liyan Xu1, Jing Li4, Yue Yu5, Jie Zhou1
1 Pattern Recognition Center, WeChat AI, 2 Xi'an Jiaotong University,
3 Syracuse University, 4 New Jersey Institute of Technology, 5 Lehigh University
* Indicates Equal Contribution
Paper arXiv Data

# Abstract

When reading a story, humans can quickly understand new fictional characters with a few observations, mainly by drawing analogies to fictional and real people they already know. This reflects the few-shot and meta-learning essence of humans' inference of characters' mental states, i.e., theory-of-mind (ToM), which is largely ignored in existing research. We fill this gap with a novel NLP dataset, ToM-in-AMC, the first assessment of machines' meta-learning of ToM in a realistic narrative understanding scenario. Our dataset consists of ~1,000 parsed movie scripts, each corresponding to a few-shot character understanding task that requires models to mimic humans' ability of fast digesting characters with a few starting scenes in a new movie.

We propose a novel ToM prompting approach designed to explicitly assess the influence of multiple ToM dimensions. It surpasses existing baseline models, underscoring the significance of modeling multiple ToM dimensions for our task. Our extensive human study verifies that humans are capable of solving our problem by inferring characters' mental states based on their previously seen movies. In comparison, our systems based on either state-of-the-art large language models (GPT-4) or meta-learning algorithms lags >20% behind, highlighting a notable limitation in existing approaches' ToM capabilities.

# Dataset Overview

Table 1: Movie genres in our ToM-in-AMC.
Table 2: Statistics of our ToM-in-AMC.

# ToM Prompting Method (ToMPro)

Figure 3: Our proposed ToMPro approach. The method first (a) generates character mental descriptions along multiple ToM dimensions based on input scenes; then (b) predicts the identities of a new testing scene with the generated descriptions.

# Experiments

## Main Results

Table 3: Overall performance (%) on our ToM-in-AMC task.
(*) Evaluation was conducted on a subset of the dataset.
(†) Dataset released by Sang et al., 2022.
Table 4: Performance by difficulty levels measured the number of speakers in a scene.

## Analysis

Figure 4: Ablation of ToMPro on the 5 ToM dimensions.
Figure 5: Effects of perturbation on GPT-4 ICL.

# Citation

@misc{yu2024fewshot,
  title={Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind}, 
  author={Mo Yu and Qiujing Wang and Shunchi Zhang and Yisi Sang and Kangsheng Pu and Zekai Wei and Han Wang and Liyan Xu and Jing Li and Yue Yu and Jie Zhou},
  year={2024},
  eprint={2211.04684},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}