The Stochastic Parrot on LLM’s Shoulder: A Summative Assessment of
Physical Concept Understanding
Mo Yu*,
Lemao Liu*,
Junjie Wu*,
Tsz Ting Chung*, Shunchi Zhang*,
Jiangnan Li,
Dit-Yan Yeung,
Jie Zhou
Under Review, 2024
We propose PhysiCo, a physical concept understanding benchmark to
demonstrate the stochastic parrot phenomenon in (M)LLMs: they
can describe and recognize physical concepts well in natural
language, but fail to classify them in the form of grid
representations.
Few-Shot Character Understanding in Movies as an Assessment to
Meta-Learning of Theory-of-Mind
Mo Yu*, Qiujing Wang*, Shunchi Zhang*, Yisi Sang,
Kangsheng Pu,
Zekai Wei, Han Wang,
Liyan Xu,
Jing Li,
Yue Yu,
Jie Zhou
ICML 2024 - International Conference on Machine Learning
We introduce ToM-in-AMC, a few-shot character understanding
benchmark to assess machines' meta-learning ability of
theory-of-mind (ToM) in a realistic narrative scenario. We
propose ToMPro, a ToM-aware prompting method that improves both
interpretability and performance.
Dynamic Relation Transformer for Contextual Text Block Detection
Jiawei Wang*, Shunchi Zhang*,
Kai Hu*,
Chixiang Ma,
Zhuoyao Zhong,
Lei Sun,
Qiang Huo
ICDAR 2024 - International Conference on Document Analysis and
Recognition
Oral Presentation
We propose DRFormer to detect contextual text blocks in natural
scenes by framing the task as a graph generation problem, and
jointly addresses text detection (nodes) and relation prediction
(edges).