Yixin Zhu

Yixin Zhu

Biography

Dr. Yixin Zhu is an Assistant Professor of AI Institute at Peking University and Executive Research Director of BIGAI.

He received the Ph.D. degree (‘18) from UCLA advised by Prof. Song-Chun Zhu. His research builds interactive AI by integrating high-level common sense (functionality, affordance, physics, causality, intent) with raw sensory inputs (pixels and haptic signals) to enable richer representation and abstract reasoning on objects, scenes, shapes, numbers, and agents. He is a co-organizer of Vision Meets Cognition (FPIC) workshops, 3D Scene Understanding for Vision, Graphics, and Robotics workshops, and Virtual Reality Meets Physical Reality workshops.

During his Ph.D. and postdoc studies, his work was supported by DARPA MSEE, DARPA SIMPLEX, DARPA XAI, ONR MURI on Scene Understanding, and ONR Cognitive Systems for Human-Machine Teaming.

His group is looking for highly motivated undergrads, Ph.D. students, and postdocs with exceptional programming skills and solid math backgrounds to work on 3D computer vision, abstract reasoning, physics-based simulation, and cognitive robot. [Reading List]

Interests

  • Computer Vision
  • Artificial Intelligence
  • Human–Robot Interaction

Education

  • PhD in Statistics, 2018

    UCLA

  • MS in Computer Science, 2013

    UCLA

  • BEng in Software Engineering, 2012

    Xi'an Jiaotong University

Topics

Affordance Analogy AR Causality Communication Container Functionality Generalization Glove HRI Intent IQ Manipulation MPM Number Sense Optimization Physics Programming Language Raven Representation Learning Scene Parsing Scene Reconstruction Scene Synthesis Simulation Tool VR XAI

Recent Publications

[arXiv21] HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving

Humans learn compositional and causal abstraction, i.e., knowledge, in response to the structure of naturalistic tasks. When presented with a problem-solving task involving some objects, toddlers would first interact with these objects to reckon what they are and what can be done with them. Leveraging these concepts, they could understand the …
Sirui Xie, Xiaojian Ma, Peiyu Yu, Yixin Zhu, Ying Nian Wu, Song-Chun Zhu
PDF Code Web
[arXiv21] HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving

[arXiv21] A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics

Inspired by humans' remarkable ability to master arithmetic and generalize to unseen problems, we present a new dataset, HINT, to study machines' capability of learning generalizable concepts at three different levels: perception, syntax, and semantics. In particular, concepts in HINT, including both digits and operators, are required to learn in a …
Qing Li, Siyuan Huang, Yining Hong, Yixin Zhu, Ying Nian Wu, Song-Chun Zhu
PDF Code Web
[arXiv21] A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics

[RA-L21] Synthesizing Diverse and Physically Stable Grasps with Arbitrary Hand Structures using Differentiable Force Closure Estimator

Existing grasp synthesis methods are either analytical or data-driven. The former one is oftentimes limited to specific application scope. The latter one depends heavily on demonstrations, thus suffers from generalization issues; e.g., models trained with human grasp data would be difficult to transfer to 3-finger grippers. To tackle these …
Tengyu Liu, Zeyu Liu, Ziyuan Jiao, Yixin Zhu, Song-Chun Zhu
PDF Code Video Web
[RA-L21] Synthesizing Diverse and Physically Stable Grasps with Arbitrary Hand Structures using Differentiable Force Closure Estimator

[ICCV21] YouRefIt: Embodied Reference Understanding with Language and Gesture

We study the machine’s understanding of embodied reference: One agent uses both language and gesture to refer to an object to another agent in a shared physical environment. Of note, this new visual task requires understanding multimodal cues with perspective-taking to identify which object is being referred to. To tackle this problem, we …
Yixin Chen, Qing Li, Deqian Kong, Yik Lun Kei, Song-Chun Zhu, Tao Gao, Yixin Zhu, Siyuan Huang
PDF Code Dataset Supp Web
[ICCV21] YouRefIt: Embodied Reference Understanding with Language and Gesture

[ICCV21] Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds

To date, various 3D scene understanding tasks still lack practical and generalizable pre-trained models, primarily due to the intricate nature of 3D scene understanding tasks and their immense variations introduced by camera views, lighting, occlusions, etc. In this paper, we tackle this challenge by introducing a spatio-temporal representation …
Siyuan Huang, Yichen Xie, Song-Chun Zhu, Yixin Zhu
PDF Code Supp Web
[ICCV21] Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds

[IROS21] Consolidating Kinematic Models to Promote Coordinated Mobile Manipulations

We construct a Virtual Kinematic Chain (VKC) that readily consolidates the kinematics of the mobile base, the arm, and the object to be manipulated in mobile manipulations. Accordingly, a mobile manipulation task is represented by altering the state of the constructed VKC, which can be converted to a motion planning problem, formulated and solved …
Ziyuan Jiao, Zeyu Zhang, Xin Jiang, David Han, Song-Chun Zhu, Yixin Zhu, Hangxin Liu
PDF Code Video Web
[IROS21] Consolidating Kinematic Models to Promote Coordinated Mobile Manipulations

[IROS21] Efficient Task Planning for Mobile Manipulation: a Virtual Kinematic Chain Perspective

We present a Virtual Kinematic Chain (VKC) perspective, a simple yet effective method, to improve task planning efficacy for mobile manipulation. By consolidating the kinematics of the mobile base, the arm, and the object being manipulated collectively as a whole, this novel VKC perspective naturally defines abstract actions and eliminates …
Ziyuan Jiao, Zeyu Zhang, Weiqi Wang, David Han, Song-Chun Zhu, Yixin Zhu, Hangxin Liu
PDF Code Video Web
[IROS21] Efficient Task Planning for Mobile Manipulation: a Virtual Kinematic Chain Perspective

[IROS21] Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene

Human-robot collaboration is an essential research topic in artificial intelligence (AI), enabling researchers to devise cognitive AI systems and affords an intuitive means for users to interact with the robot. Of note, communication plays a central role. To date, prior studies in embodied agent navigation have only demonstrated that human …
Qi Wu, Cheng-Ju Wu, Yixin Zhu, Jungseock Joo
PDF Code Video Web
[IROS21] Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene

[CogSci21] Individual vs. Joint Perception: a Pragmatic Model of Pointing as Communicative Smithian Helping

The simple gesture of pointing can greatly augment one’s ability to comprehend states of the world based on observations. It triggers additional inferences relevant to one’s task at hand. We model an agent’s update to its belief of the world based on individual observations using a partially observable Markov decision process (POMDP), a …
Kaiwen Jiang, Stephanie Stacy, Chuyu Wei, Adelpha Chan, Federico Rossano, Yixin Zhu, Tao Gao
PDF
[CogSci21] Individual vs. Joint Perception: a Pragmatic Model of Pointing as Communicative Smithian Helping

[ACL-Findings21] GRICE: A Grammar-based Dataset for Recovering Implicature and Conversational rEasoning

Understanding what we genuinely mean instead of what we literally say in conversations is challenging for both humans and machines; yet, this direction is mostly left untouched in modern open-ended dialogue systems. To fill in this gap, we present a grammar-based dialogue dataset, GRICE, designed to bring implicature into pragmatic reasoning in the …
Zilong Zheng, Shuwen Qiu, Lifeng Fan, Yixin Zhu, Song-Chun Zhu
PDF Web
[ACL-Findings21] GRICE: A Grammar-based Dataset for Recovering Implicature and Conversational rEasoning

[CVPR21] Learning Triadic Belief Dynamics in Nonverbal Communication from Videos

Humans possess a unique social cognition capability; nonverbal communication can convey rich social information among agents. In contrast, such crucial social characteristics are mostly missing in the existing sceneunderstanding literature. In this paper, we incorporate different nonverbal communication cues (e.g., gaze, humanposes, and gestures) …
Lifeng Fan, Shuwen Qiu, Zilong Zheng, Tao Gao, Song-Chun Zhu, Yixin Zhu
PDF Code Dataset Video Supp
[CVPR21] Learning Triadic Belief Dynamics in Nonverbal Communication from Videos

[CVPR21] ACRE: Abstract Causal Reasoning Beyond Covariation

Causal induction, i.e., identifying unobservable mechanisms that lead to the observable relations among variables, has played a pivotal role in modern scientific discovery, especially in scenarios with only sparse and limited data. Humans, even young toddlers, can induce causal relationships surprisingly well in various settings despite its …
Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, Yixin Zhu
PDF Code Dataset Supp Blog Web
[CVPR21] ACRE: Abstract Causal Reasoning Beyond Covariation

[CVPR21] Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution

Spatial-temporal reasoning is a challenging task in Artificial Intelligence (AI) due to its demanding but unique nature: a theoretic requirement on representing and reasoning based on spatial-temporal knowledge in mind, and an applied requirement on a high-level cognitive system capable of navigating and acting in space and time. Recent works have …
Chi Zhang, Baoxiong Jia, Song-Chun Zhu, Yixin Zhu
PDF Code Supp Blog Web
[CVPR21] Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution

[ICRA21] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignments

In this paper, we rethink the problem of scene reconstruction from an embodied agent’s perspective: While the classic view focuses on the reconstruction accuracy, our new perspective emphasizes the underlying functions and constraints such that the reconstructed scenes provide actionable information for simulating interactions with agents. …
Muzhi Han, Zeyu Zhang, Ziyuan Jiao, Xu Xie, Yixin Zhu, Song-Chun Zhu, Hangxin Liu
PDF Code Video Web
[ICRA21] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignments

[ICRA21] Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance

Predicting agents' future trajectories plays a crucial role in modern AI systems, yet it is challenging due to intricate interactions exhibited in multi-agent systems, especially when it comes to collision avoidance. To address this challenge, we propose to learn congestion patterns as contextual cues explicitly and devise a novel …
Xu Xie, Chi Zhang, Yixin Zhu, Ying Nian Wu, Song-Chun Zhu
PDF Code Video Web
[ICRA21] Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance
See all publications

Downloads

Demo

Students Mentored

  • Wenhe Zhang, Master, Computer Science, UCLA, 2020 Fall
  • Xiaojian Ma, Master, Computer Science, UCLA, 2019 Fall
  • Xiaolin Fang, Ph.D., CSAIL, MIT, 2019 Fall
  • Shu Wang, Ph.D., Statistics, UCLA, 2018 Fall
  • Wenwen Si, Master, Computer Vision, CMU, 2018 Fall
  • Hangxin Liu, Ph.D., Computer Science, UCLA, 2018 Spring
  • Jenny Lin, Ph.D., Computer Science, CMU, 2017 Fall
  • Mark Edmonds, Ph.D., Computer Science, UCLA, 2017 Fall
  • Tian Ye, Master, Robotics, CMU, 2017 Fall
  • Feng Gao, Master, Statistics, UCLA, 2017 Fall
  • Xu Xie, Master, Statistics, UCLA, 2017 Fall
  • Xingwen Guo, Master, Computer Science, Yale, 2017 Fall
  • Chi Zhang, Master, Computer Science, UCLA, 2017 Fall
  • Jingyu Shao, Master, Statistics, UCLA, 2016 Winter
  • Yutong Zhang, Master in Computer Science, UCLA, 2015 Fall