Yixin Zhu

Yixin Zhu

Biography

My research builds interactive AI by integrating high-level common sense (functionality, affordance, physics, causality, intent) with raw sensory inputs (pixels and haptic signals) to enable richer representation and abstract reasoning on objects, scenes, shapes, numbers, and agents. I am a co-organizer of Vision Meets Cognition (FPIC) workshops, 3D Scene Understanding for Vision, Graphics, and Robotics workshops, and Virtual Reality Meets Physical Reality workshops.

Interests

  • Computer Vision
  • Artificial Intelligence
  • Human–Robot Interaction

Education

  • PhD in Statistics, 2018

    UCLA

  • MS in Computer Science, 2013

    UCLA

  • BEng in Software Engineering, 2012

    Xi'an Jiaotong University

Topics

Affordance Analogy AR Causality Communication Container Functionality Generalization Glove HRI Intent IQ Manipulation MPM Number Sense Optimization Physics Programming Language Raven Scene Parsing Scene Reconstruction Scene Synthesis Simulation Tool VR XAI

Recent Publications

[arXiv21] HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving

Humans learn compositional and causal abstraction, i.e., knowledge, in response to the structure of naturalistic tasks. When presented with a problem-solving task involving some objects, toddlers would first interact with these objects to reckon what they are and what can be done with them. Leveraging these concepts, they could understand the …
Sirui Xie, Xiaojian Ma, Peiyu Yu, Yixin Zhu, Ying Nian Wu, Song-Chun Zhu
PDF Code Web
[arXiv21] HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving

[arXiv21] A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics

Inspired by humans' remarkable ability to master arithmetic and generalize to unseen problems, we present a new dataset, HINT, to study machines' capability of learning generalizable concepts at three different levels: perception, syntax, and semantics. In particular, concepts in HINT, including both digits and operators, are required to learn in a …
Qing Li, Siyuan Huang, Yining Hong, Yixin Zhu, Ying Nian Wu, Song-Chun Zhu
PDF Code Web
[arXiv21] A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics

[CVPR21] Learning Triadic Belief Dynamics in Nonverbal Communication from Videos

Humans possess a unique social cognition capability; nonverbal communication can convey rich social information among agents. In contrast, such crucial social characteristics are mostly missing in the existing sceneunderstanding literature. In this paper, we incorporate different nonverbal communication cues (e.g., gaze, humanposes, and gestures) …
Lifeng Fan, Shuwen Qiu, Zilong Zheng, Tao Gao, Song-Chun Zhu, Yixin Zhu
PDF Code Dataset Video Supp
[CVPR21] Learning Triadic Belief Dynamics in Nonverbal Communication from Videos

[CVPR21] ACRE: Abstract Causal Reasoning Beyond Covariation

Causal induction, i.e., identifying unobservable mechanisms that lead to the observable relations among variables, has played a pivotal role in modern scientific discovery, especially in scenarios with only sparse and limited data. Humans, even young toddlers, can induce causal relationships surprisingly well in various settings despite its …
Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, Yixin Zhu
PDF Code Dataset Supp Blog Web
[CVPR21] ACRE: Abstract Causal Reasoning Beyond Covariation

[CVPR21] Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution

Spatial-temporal reasoning is a challenging task in Artificial Intelligence (AI) due to its demanding but unique nature: a theoretic requirement on representing and reasoning based on spatial-temporal knowledge in mind, and an applied requirement on a high-level cognitive system capable of navigating and acting in space and time. Recent works have …
Chi Zhang, Baoxiong Jia, Song-Chun Zhu, Yixin Zhu
PDF Code Supp Blog Web
[CVPR21] Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution

[ICRA21] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignments

In this paper, we rethink the problem of scene reconstruction from an embodied agent’s perspective: While the classic view focuses on the reconstruction accuracy, our new perspective emphasizes the underlying functions and constraints such that the reconstructed scenes provide actionable information for simulating interactions with agents. …
Muzhi Han, Zeyu Zhang, Ziyuan Jiao, Xu Xie, Yixin Zhu, Song-Chun Zhu, Hangxin Liu
PDF Code Video Web
[ICRA21] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignments

[ICRA21] Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance

Predicting agents' future trajectories plays a crucial role in modern AI systems, yet it is challenging due to intricate interactions exhibited in multi-agent systems, especially when it comes to collision avoidance. To address this challenge, we propose to learn congestion patterns as contextual cues explicitly and devise a novel …
Xu Xie, Chi Zhang, Yixin Zhu, Ying Nian Wu, Song-Chun Zhu
PDF Code Video Web
[ICRA21] Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance

[IJNME21] Lagrangian‐Eulerian Multi‐Density Topology Optimization with the Material Point Method

In this paper, a hybrid Lagrangian‐Eulerian topology optimization (LETO) method is proposed to solve the elastic force equilibrium with the Material Point Method (MPM). LETO transfers density information from freely movable Lagrangian carrier particles to a fixed set of Eulerian quadrature points. The transfer is based on a smooth radial kernel …
Yue Li, Xuan Li, Minchen Li, Yixin Zhu, Bo Zhu, Chenfanfu Jiang
PDF Code Video
[IJNME21] Lagrangian‐Eulerian Multi‐Density Topology Optimization with the Material Point Method

[ECCV20] LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities

Understanding and interpreting human actions is a long-standing challenge and a critical indicator of perception in artificial intelligence. However, a few imperative components of daily human activities are largely missed in prior literature, including the goal-directed actions, concurrent multi-tasks, and collaborations among multi-agents. We …
Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, Song-Chun Zhu
PDF Code Dataset Supp Presentation Web
[ECCV20] LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities

[IROS20] Human-Robot Interaction in a Shared Augmented Reality Workspace

We design and develop a new shared Augmented Reality (AR) workspace for Human-Robot Interaction (HRI), which establishes a bi-directional communication between human agents and robots. In a prototype system, the shared AR workspace enables a shared perception, so that a physical robot not only perceives the virtual elements in its own view but also …
Shuwen Qiu, Hangxin Liu, Zeyu Zhang, Yixin Zhu, Song-Chun Zhu
PDF Code Video Presentation Web
[IROS20] Human-Robot Interaction in a Shared Augmented Reality Workspace

[IROS20] Graph-based Hierarchical Knowledge Representation for Robot Task Transfer from Virtual to Physical World

[Best Paper Finalist]
We study the hierarchical knowledge transfer problem using a cloth-folding task, wherein the agent is first given a set of human demonstrations in the virtual world using an Oculus Headset, and later transferred and validated on a physical Baxter robot. We argue that such an intricate robot task transfer across different …
Zhenliang Zhang, Yixin Zhu, Song-Chun Zhu
PDF Code Video Presentation Web
[IROS20] Graph-based Hierarchical Knowledge Representation for Robot Task Transfer from Virtual to Physical World

[Engineering20] Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense

Recent progress in deep learning is essentially based on a “big data for small tasks” paradigm, under which massive amounts of data are used to train a classifier for a single narrow task. In this paper, we call for a shift that flips this paradigm upside down. Specifically, we propose a “small data for big tasks” paradigm, …
Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, Josh Tenenbaum, Song-Chun Zhu
PDF
[Engineering20] Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense

[SIGGRAPH20] A Massively Parallel and Scalable Multi-GPU Material Point Method

Harnessing the power of modern multi-GPU architectures, we present a massively parallel simulation system based on the Material Point Method (MPM) for simulating physical behaviors of materials undergoing complex topological changes, self-collision, and large deformations. Our system makes three critical contributions. First, we introduce a new …
Xinlei Wang, Yuxing Qiu, Stuart Slattery, Yu Fang, Minchen Li, Song-Chun Zhu, Yixin Zhu, Min Tang, Dinesh Manocha, Chenfanfu Jiang
PDF Code Video Supp Presentation Web
[SIGGRAPH20] A Massively Parallel and Scalable Multi-GPU Material Point Method

[SIGGRAPH20] IQ-MPM: An Interface Quadrature Material Point Method for Non-sticky Strongly Two-Way Coupled Nonlinear Solids and Fluids

We propose a novel scheme for simulating two-way coupled interactions between nonlinear elastic solids and incompressible fluids. The key ingredient of this approach is a ghost matrix operator-splitting scheme for strongly coupled nonlinear elastica and incompressible fluids through the weak form of their governing equations. This leads to a stable …
Yu Fang, Ziyin Qu, Minchen Li, Xinxin Zhang, Yixin Zhu, Mridul Aanjaneya, Chenfanfu Jiang
PDF Code Video Presentation
[SIGGRAPH20] IQ-MPM: An Interface Quadrature Material Point Method for Non-sticky Strongly Two-Way Coupled Nonlinear Solids and Fluids

[ICRA20] Joint Inference of States, Robot Knowledge, and Human (False-)Beliefs

Aiming to understand how human (false-)belief—a core socio-cognitive ability—would affect human interactions with robots, this paper proposes to adopt a graphical model to unify the representation of object states, robot knowledge, and human (false-)beliefs. Specifically, a parse graph (PG) is learned from a single-view spatiotemporal …
Tao Yuan, Hangxin Liu, Lifeng Fan, Zilong Zheng, Tao Gao, Yixin Zhu, Song-Chun Zhu
PDF Video Presentation
[ICRA20] Joint Inference of States, Robot Knowledge, and Human (False-)Beliefs
See all publications

Downloads

Demo

Students Mentored

  • Wenhe Zhang, Master, Computer Science, UCLA, 2020 Fall
  • Xiaojian Ma, Master, Computer Science, UCLA, 2019 Fall
  • Xiaolin Fang, Ph.D., CSAIL, MIT, 2019 Fall
  • Shu Wang, Ph.D., Statistics, UCLA, 2018 Fall
  • Wenwen Si, Master, Computer Vision, CMU, 2018 Fall
  • Hangxin Liu, Ph.D., Computer Science, UCLA, 2018 Spring
  • Jenny Lin, Ph.D., Computer Science, CMU, 2017 Fall
  • Mark Edmonds, Ph.D., Computer Science, UCLA, 2017 Fall
  • Tian Ye, Master, Robotics, CMU, 2017 Fall
  • Feng Gao, Master, Statistics, UCLA, 2017 Fall
  • Xu Xie, Master, Statistics, UCLA, 2017 Fall
  • Xingwen Guo, Master, Computer Science, Yale, 2017 Fall
  • Chi Zhang, Master, Computer Science, UCLA, 2017 Fall
  • Jingyu Shao, Master, Statistics, UCLA, 2016 Winter
  • Yutong Zhang, Master in Computer Science, UCLA, 2015 Fall