Yixin Zhu
Biography
Dr. Yixin Zhu is an Assistant Professor of AI Institute at Peking University and Executive Research Director of BIGAI.
He received the Ph.D. degree (‘18) from UCLA advised by Prof. Song-Chun Zhu. His research builds interactive AI by integrating high-level common sense (functionality, affordance, physics, causality, intent) with raw sensory inputs (pixels and haptic signals) to enable richer representation and abstract reasoning on objects, scenes, shapes, numbers, and agents. He is a co-organizer of Vision Meets Cognition (FPIC) workshops, 3D Scene Understanding for Vision, Graphics, and Robotics workshops, and Virtual Reality Meets Physical Reality workshops.
During his Ph.D. and postdoc studies, his work was supported by DARPA MSEE, DARPA SIMPLEX, DARPA XAI, ONR MURI on Scene Understanding, and ONR Cognitive Systems for Human-Machine Teaming.
His group is looking for highly motivated undergrads, Ph.D. students, and postdocs with exceptional programming skills and solid math backgrounds to work on 3D computer vision, abstract reasoning, physics-based simulation, and cognitive robot. [Reading List]
Interests
- Computer Vision
- Artificial Intelligence
- Human–Robot Interaction
Education
-
PhD in Statistics, 2018
UCLA
-
MS in Computer Science, 2013
UCLA
-
BEng in Software Engineering, 2012
Xi'an Jiaotong University
Recent Publications
[arXiv21] Emergent Graphical Conventions in a Visual Communication Game
Humans communicate with graphical sketches apart from symbolic languages. While recent studies of emergent communication primarily focus on symbolic languages, their settings overlook the graphical sketches existing in human communication; they do not account for the evolution process through which symbolic sign systems emerge in the trade-off …
![[arXiv21] Emergent Graphical Conventions in a Visual Communication Game](/publication/communication2021arxiv/featured_hucf967c33f389130ab619766a81118218_217647_150x0_resize_q100_lanczos.jpg)
[arXiv21] Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning
Is intelligence realized by connectionist or classicist? While connectionist approaches have achieved superhuman performance, there has been growing evidence that such task-specific superiority is particularly fragile in systematic generalization. This observation lies in the central debate between connectionist and classicist, wherein the latter …
![[arXiv21] Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning](/publication/iq2021arxiv/featured_hu0049c2bc7c1065ae5e0afe51b43d2046_21033_150x0_resize_lanczos_2.png)
[AAIL21] Patching interpretable And-Or-Graph knowledge representation using augmented reality
[DARPA XAI Special Issue]
We present a novel augmented reality (AR) interface to provide effective means to diagnose a robot’s erroneous behaviors, endow it with new skills, and patch its knowledge structure represented by an And-Or-Graph (AOG). Specifically, an AOG representation of opening medicine bottles is learned from human demonstration …
We present a novel augmented reality (AR) interface to provide effective means to diagnose a robot’s erroneous behaviors, endow it with new skills, and patch its knowledge structure represented by an And-Or-Graph (AOG). Specifically, an AOG representation of opening medicine bottles is learned from human demonstration …
![[AAIL21] Patching interpretable And-Or-Graph knowledge representation using augmented reality](/publication/ar2021aail/featured_hu760b08af59e7fb493cd03e6e915adefe_496385_150x0_resize_lanczos_2.png)
[NeurIPS21] Unsupervised Foreground Extraction via Deep Region Competition
We present Deep Region Competition (DRC), an algorithm designed to extract foreground objects from images in a fully unsupervised manner. Foreground extraction can be viewed as a special case of generic image segmentation that focuseson identifying and disentangling objects from the background. In this work, we rethink the foreground extraction by …
![[NeurIPS21] Unsupervised Foreground Extraction via Deep Region Competition](/publication/segmentation2021neurips/featured_huc0dbe18711d462a0e0d6fd4bb41cce43_379646_150x0_resize_q100_lanczos.jpg)
[arXiv21] HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
Humans learn compositional and causal abstraction, i.e., knowledge, in response to the structure of naturalistic tasks. When presented with a problem-solving task involving some objects, toddlers would first interact with these objects to reckon what they are and what can be done with them. Leveraging these concepts, they could understand the …
![[arXiv21] HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving](/publication/generalization-halma2021arxiv/featured_hu4c2238623bbb7feccf8e0ecbc09c7740_130036_150x0_resize_q100_lanczos.jpg)
[arXiv21] A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics
Inspired by humans' remarkable ability to master arithmetic and generalize to unseen problems, we present a new dataset, HINT, to study machines' capability of learning generalizable concepts at three different levels: perception, syntax, and semantics. In particular, concepts in HINT, including both digits and operators, are required to learn in a …
![[arXiv21] A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics](/publication/generalization-hint2021arxiv/featured_huf1cd1c1b59220aee6296caa95cb97389_579382_150x0_resize_q100_lanczos.jpg)
[RA-L21] Synthesizing Diverse and Physically Stable Grasps with Arbitrary Hand Structures using Differentiable Force Closure Estimator
Existing grasp synthesis methods are either analytical or data-driven. The former one is oftentimes limited to specific application scope. The latter one depends heavily on demonstrations, thus suffers from generalization issues; e.g., models trained with human grasp data would be difficult to transfer to 3-finger grippers. To tackle these …
![[RA-L21] Synthesizing Diverse and Physically Stable Grasps with Arbitrary Hand Structures using Differentiable Force Closure Estimator](/publication/grasp2021ral/featured_huf1cd1c1b59220aee6296caa95cb97389_6300966_150x0_resize_q100_lanczos.jpg)
[ICCV21] YouRefIt: Embodied Reference Understanding with Language and Gesture
We study the machine’s understanding of embodied reference: One agent uses both language and gesture to refer to an object to another agent in a shared physical environment. Of note, this new visual task requires understanding multimodal cues with perspective-taking to identify which object is being referred to. To tackle this problem, we …
![[ICCV21] YouRefIt: Embodied Reference Understanding with Language and Gesture](/publication/pointing2021iccv/featured_hucf967c33f389130ab619766a81118218_3223729_150x0_resize_q100_lanczos.jpg)
[ICCV21] Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds
To date, various 3D scene understanding tasks still lack practical and generalizable pre-trained models, primarily due to the intricate nature of 3D scene understanding tasks and their immense variations introduced by camera views, lighting, occlusions, etc. In this paper, we tackle this challenge by introducing a spatio-temporal representation …
![[ICCV21] Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds](/publication/representationlearning2021iccv/featured_hucdc9ee304666cbf91f145cf7dfd0a98f_1363932_150x0_resize_lanczos_2.png)
[IROS21] Consolidating Kinematic Models to Promote Coordinated Mobile Manipulations
We construct a Virtual Kinematic Chain (VKC) that readily consolidates the kinematics of the mobile base, the arm, and the object to be manipulated in mobile manipulations. Accordingly, a mobile manipulation task is represented by altering the state of the constructed VKC, which can be converted to a motion planning problem, formulated and solved …
![[IROS21] Consolidating Kinematic Models to Promote Coordinated Mobile Manipulations](/publication/vkc-motion2021iros/featured_huc9e2d482fa00f85dcffa7d9ef3eaa470_297982_150x0_resize_lanczos_2.png)
[IROS21] Efficient Task Planning for Mobile Manipulation: a Virtual Kinematic Chain Perspective
We present a Virtual Kinematic Chain (VKC) perspective, a simple yet effective method, to improve task planning efficacy for mobile manipulation. By consolidating the kinematics of the mobile base, the arm, and the object being manipulated collectively as a whole, this novel VKC perspective naturally defines abstract actions and eliminates …
![[IROS21] Efficient Task Planning for Mobile Manipulation: a Virtual Kinematic Chain Perspective](/publication/vkc-task2021iros/featured_huf1cd1c1b59220aee6296caa95cb97389_5094973_150x0_resize_q100_lanczos.jpg)
[IROS21] Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene
Human-robot collaboration is an essential research topic in artificial intelligence (AI), enabling researchers to devise cognitive AI systems and affords an intuitive means for users to interact with the robot. Of note, communication plays a central role. To date, prior studies in embodied agent navigation have only demonstrated that human …
![[IROS21] Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene](/publication/pointing2021iros/featured_huf1cd1c1b59220aee6296caa95cb97389_704212_150x0_resize_q100_lanczos.jpg)
[CogSci21] Individual vs. Joint Perception: a Pragmatic Model of Pointing as Communicative Smithian Helping
The simple gesture of pointing can greatly augment one’s ability to comprehend states of the world based on observations. It triggers additional inferences relevant to one’s task at hand. We model an agent’s update to its belief of the world based on individual observations using a partially observable Markov decision process (POMDP), a …
![[CogSci21] Individual vs. Joint Perception: a Pragmatic Model of Pointing as Communicative Smithian Helping](/publication/pointing2021cogsci/featured_hub10688d9ea7923e3c59c712b17b93dae_76406_150x0_resize_lanczos_2.png)
[ACL-Findings21] GRICE: A Grammar-based Dataset for Recovering Implicature and Conversational rEasoning
Understanding what we genuinely mean instead of what we literally say in conversations is challenging for both humans and machines; yet, this direction is mostly left untouched in modern open-ended dialogue systems. To fill in this gap, we present a grammar-based dialogue dataset, GRICE, designed to bring implicature into pragmatic reasoning in the …
![[ACL-Findings21] GRICE: A Grammar-based Dataset for Recovering Implicature and Conversational rEasoning](/publication/implicature2021acl-findings/featured_huf1cd1c1b59220aee6296caa95cb97389_1013821_150x0_resize_q100_lanczos.jpg)
[CVPR21] Learning Triadic Belief Dynamics in Nonverbal Communication from Videos
Humans possess a unique social cognition capability; nonverbal communication can convey rich social information among agents. In contrast, such crucial social characteristics are mostly missing in the existing sceneunderstanding literature. In this paper, we incorporate different nonverbal communication cues (e.g., gaze, humanposes, and gestures) …
![[CVPR21] Learning Triadic Belief Dynamics in Nonverbal Communication from Videos](/publication/mind2021cvpr/featured_hucf967c33f389130ab619766a81118218_1253322_150x0_resize_q100_lanczos.jpg)
Downloads
- [NeurIPS21][Code] Unsupervised Foreground Extraction via Deep Region Competition
- [ICCV21][Code] YouRefIt: Embodied Reference Understanding with Language and Gesture
- [ICCV21][Dataset] YouRefIt: Embodied Reference Understanding with Language and Gesture
- [ICCV21][Code] Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds
- [RA-L21][Code] Synthesizing Diverse and Physically Stable Grasps with Arbitrary Hand Structures using Differentiable Force Closure Estimator
- [IROS21][Code] Consolidating Kinematic Models to Promote Coordinated Mobile Manipulations
- [IROS21][Code] Efficient Task Planning for Mobile Manipulation: a Virtual Kinematic Chain Perspective
- [IROS21][Code] Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene
- [CVPR21][Code] Learning Triadic Belief Dynamics in Nonverbal Communication from Videos
- [CVPR21][Dataset] Learning Triadic Belief Dynamics in Nonverbal Communication from Videos
- [CVPR21][Code] ACRE: Abstract Causal Reasoning Beyond Covariation
- [CVPR21][Dataset] ACRE: Abstract Causal Reasoning Beyond Covariation
- [CVPR21][Code] Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution
- [ICRA21][Code] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignments
- [ICRA21][Code] Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance
- [IJNME21][Code] Lagrangian‐Eulerian Multi‐Density Topology Optimization with the Material Point Method
- [ECCV20][Code] LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities
- [ECCV20][Dataset] LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities
- [IROS20][Code] Human-Robot Interaction in a Shared Augmented Reality Workspace
- [IROS20][Code] Graph-based Hierarchical Knowledge Representation for Robot Task Transfer from Virtual to Physical World
- [SIGGRAPH20][Code] A Massively Parallel and Scalable Multi-GPU Material Point Method
- [SIGGRAPH20][Code] IQ-MPM: An Interface Quadrature Material Point Method for Non-sticky Strongly Two-Way Coupled Nonlinear Solids and Fluids
- [ICRA20][Code] Congestion-aware Evacuation Routing using Augmented Reality Devices
- [ScienceRobotics19][Code] A tale of two explanations: Enhancing human trust by explaining robot behavior
- [AAAI20][Code] Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning
- [AAAI20][Code] Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning
- [AAAI20][Dataset] Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning
- [NeurIPS19][Code] Learning Perceptual Inference by Contrasting
- [ICCV19][Code] Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
- [IROS19][Code] Learning Virtual Grasp with Failed Demonstrations via Bayesian Inverse Reinforcement Learning
- [TURC19][Code] VRGym: A Virtual Testbed for Physical and Interactive AI
- [CVPR19][Code] RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
- [CVPR19][Dataset] RAVEN-10000
- [ICRA19][Code] High-Fidelity Grasping in Virtual Reality using a Glove-based System
- [ICRA19][Code] Self-Supervised Incremental Learning for Sound Source Localization in Complex Indoor Environment
- [AAAI19][Code] MetaStyle: Three-Way Trade-Off Among Speed, Flexibility and Quality in Neural Style Transfer
- [NeurIPS18][Code] Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation
- [ECCV18][Code] Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image
- [CogSci18][Code] Human Causal Transfer: Challenges for Deep Reinforcement Learning
- [CVPR18][Code] Human-centric Indoor Scene Synthesis Using Stochastic Grammar
- [SIGGRAPH18][Code] A Moving Least Squares Material Point Method with Displacement Discontinuity and Two-Way Rigid Body Coupling
- [ICRA18][Code] Interactive Robot Knowledge Patching using Augmented Reality
- [ICRA18][Code] Unsupervised Learning of Hierarchical Models for Hand-Object Interactions
- [IROS17][Code] Feeling the Force: Integrating Force and Pose for Fluent Discovery through Imitation Learning to Open Medicine Bottles
- [IROS17][Code] A Glove-based System for Studying Hand-Object Manipulation via Joint Pose and Force Sensing
- [SIGGRAPHAsia16Workshop][Code] A Virtual Reality Platform for Dynamic Human-Scene Interaction
- [CVPR16][Code] FEM Simulation for Inferring Forces during Human-scene Interactions
- [CVPR16][Code] Human Reconstruction from Kinect Skeleton
- [CVPR16][Code] Watertight 3D Scene Dataset
- [CVPR15][Code] Kinect2 Toolbox
- [CVPR15][Dataset] Tool 3D Dataset
Demo
Students Mentored
- Wenhe Zhang, Master, Computer Science, UCLA, 2020 Fall
- Xiaojian Ma, Master, Computer Science, UCLA, 2019 Fall
- Xiaolin Fang, Ph.D., CSAIL, MIT, 2019 Fall
- Shu Wang, Ph.D., Statistics, UCLA, 2018 Fall
- Wenwen Si, Master, Computer Vision, CMU, 2018 Fall
- Hangxin Liu, Ph.D., Computer Science, UCLA, 2018 Spring
- Jenny Lin, Ph.D., Computer Science, CMU, 2017 Fall
- Mark Edmonds, Ph.D., Computer Science, UCLA, 2017 Fall
- Tian Ye, Master, Robotics, CMU, 2017 Fall
- Feng Gao, Master, Statistics, UCLA, 2017 Fall
- Xu Xie, Master, Statistics, UCLA, 2017 Fall
- Xingwen Guo, Master, Computer Science, Yale, 2017 Fall
- Chi Zhang, Master, Computer Science, UCLA, 2017 Fall
- Jingyu Shao, Master, Statistics, UCLA, 2016 Winter
- Yutong Zhang, Master in Computer Science, UCLA, 2015 Fall