Yixin Zhu

Yixin Zhu

Postdoc

VCLA lab, UCLA

Biography

I am a postdoctoral scholar in Prof. Song-Chun Zhu’s VCLA lab at UCLA.

My research builds interactive AI by integrating high-level common sense (functionality, affordance, physics, causality) with raw sensory inputs (pixels and haptic signals) to enable richer representation and abstract reasoning on objects, scenes, shapes, numbers, and agents.

My work is currently supported by ONR MURI on Scene Understanding, DARPA XAI, and ONR Cognitive Systems for Human-Machine Teaming. I am a co-organizer of Vision Meets Cognition (FPIC) workshops, 3D Scene Understanding for Vision, Graphics, and Robotics workshops, and Virtual Reality Meets Physical Reality workshops.

We are looking for highly motivated students with exceptional programming skills and solid math background to work on 3D computer vision, abstract reasoning, physics-based simulation, and robotics. If you are a UCLA student and interested in working with me, please read some papers from the reading list before sending me an email. For a partial list of ongoing projects in the lab, see VCLA Project Bulletin.

Interests

  • Computer Vision
  • Artificial Intelligence
  • Human–Robot Interaction

Education

  • PhD in Statistics, 2018

    UCLA

  • MS in Computer Science, 2013

    UCLA

  • BEng in Software Engineering, 2012

    Xi'an Jiaotong University

Topics

3D Object Detection Activity Affordance Analogy AR Causality Container Functionality Glove HRI IQ Manipulation MPM Optimization Physics Raven Scene Parsing Scene Synthesis Simulation VR

Recent Publications

[ECCV20] LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities

Understanding and interpreting human actions is a long-standing challenge and a critical indicator of perception in artificial intelligence. However, a few imperative components of daily human activities are largely missed in prior literature, including the goal-directed actions, concurrent multi-tasks, and collaborations among multi-agents. We …
Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, Song-Chun Zhu
PDF Code Dataset Supp Presentation Web
[ECCV20] LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities

[IROS20] Human-Robot Interaction in a Shared Augmented Reality Workspace

We design and develop a new shared Augmented Reality (AR) workspace for Human-Robot Interaction (HRI), which establishes a bi-directional communication between human agents and robots. In a prototype system, the shared AR workspace enables a shared perception, so that a physical robot not only perceives the virtual elements in its own view but also …
Shuwen Qiu, Hangxin Liu, Zeyu Zhang, Yixin Zhu, Song-Chun Zhu
PDF Code Video Presentation Web
[IROS20] Human-Robot Interaction in a Shared Augmented Reality Workspace

[IROS20] Graph-based Hierarchical Knowledge Representation for Robot Task Transfer from Virtual to Physical World

[Best Paper Finalist]
We study the hierarchical knowledge transfer problem using a cloth-folding task, wherein the agent is first given a set of human demonstrations in the virtual world using an Oculus Headset, and later transferred and validated on a physical Baxter robot. We argue that such an intricate robot task transfer across different …
Zhenliang Zhang, Yixin Zhu, Song-Chun Zhu
PDF Code Video Presentation Web
[IROS20] Graph-based Hierarchical Knowledge Representation for Robot Task Transfer from Virtual to Physical World

[Engineering20] Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense

Recent progress in deep learning is essentially based on a “big data for small tasks” paradigm, under which massive amounts of data are used to train a classifier for a single narrow task. In this paper, we call for a shift that flips this paradigm upside down. Specifically, we propose a “small data for big tasks” paradigm, …
Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, Josh Tenenbaum, Song-Chun Zhu
PDF
[Engineering20] Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense

[SIGGRAPH20] A Massively Parallel and Scalable Multi-GPU Material Point Method

Harnessing the power of modern multi-GPU architectures, we present a massively parallel simulation system based on the Material Point Method (MPM) for simulating physical behaviors of materials undergoing complex topological changes, self-collision, and large deformations. Our system makes three critical contributions. First, we introduce a new …
Xinlei Wang, Yuxing Qiu, Stuart Slattery, Yu Fang, Minchen Li, Song-Chun Zhu, Yixin Zhu, Min Tang, Dinesh Manocha, Chenfanfu Jiang
PDF Code Video Supp Presentation Web
[SIGGRAPH20] A Massively Parallel and Scalable Multi-GPU Material Point Method

[SIGGRAPH20] IQ-MPM: An Interface Quadrature Material Point Method for Non-sticky Strongly Two-Way Coupled Nonlinear Solids and Fluids

We propose a novel scheme for simulating two-way coupled interactions between nonlinear elastic solids and incompressible fluids. The key ingredient of this approach is a ghost matrix operator-splitting scheme for strongly coupled nonlinear elastica and incompressible fluids through the weak form of their governing equations. This leads to a stable …
Yu Fang, Ziyin Qu, Minchen Li, Xinxin Zhang, Yixin Zhu, Mridul Aanjaneya, Chenfanfu Jiang
PDF Code Video Presentation
[SIGGRAPH20] IQ-MPM: An Interface Quadrature Material Point Method for Non-sticky Strongly Two-Way Coupled Nonlinear Solids and Fluids

[arXiv20] LETO: Hybrid Lagrangian-Eulerian Method for Topology Optimization

We propose LETO, a new hybrid Lagrangian-Eulerian method for topology optimization. At the heart of LETO lies in a hybrid particle-grid Material Point Method (MPM) to solve for elastic force equilibrium. LETO transfers density information from freely movable Lagrangian carrier particles to a fixed set of Eulerian quadrature points. The quadrature …
Yue Li, Xuan Li, Minchen Li, Yixin Zhu, Bo Zhu, Chenfanfu Jiang
PDF Code Video
[arXiv20] LETO: Hybrid Lagrangian-Eulerian Method for Topology Optimization

[ICRA20] Joint Inference of States, Robot Knowledge, and Human (False-)Beliefs

Aiming to understand how human (false-)belief—a core socio-cognitive ability—would affect human interactions with robots, this paper proposes to adopt a graphical model to unify the representation of object states, robot knowledge, and human (false-)beliefs. Specifically, a parse graph (PG) is learned from a single-view spatiotemporal …
Tao Yuan, Hangxin Liu, Lifeng Fan, Zilong Zheng, Tao Gao, Yixin Zhu, Song-Chun Zhu
PDF Video Presentation
[ICRA20] Joint Inference of States, Robot Knowledge, and Human (False-)Beliefs

[ICRA20] Congestion-aware Evacuation Routing using Augmented Reality Devices

We present a congestion-aware routing solution for indoor evacuation, which produces real-time individual-customized evacuation routes among multiple destinations while keeping tracks of all evacuees' locations. A population density map, obtained on-the-fly by aggregating locations of evacuees from user-end AR devices, is used to model the …
Zeyu Zhang, Hangxin Liu, Ziyuan Jiao, Yixin Zhu, Song-Chun Zhu
PDF Code Video Presentation
[ICRA20] Congestion-aware Evacuation Routing using Augmented Reality Devices

[ScienceRobotics19] A tale of two explanations: Enhancing human trust by explaining robot behavior

The ability to provide comprehensive explanations of chosen actions is a hallmark of intelligence. Lack of this ability impedes the general acceptance of AI and robot systems in critical tasks. This paper examines what forms of explanations best foster human trust in machines and proposes a framework in which explanations are generated from both …
Mark Edmonds, Feng Gao, Hangxin Liu, Xu Xie, Siyuan Qi, Brandon Rothrock, Yixin Zhu, Ying Nian Wu, Hongjing Lu, Song-Chun Zhu
PDF Code Video Summary Video Supp Smithsonian IEEE Spectrum UCLA Samueli School of Engineering Web
[ScienceRobotics19] A tale of two explanations: Enhancing human trust by explaining robot behavior

[AAAI20] Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning

Learning transferable knowledge across similar but different settings is a fundamental component of generalized intelligence. In this paper, we approach the transfer learning challenge from a causal theory perspective. Our agent is endowed with two basic yet general theories for transfer learning: (i) a task shares a common abstract structure that …
Mark Edmonds, Xiaojian Ma, Siyuan Qi, Yixin Zhu, Hongjing Lu, Song-Chun Zhu
PDF Code Poster Supp Web
[AAAI20] Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning

[AAAI20] Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning

As a comprehensive indicator of mathematical thinking and intelligence, the number sense (Dehaene 2011) bridges the induction of symbolic concepts and the competence of problem-solving. To endow such a crucial cognitive ability to machine intelligence, we propose a dataset, Machine Number Sense (MNS), consisting of visual arithmetic problems …
Wenhe Zhang, Chi Zhang, Yixin Zhu, Song-Chun Zhu
PDF Code Dataset Poster Web
[AAAI20] Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning

[NeurIPS19] Learning Perceptual Inference by Contrasting

‘Thinking in pictures,’ [1] i.e., spatial-temporal reasoning, effortless and instantaneous for humans, is believed to be a significant ability to perform logical induction and a crucial factor in the intellectual history of technology development. Modern Artificial Intelligence (AI), fueled by massive datasets, deeper models, and mighty …
Chi Zhang, Baoxiong Jia, Feng Gao, Yixin Zhu, Hongjing Lu, Song-Chun Zhu
PDF Code Poster Slides Blog Web
[NeurIPS19] Learning Perceptual Inference by Contrasting

[NeurIPS19] PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points

Detecting 3D objects from a single RGB image is intrinsically ambiguous, thus requiring appropriate prior knowledge and intermediate representations as constraints to reduce the uncertainties and improve the consistencies between the 2D image plane and the 3D world coordinate. To address this challenge, we propose to adopt perspective points as a …
Siyuan Huang, Yixin Chen, Tao Yuan, Siyuan Qi, Yixin Zhu, Song-Chun Zhu
PDF Code Poster Supp Web
[NeurIPS19] PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points

[ICCV19] Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense

We propose a new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction—3D estimations of object bounding boxes, camera pose, and room layout, and (ii) 3D human pose estimation. The intuition behind is to leverage the coupled nature of these two tasks …
Yixin Chen, Siyuan Huang, Tao Yuan, Yixin Zhu, Siyuan Qi, Song-Chun Zhu
PDF Code Poster Video Supp Web
[ICCV19] Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
See all publications

Downloads

Demo

Students Mentored

  • Wenhe Zhang, Master, Computer Science, UCLA, 2020 Fall
  • Xiaojian Ma, Master, Computer Science, UCLA, 2019 Fall
  • Xiaolin Fang, Ph.D., CSAIL, MIT, 2019 Fall
  • Shu Wang, Ph.D., Statistics, UCLA, 2018 Fall
  • Wenwen Si, Master, Computer Vision, CMU, 2018 Fall
  • Hangxin Liu, Ph.D., Computer Science, UCLA, 2018 Spring
  • Jenny Lin, Ph.D., Computer Science, CMU, 2017 Fall
  • Mark Edmonds, Ph.D., Computer Science, UCLA, 2017 Fall
  • Tian Ye, Master, Robotics, CMU, 2017 Fall
  • Feng Gao, Master, Statistics, UCLA, 2017 Fall
  • Xu Xie, Master, Statistics, UCLA, 2017 Fall
  • Xingwen Guo, Master, Computer Science, Yale, 2017 Fall
  • Chi Zhang, Master, Computer Science, UCLA, 2017 Fall
  • Jingyu Shao, Master, Statistics, UCLA, 2016 Winter
  • Yutong Zhang, Master in Computer Science, UCLA, 2015 Fall