PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning

NeurIPS 2021

Yining Hong1 Li Yi2 Joshua B. Tenenbaum4 Antonio Torralba4 Chuang Gan3
1University of California, Los Angeles       2Stanford University         3MIT-IBM Watson AI Lab       4MIT

A critical aspect of human visual perception is the ability to parse visual scenes into individual objects and further into object parts, forming part-whole hierarchies. Such composite structures could induce a rich set of semantic concepts and relations, thus playing an important role in the interpretation and organization of visual signals as well as for the generalization of visual perception and reasoning. However, existing visual reasoning benchmarks mostly focus on objects rather than parts. Visual reasoning based on the full part-whole hierarchy is much more challenging than object-centric reasoning due to finer-grained concepts, richer geometry relations, and more complex physics. Therefore, to better serve for part-based conceptual, relational and physical reasoning, we introduce a new large-scale diagnostic visual reasoning dataset named PTR. PTR contains around 70k RGBD synthetic images with ground truth object and part level annotations regarding semantic instance segmentation, color attributes, spatial and geometric relationships, and certain physical properties such as stability. These images are paired with 700k machine-generated questions covering various types of reasoning types, making them a good testbed for visual reasoning models. We examine several state-of-the-art visual reasoning models on this dataset and observe that they still make many surprising mistakes in situations where humans can easily infer the correct answer. We believe this dataset will open up new opportunities for part-based reasoning.

The PTR Universe



(a) Conceptual Reasoning

(b) Physical Reasoning


(c) Relational Reasoning

(d) Analogical Reasoning

Evaluation Server

The evaluation server is set up on EvalAI


@inproceedings{hong2021ptr, author = {Hong, Yining and Yi, Li and Tenenbaum, Joshua B and Torralba, Antonio and Gan, Chuang}, title = {PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning}, booktitle = {Advances In Neural Information Processing Systems}, year = {2021} }

Paper     Code