IDEAL-Bench: Indoor Dataset and Evaluation Suite for Analyzing 3D Layout Reasoning

We introduce IDEAL-Bench, an evaluation suite that requires VLMs to predict structured 3D layouts on photorealistic indoor scenes, capturing each visible object's position, orientation, and extent rather than just answering questions about them. Predictions are scored across eleven metrics spanning five numerical dimensions, paired with a perceptual render-and-compare protocol that substitutes predicted poses into the original scene and re-renders it for comparison against the reference image.

• We introduce holistic 3D layout inference as a new evaluation paradigm for VLM spatial competence, requiring structured, geometrically verifiable estimations of semantic-aware object pose and extent from a single image.
• We release IDEAL-Scenes, a dataset of 1,000 indoor scenes across 10 room types with programmatically extracted ground-truth layouts and re-renderable Blender source files, together with IDEAL-Bench, an evaluation suite combining numerical geometric metrics with perceptual render-and-compare assessment validated against human judgments.
• Our evaluation of 15 VLMs uncovers a critical gap between linguistic proficiency and geometric precision, identifying coordinate estimation as a primary bottleneck. These findings demonstrate that IDEAL-Bench serves as a necessary complement to existing benchmarks by surfacing hidden geometric deficiencies in holistic 3D reasoning.

Holistic 3D Layout Inference

From a single RGB image, a candidate object-category list, and a set of candidate room types, a model must produce a globally consistent, structured 3D layout: the room type, the room dimensions, and for every visible object its category, 3D center position, bounding-box extent, and yaw rotation. No depth, camera parameters, or instance masks are provided, and the model reasons about 3D structure purely from monocular appearance.

IDEAL-Scenes

A synthetic indoor dataset of 1,000 scenes spanning 10 room types (bedroom, home studio, living room, kitchen, bathroom, dining room, office, meeting room, classroom, and library), with 100 scenes per type. Built on a procedural pipeline based on InfiniGen, each scene is a self-contained bundle: a rendered RGB image, programmatically exact ground-truth annotations, the native scene.blend source file, an instance segmentation map, and scene metadata. The generation pipeline is released on GitHub and designed to be extensible, supporting additional room types, asset libraries, and rendering from arbitrary camera viewpoints.

Evaluation Protocol

IDEAL-Bench scores every prediction through two parallel pathways: a numerical comparison against the ground-truth layout, and a perceptual comparison against a re-rendered reconstruction.

Numerical evaluation

Predicted objects are matched to ground-truth instances. For non-grid scenes, we use category-wise Hungarian matching to assign each predicted object to its closest-matching ground-truth instance and compare position, dimensions, and rotation. For grid-structured scenes (classrooms, libraries, offices) where per-instance correspondence is ill-posed, we assess structural regularity directly instead. Predictions are then scored along five functionally distinct dimensions, each normalised to [0, 100]. The Overall score is the unweighted mean of all eleven applicable rate metrics, so no single dimension is privileged a priori.

Perceptual evaluation

For every object in view, the predicted position, yaw, and dimensions replace the ground-truth values in the original Blender scene, and the result is re-rendered from the source camera. Hallucinated objects are rendered as red spheres to make false positives explicit. A Judge VLM (Claude Opus 4.7) then compares each reconstruction against the reference image, assigning a 1–5 spatial-similarity score together with a strict total ranking, both repeated five times and averaged. Because weaker models often produce reconstructions with missing assets or failed composition, this pathway is restricted to the six top-scored models across 200 scenes (20 per room type).

Leaderboard

Numerically, D1 (Scene Validity) is near-saturated and D4 (Object Recognition) remains uniformly strong, while performance collapses on D3 (Geometric Accuracy). Even the leader, Gemini 2.5 Pro, reaches only 62.1/100 overall. Parameter count is also a weak predictor of rank: Gemma-4-31B-IT ranks third overall, ahead of open-source models with far more parameters, and GPT-4o outranks the newer GPT-5.4. Two models, InternVL3.5-30B-A3B and Janus-Pro-7B, fail to parse valid output altogether.

The full breakdown above reports Overall, competition Rank, Geo. (the mean of D3 and D5, isolating pure geometric reasoning from task-compliance), and every sub-metric behind D1–D5. Bold marks the best value per metric, underlined marks the second-best.

Perceptual leaderboard

The perceptual evaluation is restricted to the six top-scored models, judged across 200 scenes (20 per room type). The resulting ranking largely mirrors the numerical leaderboard, with one notable exception: GPT-4o ranks second overall numerically but drops to second-to-last here. Its high numerical score comes from getting individual objects roughly right, which is not the same as producing a layout that looks coherent as a whole scene. Together with the numerical metrics, this perceptual check offers a more intuitive, visual way to assess a model's 3D understanding and spatial reasoning as a whole.

Mean Perceptual: average 1–5 Likert score from the Judge VLM, repeated five times. Mean Rank: average strict total ranking per scene. Top-1: share of scenes where the model's reconstruction was judged best.

Conclusion

We present IDEAL-Scenes and IDEAL-Bench, a dataset and evaluation suite for testing whether vision-language models can recover a coherent 3D layout. Our evaluation of fifteen such models reveals a consistent pattern of strengths and bottlenecks in how they reason about space.

Together, these findings position IDEAL-Bench as a diagnostic complement to existing benchmarks, surfacing geometric reasoning failures that QA-based evaluation cannot.

BibTeX

@misc{idealbench2026,
    title={IDEAL-Bench: Indoor Benchmark Dataset for Evaluating & Analysis by 3D Layout Reasoning},
    author={Yuening Cai and Junwei Zhou and Youran Qu and Yu-Wing Tai},
    year={2026},
    eprint={xxxx},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/xxxx}
  }