LangLoc: “Tell Me What You See”

ECCV 2026

Fine-grained indoor localization from a natural-language description. No camera, no GPS.

Shaurya Kishore Panwar1,2,*, Roham Zendehdel Nobari1,2,*, Shirley Feng Yi Lau1,2,*, Abu Bakr Rahman Shaik1,2,*, Manuel Günther2, Marc Pollefeys1,3, Daniel Barath1,4
1ETH Zürich   2University of Zürich   3Microsoft   4HUN-REN SZTAKI
* Equal contribution
Code Paper (coming soon) arXiv (coming soon) Dataset (coming soon)
A free-form description (left, as a dialog) localizes the observer inside a known 3D living-room scene (right), shown with the predicted viewing frustum and a scene-graph overlay of the objects.
From a free-form description of the surroundings, LangLoc estimates the observer’s 2D floor position and heading inside a known 3D scene, using no camera. (click to enlarge)

TL;DR. LangLoc is the first to resolve an intra-scene pose from language alone; prior work stops at coarse scene retrieval. It (1) retrieves the scene with a dual-branch GATv2 + CLIP encoder, (2) estimates position and heading by scoring a dense floor grid through ray-cast object visibility, and (3) resolves residual ambiguity with a Bayesian yes/no dialog.

No camera

Localize from a sentence: private and lightweight, with no images to send.

Scene retrieval

A dual-branch GATv2 + CLIP encoder finds the right scene from the description.

Fine localization

Ray-cast floor-grid scoring recovers the 2D position and heading.

Yes/no dialog

A Bayesian module asks targeted questions to pin down ambiguous poses.

Abstract

We tackle fine-grained indoor localization from natural language: given a free-form description of one’s surroundings, estimate the observer’s 2D position and heading within a known 3D environment. Language queries are lightweight, privacy-preserving, and need no camera. Yet prior work stops at coarse scene retrieval and cannot resolve an intra-scene pose. We close this gap with LangLoc, a three-stage pipeline that (i) retrieves the correct scene via a dual-branch GATv2 encoder with CLIP semantic features, surpassing the previous best by 8 percentage points in Top-1 recall; (ii) estimates position and heading by scoring a dense floor grid through ray-cast object visibility, reaching a median error of 0.95 m; and (iii) resolves residual ambiguity through a Bayesian dialog module that asks targeted yes/no questions and updates a pose posterior until the location is pinpointed. To support this task we contribute a benchmark of 13,000+ pose-indexed natural-language descriptions over 1,300+ indoor 3D scans.

Method

LangLoc overview: a free-form description is parsed into a text scene graph, matched against a database of 3D scene graphs to retrieve the scene, then localized within it and refined through a yes/no dialog.
LangLoc overview. A free-form description is parsed into a text scene graph, matched against a database of 3D scene graphs for retrieval, then localized within the chosen scene and refined through dialog. (click to enlarge)

1. Scene retrieval. A dual-branch GATv2 encoder with CLIP semantic features embeds the text-derived query graph and the database scene graphs into a shared space, ranking candidate scenes by a blend of learned-embedding, global-CLIP, and label-overlap similarity.

2. Fine localization. Within the retrieved scene, LangLoc samples a dense floor grid and scores each candidate viewpoint by ray-casting which described objects are visible, yielding a 2D position and a heading direction, all without any image.

3. Dialog disambiguation. When a description matches several viewpoints, a Bayesian module asks targeted yes/no questions (e.g. “Is there a chair to the left of the table?”) and updates a pose posterior until the location is resolved.

Fine-localization pipeline: the parsed query is matched to 3D objects, a dense floor grid is scored by ray-cast visibility, and a field-of-view aggregation step estimates the final position and heading.
Fine-localization pipeline. The parsed query is matched to 3D objects; a dense floor grid is scored by ray-cast visibility, and a field-of-view aggregation step estimates the final position and heading. (click to enlarge)

Qualitative Results

Predicted (yellow) vs. ground-truth (red) viewing frustums on 3RScan and ScanNet. LangLoc recovers both the floor position and the heading from language alone.

3RScan example: predicted (yellow) and ground-truth (red) viewing frustums overlaid on the reconstructed indoor scene.
3RScan
ScanNet example: predicted (yellow) and ground-truth (red) viewing frustums overlaid on the reconstructed indoor scene.
ScanNet

BibTeX

@inproceedings{langloc2026,
  title     = {LangLoc: Tell Me What You See},
  author    = {Panwar, Shaurya Kishore and Zendehdel Nobari, Roham and
               Lau, Shirley Feng Yi and Shaik, Abu Bakr Rahman and
               G\"unther, Manuel and Pollefeys, Marc and Barath, Daniel},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2026},
}

The final entry (pages, DOI) will be added once the ECCV 2026 proceedings are published.