Updated

Updated · Google Research · Aug 25

Google XR Unveils AgentHands Prototype, Beating Speech-Only Baseline in 12-User Study

Updated

Updated · Google Research · Aug 25

Google XR Unveils AgentHands Prototype, Beating Speech-Only Baseline in 12-User Study

1 articles · Updated · Google Research · Aug 25

Google XR introduced AgentHands, an LLM-powered XR prototype that adds synchronized hand gestures to conversational agents so they can give spatially grounded guidance during physical tasks.
A 12-person within-subjects study found AgentHands outperformed a speech-only baseline on location understanding, action understanding and warning noticeability while using the same scripted verbal content.
The system works by registering real-world objects with eye gaze and scene reconstruction, then turning LLM-generated GestureEvents into word-timed hand motions synchronized with text-to-speech on the headset.
Google demonstrated the prototype in orchid care, 3D-printer walkthroughs and wellness coaching, arguing that embodied gestures can reduce cognitive load and make Android XR assistance more natural.

Sources

Are human-like gestures the ultimate XR interface, or just a temporary crutch masking deeper AI communication flaws?

Could giving AI virtual hands actually clutter your vision instead of helping you navigate the physical world?

Will constantly scanning your room to power these AI gestures secretly compromise your privacy in immersive spaces?

1 total