Platform-Level Integration
On Android XR, Gemini is integrated into the core operating system rather than being a standalone application. This allows it to maintain environment awareness — "seeing" what the user sees through the passthrough cameras or smart glasses lenses.
Core Capabilities
- Visual Understanding: Identify objects, text, and landmarks in real-time.
- Live Translation: Translate spoken language or physical text (signs, menus) into visual overlays.
- Spatial Assistance: Provide turn-by-turn navigation or instructions anchored to the physical environment.
- Multimodal Interaction: Users can interact via voice, eye tracking, and hand gestures.
Privacy & Security
Google has implemented a "Privacy Shield" architecture for Android XR, where sensor data processed by Gemini is handled on-device whenever possible. Visual data used for environment understanding is strictly controlled via user permissions.