~/projects/rokidJesse Zeng / portfolio
All projects

Rokid AR Guide

Role
Project lead / device integration
When
Jul–Aug 2026
Stack
Android / Kotlin / Python / FastAPI / Dify
Status
Internship

Why I built it

An exhibition has more information than a visitor can absorb at once. I wanted to make the explanation available beside the object itself, rather than send people back and forth between a display and a screen.

Who it is for

The intended benefit is a more continuous visit: look at an exhibit, hear a reviewed introduction, then ask a question in the same context. The interaction should support the visitor's attention, not compete with it.

My contribution

I led the guide and integrated device video, recognition, exhibit context and HUD/TTS presentation. With He Yutong, I worked through connection failures and the transition from reviewed introductions to grounded dialogue.

Connection & context

A connection is more than a Bluetooth success message. I traced a callback/state conflict that stopped the next connection stage, then revised the transition and error handling so the video path could continue.

The latest useful frame takes priority over a backlog. Once an exhibit is stable, the backend maintains the conversation context; switching exhibits invalidates old answers rather than letting them appear beside a new object.

Information & dialogue

Reviewed introductions and generated answers follow different paths. The introduction finishes before listening begins. Follow-up answers use the current exhibit's reviewed material; a knowledge gap is acknowledged instead of filled with unsupported details.

  1. CaptureGlasses video / Android decoding
  2. RecognizeRecent frame / Python recognition
  3. PresentReviewed knowledge / HUD / speech
Device-to-visitor flow. Diagram, not an internal interface screenshot.

Failures & presentation

When a model-service fault leaked internal status into subtitles, I separated service logs from the visitor's answer. The guide can use reviewed material as a fallback without presenting operational noise as exhibition content.

Where it stands

Device demonstrations exercised the recognition-to-answer-to-display flow. The final software handoff included regression, dialogue and retrieval checks. The next stage is broader field evaluation, separate from those development results.