Rokid AR Guide
- Role
- Project lead / device integration
- When
- Jul–Aug 2026
- Stack
- Android / Kotlin / Python / FastAPI / Dify
- Status
- Internship
Why I built it
An exhibition has more information than a visitor can absorb at once. I wanted to make the explanation available beside the object itself, rather than send people back and forth between a display and a screen.
Who it is for
The intended benefit is a more continuous visit: look at an exhibit, hear a reviewed introduction, then ask a question in the same context. The interaction should support the visitor's attention, not compete with it.
My contribution
I led the guide and integrated device video, recognition, exhibit context and HUD/TTS presentation. With He Yutong, I worked through connection failures and the transition from reviewed introductions to grounded dialogue.
Connection & context
A connection is more than a Bluetooth success message. I traced a callback/state conflict that stopped the next connection stage, then revised the transition and error handling so the video path could continue.
The latest useful frame takes priority over a backlog. Once an exhibit is stable, the backend maintains the conversation context; switching exhibits invalidates old answers rather than letting them appear beside a new object.
Information & dialogue
Reviewed introductions and generated answers follow different paths. The introduction finishes before listening begins. Follow-up answers use the current exhibit's reviewed material; a knowledge gap is acknowledged instead of filled with unsupported details.
- CaptureGlasses video / Android decoding
- RecognizeRecent frame / Python recognition
- PresentReviewed knowledge / HUD / speech
Failures & presentation
When a model-service fault leaked internal status into subtitles, I separated service logs from the visitor's answer. The guide can use reviewed material as a fallback without presenting operational noise as exhibition content.
Where it stands
Device demonstrations exercised the recognition-to-answer-to-display flow. The final software handoff included regression, dialogue and retrieval checks. The next stage is broader field evaluation, separate from those development results.