Teaching a collaborative robot is often more difficult than simply showing it what to do. In practice, novice users often provide demonstrations that are incomplete, inconsistent, or poorly suited to the robot’s learning process. A key reason is that human teachers often struggle to understand what the robot has actually learned at any given moment. Unlike human students, robots do not naturally signal confusion, uncertainty, or partial understanding. To address this gap, robot learning must be treated as a dialogue in which the robot actively communicates its current state to the teacher through clear and accessible feedback, such as visual, haptic, or auditory signals. This research focuses on new communication interfaces that help robot “students” express what they understand, what they do not understand, and when they need help, making teaching a shared and responsive process rather than a guessing game. In this work, we address the overarching question:
How can robots communicate internal constraints to users in human-robot kinesthetic interactions?
User Experiment
We first conducted a focus group study (N = 9) to identify effective ways of visualizing key robot information, including joint limits, self-collisions, and manipulability. Guided by these insights, we designed an AR-based real-time feedback system and evaluated it in a between-subjects user study (N = 36) on a 7-DoF collaborative robot. Participants performed two tasks, including insertion and drink pouring, with the second task enabling assessment of participants’ learning across tasks.
We designed the following three conditions to test the proposed feedback system:
Offline Feedback — Participants demonstrated tasks without real-time guidance. After the first three demonstrations, they observed the robot’s reproduced execution and used this offline feedback to adjust subsequent demonstrations.
Real-Time Feedback — Participants received continuous AR feedback during demonstrations, showing key task-relevant states to guide movements in real time. No execution playback was provided between rounds.
Combined Feedback — Participants received both real-time AR cues and offline execution playback, enabling immediate guidance during demonstrations and reflection on the robot’s reproduced motion afterward.
Key Findings
- Novice users who received real-time AR feedback not only performed better on the initial task but also retained these benefits when the feedback was removed in the second task, demonstrating measurable learning.
- Feedback supported users in developing a better understanding of the robot’s motion constraints, leading to improved robot task performance even without visual assistance.
Resources
Citation
Muhammad Bilal, Tharaka Ratnayake, D. Antony Chacon, Nir Lipovetzky, Denny Oetomo, and Wafa Johal. Design and Evaluation of AR-Based Real-Time Feedback System for Kinesthetic Robot Teaching. In Proceedings of the ACM Designing Interactive Systems Conference (DIS), pp. 332–356, 2026. DOI: 10.1145/3800645.3812951
This paper received Honourable Mention Award.