Research

overlay
18 Jun

Real-Time Feedback System for Kinesthetic Robot Teaching

Teaching a collaborative robot is often more difficult than simply showing it what to do. In practice, novice users often provide demonstrations that are incomplete, inconsistent, or poorly suited to the robot’s learning process. A key reason is that human teachers often struggle to understand what the robot has actually learned at any given moment. Unlike human students, robots do not naturally signal confusion, uncertainty, or partial understanding. To address this gap, robot learning must be treated as a dialogue in which the robot actively communicates its current state to the teacher through clear and accessible feedback, such as visual, haptic, or auditory signals. This research focuses on new communication interfaces that help robot “students” express what they understand, what they do not understand, and when they need help, making teaching a shared and responsive process rather than a guessing game. In this work, we address the overarching question:

Research Question: How can robots communicate internal constraints to users in human-robot kinesthetic interactions?

Detailed Outline

 

User Experiment

We first conducted a focus group study (N = 9) to identify effective ways of visualizing key robot information, including joint limits, self-collisions, and manipulability. Guided by these insights, we designed an AR-based real-time feedback system and evaluated it in a between-subjects user study (N = 36) on a 7-DoF collaborative robot. Participants performed two tasks, including insertion and drink pouring, with the second task enabling assessment of participants’ learning across tasks.

We designed the following three conditions to test the proposed feedback system:

Offline Feedback — Participants demonstrated tasks without real-time guidance. After the first three demonstrations, they observed the robot’s reproduced execution and used this offline feedback to adjust subsequent demonstrations.

Real-Time Feedback — Participants received continuous AR feedback during demonstrations, showing key task-relevant states to guide movements in real time. No execution playback was provided between rounds.

Combined Feedback — Participants received both real-time AR cues and offline execution playback, enabling immediate guidance during demonstrations and reflection on the robot’s reproduced motion afterward.

The study compares three feedback conditions during kinesthetic robot teaching: offline feedback (red), real-time feedback (blue), and combined feedback (green). Participants first performed a cuboid block insertion task by providing three kinesthetic demonstrations to a 7-DoF robot, followed by a second demonstration round depending on the feedback condition. A second drink-pouring task was then performed with no feedback for any group, enabling assessment of participants’ learning across tasks.

Key Findings

  • Novice users who received real-time AR feedback not only performed better on the initial task but also retained these benefits when the feedback was removed in the second task, demonstrating measurable learning.
  • Feedback supported users in developing a better understanding of the robot’s motion constraints, leading to improved robot task performance even without visual assistance.

Resources

PDF | Video

Citation

Muhammad Bilal, Tharaka Ratnayake, D. Antony Chacon, Nir Lipovetzky, Denny Oetomo, and Wafa Johal. Design and Evaluation of AR-Based Real-Time Feedback System for Kinesthetic Robot Teaching. In Proceedings of the ACM Designing Interactive Systems Conference (DIS), pp. 332–356, 2026. DOI: 10.1145/3800645.3812951

This paper received Honourable Mention Award.

overlay
15 Mar

Robot Kinematic Redundancy in Learning from Demonstration

Kinematic redundancy, where a robot has more joint degrees of freedom than required for a given task, provides greater flexibility during kinesthetic demonstrations. However, this flexibility also introduces additional complexity, as novice users must navigate a larger set of feasible robot configurations. This project investigates how the robot's degree of redundancy (DoR) affects human performance during kinesthetic demonstrations and, in turn, how the resulting demonstrations influence robot learning and task execution.

Study overview showing the robotic platform, motion-capture system, and experimental setup.

To study the impact of robot redundancy, we artificially reduced the robot's DoR by locking one of its joints. We then conducted a within-subject user study with 24 participants, comparing two conditions: high DoR (unconstrained) and low DoR (constrained). Our experimental setup also incorporated a motion capture system to record participants' hand interactions with individual robot joints, enabling detailed joint-level analysis of physical interaction during teaching.

Participants completed two manipulation tasks: button pressing and cuboid block insertion.

Subfigures (a–b) show a participant kinesthetically demonstrating the button task. Subfigures (c–d) depict the insertion task, with (c) showing block pickup and (d) partial insertion.

 

User Experiment Videos

This participant first performed button pressing task under constrained condition followed by unconstrained condition.

 

This participant first performed insertion task under unconstrained condition followed by constrained condition.

 

Key Findings

  • Constraining the robot's redundancy significantly increased participants' mental workload.
  • Demonstrations took longer to complete under the low-DoR condition.
  • Reduced redundancy resulted in more failed demonstration attempts.
  • Participants physically interacted with the robot's joints more frequently when redundancy was reduced.
  • Demonstrations collected under the low-DoR condition led to poorer subsequent robot learning and task execution.

To support reproducibility and future research, we also released InteractLfD, a comprehensive dataset of human kinesthetic demonstrations collected during this study.

Resources

PDF | InteractLfD (Dataset)

Citation

Muhammad Bilal, D. Antony Chacon, Nir Lipovetzky, Denny Oetomo, and Wafa Johal. Investigating the Impact of Robot Degree of Redundancy on Learning from Demonstration. In Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 825–833, 2026. DOI:10.1145/3757279.3785606

overlay
22 Oct

Quantifying Demonstration Quality in Learning from Demonstration

Learning from Demonstration empowers novice users to teach robots daily life tasks without writing sophisticated code, thereby promoting the democratization of robotics. However, novice users often provide sub-optimal demonstrations, which can potentially impact the robot’s ability to efficiently learn and execute the tasks. Prior research has assessed the quality of demonstrations by evaluating the robot’s task performance; however, the approach remains insufficient to qualify individual demonstrations, leaving the reason for classifying demonstrations as high- or low-quality unknown. Therefore, this simulation-based study aims to quantify the quality of individual demonstration at each step by incorporating motion-related quality features such as manipulability and joint-space jerk.

The given datasets have been evaluated based on the quality features, manipulability and joint-space jerk, and ranked them from high to low quality. At next step, we trained an LfD model for each group, where N denotes the total number of groups, followed by rollouts the learned LfD model in order to compute the task performance and assess the quality of task execution based on the quality features.

Our results illustrate a strong correlation between ranked demonstrations and the quality of task execution. Interestingly, we observed that the quality features have a significant impact on task performance, particularly when the provided demonstrations exhibit diversity in terms of quality. Overall, this analysis enables quantifying the quality of individual demonstrations based on motion-related quality features, thus improving learning from demonstration.

Key Findings

This study makes three primary contributions:

  • Utilisation of motion-based features to quantify the quality of individual demonstrations before learning process.
  • Transferability of quality features from provided demonstrations to the corresponding LfD model generated motions.
  • Quality features can serve as an indicator of task performance when used in a diverse set of demonstrations.

Resources

PDF

Citation

Muhammad Bilal, Nir Lipovetzky, Denny Oetomo, and Wafa Johal. Beyond Success: Quantifying Demonstration Quality in Learning from Demonstration. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5120–5127, 2024. DOI:10.1109/IROS58592.2024.10802187

 

overlay
8 Apr

Adaptive Variable Impedance Control for Force Tracking

Robots operating in contact-rich environments, such as grinding, polishing, surface finishing, cleaning, and assembly, require advanced force control strategies to interact safely and accurately with their surroundings. Traditional motion control approaches are often insufficient for these applications, as variations in environmental properties can lead to force tracking errors, excessive contact forces, or damage to the robot and workpiece.

This project investigates an adaptive variable impedance control framework that enables a robot to dynamically regulate interaction forces under uncertain environmental conditions. Unlike conventional impedance controllers that rely on fixed stiffness and damping parameters, the proposed approach adapts the impedance parameters based on the interaction state, allowing the robot to maintain stable force tracking despite changes in surface properties.


Block diagram of position-based impedance control for force tracking. The reference trajectory along with the net force computes the commanded position followed by inverse kinematics calculation. The PID controller is used to control the robot arm.

The controller models the robot–environment interaction using a virtual mass–spring–damper system and adjusts the desired motion response according to the external contact forces. This enables accurate force regulation without requiring complete knowledge of the environment dynamics.

Interaction between the robot end-effector and the environment where the environment model is represented by the spring element while impedance model is represented by mass-spring-damper system.

Singularity Avoidance for Redundant Robots

For redundant robotic systems, additional degrees of freedom provide flexibility to avoid singular configurations and improve motion capability. This project integrates a singularity avoidance strategy into the inverse kinematics framework, allowing the robot to maintain high manipulability while executing complex contact trajectories.

The inverse kinematics solution is obtained using the Damped Least Squares method, combined with a manipulability-based optimisation strategy to prevent singular configurations during task execution.

Key Results

  • Developed an adaptive variable impedance controller for force tracking under uncertain environmental conditions.
  • Reduced force-error overshoot during environmental stiffness transitions by up to 56.13% compared with constant impedance control.
  • Improved robot manipulability by 5.95% through an integrated singularity avoidance strategy.
  • Demonstrated stable multi-axis force tracking for redundant robotic manipulators performing complex contact trajectories.

Citation

Muhammad Bilal, M. Nadeem Akram, Mohsin Rizwan. "Adaptive Variable Impedance Control for Multi-Axis Force Tracking in Uncertain Environment Stiffness with Redundancy Exploitation." Journal of Control Engineering and Applied Informatics, vol. 24, no. 2, pp. 35–45, 2022.