UR Robotic Arm Human-Machine Interface

About the UR´s HMI Project

This project develops a Human-Machine Interface (HMI) for controlling a UR3 robotic arm using predefined hand gestures for tracking hand trajectory and after filtering the data it is used to give trajectory input to the robot. Using a RealSense D455 camera and OpenCV image processing library. A Kalman filter (KF) is implemented to enhance the precision and decrease the error of the hand trajectory, by using sensorial fusion concept with multiple instances of the hand position, captured during the tracking process and a constant velocity state space model to predict a more accurate and stable trajectory for the robot’s input. Simulation and lab test optimized the filter’s adjustable parameters, resulting in a more reliable HMI in multiple scenarios, having an overall standard deviation of 3.3 mm and accuracy of +/- 5 mm. This project demonstrates the potential of intuitive human-robot interaction in collaboration environments, and further capabilities with the suggested improvements.

Robotic Arm HMI Trayectory Programing Pathway Showcase

Hardware and Connections

The tripod is fixed so that the camera lens height from the table surface is approximately 1 meter. This configuration allows the tripod and other structural parts to be placed outside the workspace area preventing any visual or physical obstruction, while providing a stable mounting of the camera sensor.

The communication are stablished from the camera to the computer using a USB 3.0 cables. This connection ensures high quality of image data transfer with low latency. The UR robot control box is connected through an Ethernet cable, to the computer ethernet port using TCP/IP protocol.

Algorithm Design and Architecture

To provide a simple and reliable user navigation through the HMI algorithm’s functionalities and options, a Finite State Machine (FSM) architecture is adopted. This approach organizes the HMI into different states, each handling a specific operation or group of tasks and managing the user input whether from the computer terminal or through the camera sensor.

Image Preprocessing and Hand Segmentation Module

The Intel RealSense D455 RGB and Depth camera is fixed above the workspace, facing down, to capture the operator’s hand movement and other gesture commands. This type of camera provides a detailed 3D data, which is processed using image processing techniques, implementing the Python library OpenCV. These techniques include thresholding to create accurate hand segmentation and extracting relevant features using the convex hull and defect approach to extract geometrical features from the hand parts. These features are used to recognise various predefined hand gestures.

Hand Tracking and Kalman Filter Module

Pixel position to spatial distance conversion converts the pixel position to spatial coordinate within the workspace using camera’s known field of view and depth information.
Raw trajectory array variable storing store the raw trajectory data in an array for further processing.
Kalman filtering applies Kalman filtering to the raw position measurements to reduce noise and produce a smooth trajectory.
Filtered trajectory storage stores the filtered trajectory in a data array for further use.
Filtered trajectory retrieval retrieves the filtered trajectory data to use in subsequences modules.

Robot-HMI Communication Interface

Stablishes adequate communication with the robotic controller and terminate the communication correctly when needed.

Robot TCP and joint movement command generation generates for the robot movement and transmit using the RTDE interface.
Position Data and Kinematics information: manages communication with the robot using RTDE to receive position, inverse and forward calculation and error message handling.
Implement safety features to limit the robot joint velocity, stop command and other.

Other Showcases and Conclusion of The Project

This project successfully implemented an HMI system that uses image processing and hand segmentation to identify user-defined trajectories, which is used to control the UR3 robotic arm in a collaborative workspace. This approach demonstrates the potential for controlling robotics arms in collaborative workspace using intuitive human gestures.

A critical component was the Kalman filter, which enhance the reliability of hand position measurement by fusion multiple instances of the hand and predicting a more precis position based on a linear motion model. Future improvements could refine the model to better handle change in direction, reducing errors in complex trajectories.

For more information about this project please contact.

Google Sites

Report abuse