Performance Measures for Quantifying Safe and Reliable Operation of Professional Service Robots in Unstructured, Dynamic Environments

Date and venue

ICAR 2011, Tallinn, Estonia.
June 20, 2011, 9.00–17.00.


To enable true human‐machine collaboration of service robots, AGVs, and forklifts working in close proximity in semi‐ and unstructured environments, it is imperative that certain requisite safety and reliability constraints be met. Consequently, designing relevant experiments and developing test methods/metrics that provide statistically significant results and quantifiable performance data plays a significant role in achieving this goal, thus enabling performance evaluation and benchmarking of constituent components of navigation and world modeling systems. The dynamic nature of real‐world environments presents additional challenges for establishing evaluation procedures, including standardized description of the environment, selection of representative test scenarios, and the development of suitable benchmarks regarding autonomy and robustness to changes in the environment.

This full‐day workshop aims to bring together leading experts from academia, industry, and government who have an interest in robotic solutions that require safe operation in professional, semi‐ and unstructured environments. As part of the workshop, the organizers invite input and feedback from the participants to formulate test methods and performance metrics that take into account industry requirements.


Opening and welcome
Rolf Lakaemper Temple University and Raj Madhavan University of Maryland College Park and NIST: Evaluating World Modeling and Navigation using CLEO (Complexity Levels for Environments and Obstacles) Presentation material
Ola B Pettersson Atlas Copco: Safe Operation of Multiple Mining Machines Presentation material
Emil Hällstig Optronic: Optical Metrology for Industrial Environments Presentation material
Coffee break
Yury Brodskiy, Jan Broenink, Angelika Mader and Stefano Stramigioli University of Twente: A Survey on Robust Autonomy Metrics for Robots Presentation material
Rikard Mäki Volvo Construction Equipment: Autonomous Functions in Potentially Harmful Construction Equipment
Henrik Andreasson Örebro University: Vision-Based People Detection Utilizing Reflective Vests for Autonomous Transportation Applications Presentation material
Lunch break
Gerald Steinbauer and Franz Wotawa Graz University of Technology: Evaluating the Robustness of the Perception-Decision-Execution Cycle of Autonomous Robots Presentation material
Ross Creed, Kristiyan Georgiev, and Rolf Lakaemper Temple University: A Visualization Tool for Hybrid Pose Map Evaluation Presentation material
Martin Magnusson and Todor Stoyanov Örebro University: Comparative Accuracy Evaluation of 3D Representations and Range Sensors Presentation material
Coffee break

Important dates

May 23, 2011
Deadline for initial submissions (one-page abstract)
May 27, 2011
Notification of acceptance
June 13, 2011
Deadline for final manuscripts
June 20, 2011


Topics and submissions

The workshop will provide a forum for peer‐reviewed papers. In addition, invited speakers from industry and academia will present their view on key aspects of safe and reliable professional service robot operation based on their experience in relevant industrial and research projects.

Authors interested in presenting at the workshop are requested to submit a one page abstract with title, author names and affiliations to Martin Magnusson []. Based on a review of the submitted information, presenters will be selected.

Papers will be included in the workshop notes, available for download online. Please note that this material will not appear in the conference proceedings.

Topics of interest include:

Intended audience

The workshop addresses people from academia, industry, governmental and other institutions who are developing or intend to use robotic solutions that require safe operation in professional, semi‐ and unstructured environments.