Development of a cross-platform, modular assistance system for navigation and intuitive control using AI-based decision-making for autonomous in-plant transport exemplified by bridge cranes

The KraNavi research project aims to develop an augmented reality (AR)-based assistance system for the autonomous transport of loads using bridge cranes. By enabling intuitive control through speech and gestures, the system simplifies crane operation significantly and helps prevent accidents as well as personal injury and material damage caused by operator errors. An environment recognition system identifies and classifies objects, people, and obstacles in real time, providing all relevant information to the operator via AR.

At the core of the research project is the exploration of an intuitive, multimodal interaction concept. By combining gesture- and speech-based control with AI-supported analysis, communication between humans and machines becomes more natural and intuitive. Principles of human interaction—such as pointing with a finger—are directly transferred to crane control. This creates a previously unexplored control paradigm for stationary transport systems, significantly enhancing both safety and productivity.

The solution approach integrates an AR device together with multiple sensors mounted on the crane for environment perception. A central control unit manages route planning and trajectory guidance. Using speech commands and gestures, the crane operator selects the load, confirms the transport start, and approves the automatically generated route. The planned path is visualized in the operator’s field of view, hazards are highlighted, and the subsequent transport is executed autonomously. In this way, KraNavi addresses a major gap in the state of the art, as current systems lack comparable human-machine interaction and comprehensive environment perception.