Robots increasingly have to collaborate with humans or work in places where people can’t always reach them. In some places, a network connection can’t be relied on. A maintenance robot in a railway tunnel is a clean example of what that actually demands: no GPS, no cloud connection, and it still has to know its location, check the hardware around it, and collaborate and communicate with a person nearby. All of this must happen in real time, on a single onboard computer.
That’s the scenario XLAB set out to solve in FORTIS, a project developing multimodal interaction between human and robot for industrial settings, such as construction, railway maintenance, and manufacturing. As part of the consortium, we built four solutions that address this chain of problems in order: seeing the world, placing itself in it, inspecting it, and talking about what it finds. All four run end to end on an NVIDIA Jetson AGX Orin, packaged in ROS 2, with nothing sent to the cloud.
1. Seeing: object detection and segmentation
The first solution finds objects in the scene in real time in two ways, then outlines them. Open-vocabulary detection (GroundingDINO) locates arbitrary objects straight from a text description, so the next object worth detecting doesn’t have to be something seen only during training. Where accuracy and speed matter more than flexibility, a lightweight supervised model (YOLOv11n) is fine-tuned on exactly the objects a given use case cares about.
Detections then feed a segmentation model (SAM, or the leaner FastSAM) for pixel-wise masks. Fused with depth from an RGBD camera, those masks tell the robot how far away each object is, which is enough for performing obstacle detection on its path. The solution has already been validated on real railway and factory recordings, running in real time in ROS 2 on the Jetson.
Example output of open-vocabulary detection followed by segmentation on a factory recording.
2. Positioning: 2D marker detection
With object detection in place, the robot still needs to know where it is. As explained before, railway tunnels are GPS-denied by design. XLAB reframed localization as a focused OCR problem. The solution reads the kilometric point markers already on the tunnel wall: signs like KM. 1+383, meaning 1 kilometre and 383 metres from a defined origin. Reading such markers gives the robot a global piece of information it can fuse with whatever local information it already has, so it always knows where it stands underground.
Built on proven OCR engines (RapidOCR, EasyOCR) and tuned for real-time on-site processing, this runs on the Jetson. The solution has already been validated on a closely matching public benchmark, and evaluation on pilot recordings is ongoing. On-site testing is planned by the end of the project.
Example output of RapidOCR engine on a kilometric point marker.
3. Inspecting: 2D anomaly detection
Once the robot knows its location, it can check the hardware around it for any faults and report not just what is wrong, but also where. Fastener inspection is repetitive, safety-critical, and easy to get wrong when someone is tired. This solution automates fastener inspection with a three-stage pipeline. First, a supervised detector (YOLOv11n) locates each clip and bolt. Secondly, an unsupervised model (DFM), trained only on normal (anomaly-free) samples, flags anything that deviates. Finally, a Vision Language Model (Gemma3) answers the question what is wrong here? In this way, the system categorizes the anomaly and writes a short, plain-language explanation the worker can act on.
Training the second stage on normal samples only is done as real infrastructures faults are rare and usually unlabeled. This means the pipeline catches uncategorized faults, which is why the third, explanation stage is required, especially for the worker. The pipeline has been validated end to end on real pilot recordings and demonstrated live to the consortium on the edge hardware.
An example of an anomaly-free fastener. The pipeline will first detect it, then classify, before producing an explanatory output: The fastener is anomaly-free. There are no missing clips, bolts, broken parts, or misplaced components visible in the image.
Talking: Human–robot context communication
Perception is only useful if the robot does something sensible with it. This solution is a context-aware reasoning engine that decides the content of every human–robot exchange. It continuously maintains a picture of the workspace — human and scene status, robot status, incoming messages, planned actions — against a base of fixed knowledge like available commands and pilot-specific behavioural rules. Based on the current situation and context, a locally deployed LLM decides, in a single structured-output call, whether to command the robot, respond to the human, or both.
One of the most important aspects of this solution is that safety-critical situations never wait on the language model. They bypass it entirely through predefined, deterministic responses in roughly 20 milliseconds, compared to more than a second for an LLM call. This means the system is natural and adaptive when there is time, but fast and predictable when there is not. The solution has been demonstrated live on the edge hardware, reasoning over real inputs from the perception solutions above.
One box, one pipeline
All four solutions share one target: a single Jetson AGX Orin running ROS 2, on-site and in real-time. Detection and marker reading tell the robot where it is and what is around it. Then, anomaly detection tells it what is wrong. Finally, the reasoning engine turns all of it into a command or a sentence a person can act upon. That is the integration story FORTIS is built on, and where XLAB does its work.
What’s next
Over the coming months, all four solutions move from recorded data and controlled demos into on-site pilot testing. That means integrating with the solutions our partners are building (the part XLAB coordinates), and running the whole pipeline on real fasteners, real markers, and a person actually walking in front of a robot rather than a recording of one. Alongside that, the reasoning engine will get a voice. We’re also working on a text-to-speech solution that runs on the same box as everything else, so what the robot decides to say comes back out loud rather than just on a screen.
By the project’s close we want collaborative robots that can be trusted to work alongside humans and that can dynamically adapt to the current environment. We’ll share updates as on-site results come in — follow along at fortis-project.eu.
FORTIS has received funding from the European Union’s Horizon Europe research and innovation programme under Grant Agreement No. 101135707.
