Autonomous and Intelligent Systems (AIS) Laboratory

eM/S Salama, our Unmanned Surface Vessel (USV), signifies a remarkable advancement in maritime technology. Our expedition commenced in 2022, and we have presently attained a phase where Salama can be maneuvered  remotely, with semi-autonomous mode, and further features under development. Our initiative is firmly rooted in the Applied Research Platform for Autonomous Systems (ARPA), a testbed for maritime automation, autonomy, and remote control, bolstering the blue growth sector in Southwest Finland.

The considerable dimensions of eM/S Salama enable it to function under strenuous weather conditions and accumulate multi-modal data for deep learning from a diverse range of scenarios. Salama employs a multi-modal sensing system, including RGB cameras, thermal cameras, and LiDAR sensors. We have gathered and annotated data, which we utilize in deep learning algorithms to attain situational awareness of the vessel’s environment. Formulated sensor auto-calibration and registration methods ensure precise sensor measurements and dependable operation.

EM/S Salama constitutes a component of a test platform conceived and constructed by Turku University of Applied Sciences. This platform aids the research community and industry in effectively employing deep learning to expedite the development of autonomous maritime vessels. It facilitates the development and performance verification of deep learning algorithms, bridges the digitalization skills gap, and encourages collaboration between academia and industry.

We have independently developed and constructed the test platform, incorporating commercial off-the-shelf equipment into the comprehensive system when available.

We invite you to accompany us as we traverse the challenges and opportunities of applying deep learning to remote sensing problems in intricate marine environments. We extend a warm welcome aboard.

A Remote Operations Center (ROC) for the USV, when integrated with deep learning algorithms, is a potent tool that brings numerous advantages. Deep learning algorithms can analyze and process precise situational data collected from the USV multi-modal sensoring system and increase the efficiency and safety of the USV. Efficiency improves by optimizing processes and safety improves when people can be removed from dangerous operating environments or when human errors can be reduced in operations.

In the ROC, operators can monitor, support, assist, supervise, and control the USV. The ROC can monitor the USV or directly control the USV’s systems. The key components for remote operation of the USV are the USV multi-modal sensoring system, data communication links, and the ROC itself. The USV multi-modal sensoring system provides information about both the USV itself and its environment. The sensor data of the USV transferred to the ROC can also be utilized directly in a digital twin of the vessel.

The ROC is equipped with a comprehensive display setup, which includes five 55-inch screens and three 24-inch screens. These screens provide 180-degree front and back views, telemetry data, and map data. The system is designed to be flexible, allowing for the addition of extra screens to display information from other sensors, such as side-view, thermal cameras, and LiDARs. For instance, two additional screens can be incorporated to present views from both sides of the vessel. Further Metaverse implementation of the ROC will be provided.

Tablets are employed to control the vessel, offering a more convenient method to modify and test different control systems compared to using a physical controller or an exact replica of the vessel dashboard. The telemetry data from the vessel is transmitted using NMEA2000 messages, and the video streams are facilitated via RTSP.

The ROC software was obtained from a commercial supplier Brighthouse Intelligence, ensuring reliable and efficient operation of the system. Currently the ROC is used to control eM/S Salama, but in the future we plan to extend it to control fleets of vessels and also extend into scenarios other than maritime.

The 5G Test Network Turku, operated by the Turku University of Applied Sciences, is a wireless communications test network built from commercial standard hardware and software¹. The network, which has its own operator ID and SIM cards, uses 4G and 5G technologies. Its indoor and outdoor base stations and network elements can be flexibly configured according to testing and research needs¹.

The laboratory has over 30 years of experience in radio engineering applications and equipment, RF electronics, and measurement technology¹. It offers comprehensive measurements and tests for telecommunication systems including 4G LTE, 5G NSA, and 5G SA. The laboratory also has a shielded and attenuated room for radio interface measurements¹.

A unique feature of the test network is the ability to quickly build a 4G/5G private network (Rapidly Deployable Network) that can be taken to the customer’s test area and environment¹. For instance, such a network has been built in the Port of Turku for research and development use.

In the context of maritime environments, 5G technology can significantly enhance the deployment and adoption of autonomous or remotely controlled unmanned surface vessels. The high-speed, low-latency communication provided by 5G is crucial for the real-time control and monitoring of the vessels like our eM/S Salama. Furthermore, the 5G Test Network Turku’s ability to rapidly deploy a private network can be particularly useful in creating a dedicated communication infrastructure for these vessels in specific test areas.

The unmanned surface vessel eM/S Salama faces two primary challenges: limited local data storage capacity and the resource-intensive nature of developing deep learning models. To address these issues, researchers have created an on-shore data platform specifically tailored for storing the multi-modal data collected by the eM/S Salama. This platform also allows enrichment of the data by incorporating information from external sources, such as accurate weather data and Automatic Identification System (AIS) data.

Given the diverse sensor data types that need storage, the platform adopts a hybrid data storage approach. It combines different databases optimized for various data formats and use cases. The two main components are:

PostgreSQL Relational Database:

• Used for indexing all data objects and their associated metadata.
• Currently employed to store AIS data and data object metadata.
• As the project progresses, this relational database will accommodate other structured data types.

Ambry Object Storage:

• Specifically designed for unstructured data (ex. video).
• Provides efficient storage for immutable data accessible via an HTTP API.

The data platform is deployed on the Turku UAS on-shore Proxmox-based cloud, which enhances security by isolating data platform services from the Internet by default. Only exposing the gateway API services for external use.

Within the Proxmox environment, a microservice architecture is employed. The data source gateway serves as the entry point for data into the system, supporting multiple interfaces for ingesting data with various protocols (including MQTT and HTTPs). Communication between microservices and data delivery from the gateway to storage handlers utilize Apache Pulsar, which meets performance, durability, and multi-tenancy requirements. Keycloak manages identity and access throughout the system, with Pulsar endpoints serving as key points.

The storage handler services ensure that data acquired from the gateway is associated with metadata. Initially, data is stored in local hot storage (Ambry and PostgreSQL) and later moved to cold storage. Metadata includes fields describing the data object and its current storage location, enabling efficient data search queries. The metadata conforms to Dublin Core metadata standard.

Data platform works as a central data hub through which the streams of our collected raw sensor data and derived enriched data flow. We publish datasets carefully aggregated from these data streams. Machine-to-machine access to this data is offered through multiple interfaces.