Faced with the increasing exploitation and artificialisation of the oceans, political players and society recognize the urgent need for a global assessment of the state of marine ecosystems. Our ability to distinguish natural variations in ecosystems from changes resulting from anthropogenic activities requires multidisciplinary monitoring of these environments. This thinking led to the development of monitoring programs at instrumented sites in coastal environments, and to the establishment of observatories that provide continuous access to both coastal and deep-sea ecosystems, in which the Research Infrastructure EMSO plays a leading role at the European level. These advances have been made possible by the technological developments of recent decades. In particular, underwater imaging has rapidly emerged as a non-destructive method for examining biodiversity on unprecedented time and space scales. The exponential increase in resolution and quality of optical sensors means that species can be identified with ever greater precision, and the reduction in their cost facilitates their acquisition. These advancements demonstrate the potential of imaging tools for ecological monitoring of marine ecosystems.

Yet the success of imagery data for scientific purposes leads to new challenges linked to the processing of the exponential amount of data collected, which can be time-consuming and tedious. The advent of Artificial Intelligence (AI) has enabled the development of powerful algorithms that should facilitate the processing of large imaging datasets. However, the capacity of machines to convert large volumes of raw optical signals into usable data for studying marine habitats is conditioned by a learning phase on large standardized reference databases resulting from manual processing based on human decisions. These reference databases are generated by scientists, students, technical staff in laboratories, as well as by citizens through online platforms (e.g. Deep Sea Spy citizen science annotation platform developed by IFREMER). Innovative AI algorithms should be developed to improve the human-intensive work of annotations.

In the framework of the iMagine Horizon Europe project (Grant number 101017567), IFREMER is participating in the “Ecosystem monitoring at EMSO sites by video imagery” use case. This case study aims to create an operational and integrated service, based on AI models, for automatic processing of images collected by cameras at EMSO underwater sites, identifying and analysing fauna and habitats for ecosystem monitoring purposes.

In this context, the primary aim is to develop an automated object detection system for identifying species in the images using supervised machine learning techniques. This supervised learning approach relies on labelled data, and in this case, we used the citizen science annotations from Deep Sea Spy to train the model. Indeed, Deep Sea Spy is a participative science platform launched in 2017, that provides access to images from EMSO-Azores and Ocean Networks Canada observatories for annotation purposes.

Convolutional Neural Networks (CNN) are frequently used for object detection tasks due to their ability to extract relevant features from images. Consequently, a CNN-based object detection algorithm called Yolov8 was specifically employed to automatically identify species present in the images.

Significant efforts have been made to prepare the training dataset for automatic identification, which has been challenging due to the diversity of species and the variable methods used for labelling the data. Indeed, the annotations were made by multiple annotators (each image is analyzed by 10 persons), which resulted in varying levels of accuracy and consistency in the labelling of the different species. Furthermore, the data were labelled differently depending on the species (polygons, lines, points). This has required additional effort to standardise and clean the data in order to make it suitable for use with the YOLOv8 model.

The final result of this work is the development of a user-friendly pipeline that comprises several interconnected components and functions, enabling efficient training data preparation, cleaning, and effective model training. Key elements of the pipeline include:

• Training data Conversion: The pipeline starts by converting various annotation types (polygons, lines, and points) into regular bounding boxes. Corrections are applied to compensate for potential conversion errors, and adjustments are made based on the original label and species.
• Training data Unification: A Python function unifies overlapping bounding boxes, leveraging redundant information from multiple annotators to ensure accurate object labelling. The unification process considers the Intersection over Union (IoU) metric and is species-dependent, reducing incorrect annotations.
• Training data Visualization and Validation: A function generates thumbnails from bounding boxes, allowing users to visually inspect and verify data quality. Unreliable or incorrect thumbnails can be deleted, removing their corresponding bounding boxes from the dataset and ensuring precise control over the data used for Yolov8 model training.
• Yolov8 learning set formatting: The pipeline prepares the cleaned dataset in the required format for training the Yolov8 model, automating the process and facilitating efficient model development.

As a final step, Yolov8 was trained with a cleaned learning dataset and results are now under revision and validation in order to further optimise the algorithm.

Once finalized, the pipeline will be integrated into the iMagine platform as a service, providing researchers, marine biologists and other stakeholders with tools for cleaning of datasets (especially citizen science image datasets) and an AI model capable of automatic marine species classification.

The presentation at IMDIS 2024 will give an overview of the final annotation pipeline, from training data preparation and cleaning, to model learning and inference results.