Passive hydroacoustic studies have underscored the efficiency and relevance of deploying autonomous hydrophones for the surveillance of underwater geophony. In particular, monitoring networks have been deployed for detecting SOFAR-propagating hydroacoustic waves generated by seismic events and locating their sources. The technique has been extended to study other hydroacoustic signals, such as P-waves from teleseismic events or impulsive waves generated by sea water-lava interactions. A significant challenge in this endeavor lies in the time required for the manual detection and annotation of these signals in long-term records. To address this issue, we tested the feasibility of implementing automated algorithms based on machine learning to detect and identify these various signals, and obtained satisfying classification and time picking accuracies. We incorporated those models in a benchmarking framework, proposing a training dataset, two evaluation datasets, two tasks to solve and the evaluations of the mentionned models on them. The goal of this framework is to foster the development of new models in the community, as it gives a clear way to evaluate them.