Digital holography is an imaging process that encodes the 3D information of objects into a single intensity image. In recent years, this technology has been used to detect and count various microscopic objects and has been applied in submersible equipment to monitor the in situ distribution of plankton. To count and classify plankton, conventional methods require a holographic reconstruction step to decode the hologram before identifying the objects. However, this iterative and time-consuming step must be performed at each frame of a video, which makes it difficult to support real-time processing. We propose a real-time object detection based approach that simultaneously performs the detection, classification and counting of all plankton within videos of raw holograms. Experiments show that our pipeline based on YOLOv5 and SORT is fast (44 FPS) and can accurately detect and identify the plankton among 13 classes (97.6% mAP@0.5, 92% MOTA). Our method can be implemented to detect and count other microscopic objects in raw holograms.