Digital holographic microscopy is an imaging process that encodes the 3D information of a sample into a single 2D hologram. The holographic reconstruction that decodes the hologram is conventionally based on the diffraction formula and involves various iterative steps in order to recover the lost phase information of the hologram. In the past few years, the deep-learning-based model has shown great potential to perform holographic reconstruction directly on a single hologram. However, preparing a large and high-quality dataset to train the models remains a challenge, especially when the holographic reconstruction images that serve as ground truth are difficult to obtain and can have a deteriorated quality due to various interferences of the imaging device. A cycle generative adversarial network is first trained with unpaired brightfield microscope images to restore the visual quality of the holographic reconstructions. The enhanced holographic reconstructions then serve as ground truth for the supervised learning of a U-Net that performs the holographic reconstruction on a single hologram. The proposed method was evaluated on plankton images and could also be applied to achieve super-resolution or colorization of the holographic reconstructions.