Phages depend on their bacterial host to replicate. The habitat, density and genetic diversity of host populations are therefore key factors in phage ecology, but our ability to explore their biology depends on the isolation of a diverse and representative collection of phages from different sources. Here, we compared two populations of marine bacterial hosts and their phages collected during a time series sampling program in an oyster farm. The population of Vibrio crassostreae, a species associated specifically to oysters, was genetically structured into clades of near clonal strains, leading to the isolation of closely related phages forming large modules in phage-bacterial infection networks. For Vibrio chagasii, which blooms in the water column, a lower number of closely related hosts and a higher diversity of isolated phages resulted in small modules in the phage-bacterial infection network. Over time, phage load was correlated with V. chagasii abundance, indicating a role of host blooms in driving phage abundance. Genetic experiments further demonstrated that these phage blooms can generate epigenetic and genetic variability that can counteract host defense systems. These results highlight the importance of considering both the environmental dynamics as well as the genetic structure of the host when interpreting phage-bacteria networks.