Phytoplanktonic communities maintain a high diversity in a seemingly homogeneous environment, competing for the same set of resources. Many theories have been proposed to explain this coexistence despite likely competition, such as contrasted responses to temporal environmental variation. However, theory has developed at a faster pace than empirical evaluation using field data, which requires inferring drivers of community dynamics from observational time series. Here, we combine autoregressive models with a data set spanning more than 20 years of phytoplankton counts every two weeks, together with nutrients and physical variables. By comparing models dominated by nutrients or physical variables (hydrodynamics and climate), we first explore which abiotic factors contribute more to phytoplankton growth and decline. We find that physical drivers – such as irradiance, wind, and salinity – explain some of the variability in abundances unexplained by biotic interactions. In contrast, responses to nutrients explain less of the phytoplankton variability. Concerning biotic drivers of community dynamics, multivariate autoregressive models reveal that competition between different groups (genera) has a much weaker effect on population growth rates than competition within a group. In fact, the few biotic interactions between genera that are detected are frequently positive. Hence, our system is unlikely to be best represented as a set of competitors whose differing responses to fluctuating environments allow coexistence, as in ‘paradox of the plankton’ models with a storage effect or a relative nonlinearity of competition. Coexistence is more likely to result from stabilizing niche differences, manifested through high intragroup density‐dependence. Competition between phytoplanktonic groups and nutrient limitation are often invoked as drivers of phytoplankton dynamics; our findings suggest instead that more attention should be given to the physical structure of the environment and natural enemies such as grazers and pathogens, that could explain the high intraspecific density dependence found here.