Fishing may induce neutral and adaptive evolution affecting life-history traits, and molecular evidence has shown that neutral genetic diversity has declined in some exploited populations. Here we theoretically study the interplay between neutral and adaptive evolution caused by fishing. An individual-based eco-genetic model is devised that includes neutral and functional loci in a realistic ecological setting. In line with theoretical expectations, we find that fishing induces evolution towards slow growth, early maturation at small size, and higher reproductive investment. We show, first, that the choice of genetic model (based on either quantitative genetics or gametic inheritance) influences the evolutionary recovery of traits after fishing ceases. Second, we analyse the influence of three factors possibly involved in the lack of evolutionary recovery: the strength of selection, the effect of genetic drift, and the loss of adaptive potential. We find that evolutionary recovery is hampered by an association of weak selection differentials with reduced additive genetic variances. Third, the contribution of fisheries-induced selection to the erosion of functional genetic diversity clearly dominates that of genetic drift only for the traits related to maturation. Together, our results highlight the importance of taking into account population genetic variability in predictions of eco-evolutionary dynamics.