The differential response of marine populations to climate change remains poorly understood. Here, we combine common garden thermotolerance experiments in aquaria and population genetics to disentangle the factors driving the population response to thermal stress in a temperate habitat-forming species: the octocoral Paramuricea clavata. Using eight populations separated from tens of meters to hundreds of kilometers, which were differentially impacted by recent mortality events, we identify 25 °C as a critical thermal threshold. After one week of exposure at this temperature, seven of the eight populations were affected by tissue necrosis and after 30 days of exposure at this temperature, the mean % of affected colonies increased gradually from 3 to 97%. We then demonstrate the weak relation between the observed differential phenotypic responses and the local temperature regimes experienced by each population. A significant correlation was observed between these responses and the extent of genetic drift impacting each population. Local adaptation may thus be hindered by genetic drift, which seems to be the main driver of the differential response. Accordingly, conservation measures should promote connectivity and control density erosion in order to limit the impact of genetic drift on marine populations facing climate change.