As climate change accelerates, species are shifting poleward and subtropical and tropical species are colonizing temperate environments. A popular approach for characterizing such responses is the community temperature index (CTI), which tracks the mean thermal affinity of a community. Studies in marine, freshwater, and terrestrial ecosystems have documented increasing CTI under global warming. However, most studies have only linked increasing CTI to increases in warm-affinity species. Here, using long-term monitoring of marine fishes across the Northern Hemisphere, we decomposed CTI changes into four underlying processes—tropicalization (increasing warm-affinity), deborealization (decreasing cold-affinity), borealization (increasing cold-affinity), and detropicalization (decreasing warm-affinity)—for which we examined spatial variability and drivers. CTI closely tracked changes in sea surface temperature, increasing in 72% of locations. However, 31% of these increases were primarily due to decreases in cold-affinity species, i.e., deborealization. Thus, increases in warm-affinity species were prevalent, but not ubiquitous. Tropicalization was stronger in areas that were initially warmer, experienced greater warming, or were deeper, while deborealization was stronger in areas that were closer to human population centers or that had higher community thermal diversity. When CTI (and temperature) increased, species that decreased were more likely to be living closer to their upper thermal limits or to be commercially fished. Additionally, warm-affinity species that increased had smaller body sizes than those that decreased. Our results show that CTI changes arise from a variety of underlying community responses that are linked to environmental conditions, human impacts, community structure, and species characteristics.