Chemical weathering of silicate rocks on continents is thought to have played an important role in the evolution of past atmospheric carbon dioxide over geologic timescales. However, the detailed links between continental weathering and climate change over shorter timescales, and their potential impact on sediment records deposited in the ocean, remain poorly understood. Here, we present clay mineralogy and strontium‐neodymium isotopic data for marine sediment records from the Northern Indian Ocean, with the aim of investigating the weathering response of large Himalayan river basins to orbital and millennial climate forcing. We show that past glaciated episodes of the late Quaternary corresponded to periods of increased physical erosion, associated with the preferential export of illite and chlorite assemblages from the Himalayan highlands having relatively radiogenic Sr isotopic signatures. In contrast, the warm periods of enhanced monsoon rainfall coincided with the transport of intensively weathered smectite‐dominated soils derived from the floodplains, characterized by lower 87Sr/86Sr signatures. This finding suggests that the short‐term climatic variability over Late Quaternary timescales was accompanied by concomitant changes between high mountain‐ versus floodplain‐dominated weathering regimes, with possible impact on the nature of weathered rocks and, as a consequence, on the carbon cycle.