We present an observation-based heat budget analysis for seasonal and interannual variations of ocean heat content (H) in the mixed layer (H-mld) and full-depth ocean (H-tot). Surface heat flux and ocean heat content estimates are combined using a novel Kalman smoother-based method. Regional contributions from ocean heat transport convergences are inferred as a residual and the dominant drivers of H-mld and H-tot are quantified for seasonal and interannual time scales. We find that non-Ekman ocean heat transport processes dominate H-mld variations in the equatorial oceans and regions of strong ocean currents and substantial eddy activity. In these locations, surface temperature anomalies generated by ocean dynamics result in turbulent flux anomalies that drive the overlying atmosphere. In addition, we find large regions of the Atlantic and Pacific oceans where heat transports combine with local air-sea fluxes to generate mixed layer temperature anomalies. In all locations, except regions of deep convection and water mass transformation, interannual variations in H-tot are dominated by the internal rearrangement of heat by ocean dynamics rather than the loss or addition of heat at the surface. Our analysis suggests that, even in extratropical latitudes, initialization of ocean dynamical processes could be an important source of skill for interannual predictability of H-mld and H-tot. Furthermore, we expect variations in H-tot (and thus thermosteric sea level) to be more predictable than near surface temperature anomalies due to the increased importance of ocean heat transport processes for full-depth heat budgets.