The space-time reconstruction of sea surface dynamics from satellite observations is a challenging inverse problem due to the associated irregular sampling. Satellite altimetry provides a direct observation of the sea surface height (SSH), which relates to the divergence-free component of sea surface currents. The associated sampling pattern prevents operational schemes from retrieving fine-scale dynamics, typically below 10 days. By contrast, other satellite sensors provide higher-resolution observations of sea surface tracers such as sea surface temperature (SST). Multimodal inversion schemes then arise as appealing approaches. Though theoretical evidence supports the existence of an explicit relationship between sea surface temperature and sea surface dynamics under specific dynamical regimes, the generalization to the variety of upper ocean dynamical regimes is complex. Here, we investigate this issue from a physics-informed learning perspective. We introduce a trainable multimodal inversion scheme for the reconstruction of sea surface dynamics from multi-source satellite-derived observations, namely satellite-derived SSH and SST data. The proposed multimodal 4DVarNet schemes combine a variational formulation involving trainable observation and a priori terms with a trainable gradient-based solver. An observing system simulation experiment for a Gulf Stream region supports the relevance of our approach compared with state-of-the-art schemes. We report a relative improvement greater than 60% compared with the operational altimetry product in terms of root mean square error and resolved space-time scales. We discuss further the potential and the limitations of the proposed approach for the reconstruction and forecasting of geophysical dynamics from irregularly-sampled satellite observations.