Field-tower-based observations were used to estimate the Doppler velocity of deep water plunging breaking waves. About 1000 breaking wave events observed by a synchronized video camera and dual-polarization Doppler continuous-wave Ka-band radar at incidence angles varying from 25 to 55 degrees and various azimuths were analyzed using computer vision methods. Doppler velocities (DVs) associated with breaking waves were, for the first time, directly compared to whitecap optical velocities measured as the line-of-sight projection of the whitecap velocity vector (LOV). The DV and LOV were found correlated; however, the DV was systematically less than the LOV with the ratio dependent on the incidence angle and azimuth. The largest DVs observed at up-wave and down-wave directions were accompanied by an increase of the cross-section polarization ratio, HH/VV, up to 1, indicating a non-polarized backscattering mechanism. The observed DV was qualitatively reproduced in terms of a combination of fast specular (coherent) and slow non-specular (incoherent) returns from two planar sides of an asymmetric wedge-shaped breaker. The difference in roughness and tilt between breaker sides (the front face was rougher than the rear face) explained the observed DV asymmetry and was consistent with previously reported mean sea surface Doppler centroid data and normalized radar cross-section measurements.