Cross-correlation between weak lensing of the Cosmic Microwave Background (CMB) and weak lensing of galaxies offers a way to place robust constraints on cosmological and astrophysical parameters with reduced sensitivity to certain systematic effects affecting individual surveys. We measure the angular cross-power spectrum between the Atacama Cosmology Telescope (ACT) DR4 CMB lensing and the galaxy weak lensing measured by the Dark Energy Survey (DES) Y3 data. Our baseline analysis uses the CMB convergence map derived from ACT-DR4 and Planck data, where most of the contamination due to the thermal Sunyaev Zel'dovich effect is removed, thus avoiding important systematics in the cross-correlation. In our modelling, we consider the nuisance parameters of the photometric uncertainty, multiplicative shear bias and intrinsic alignment of galaxies. The resulting cross-power spectrum has a signal-to-noise ratio = 7.1 and passes a set of null tests. We use it to infer the amplitude of the fluctuations in the matter distribution (S8 ≡ σ8(Ωm/0.3)0.5 = 0.782 ± 0.059) with informative but well-motivated priors on the nuisance parameters. We also investigate the validity of these priors by significantly relaxing them and checking the consistency of the resulting posteriors, finding them consistent, albeit only with relatively weak constraints. This cross-correlation measurement will improve significantly with the new ACT-DR6 lensing map and form a key component of the joint 6×2pt analysis between DES and ACT.