Here we investigate the origin of the dust mass (Mdust) observed in the Milky Way (MW) and of dust scaling relations found in a sample of local galaxies from the DGS and KINGFISH surveys. To this aim, we model dust production from Asymptotic Giant Branch (AGB) stars and supernovae (SNe) in simulated galaxies forming along the assembly of a MW-like halo in a well-resolved cosmic volume of 4 cMpc using the GAMESH pipeline. We explore the impact of different sets of metallicity and mass-dependent AGB and SN dust yields on the predicted Mdust. Our results show that models accounting for grain destruction by the SN reverse shock predict a total dust mass in the MW, that is a factor of ∼4 less than observed, and cannot reproduce the observed galaxy-scale relations between dust and stellar masses, and dust-to-gas ratios and metallicity, with a smaller discrepancy in galaxies with low metallicity (12 + log(O/H) < 7.5) and low stellar masses (Mstar < 107 M⊙). In agreement with previous studies, we suggest that competing processes in the interstellar medium must be at play to explain the observed trends. Our result reinforces this conclusion by showing that it holds independently of the adopted AGB and SN dust yields.