Quantifying the contribution of mergers to the stellar mass of galaxies is key for constraining the mechanisms of galaxy assembly across cosmic time. However, the mapping between observable galaxy properties and merger histories is not trivial: cosmological galaxy simulations are the only tools we have for calibration. We study the robustness of a simulation-based inference of the ex situ stellar mass fraction of nearby galaxies to different observables - integrated and spatially resolved - and to different galaxy formation models - IllustrisTNG and EAGLE - with machine learning. We find that at fixed simulation, the fraction of accreted stars can be inferred with very high accuracy, with an error ~5 per cent (10 per cent) from 2D integral-field spectroscopic maps (integrated quantities) throughout the considered stellar mass range. A bias (> 5 per cent) and an increase in scatter by a factor of 2 are introduced when testing with a different simulation, revealing a lack of generalization to distinct galaxy-formation models. Interestingly, upon using only stellar mass and kinematics maps in the central galactic regions for training, we find that this bias is removed and the ex situ stellar mass fraction can be recovered in both simulations with <15 per cent scatter, independently of the training set's origin. This opens up the door to a potential robust inference of the accretion histories of galaxies from existing integral field unit surveys, such as MaNGA, covering a similar field of view and containing spatially resolved spectra for tens of thousands of nearby galaxies.