High-resolution galaxy spectra encode information about the stellar populations within galaxies. The properties of the stars, such as their ages, masses, and metallicities, provide insights into the underlying physical processes that drive the growth and transformation of galaxies over cosmic time. We explore a simulation-based inference (SBI) workflow to infer from optical absorption spectra the posterior distributions of metallicities and the star formation histories (SFHs) of galaxies (i.e. the star formation rate as a function of time). We generated a dataset of synthetic spectra to train and test our model using the spectroscopic predictions of the MILES stellar population library and non-parametric SFHs. We reliably estimate the mass assembly of an integrated stellar population with well-calibrated uncertainties. Specifically, we reach a score of 0.97 R2 for the time at which a given galaxy from the test set formed 50% of its stellar mass, obtaining samples of the posteriors in only 10‑4 s. We then applied the pipeline to real observations of massive elliptical galaxies, recovering the well-known relationship between the age and the velocity dispersion, and show that the most massive galaxies (σ ∼ 300 km s‑1) built up to 90% of their total stellar masses within 1 Gyr of the Big Bang. The inferred properties also agree with the state-of-the-art inversion codes, but the inference is performed up to five orders of magnitude faster. This SBI approach coupled with machine learning and applied to full spectral fitting makes it possible to address large numbers of galaxies while performing a thick sampling of the posteriors. It will allow both the deterministic trends and the inherent uncertainties of the highly degenerated inversion problem to be estimated for large and complex upcoming spectroscopic surveys, such as DESI, WEAVE, or 4MOST.