At a fixed stellar mass, the size of low-redshift early-type galaxies is found to be a factor of 2 larger than that of their counterparts at z˜ 1, a result with important implications for galaxy formation models. In this paper, we have explored the buildup of the local mass-size relation of elliptical galaxies using two visually classified samples. At low redshift, we compiled a subsample of 2656 elliptical galaxies from the Sloan Digital Sky Survey, whereas at higher redshift (up to z˜ 1), we extracted a sample of 228 objects from the Hubble Space Telescope/Advanced Camera for Surveys images of the Great Observatories Origins Deep Survey. All the galaxies in our study have spectroscopic data, allowing us to determine the age and mass of the stellar component. Contrary to previous claims in the literature, using the fossil record information contained in the stellar populations of our local sample, we do not find any evidence for an age segregation at a given stellar mass, depending on the size of the galaxies. At a fixed dynamical mass, there is only a ≲9 per cent size difference in the two extreme age quartiles of our sample. Consequently, the local evidence does not support a scenario whereby the present-day mass-size relation has been progressively established via a bottom-up sequence, where older galaxies occupy the lower part of this relation, remaining in place since their formation. We do not find any age-segregation difference in our high-z sample either. Therefore, we find a trend in size that is insensitive to the age of the stellar populations, at least since z˜ 1. This result supports the idea that the stellar mass-size relation is formed at z˜ 1, with all galaxies populating a region which roughly corresponds to 1/2 of the present size distribution. We have explored two possible scenarios for size growth: puffing up or minor merging. The fact that the evolution in size is independent of the stellar age, together with the absence of an increase in the scatter of the relationship with redshift does not support the puffing-up mechanism. The observational evidence, however, cannot reject at this stage the minor-merging hypothesis. We have made an estimation of the number of minor-merger events necessary to bring the high-z galaxies into the local relation compatible with the observed size evolution. Since z= 0.8, if the mass ratio of the merger is 1:3, then we estimate ˜3 ± 1 minor mergers and if the ratio is 1:10, then we obtain ˜8 ± 2 events.