Ten years of continuous measurements of the radial velocity of the Sun have been used to estimate the contribution of various solar surface phenomena to the observed background solar velocity spectrum (BSVS). The characteristics of this spectrum are of great importance, as they represent the ultimate limit on the sensitivity of measurements of solar oscillations. A precise determination of this spectrum from the ground is invariably contaminated by the discontiniuity in the observations, the unavoidable effect of the Earth's atmosphere and, possibly, by the instrumentation itself. The present analysis uses observations made with a very stable instrument to allow comparison of several sets of data, collected along different phases of the solar activity cycle. The results show a high stability of the BSVS throughout the frequency ranges of interest for helioseismology and, moreover, its profile cannot be modeled using a single exponential function, as the exponent changes with frequency. The roughness of the spectrum is calculated, allowing an estimate of the required signal-to-noise ratio in order to detect an oscillation with a given amplitude. Finally, following the modeling of the expected background spectrum proposed by Harvey, the rms full disk velocity of the main solar atmospheric phenomena (granulation, mesogranulation, supergranulation, and active regions) are calculated. Despite the limitations of the procedure employed in this analysis, the main conclusion is the overall good agreement with the model, although mesogranulation seems to have been underestimated. Comparison with the different phases of the solar activity cycle reveals a different behavior of the mesogranulation, while granulation appears to be stable.