We present the evolution of the rest-frame optical luminosity density jrestλ, the integrated rest-frame optical color, and the stellar mass density, ρ*, for a sample of Ks band-selected galaxies in the Hubble Deep Field-South (HDF-S). We derived jrestλ in the rest-frame U, B, and V bands and found that jrestλ increases by a factor of 1.9+/-0.4, 2.9+/-0.6, and 4.9+/-1.0 in the V, B, and U rest-frame bands, respectively, between redshifts of 0.1 and 3.2. We derived the luminosity-weighted mean cosmic (U-B)rest and (B-V)rest colors as a function of redshift. The colors bluen almost monotonically with increasing redshift; at z=0.1, the (U-B)rest and (B-V)rest colors are 0.16 and 0.75, respectively, while at z=2.8 they are -0.39 and 0.29, respectively. We derived the luminosity-weighted mean M/L*V, using the correlation between (U-V)rest and logM/L*V that exists for a range in smooth star formation histories (SFHs) and moderate extinctions. We have shown that the mean of individual M/L*V estimates can overpredict the true value by ~70%, while our method overpredicts the true value by only ~35%. We find that the universe at z~3 had ~10 times lower stellar mass density than it does today in galaxies with LrestV>1.4×1010h-270 Lsolar. Half of the stellar mass of the universe was formed by z~1-1.5. The rate of increase in ρ* with decreasing redshift is similar to but above that for independent estimates from the HDF-N, but it is slightly less than that predicted by the integral of the SFR(z) curve. Based on observations with the NASA/ESA Hubble Space Telescope, obtained at the Space Telescope Science Institute, which is operated by the the Association of Universities for Research in Astronomy (AURA), Inc., under NASA contract NAS5-26555. Also based on observations collected at the European Southern Observatories on Paranal, Chile as part of the ESO program 164.O-0612.