We present the evolution of the volume-averaged properties of the rest-frame optically luminous (LV>3×1010 h-270 Lsolar) galaxy population to z~3, determined from four disjoint deep fields. We characterize their rest-frame UV through optical properties via the mean SED. To measure evolution, we apply the selection criteria to a sample of galaxies from the SDSS and COMBO-17 survey. The mean rest-frame 2200 Å through V-band SED becomes steadily bluer with increasing redshift, but at all redshifts z<3 the mean SED falls within the range defined by ``normal'' galaxies in the nearby universe. We measure the mean stellar mass-to-light ratios (M*/L) and stellar mass densities (ρ*) by fitting models to the mean rest-frame UV-optical SEDs. The ρ* in galaxies selected at a fixed luminosity has increased by a factor of 3.5-7.9 from z=3 to 0.1. If we instead use our observed M*/LV evolution to select galaxies at a fixed mass, ρ* evolves by a factor of 5.3-16.7. After correcting to total, the measured ρ* at z<2 lie below the integral of the star formation rate density as a function of redshift as derived from UV-selected samples after a standard correction for extinction. We find large discrepancies between recent model predictions for the evolution of ρ* and our results, even when our observational selection is applied to the models. Finally, we determine that distant red galaxies (selected to have Js-Ks>2.3) in our LrestV-selected samples contribute 30% and 64% of the stellar mass budget at z~2 and z~2.8, respectively. These galaxies are largely absent from UV surveys, and this result highlights the need for mass selection of high-redshift galaxies. Based on observations with the NASA/ESA Hubble Space Telescope, obtained at the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy (AURA), Inc., under NASA contract NAS5-26555. Also based on observations collected at the European Southern Observatories on Paranal, Chile as part of the ESO program 164.O-0612.