Aims: We study the variation in the magnetic field strength and the umbral intensity of sunspots during the declining phase of the solar cycle No. 23 and in the beginning of cycle No. 24. Methods: We analyze a sample of 183 sunspots observed from 1999 until 2011 with the Tenerife Infrared Polarimeter (TIP) at the German Vacuum Tower Telescope (VTT). The magnetic field strength is derived from the Zeeman splitting of the Stokes-V signal in one near-infrared spectral line, either Fe i 1564.8 nm, Fe i 1089.6 nm, or Si i 1082.7 nm. This avoids the effects of the unpolarized stray light from the field-free quiet Sun surroundings that can affect the splitting seen in Stokes-I in the umbra. The minimum umbral continuum intensity and umbral area are also measured. Results: We find that there is a systematic trend for sunspots in the late stage of the solar cycle No. 23 to be weaker, i.e., to have a smaller maximum magnetic field strength than those at the start of the cycle. The decrease in the field strength with time of about 94 Gyr-1 is well beyond the statistical fluctuations that would be expected because of the larger number of sunspots close to cycle maximum (14 Gyr-1). In the same time interval, the continuum intensity of the umbra increases with a rate of 1.3 (±0.4)% of Ic yr-1, while the umbral area does not show any trend above the statistical variance. Sunspots in the new cycle No. 24 show higher field strengths and lower continuum intensities than those at the end of cycle No. 23, interrupting the trend. Conclusions: Sunspots have an intrinsically weaker field strength and brighter umbrae at the late stages of solar cycles compared to their initial stages, without any significant change in their area. The abrupt increase in field strength in sunspots of the new cycle suggests that the cyclic variations are dominating over any long-term trend that continues across cycles. We find a slight decrease in field strength and an increase in intensity as a long-term trend across the cycles.