Context. In the solar atmosphere, the acoustic cutoff frequency is a local quantity that depends on atmospheric height. It separates low-frequency evanescent waves from high-frequency propagating waves. Aims: We measure the cutoff frequency of slow magnetoacoustic waves at various heights of a sunspot umbra and compare the results with the estimations from several analytical formulae. Methods: We analyzed the oscillations in the umbra of a sunspot belonging to active region NOAA 12662 observed in the 10 830 Å spectral region with the GREGOR Infrared Spectrograph and in the Fe I 5435 Å line with the GREGOR Fabry-Pérot Interferometer. Both instruments are attached to the GREGOR telescope at the Observatorio del Teide, Tenerife, Spain. We computed the phase and amplification spectra between the velocity measured from various pairs of lines that sample various heights of the solar atmosphere. The cutoff frequency and its height variation were estimated from the inspection of the spectra. Results: At the deep umbral photosphere the cutoff frequency is around 5 mHz and it increases to 6 mHz at higher photospheric layers. At the chromosphere the cutoff is 3.1 mHz. A comparison of the observationally determined cutoff with the theoretically predicted values reveals an agreement in the general trend and a reasonable match at the chromosphere, but also significant quantitative differences at the photosphere. Conclusions: Our analyses show strong evidence of the variation of the cutoff frequency with height in a sunspot umbra, which is not fully accounted for by current analytical estimations. This result has implications for our understanding of wave propagation, the seismology of active regions, and the evaluation of heating mechanisms based on compressible waves.