We report on high angular resolution, high-precision spectropolarimetric measurements of a decaying sunspot. The spot gradually lost its penumbra during the course of 3 days. In the late stages of evolution, where the only remnant of the spot is a naked umbra, we find small-scale inhomogeneities in the magnetic canopy surrounding it. The inhomogeneities are observed as finger-like structures of weak and nearly horizontal magnetic fields extending 1"-2" from the border of the umbra. These fields are not associated with filamentary structures in continuum intensity or with conspicuous Evershed flows. The Stokes profiles emerging from the fingers exhibit blueshifts, which we interpret as upward motions. This previously unknown fine structure may be related to penumbral field lines that no longer carry strong Evershed flows and rise to the chromosphere, producing the disappearance of the penumbra at photospheric levels.