Observations with the balloon-borne Sunrise/ Imaging Magnetograph eXperiment (IMaX) provide high spatial resolution (roughly 100 km at disk center) measurements of the magnetic field in the photosphere of the quiet Sun. To investigate the magnetic structure of the chromosphere and corona, we extrapolate these photospheric measurements into the upper solar atmosphere and analyze a 22-minute long time series with a cadence of 33 seconds. Using the extrapolated magnetic-field lines as tracer, we investigate temporal evolution of the magnetic connectivity in the quiet Sun's atmosphere. The majority of magnetic loops are asymmetric in the sense that the photospheric field strength at the loop foot points is very different. We find that the magnetic connectivity of the loops changes rapidly with a typical connection recycling time of about 3±1 minutes in the upper solar atmosphere and 12±4 minutes in the photosphere. This is considerably shorter than previously found. Nonetheless, our estimate of the energy released by the associated magnetic-reconnection processes is not likely to be the sole source for heating the chromosphere and corona in the quiet Sun.