The frequencies of solar p-modes are known to change over the solar cycle. There is also recent evidence that the relation between frequency shift of low-degree modes and magnetic flux or other activity indicators differs between the rising and falling phases of the solar cycle, leading to a hysteresis in such diagrams. We consider the influence of the changing large-scale surface distribution of the magnetic flux on low-degree (l<=3) p-mode frequencies. To that end, we use time-dependent models of the magnetic flux distribution and study the ensuing frequency shifts of modes with different order and degree as a function of time. The resulting curves are periodic functions (in simple cases just sine curves) shifted in time by different amounts for the different modes. We show how this may easily lead to hysteresis cycles comparable to those observed. Our models suggest that high-latitude fields are necessary to produce a significant difference in hysteresis between odd- and even-degree modes. Only magnetic field distributions within a small parameter range are consistent with the observations by Jiménez-Reyes et al. Observations of p-mode frequency shifts are therefore capable of providing an additional diagnostic of the magnetic field near the solar poles. The magnetic distribution that is consistent with the p-mode observations also appears reasonable compared with direct measurements of the magnetic field.