We present high time resolution ULTRACAM optical and NOTCam infrared observations of the edge-on black hole X-ray transient Swift J1357.2-0933. Our data taken in 2012 and 2013 show the system to be at its pre-outburst magnitude and so the system is in quiescence. In contrast to other X-ray transients, the quiescent light curves of Swift J1357.2-0933 do not show the secondary star's ellipsoidal modulation. The optical light curve is dominated by variability with an optical fractional rms of ˜35 per cent, a factor of >3 larger than what is observed in other systems at similar time resolution. Optical flare events lasting 2-10 min with amplitudes of up to ˜1.5 mag are seen as well as numerous rapid ˜0.8 mag dip events which are similar to the optical dips seen in outburst. Similarly, the infrared J-band light curve is dominated by variability with a fractional rms of ˜21 per cent, and flare events lasting 10-30 min with amplitudes of up to ˜1.5 mag are observed. The quiescent optical to mid-infrared spectral energy distribution (SED) in quiescence is dominated by a non-thermal component with a power-law index of -1.4 (the broad-band rms SED has a similar index) which arises from optically thin synchrotron emission most likely originating in a weak jet; the lack of a peak in the SED rules out advection-dominated models. Using the outburst amplitude-period relation for X-ray transients, we estimate the quiescent magnitude of the secondary star to lie in the range Vmin = 22.7-25.6, which when combined with the absolute magnitude of the expected M4.5 V secondary star allows us to constrain the distance to lie in the range 0.5-6.3 kpc. The short orbital period argues for a nuclearly evolved star with an initial mass ˜1.5 M⊙, which has evolved to a 0.17 M⊙ star. The high Galactic latitude of Swift J1357.2-0933 implies a scaleheight in the range 0.4-4.8 kpc above the Galactic plane, possibly placing Swift J1357.2-0933 in a sub-class of high-z short-period black hole X-ray transients in the Galactic halo.