We use spectral energy distribution (SED) fitting to place constraints on the stellar population properties of 29 quiescent ultra-diffuse galaxies (UDGs) across different environments. We use the fully Bayesian routine PROSPECTOR coupled with archival data in the optical, near, and mid-infrared from Spitzer and Wide-field Infrared Survey Explorer under the assumption of an exponentially declining star formation history. We recover the stellar mass, age, metallicity, dust content, star formation time scales, and photometric redshifts (photo-zs) of the UDGs studied. Using the mid-infrared data, we probe the existence of dust in UDGs. Although its presence cannot be confirmed, we find that the inclusion of small amounts of dust in the models brings the stellar populations closer to those reported with spectroscopy. Additionally, we fit the redshifts of all galaxies. We find a high accuracy in recovering photo-zs compared to spectroscopy, allowing us to provide new photo-z estimates for three field UDGs with unknown distances. We find evidence of a stellar population dependence on the environment, with quiescent field UDGs being systematically younger than their cluster counterparts. Lastly, we find that all UDGs lie below the mass-metallicity relation for normal dwarf galaxies. Particularly, the globular cluster (GC)-poor UDGs are consistently more metal-rich than GC-rich ones, suggesting that GC-poor UDGs may be puffed-up dwarfs, while most GC-rich UDGs are better explained by a failed galaxy scenario. As a byproduct, we show that two galaxies in our sample, NGC 1052-DF2 and NGC 1052-DF4, share equivalent stellar population properties, with ages consistent with 8 Gyr. This finding supports formation scenarios where the galaxies were formed together.