We present a uniform analysis of the stellar initial mass function (IMF) from integrated light spectroscopy of 15 compact stellar systems (11 globular clusters in M31 and 4 ultra compact dwarfs in the Virgo cluster, UCDs) and two brightest Coma cluster galaxies (BCGs), covering a wide range of metallicities (-1.7 < [Fe/H] < 0.01) and velocity dispersions (7.4 km s-1 <σ < 275 km s-1). The S/N ~100 Å-1 Keck LRIS spectra are fitted over the range 4000 < λ/Å < 10 000 with flexible full-spectrum stellar population synthesis models. We use the models to fit simultaneously for ages, metallicities, and individual elemental abundances of the population, allowing us to decouple abundance variations from variations in IMF slope. We show that compact stellar systems do not follow the same trends with physical parameters that have been found for early-type galaxies. Most globular clusters in our sample have an IMF consistent with that of the Milky Way, over a wide range of [Fe/H] and [Mg/Fe]. There is more diversity among the UCDs, with some showing evidence for a bottom-heavy IMF, but with no clear correlation with metallicity, abundance, or velocity dispersion. The two Coma BCGs have similar velocity dispersion and metallicity, but we find the IMF of NGC 4874 is consistent with that of the Milky Way while NGC 4889 presents evidence for a significantly bottom-heavy IMF. For this sample, the IMF appears to vary between objects in a way that is not explained by a single metallicity-dependent prescription.