We present new near-infrared spectra of 55 objects observed using the NASA InfraRed Telescope Facility and the Telescopio Nazionale Galileo, along with visible spectra of 21 objects obtained from the SMASS and S3OS2 surveys, to explore the differences in spectral slope and curvature between the background and the families and to show that the background is a possible source for both Bennu and Ryugu. Within the background population there is spectral diversity in taxonomy, spectral slope, and absorption band parameters. Our sample of asteroids shows that the background looks spectrally similar to the families in the same region, i.e., the background and families may have originated from the same or similar composition parent bodies. Average band center (0.69 ± 0.02 μm, depth: 2.3 ± 0.9%) of an ~0.7 μm absorption feature attributed to aqueous alteration is present in 30% of our primitive background asteroid sample, similar to abundances observed in other primitive inner belt asteroid families. Both near-Earth asteroid sample return mission targets, (101955) Bennu and (162173) Ryugu, are thought to have originated from primitive asteroid populations in the inner main belt, specifically from the low inclination asteroid families. A population that has not been explored spectrally but is dynamically able to deliver asteroid fragments to near-Earth space is the background population, i.e., asteroids that do not cluster into families. Based on our spectral comparisons, the primordial background is a possible source for (162173) Ryugu, but not for (101955) Bennu.